testthat 2.0.0

New default reporter

A new default reporter, ReporterProgress, revamps the output to make use of colour and reveal details of failures as they occur:

If you prefer the previous version, you can run it once with devtools::test(reporter = "summary") or turn back time by setting option(testthat.default_reporter = "summary").

Setup and teardown

There are two new ways to setup and teardown code:

• New setup() and teardown() functions specify code to be run at the beginning and end of each test file. Write them next to each other so you can easily confirm that each setup() is paired with a teardown() that cleans up any changes to the global state.

  tmp <- tempfile()
  setup({
    writeLines("TEST DATA", tmp)
  })
  teardown({
    unlink(tmp)
  })

• tests/testthat/setup-xyz.R files are run before the first test file is executed. They are similar to the existing helpers-xyz.R files, but are not run by devtools::load_all(). Similarly, test/teststhat/teardown-xyz.R files are run after all tests are complete; use these to clean up any global changes made by the setup files.

New and improved expectations

We have identified a new family of expectations that compares the results of an expression to a known good value stored in a file. They are designed to be used in conjunction with git so that you can see what precisely has changed, and revert it if needed:

• expect_known_output() saves the output of an expression and will fail the first time the output changes. The file is updated on each run, so needs to be used in conjunction with git. It replaces expect_output_file() which is now soft-deprecated.

  # File to save results: would usually be filename, which will be stored 
  # in tests/testthat.
  tmp <- tempfile()
  
  # The first run always succeeds, but warns
  expect_known_output(mtcars[1:10, ], tmp, print = TRUE)
  #> Warning: Creating reference output
  
  # Subsequent runs will suceed only if the file is unchanged
  # This will succeed:
  expect_known_output(mtcars[1:10, ], tmp, print = TRUE)
  
  # This will fail
  expect_known_output(mtcars[1:9, ], tmp, print = TRUE)
  #> Error: `mtcars[1:9, ]` has changed from known value recorded in '/tmp/Rtmphlp98S/file481a3697072d'.
  #> Lengths differ: 10 is not 11

• expect_known_value() replaces expect_equal_to_reference(), which has been soft-deprecated. It gains an update argument defaulting to TRUE. This changes behaviour from the previous version, and soft-deprecated expect_equal_to_reference() gets update = FALSE.

We’ve also improved tools for testing for failures:

• expect_condition() works like expect_error() but captures any condition, not just error conditions.

• expect_error() gains a class argument that allows you to make an assertion about the class of the error object.

We’ve also added expect_setequal() to compares two sets (stored in vectors), ignoring duplicates and differences in order. Finally, we added a few new helpers for skipping tests:

• skip_if() makes it easy to skip a test when a condition is true. For example, use skip_if(getRversion() <= 3.1) to skip a test in older R versions.

• skip_if_translated() skips tests if you’re running in an locale where translations are likely to occur. Use this to avoid spurious failures when checking the text of error messages in non-English locales.

• skip_if_not_installed() gains new minimum_version argument. This allows you to only test if a minimum version requrement is met, e.g.

  skip_if_not_installed("ggplot2", "2.0.0")

Quasiquotation support

All expectations can now make use of unquoting, with !!. This makes it much easier to generate informative failure messages when running tests in a for loop or function. For example take this test:

f <- function(i) if (i > 3) i * 9 else i * 10

for (i in 1:5) {
  expect_equal(f(i), i * 10)
}
#> Error: f(i) not equal to i * 10.
#> 1/1 mismatches
#> [1] 36 - 40 == -4

The error message is not great because you don’t know which iteration caused the problem! You can resolve that problem by using the unquoting operator !! (pronounced bang-bang):

for (i in 1:5) {
  expect_equal(f(!!i), !!(i * 10))
}
#> Error: f(4L) not equal to 40.
#> 1/1 mismatches
#> [1] 36 - 40 == -4

(Note that this is not tidy evaluation per se, but is closely related. It works a little differently compared to tidyverse packages because quoting is only used to generate the failure messages: see ?quasi_label() for more details. At this time you can not unquote quosures.)

Breaking changes

Unfortunately it was necessary to make a few API breaking changes in testthat in order to make testthat more flexible for the future, and deal with some changes made to R. These changes affected around 1 in 40 packages using testthat on CRAN, and all maintainers were warned a month in advance.

If you have a non-CRAN package, read the following list of symptoms and remedies in order to get your package working with testthat 2.0.0:

• “Can’t mock functions in base packages”: You can no longer use with_mock() to mock functions in base packages, because this no longer works in R-devel due to changes with the byte code compiler. I recommend using mockery or mockr instead.

• The order of arguments to expect_equivalent() and expect_error() was changed slightly as both now pass ... on to another function. This reveals itself with a number of different errors, like:

  • ‘what’ must be a character vector
  • ‘check.attributes’ must be logical
  • ‘tolerance’ should be numeric
  • argument is not interpretable as logical
  • threw an error with unexpected class
  • argument “quo” is missing, with no default
  • argument is missing, with no default

  If you see one of these errors, check the number, order, and names of arguments to the expectation.

• “Failure: (unknown)”. The last release mistakenly failed to test bare expectations not wrapped inside test_that(). If you see “(unknown)” in a failure message, this is a failing expectation that you previously weren’t seeing. As well as fixing the failure, please also wrap inside a test_that() with an informative name.

• “Error: the argument has already been evaluated”: the way in which expectations now create labels has changed, which caused a couple of failures with unusual usage when combined with Reduce, lapply(), and Map(). Avoid these functions in favour of for loops. I also recommend reading the section on quasiquotation support (above) in order to create more informative failure messages.

• is_null() and matches() have been deprecated because they conflict with other functions in the tidyverse. Use the modern expect_null() and expect_matches() instead.

Acknowledgements

A big thanks goes to Kirill Müller for his help running R CMD check on all the packages that use testthat - in total he ran R CMD check over 10,000 times! Likewise, a big thanks to the CRAN team who also have to run these checks to ensure that other packages are not broken inadvertently.

A whopping 103 people helped to make this release happen. Thanks to everyone who opened issues and contributed code: @3psil0n, @agricolamz, @akbertram, @AmeliaMN, @andrie, @aronatkins, @BarkleyBG, @bbolker, @bc, @bdwyer2, @billchenxi, @billdenney, @BillDunlap, @boennecd, @bradleyjeck, @briencj, @brodieG, @carlganz, @cbare, @cderv, @cdriveraus, @cfhammill, @chambm, @Christoph999, @cogmind, @colearendt, @dchudz, @dlindelof, @dmenne, @dougpagani, @egnha, @epurdom, @fangly, @floybix, @FrancoisGuillem, @frankandrobot, @fritzo, @gaborcsardi, @george-weingart, @hadley, @hansharhoff, @hesamaseh, @HyukjinKwon, @jackolney, @jackwasey, @jcheng5, @jdblischak, @jdshkolnik, @jefferis, @jennybc, @jeroen, @jimhester, @joethorley, @jspitzen, @kalibera, @katrinleinweber, @kenahoo, @kenahoo-windlogics, @kendonB, @kevinushey, @kevinykuo, @kforner, @klmr, @krlmlr, @lbartnik, @lc0, @lionel-, @lorenzwalthert, @mahaiyer, @MarkEdmondson1234, @maxheld83, @micstr, @mllg, @mlysy, @mschubert, @nabilabd, @nealrichardson, @p-smirnov, @paciorek, @pat-s, @patperry, @r2evans, @rdiaz02, @rgknight, @richierocks, @robwe, @ruaridhw, @s6mike, @schoettl, @scottkosty, @Stan125, @stla, @swt30, @twolodzko, @unDocUMeantIt, @vermouthmjl, @wch, @wlandau-lilly, @wligtenberg, @wsloand, @wush978, @zachcp, and @zer0hedge