Repository navigation
cargo test --tests also runs unit tests -- target filters are confusing #10936
Description
Problem
cargo test has flags --libs, --bins, --tests, --doc, which I thought (based on their documentation) would control whether the lib crate unit tests / bin crate unit tests / integration tests / doc tests, respectively, are being run. But that does not seem to be what happens, so either the documentation is wrong or it is very confusing.
Steps
- Run
cargo test --tests. - Notice how it runs not just the integration tests, but also the unit tests!
- However,
cargo test --binsruns only the binary crate unit tests.
This seems clearly inconsistent. I would expect cargo test --tests to be the same as cargo test --test a --test b (manually listing all tests), which is how cargo test --bins seems to behave.
Possible Solution(s)
No response
Notes
Generally the cargo test documentation for all these many flags that control which tests are being run, is unclear on whether the flag adds tests to the default test set, or remove the default test set to only run a small select set of tests. (This also affects other commands like cargo check.) For instance, cargo test --benches tests more than the default test set, but cargo test --bins tests less than the default test set, but the documentation for these two flags reads exactly the same. Some flags use the word 'only', which is maybe meant to indicate "reduce default test set", but --bins does not use that word and anyway this is not at all clear.
The behavior I would have expected is along the lines of "if any of these flags is present, then only what is specified by the flags is being run, no default test set", but that does not match implemented behavior. This then makes me wonder how I would even say "run only integration tests" (cargo test --tests doesn't do that), or "check only the benchmarks" (cargo check --benches doesn't do that).
Version
cargo 1.64.0-nightly (85b500cca 2022-07-24)
release: 1.64.0-nightly
commit-hash: 85b500ccad8cd0b63995fd94a03ddd4b83f7905b
commit-date: 2022-07-24
host: x86_64-unknown-linux-gnu
libgit2: 1.4.2 (sys:0.14.2 vendored)
libcurl: 7.83.1-DEV (sys:0.4.55+curl-7.83.1 vendored ssl:OpenSSL/1.1.1q)
os: Debian testing (bookworm) [64-bit]
Activity
weihanglo commentedon Aug 7, 2022
Totally agree it is confusing. People usually invoke
cargo test --helpand see it sayTest all tests. Then guessing like "I know cargo tests. Tests here must refer to all test targets". And it turns out to be all targets that has default test flag set.It's hard to encourage people to use
cargo help test, read all the lengthy doc1, and figure out how it works. I believe changing the behaviour is a breaking change, so there indeed is an inconsistency between--testsand other target options such as--bins. I would suggest improve the short help message first and see how to expand the long version, though I currently have no thought about a better help message.Footnotes
--testsTest all targets in test mode that have the
test = truemanifest flag set. By default this includes the library and binaries built as unittests, and integration tests. Be aware that this will also build any required dependencies, so the lib target may be built twice (once as a unittest, and once as a dependency for binaries, integration tests, etc.). Targets may be enabled or disabled by setting the test flag in the manifest settings for the target. ↩RalfJung commentedon Aug 7, 2022
So is there even a way to ask cargo to test (or check or ...) all test targets (the things that are by default in the
tests/directory, the things that can be selected via--test name) but nothing else? I expect this to be possible somehow, and there's no other flag which does this, so it must be--tests-- that was my reasoning.RalfJung commentedon Aug 7, 2022
Oh wow, I had no idea that this command even existed! Thanks for pointing it out, I will try to remember. :)
I now see it is mentioned at the end of
cargo test --help. Not sure how I missed it. I guess I didn't expect there to be a short and long version of the help, so I didn't even go looking.What's the reason why
cargo test --helpdoes not do the same thing ascargo help test?weihanglo commentedon Aug 7, 2022
Underneath the hod, integration tests and unit tests are handled by the same mechanism. They are all collected from functions with
#[test]attribute by libtest. Here is an old write-up may help. #10361 (comment). If you do want do run all integration tests but not unit test in library target, try to leverage glob syntax such ascargo test --test '*'.I got a impression that @epage had mentioned it somewhere but can't remember.
RalfJung commentedon Aug 7, 2022
Ah, didn't know that was possible, thanks. I should really read those long docs in their entirety.
Still I feel like a flag is missing here, since we do have such a flag for all other similar cases, AFAIK.
heisen-li commentedon Dec 29, 2023
The description of
--testsin the cargo book:But there seems to be a problem with this statement, the results of my tests using
cargo test --binsto testcargowill not show up incargo test --tests, implying that--testsdoesn't test the binary unit tests? So it would make more sense for--teststo be modified to only accomplish integration tests and library unit tests?Also, in
cargo test -h, in order to maintain uniformity, should--binsand--binbe described similarly to--lib, modified to read:--bin: Test only the specified binary's unit tests--bins: Test all binaries' unit testsepage commentedon Jan 2, 2024
In case its relevant, #12915 made some minor wording tweaks in 1.75
Auto merge of #13236 - heisen-li:test, r=epage
RalfJung commentedon Oct 11, 2024
I think the main confusion is the fact that this is not equivalent to
cargo test --tests. It certainly feels like it should be equivalent. Thecargo help testdocs now explain this, butcargo test --helpstill doesn't even hint at this. I should try giving this a shot...Auto merge of #14675 - RalfJung:help, r=weihanglo
fix(gate): the std-first audit is the Rust binary, with the exemption…