Cargo runs unit tests when they are disabled and --lib is passed

Problem

When a crate has no unit tests (via test = false in its manifest), I would expect cargo to never ever try to run unit tests for that crate.

And yet it turns out that cargo test --lib will run the unit tests.

Steps

Run cargo test --lib --bins --examples --tests --benches (in an attempt to run "everything except for doc tests"), or just cargo test --lib
Note how even if the library has unit tests are disabled, cargo will run them

     Running unittests src/lib.rs (build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/core-04ceb2ced8c36de5)

running 0 tests


test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 7.54µs

Possible Solution(s)

--lib should just be a filter, removing from the usual test set everything that's not a "library unit test". It should not add things that were not present to begin with.

Notes

This is related to but different from #10936

Version

No response

Something similar seems to happen with benchmarks. With a manifest like this

[[bench]]
name = "allocbenches"
path = "benches/lib.rs"
test = true

[[bench]]
name = "vec_deque_append_bench"
path = "benches/vec_deque_append.rs"
harness = false

Running cargo test will only run the first benchmark, but running cargo test --lib --bins --examples --tests --benches will run both -- even though it's not a test and shouldn't be run as a test.

This seems to be covered by rust-lang/testing-devex-team#1> Once we have a new JSON output format, Cargo could get more controls over tests than relying on libtest.

I don't think this issue has anything to do with handling the output from the tests. It's all about which tests cargo decides to run.

I see. Sorry I understood it wrong.

--lib is a flag for user to explicitly requests tests in library target to be run, so that is expected. So does other flags like --test, --bench, etc.

test = false controls only the default set of tests being run. See the target selection section.

Setting targets to test = false will stop them from being tested by default. Target selection options that take a target by name (such as --example foo) ignore the test flag and will always test the given target.

--lib should just be a filter, removing from the usual test set everything that's not a "library unit test". It should not add things that were not present to begin with.

The proposed solution can only be solved with rust-lang/testing-devex-team#1, from my understanding, because Cargo needs to run libtest to know if there is any unit test defined.

test = false controls only the default set of tests being run. See the target selection section.

That's very surprising... but it explains a lot, thanks.

Auto merge of #3430 - RalfJung:doc, r=RalfJung

Auto merge of rust-lang#3430 - RalfJung:doc, r=RalfJung

Navigation Menu

Repository navigation

Cargo runs unit tests when they are disabled and `--lib` is passed #13668

Problem

Steps

Possible Solution(s)

Notes

Version

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Cargo runs unit tests when they are disabled and --lib is passed #13668

Description

Problem

Steps

Possible Solution(s)

Notes

Version

Activity

RalfJung commented on Mar 29, 2024

weihanglo commented on Mar 29, 2024

RalfJung commented on Mar 29, 2024

weihanglo commented on Mar 29, 2024

weihanglo commented on Mar 29, 2024

RalfJung commented on Mar 29, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Cargo runs unit tests when they are disabled and `--lib` is passed #13668