Cargo book doesn't specify which tests run in parallel and which tests run in fresh environment #10361

Activity

1. safinaskar

   added

   C-bugCategory: bugCategory: bug

   on Feb 4, 2022
2. weihanglo commented on Feb 18, 2022

   

   weihanglo

   on Feb 18, 2022

   Last edited by weihanglo

   Member

   More actions

   Firstly, in cargo each target compiles down to a crate, which usually refers to a compilation artifact such like a library or a executable.

   For unit test, cargo invokes rustc to collect all #[test] items and link them to libtest for building final executable. This executable runs all your unit tests inside a target and by default in parallel unless you pass cargo test -- --test-threads 1. For instance, if you have one lib target and one bin. When you run cargo test, Cargo produces two unittest executables: One for lib and the other for bin. The two executables are run one by one, just like how cargo-nextest describes in its own book. They are two processes spawn by cargo, so they might inherit a part of environment from cargo. But all #[test] inside the same target are just run under the same process. You can consult the relevant code here for more info.

   The aforementioned rule also applies on integration tests, all #[test] items inside on test target are compiled into on libtest executable. Each integration test produces a crate so that each is run in different process. This seems somehow inefficient and people sometimes combine all integration tests into one target to exploit libtest's parallelism. (cargo itself does that as well)

   Things get interesting in doc tests. Cargo permits only one lib target for each package, and only compiles doc test for lib target. You might ask, does that implies a package gets exactly one executable for running doc tests? The answer is No. Doc tests current are controlled by rustdoc, which leverages library artifact such like rlib to compile each code block into an executable respectively. So, instead of one executable, you get multiple exetuables if you have multiple code blocks to test. These exectuables are located in a temporary directory by default, so you won't see them in the target dir. The relevant code lies in here. Interestingly, code blocks of doctest are run in parallel although they are different executables.

   Here is a table of my summary:

   	Unit tests	Integration tests	Doctests
   Run in parallel?	✅ for #[test] ❌ for target	✅ for #[test] ❌ for target	✅ for each code block
   Share environment?	One target per process	One target per process	One code block per process

   My knowledge might be rusty. Correct me if you find anything incorrect.

   👍React with 👍1Reacted by heisen

   ❤️React with ❤️3Reacted by Jacob Finkelman, Andy Li and Ramzi Sabra
3. weihanglo

   added

   A-documenting-cargo-itselfArea: Cargo's documentationArea: Cargo's documentation

   Command-test

   on Feb 25, 2022
4. weihanglo commented on Jun 5, 2022

   

   weihanglo

   on Jun 5, 2022

   Member

   More actions

   Close as #10726 adds more sentences on this topic. Thanks for the report!

   cargo/src/doc/man/cargo-test.md

   Lines 30 to 54 in cce487e

   	Tests are built with the `--test` option to `rustc` which creates a special
   	executable by linking your code with libtest. The executable automatically
   	runs all functions annotated with the `#[test]` attribute in multiple threads.
   	`#[bench]` annotated functions will also be run with one iteration to verify
   	that they are functional.
   	If the package contains multiple test targets, each target compiles to a
   	special executable as aforementioned, and then is run serially.
   	The libtest harness may be disabled by setting `harness = false` in the target
   	manifest settings, in which case your code will need to provide its own `main`
   	function to handle running tests.
   	### Documentation tests
   	Documentation tests are also run by default, which is handled by `rustdoc`. It
   	extracts code samples from documentation comments of the library target, and
   	then executes them.
   	Different from normal test targets, each code block compiles to a doctest
   	executable on the fly with `rustc`. These executables run in parallel in
   	separate processes. The compilation of a code block is in fact a part of test
   	function controlled by libtest, so some options such as `--jobs` might not
   	take effect. Note that this execution model of doctests is not guaranteed
   	and may change in the future; beware of depending on it.
5. weihanglo

   closed this as completedon Jun 5, 2022