Repository navigation
Missing source-based coverage for a source file which is supposed to be covered #79456
Description
As I mentioned in rust-lang/compiler-team#278 (comment), I've implemented support for source-based code coverage in grcov, and started using it to collect coverage for grcov itself.
There is a file that is definitely covered, but doesn't appear to be in the report. It is in the report when using gcov-based coverage.
STR:
- Clone https://github.com/mozilla/grcov;
- Run
LLVM_PROFILE_FILE="grcov-%p-%m.profraw" cargo test; - Run
grcov `find . \( -name "grcov-*.profraw" \) -print\` --binary-path ./target/debug/grcov -s . -t html --branch --ignore-not-existing --ignore "/*" -o coverageto generate a HTML report (or you can also generate a textual one using the llvm tools); - See that src/llvm_tools.rs appears to be totally uncovered, while it should be covered.
Activity
nagisa commentedon Nov 27, 2020
Does this properly associate the profraw with the exact binary that produced it?
--binary-path ./target/debug/grcovwould suggest that you're usinggrcov(the[[bin]]target) while theprofrawfiles seem to have been generated by various test binaries (typically have hashes in them, somewhere).Is it also reproducible when using the llvm-cov tooling?
marco-c commentedon Nov 27, 2020
Oh, you're right, I was assuming only the source code behind
#[cfg(test)]would be missed this way. It works if I use the actual test binary!This means grcov will need a way to automatically detect the binary paths, and run llvm-cov accordingly...
nagisa commentedon Nov 27, 2020
Closing as not-a-bug (at least not in rustc).
Of course, I agree that having to associate the exact binary with the profile file it produced is sub-optimal, and cargo makes it harder still by appending hashes to everything. That said, this seems like the issue to be fixed in either the LLVM-provided instrumentation runtime (maybe just having a way for it to splice binary name into the profile filename would help?) or in cargo (that's what I'm planning to do for my own use-cases).
marco-c commentedon Nov 27, 2020
Maybe using
%min LLVM_PROFILE_FILE, which "expands out to the instrumented binary’s signature". Then we just need to read the signature from the binaries to match them to the right profraw file.richkadel commentedon Dec 2, 2020
@marco-c - In case this is helpful...
When PR #79109 lands, there will be an update to the "Rust Unstable Book" section on source-based-code-coverage, including an example for coverage of tests run through
cargo test.The example does use
%mto make sure each test gets its ownprofrawresult.You can see the new version in the PR commit, here:
https://github.com/rust-lang/rust/blob/00117d712189ca703b481b471e11bcc742f0346e/src/doc/unstable-book/src/compiler-flags/source-based-code-coverage.md#test-coverage
Hope this helps.
marco-c commentedon Dec 2, 2020
@richkadel for now in grcov I'm simply running llvm-cov with the profdata file associated to all possible binary files, not a great solution, but it seems to work...
The next step will be to figure out how to use the binary signature to correctly match the profraw to the right binary, but I've looked at the binary signature code in LLVM and it is platform-specific, I'm not too keen on including something like that in grcov.
I've just now realized the binary signature is in the binary name itself! So that's going to make it easier ;)
The only constraint on the user will be the forced usage of LLVM_PROFILE_FILE.
For context, grcov is a tool to parse coverage artifacts (from Rust and other languages). Basically an alternative to
llvm-cov, but easier to use and supporting more options, input formats, and output formats. I need to make it easy to use as part of CI, so can't rely on users runningcargo test, checking the name of the binaries, and passing their names to grcov.This is what I currently suggest doing to people using grcov: