Misses in coverage

[xd009642]

Mention any false positives or false negatives in code coverage in this issue with an example where it occurred and it will be added to the list!

• An else only line https://codecov.io/gh/Smithay/wayland-rs/src/master/wayland-commons/src/socket.rs#L33
• Single logical lines split across multiple ones see Missing coverage for logical lines split on multiple lines #270 for past discussion
• match statement with enum or *self false positive when matching enum #258
• macros in function bodies https://codecov.io/gh/simd-lite/simdjson-rs/src/master/src/stage2.rs#L133
• static values can be uncovered https://codecov.io/gh/simd-lite/simdjson-rs/src/master/src/stage2.rs#L320
• const values can be uncovered https://codecov.io/gh/simd-lite/simdjson-rs/src/master/src/stage2.rs#L320
• unsafe blocks marked as uncovered https://codecov.io/gh/simd-lite/simdjson-rs/src/master/src/lib.rs#L285
• const initialisations aren't covered.
• general chained method calls (sometimes can differ in free functions versus methods see Different coverage in free function vs associated methods #238 )
• PhantomData
• Inconsistent coverage with macros like assert_eq!
• Not sure how well inline asm! works
• Code disabled by feature configs included
• struct fields in matches with enum structs Foo::X{ref bar} => {} bar can register as uncovered
• trait's ignore methods without default impl

[jamestwebber]

It looks like my coverage report is counting the lines in tests as part of coverage (including one miss, somehow). See e.g. here if you scroll down to the lines 500+. Not sure whether that's intentional or not but seems a bit odd (and the miss is a false negative I guess)

[xd009642]

There's some weirdness with macros, I need to figure out the best way to handle them. Also if you want to exclude test code you can use --exclude-tests and all lines in anything with the #[test] attribute or #[cfg(test)] will be removed from the results

[pwoolcoc]

Just curious, why is --ignore-tests not the default? Are people really writing tests for their tests? I can't imagine a situation where I'd want my test code to be counted as part of the coverable source code

[xd009642]

Honestly just because as a feature it came later and no ones asked for the default to be switched around, I've got no opposition to switching the default behaviour. I also definitely don't think people are writing tests for their tests (or at least I hope not...)

The main use I can see is for integration tests that involve things like file or network IO some people might want to make sure their tests are actually executing the code as thoroughly as they expect

[mathstuf]

I think this is another instance of the "split logical lines", but the attached report for serde.rs shows some more uncovered chain calls and unexpected (inlined?) not-to-be-covered lines at the very bottom.

tarpaulin-report.html.gz

[Kixiron]

A variable declaration with the left-hand expression on a different line is counted as untested, see this report and this one. Additionally, conditionally compiled code that is not part of the current feature set seems to flag a false negative as well, as shown here

[xd009642]

oh the feature flag is something I've definitely missed! Although I suppose I wonder if that should be filtered out by default, as ideally you should test all your features thoroughly and it is possible to organise different runs in a batch with different feature settings... Maybe I'll add a flag to add/remove them.

Also for the assign expressions thinking of something that maps to logical lines of code for solving that class of issues once and for all

[jtempest]

It looks like the inline, associated functions returning &'static str in this report only have the function signature marked and not the body that returns the string.

Also, thanks for tarpaulin, it's really handy and easy to set up :)

[recmo]

Are people really writing tests for their tests?

I also definitely don't think people are writing tests for their tests (or at least I hope not...)

I can think of two examples:

• Implementations of Quickcheck / Proptest test case generation. These implementations are sometimes complicated enough to warrant some testing.
• Reference functions. Sometimes I write a slow-but-simple textbook test version of an algorithm (for example an FFT). I then use this to compare with the fast-but-complicated real implementation. But I first add a few tests to make sure the reference is correct.

But I don't care much about coverage in both cases, --ignore-test seems like a good default.

[jonhoo]

Here's another coverage result that seems to have some odd lines marked as not covered:
https://codecov.io/gh/jonhoo/openssh-rs/src/9c82a1b11668033e2361d04f821786a41ce46615/src/lib.rs

The repository is over here: https://github.com/jonhoo/openssh-rs/

[jonhoo]

I believe all of the misses here are erroneous:
https://codecov.io/gh/jonhoo/cliff/src/811bace3d7b1d7c64bdd316ab59d4e355e3d163c/src/lib.rs

[xd009642]

GitHub API: Forbidden

😞

[jamestwebber]

^ I think that's just GitHub having issues today

[FeuRenard]

I have 2 covered lines containing doc comments using tarpaulin 0.11.1:
https://codecov.io/gh/FeuRenard/mygpoclient-rs/src/c5e4d61f21b548f2efd03429a4d35f62441c42c2/src/episode.rs

[kristiannotari]

False negatives here.

I have setup tarpaulin recently and tested it on my code and uploaded test coverage to Codecov. This is the actual report on Codecov (no custom formatting).

I have found that:

• some write!( ... ) macros arguments, when those are split on multiple lines are flagged as misses
• some assert!( ... ) macros arguments, when those are split on multiple lines are flagged as misses
• some right hand sides of let ... assignments, when on other lines (even if self is on the same line of the let ...) are flagged as misses

Here's my travis-ci build configuration, if needed:

configuration

language: rust
sudo: required # required for some configurations
# tarpaulin has only been tested on bionic and trusty other distros may have issues
dist: bionic
addons:
  apt:
      packages:
          - libssl-dev
rust:
- stable
- beta
- nightly
jobs:
allow_failures:
  - rust: nightly
fast_finish: true
cache: cargo
before_script: |
if [[ "$TRAVIS_RUST_VERSION" == stable ]]; then
  cargo install cargo-tarpaulin
fi
after_success: |
if [[ "$TRAVIS_RUST_VERSION" == stable ]]; then
  # Get coverage report and upload it for codecov.io.
  cargo tarpaulin --out Xml
  bash <(curl -s https://codecov.io/bash)
fi

[khoek]

I have a whole bunch of match self.blah { <newline> }s where the line containing the match itself is marked not-covered.

[CAD97]

In a fn foo(&mut self, ...) -> &mut Self { ...; self }, the final line returning self is always listed as uncovered.

It seems to be any trailing expression that is just a borrow from self: report

[CAD97]

The item headers for impls in statement position are incorrectly marked as uncovered. Lifting the impl blocks to item position properly does not consider these lines as uncovered. [example]

[mathstuf]

@CAD97 That link is behind a login wall. Not everyone has already given access to codecov.io.

[CAD97]

@mathstuf ah, sorry, I was unaware that that link required a login. (It should be public, the repository in question is public.) (If it was just that the second link was blank, it's because I had forgotten to pin it to a specific commit, which is fixed now.) Here's the linked snippets inline:

+   pub fn builder(&mut self) -> &mut Builder {
-       &mut self.cache
    }

+    pub fn add(&mut self, element: impl Into<NodeOrToken<Arc<Node>, Arc<Token>>) -> &mut Self {
+       self.children.push(element.into());
-       self
    }

+   fn visit_seq<Seq>(self, mut seq: Seq) -> Result<Self::Value, Seq::Error>
    where
        Seq: SeqAccess<'de>,
    {
+       if seq.size_hint().is_some() {
            struct SeqAccessExactSizeIterator<'a, 'de, Seq: SeqAccess<'de>>(
                &'a mut Builder,
                Seq,
                PhantomData<&'de ()>,
            );
-           impl<'de, Seq: SeqAccess<'de>> Iterator for SeqAccessExactSizeIterator<'_, 'de, Seq> {
-               type Item = Result<NodeOrToken<Arc<Node>, Arc<Token>>, Seq::Error>;
+               fn next(&mut self) -> Option<Self::Item> {
+                   self.1.next_element_seed(ElementSeed(self.0)).transpose()
                }

[kierankasha]

I noticed a little bug in a small crate (~100 lines) I'm working on. Comments without a newline before them are getting counted towards coverage. Adding a newline gave me +2.88% coverage. This is the command used:
cargo tarpaulin -v --release --engine ptrace --all-features --out html

This is an excerpt from the html report showing with and without the newline:

Not sure if this has been brought up before.

[clechasseur]

Found this yesterday, not sure it has already been reported, but I couldn't find any reference to it:

The code: https://github.com/clechasseur/mini_exercism/blob/c4d0075a4dad0b6c9d1eb5efdcd10f36d560d7fa/src/api/detail.rs

Resulting coverage:
https://app.codecov.io/gh/clechasseur/mini_exercism/commit/c4d0075a4dad0b6c9d1eb5efdcd10f36d560d7fa/blob/src/api/detail.rs#L117

This is inside a macro_rules! as well as inside a paste! block.

It also comes up when invoking the macro, because the macro forwards the attributes:

The code:
https://github.com/clechasseur/mini_exercism/blob/c4d0075a4dad0b6c9d1eb5efdcd10f36d560d7fa/src/api/v1.rs#L17-L23

Resulting coverage:
https://app.codecov.io/gh/clechasseur/mini_exercism/commit/c4d0075a4dad0b6c9d1eb5efdcd10f36d560d7fa/blob/src/api/v1.rs#L21

Seems like tarpaulin believes the #[derive(Debug)] is a line to be covered.

EDIT: I tried adding a test that actually uses the implementation of Debug being generated and it does solve the issue (e.g. the line is now considered covered). It's weird however because I have lots of other #[derive(Debug)] on custom types not defined through macros that do not need to be covered.

[kemitix]

Fails to record coverage properly when functions are annotated with tracing::instrumentation from tokio's tracing crate.

and

[jplatte]

For tracing, we found that it helps a lot to set RUST_LOG=trace (assuming tracing-subscriber's EnvFilter is used), otherwise expressions inside of event! / trace! / info! and so on don't get evaluated, and tarpaulin correctly shows them as not covered.

[qsantos]

I'm also seeing this behaviour. Also break statements and enum match cases are uncovered.

@eloff It might be a bit late, but was this in generic code by any chance? I have both break statements and match cases not covered in generic code.

@jplatte Is your example from generic code?

I concur with @tokcum and @blacktemplar and I think that all these might have the same root cause. Maybe discussion about this case should be regrouped in #1078.

[jplatte]

I included a link with my example. It's in an impl block with lifetime generics, no type generics are involved.

[qsantos]

I included a link with my example. It's in an impl block with lifetime generics, no type generics are involved.

Sorry, I had missed that. It seems like lifetime generics are enough to trigger the behavior I have observed:

struct S1 { v: Option<i32>, }
fn f1<'a>() {
    let s = S1 { v: Some(0) };
    Box::new(S1 {
        v: s
            .v
            .map(|v| 42),
    });
}
#[test]
fn test1() { f1(); }

struct S2 { u: i32, }
fn f2<'a>() {
    Box::new(S2 {
        u: 0,
    });
}
#[test]
fn test2() { f2(); }

fn f3<'a>() {
    Some(0)
    .map(
        |
        v
        |
        42
    );
}
#[test]
fn test3() { f3(); }

gets me

[Sajjon]

One silly, yet super important gotcha regarding accuracy: make sure you have not added:

[lib] test = false

In your Cargo.toml (and forgot about it). I had accidentally forgot about adding this and it resulted in my code coverage from 90% -> 60% (in a multi-crate-repo).

[Sajjon]

@xd009642 I use UniFFI and it is heavily macro based, and a lot of its macros usages results in Tarpaulin flagging the line where I use the macro as uncovered (they are covered, but I guess Tarpaulin does not know about it)

See https://app.codecov.io/gh/radixdlt/sargon/blob/develop/src%2Fhierarchical_deterministic%2Fbip39%2Fbip39_passphrase.rs

Also when I have declared my own macros often Eq derivation is missed:
https://app.codecov.io/gh/radixdlt/sargon/blob/develop/src%2Fprofile%2Fv100%2Fnetworks%2Fnetwork%2Fauthorized_dapp%2Fshared_with_dapp.rs#L16

[KGrewal1]

seem to get missing coverage when a chain of methods is called: may be because they're being optimised out before the final code? https://app.codecov.io/gh/KGrewal1/optimisers/blob/master/src%2Flbfgs.rs

missing coverage on struct field instantiation on lines 146 definitely doesnt seem correct however?

Also some issues where every line is covered bar a final call to collect https://app.codecov.io/gh/KGrewal1/optimisers/blob/master/src%2Fadam.rs

[tyrone-wu]

Hi 👋 , nice work with this project! 😸

I believe there's missing coverage when using include-tests on #[tokio::test]. One example of it not being covered is here, in which our GHA workflow uses include-tests = true here.
The #[test] are covered fine, it just appears to be #[tokio::test] that's not being included.

Edit: resolved #1503. tyty! 😸

[tyrone-wu]

I went through the codebase and it looks like this line https://github.com/xd009642/tarpaulin/blob/develop/src/source_analysis/items.rs#L106 doesn't return true for #[tokio::test] since it's not considered and indent when number of path segments is greater than 1.

Edit: resolved #1503. tyty! 😸

[AnthonyMichaelTDM]

I'm getting a lot of seemingly random misses in match statements:

CodeCov Link: https://app.codecov.io/gh/AnthonyMichaelTDM/mecomp/commit/0ce9868c0a0eae2458b5202e506c1250e32956b8/blob/one-or-many/src/lib.rs

Screenshot of (some of the) coverage results of interest:

the test(s) that should be resulting in those lines being covered:

#[cfg(test)]
mod tests {
    use super::*;
    use pretty_assertions::{assert_eq, assert_ne};
    use rstest::rstest;

    ...


    #[rstest]
    #[case::none(OneOrMany::<usize>::None, OneOrMany::<usize>::None, Some(std::cmp::Ordering::Equal))]
    #[case::none(OneOrMany::<usize>::None, OneOrMany::One(1), Some(std::cmp::Ordering::Less))]
    #[case::none(OneOrMany::<usize>::None, OneOrMany::Many(vec![1, 2, 3]), Some(std::cmp::Ordering::Less))]
    #[case::one(OneOrMany::One(1), OneOrMany::<usize>::None, Some(std::cmp::Ordering::Greater))]
    #[case::one(OneOrMany::One(1), OneOrMany::One(1), Some(std::cmp::Ordering::Equal))]
    #[case::one(OneOrMany::One(1), OneOrMany::One(2), Some(std::cmp::Ordering::Less))]
    #[case::one(
        OneOrMany::One(1),
        OneOrMany::One(0),
        Some(std::cmp::Ordering::Greater)
    )]
    #[case::one(OneOrMany::One(1), OneOrMany::Many(vec![1, 2, 3]), Some(std::cmp::Ordering::Less))]
    #[case::many(OneOrMany::Many(vec![1, 2, 3]), OneOrMany::<usize>::None, Some(std::cmp::Ordering::Greater))]
    #[case::many(OneOrMany::Many(vec![1, 2, 3]), OneOrMany::One(1), Some(std::cmp::Ordering::Greater))]
    #[case::many(OneOrMany::Many(vec![1, 2, 3]), OneOrMany::Many(vec![1, 2, 3]), Some(std::cmp::Ordering::Equal))]
    #[case::many(OneOrMany::Many(vec![1, 2, 3]), OneOrMany::Many(vec![2, 3]), Some(std::cmp::Ordering::Less))]
    #[case::many(OneOrMany::Many(vec![1, 2, 3]), OneOrMany::Many(vec![1, 2, 3, 4]), Some(std::cmp::Ordering::Less))]
    fn test_partial_cmp<T>(
        #[case] input: OneOrMany<T>,
        #[case] other: OneOrMany<T>,
        #[case] expected: Option<std::cmp::Ordering>,
    ) where
        T: std::fmt::Debug + PartialOrd,
    {
        let actual = input.partial_cmp(&other);
        assert_eq!(actual, expected);
    }

    #[rstest]
    #[case::none(OneOrMany::<usize>::None, OneOrMany::<usize>::None, std::cmp::Ordering::Equal)]
    #[case::none(OneOrMany::<usize>::None, OneOrMany::One(1), std::cmp::Ordering::Less)]
    #[case::none(OneOrMany::<usize>::None, OneOrMany::Many(vec![1, 2, 3]), std::cmp::Ordering::Less)]
    #[case::one(OneOrMany::One(1), OneOrMany::<usize>::None, std::cmp::Ordering::Greater)]
    #[case::one(OneOrMany::One(1), OneOrMany::One(1), std::cmp::Ordering::Equal)]
    #[case::one(OneOrMany::One(1), OneOrMany::One(2), std::cmp::Ordering::Less)]
    #[case::one(OneOrMany::One(1), OneOrMany::One(0), std::cmp::Ordering::Greater)]
    #[case::one(OneOrMany::One(1), OneOrMany::Many(vec![1, 2, 3]), std::cmp::Ordering::Less)]
    #[case::many(OneOrMany::Many(vec![1, 2, 3]), OneOrMany::<usize>::None, std::cmp::Ordering::Greater)]
    #[case::many(OneOrMany::Many(vec![1, 2, 3]), OneOrMany::One(1), std::cmp::Ordering::Greater)]
    #[case::many(OneOrMany::Many(vec![1, 2, 3]), OneOrMany::Many(vec![1, 2, 3]), std::cmp::Ordering::Equal)]
    #[case::many(OneOrMany::Many(vec![1, 2, 3]), OneOrMany::Many(vec![2, 3]), std::cmp::Ordering::Less)]
    #[case::many(OneOrMany::Many(vec![1, 2, 3]), OneOrMany::Many(vec![1, 2, 3, 4]), std::cmp::Ordering::Less)]
    fn test_cmp<T>(
        #[case] input: OneOrMany<T>,
        #[case] other: OneOrMany<T>,
        #[case] expected: std::cmp::Ordering,
    ) where
        T: std::fmt::Debug + Ord,
    {
        let actual = input.cmp(&other);
        assert_eq!(actual, expected);
    }

    ...
}

some additional context:

#[derive(Clone, Debug, PartialEq, Eq, Default)]
pub enum OneOrMany<T> {
    One(T),
    Many(Vec<T>),
    #[default]
    None,
}

[AnthonyMichaelTDM]

huh, changing my compilation profile to use opt-level 0 instead of opt-level 1 seems to have fixed it. nevermind i guess?

[neilskilling-overskilling]

Is there more to getting the tracing event! macro to be covered than setting RUST_LOG=trace?

I have done that and I am not getting the event! macro lines covered.

[xd009642]

You need an actual tracing subscriber registered, I typically use the tracing_test crate to do this.

[neilskilling-overskilling]

I think I was confusing logging and tracing - I switched to a simpler logging interface using env_logger and I achieved all of the coverage that I required. Thanks for the help.

[jendrikw]

I have another instance where an impl line inside a function is reported as not having coverage:

[jendrikw]

Another on struct initialization:

[G-M0N3Y-2503]

It seems that const fns are flagged as not covered if they are only used to initialise constants or statics.

static GLOBAL_ONE_PLUS_ONE_STATIC: u64 = add(1, 1);
const GLOBAL_ONE_PLUS_ONE_CONST: u64 = add(1, 1);

pub const fn add(left: u64, right: u64) -> u64 {
    const _UNUSED: u64 = 70;
    left + right
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn it_works() {
        assert_eq!(GLOBAL_ONE_PLUS_ONE_STATIC, 2);
        assert_eq!(GLOBAL_ONE_PLUS_ONE_CONST, 2);

        static ONE_PLUS_ONE_STATIC: u64 = add(1, 1);
        assert_eq!(ONE_PLUS_ONE_STATIC, 2);

        const ONE_PLUS_ONE_CONST: u64 = add(1, 1);
        assert_eq!(ONE_PLUS_ONE_CONST, 2);
    }
}

|| Uncovered Lines:
|| src/lib.rs: 4, 6
|| Tested/Total Lines:
|| src/lib.rs: 0/2 +0.00%
||
0.00% coverage, 0/2 lines covered, +0.00% change in coverage

appending let _unused = add(1, 1); brings the coverage to 100%

[AnthonyMichaelTDM]

that's probably because the values would be computed at compile time

[xd009642]

Yeah, values at compile time aren't evaluated at test time so not covered. const fn's can be runtime or compile time so I can't just filter them out

[G-M0N3Y-2503]

hmmm, I'd imagine the way it ideally work then is to have the compiler increment the counters.
Otherwise, I'd imagine it would be difficult to filter these constants depending on their call sites.

[mathstuf]

const evaluation happens around MIR time, long before (IIUC) tarpaulin adds any counters to the pipeline (closer to the LLVM-IR stage).

[nigeleke]

False negative on desugared statement.

Https://github.com/xd009642/tarpaulin/issues/1675