Add support for doctests

[sunshowers]

Currently, nextest doesn't support Rust doctests. This is because doctests are not exposed in stable Rust the way regular test binaries are, and are instead treated as special by cargo test.

One nightly-only fix that might work is to collect doctest executables through -Z unstable-options --persist-doctests. However, this isn't a stable approach so it must be used with care.

Note: You can run cargo test --doc as a separate step from cargo nextest run. This will not incur a performance penalty: cargo nextest run && cargo test --doc will not cause any more builds than a plain cargo test that runs doctests.

[xd009642]

So I use the --persist-doctests attribute and can share some of the issues I've faced with it, (hopefully more people using it will lead to stabilisation and improvements). So if we have a doctest with the should_panic attribute cargo test compiles and runs it and checks if the return code is non-zero, and the test passes only if it's non-zero.

However, the doctest file name is made from a normalisation of the path (all path separators replaced with _, a line number and a number to ensure the files are unique. This is because a doctest on the same line in src/foo/bar.rs and src/foo_bar.rs would otherwise generate the same filename. The line numbers are also done after macro expansion so things like include_str can make it harder to match line numbers with attributes.

I made an issue in rust about bundling attribute information into the file name to make it easier for people writing tools using this feature rust-lang/rust#92597 and if you're interested in seeing my code it's in https://github.com/xd009642/tarpaulin in src/cargo.rs, it's reasonably stable the issues mentioned above are the only parts it can fail. I've had this since the nightly flag was created and the naming convention has only changed once so I've only had one completely unexpected issue that affected all users that cause persist-doctests to be added.

Hope this is helpful, nextest looks pretty nifty 🎉

[WaffleLapkin]

There are also --test-builder/--runtool unstable flags that are used by miri to run doc tests

[nu11ptr]

Can I suggest that until this is added this is called out in the docs in a visible way? I recently was made aware how tricky doctests are to handle, but other users might not be aware. Your crate seems quite nice based on my limited usage, and it would be a shame if they didn't give it a fair chance because they thought it was broken and "skipping tests".

[sunshowers]

@nu11ptr good idea -- I've added a note to the homepage along with expanding the notes on the listing and running pages f4043b1