Ability to run non-toolstate-dependent tests only

[Manishearth]

Currently if you want to run the full Rust testsuite, you have to run ./x.py test.

However, this fails when e.g. rustfmt is broken. We're hoping to fix that in #82208 , but ideally it would be possible to ask x.py to run all the tests that Rust CI guarantees will pass.

(I may also be missing a config option)

Thoughts?

[Mark-Simulacrum]

It... doesn't fail? Or at least, the intent is that it should succeed, just printing out some failed tests prior to doing so. It may be necessary to configure a toolstate file for this to happen, I'm not sure.

[Mark-Simulacrum]

I think a "only build/test what is guaranteed to work" is likely somewhat duplicating #74709, in some sense.

[ehuss]

If a tool like rustfmt fails to build (for a test), it should be ignored (x.py should exit 0). If the tests fail to run, then it does exit with an error (and I don't think that can be overridden). The checktools script just ignores if x.py itself fails (tool errors are reported in the toolstate).

@Manishearth Is there some scenario you are running into where ./x.py test is failing in a way that seems to be different from CI?

it would be possible to ask x.py to run all the tests that Rust CI guarantees will pass.

There is no simple answer to that question, since there are dozens of configurations and different kinds of builders running different kinds of tests. ./x.py test without arguments is generally a pretty good approximation of the important things. But I also would rarely recommend anyone actually run that command (because it takes a long time, and is some testsuites are dependent on the host being configured properly).

[Manishearth]

@durin42 was hitting this whilst trying to test Rust against LLVM trunk (https://buildkite.com/llvm-project/rust-llvm-integrate-prototype).

I don't have an example of a failing build offhand, our quickfix was to select the testsuites to run manually. I believe the failure was caused by the lack of rustfmt though.

[durin42]

If I do a bare python3 x.py test today, on revision 7506228, I get:

augie% python3 x.py test
    Updating crates.io index
[...]
    Finished release [optimized] target(s) in 5.98s
tidy check
Checking which error codes lack tests...
tidy error: `rust/src/stdarch/examples/hex.rs:198` contains `#[test]`; unit tests and benchmarks must be placed into separate files or directories named `tests.rs`, `benches.rs`, `tests` or `benches`
tidy error: rust/src/stdarch/examples/Cargo.toml doesn't have `edition = "2018"` on a separate line
tidy error: rust/src/stdarch/crates/stdarch-test/Cargo.toml doesn't have `edition = "2018"` on a separate line
tidy error: `rust/src/stdarch/crates/simd-test-macro/src/lib.rs:121` contains `#[test]`; unit tests and benchmarks must be placed into separate files or directories named `tests.rs`, `benches.rs`, `tests` or `benches`

this continues for a long time, and then

Found 515 error codes
Found 0 error codes with no tests
Done!
some tidy checks failed

with an exit code of 1. So I'm not sure it's even rustfmt related, as this appears to be a rust-tidy error.

[ehuss]

Those look like remnants from when stdarch lived in a different location. It should now live in library/stdarch. That was moved back in June. When git submodules are moved, git doesn't delete the old files. If you aren't doing anything with stdarch, then I would recommend just deleting src/stdarch.

[durin42]

Ugh, I can't figure out what I was seeing. Maybe it's been fixed, maybe I misremembered.

[Manishearth]

Gonna close this for now but we can reopen if you see this again @durin42? You can try using ./x.py test again and report things if they break.