diff --git a/book.toml b/book.toml index f53ed45f61851..336e5a3695c37 100644 --- a/book.toml +++ b/book.toml @@ -24,3 +24,6 @@ follow-web-links = true exclude = [ "crates\\.io", "gcc\\.godbolt\\.org", "youtube\\.com", "youtu\\.be", "dl\\.acm\\.org", "cs\\.bgu\\.ac\\.il", "www\\.amazon\\.com", "www\\.rustaceans\\.org", "play\\.rust-lang\\.org" ] cache-timeout = 86400 warning-policy = "error" + +[output.html.redirect] +"/compiletest.html" = "tests/compiletest.html" diff --git a/src/SUMMARY.md b/src/SUMMARY.md index fa044bce5a922..3189b93644629 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -15,10 +15,16 @@ - [Documenting Compiler](./building/compiler-documenting.md) - [Rustdoc overview](./rustdoc.md) - [Adding a new target](./building/new-target.md) -- [The compiler testing framework](./tests/intro.md) +- [Testing the compiler](./tests/intro.md) - [Running tests](./tests/running.md) + - [Testing with Docker](./tests/docker.md) + - [Testing with CI](./tests/ci.md) - [Adding new tests](./tests/adding.md) - - [Using `compiletest` commands to control test execution](./compiletest.md) + - [Compiletest](./tests/compiletest.md) + - [UI tests](./tests/ui.md) + - [Test headers](./tests/headers.md) + - [Performance testing](./tests/perf.md) + - [Crater](./tests/crater.md) - [Debugging the Compiler](./compiler-debugging.md) - [Using the tracing/logging instrumentation](./tracing.md) - [Profiling the compiler](./profiling.md) diff --git a/src/compiletest.md b/src/compiletest.md deleted file mode 100644 index 2d34ea04ff71f..0000000000000 --- a/src/compiletest.md +++ /dev/null @@ -1,223 +0,0 @@ -# Using `compiletest` commands to control test execution - -## Introduction - -`compiletest` is the main test harness of the Rust test suite. It allows -test authors to organize large numbers of tests (the Rust compiler has many -thousands), efficient test execution (parallel execution is supported), and -allows the test author to configure behavior and expected results of both -individual and groups of tests. - -`compiletest` tests may check test code for success, for runtime failure, or for -compile-time failure. Tests are typically organized as a Rust source file with -annotations in comments before and/or within the test code, which serve to -direct `compiletest` on if or how to run the test, what behavior to expect, -and more. If you are unfamiliar with the compiler testing framework, -see [this chapter](./tests/intro.md) for additional background. - -The tests themselves are typically (but not always) organized into -"suites" – for example, `incremental`, a folder holding tests that check -incremental compilation behavior, `codegen`, -a folder holding tests that check code generation, and many more. The various -suites are defined in [`src/tools/compiletest/src/common.rs`] in the -`pub enum Mode` declaration. And a good introduction to the different -suites of compiler tests along with details about them can be found in -[Adding new tests](./tests/adding.md). - -## Adding a new test file - -Briefly, simply create your new test in the appropriate location under -[`src/test`]. No registration of test files is necessary as `compiletest` -will scan the [`src/test`] subfolder recursively, and will execute any -Rust source files it finds as tests. -See [Adding new tests](./tests/adding.md) for a complete guide on how to add -new tests. - -## Header Commands - -Source file annotations which appear in comments near the top of the source -file *before* any test code are known as header commands. These commands can -instruct `compiletest` to ignore this test, set expectations on whether it is -expected to succeed at compiling, or what the test's return code is expected to -be. Header commands and inline `//~ ERROR` commands are described more fully -[here](./tests/adding.md#header-commands-configuring-rustc). - -### Adding a new header command - -Header commands are defined in the `TestProps` struct in -[`src/tools/compiletest/src/header.rs`]. At a high level, there are -dozens of test properties defined here, all set to default values in the -`TestProp` struct's `impl` block. Any test can override this default value by -specifying the property in question as header command as a comment (`//`) in -the test source file, before any source code. - -#### Using a header command - -Here is an example, specifying the `must-compile-successfully` header command, -which takes no arguments, followed by the `failure-status` header command, -which takes a single argument (which, in this case is a value of 1). -`failure-status` is instructing `compiletest` to expect a failure status of 1 -(rather than the current Rust default of 101). The header command and -the argument list (if present) are typically separated by a colon: - -```rust,ignore -// must-compile-successfully -// failure-status: 1 - -#![feature(termination_trait)] - -use std::io::{Error, ErrorKind}; - -fn main() -> Result<(), Box> { - Err(Box::new(Error::new(ErrorKind::Other, "returned Box from main()"))) -} -``` - -#### Adding a new header command property - -One would add a new header command if there is a need to define some test -property or behavior on an individual, test-by-test basis. A header command -property serves as the header command's backing store (holds the command's -current value) at runtime. - -To add a new header command property: - - 1. Look for the `pub struct TestProps` declaration in - [`src/tools/compiletest/src/header.rs`] and add the new public property to - the end of the declaration. - 2. Look for the `impl TestProps` implementation block immediately following - the struct declaration and initialize the new property to its default - value. - -#### Adding a new header command parser - -When `compiletest` encounters a test file, it parses the file a line at a time -by calling every parser defined in the `Config` struct's implementation block, -also in [`src/tools/compiletest/src/header.rs`][] (note that the `Config` -struct's declaration block is found in [`src/tools/compiletest/src/common.rs`]). -`TestProps`'s `load_from()` method will try passing the current line of text to -each parser, which, in turn typically checks to see if the line begins with a -particular commented (`//`) header command such as `// must-compile-successfully` -or `// failure-status`. Whitespace after the comment marker is optional. - -Parsers will override a given header command property's default value merely by -being specified in the test file as a header command or by having a parameter -value specified in the test file, depending on the header command. - -Parsers defined in `impl Config` are typically named `parse_` -(note kebab-case `` transformed to snake-case -``). `impl Config` also defines several 'low-level' parsers -which make it simple to parse common patterns like simple presence or not -(`parse_name_directive()`), header-command:parameter(s) -(`parse_name_value_directive()`), optional parsing only if a particular `cfg` -attribute is defined (`has_cfg_prefix()`) and many more. The low-level parsers -are found near the end of the `impl Config` block; be sure to look through them -and their associated parsers immediately above to see how they are used to -avoid writing additional parsing code unnecessarily. - -As a concrete example, here is the implementation for the -`parse_failure_status()` parser, in [`src/tools/compiletest/src/header.rs`]: - -```diff -@@ -232,6 +232,7 @@ pub struct TestProps { - // customized normalization rules - pub normalize_stdout: Vec<(String, String)>, - pub normalize_stderr: Vec<(String, String)>, -+ pub failure_status: i32, - } - - impl TestProps { -@@ -260,6 +261,7 @@ impl TestProps { - run_pass: false, - normalize_stdout: vec![], - normalize_stderr: vec![], -+ failure_status: 101, - } - } - -@@ -383,6 +385,10 @@ impl TestProps { - if let Some(rule) = config.parse_custom_normalization(ln, "normalize-stderr") { - self.normalize_stderr.push(rule); - } -+ -+ if let Some(code) = config.parse_failure_status(ln) { -+ self.failure_status = code; -+ } - }); - - for key in &["RUST_TEST_NOCAPTURE", "RUST_TEST_THREADS"] { -@@ -488,6 +494,13 @@ impl Config { - self.parse_name_directive(line, "pretty-compare-only") - } - -+ fn parse_failure_status(&self, line: &str) -> Option { -+ match self.parse_name_value_directive(line, "failure-status") { -+ Some(code) => code.trim().parse::().ok(), -+ _ => None, -+ } -+ } -``` - -## Implementing the behavior change - -When a test invokes a particular header command, it is expected that some -behavior will change as a result. What behavior, obviously, will depend on the -purpose of the header command. In the case of `failure-status`, the behavior -that changes is that `compiletest` expects the failure code defined by the -header command invoked in the test, rather than the default value. - -Although specific to `failure-status` (as every header command will have a -different implementation in order to invoke behavior change) perhaps it is -helpful to see the behavior change implementation of one case, simply as an -example. To implement `failure-status`, the `check_correct_failure_status()` -function found in the `TestCx` implementation block, located in -[`src/tools/compiletest/src/runtest.rs`], was modified as per below: - -```diff -@@ -295,11 +295,14 @@ impl<'test> TestCx<'test> { - } - - fn check_correct_failure_status(&self, proc_res: &ProcRes) { -- // The value the Rust runtime returns on failure -- const RUST_ERR: i32 = 101; -- if proc_res.status.code() != Some(RUST_ERR) { -+ let expected_status = Some(self.props.failure_status); -+ let received_status = proc_res.status.code(); -+ -+ if expected_status != received_status { - self.fatal_proc_rec( -- &format!("failure produced the wrong error: {}", proc_res.status), -+ &format!("Error: expected failure status ({:?}) but received status {:?}.", -+ expected_status, -+ received_status), - proc_res, - ); - } -@@ -320,7 +323,6 @@ impl<'test> TestCx<'test> { - ); - - let proc_res = self.exec_compiled_test(); -- - if !proc_res.status.success() { - self.fatal_proc_rec("test run failed!", &proc_res); - } -@@ -499,7 +501,6 @@ impl<'test> TestCx<'test> { - expected, - actual - ); -- panic!(); - } - } -``` -Note the use of `self.props.failure_status` to access the header command -property. In tests which do not specify the failure status header command, -`self.props.failure_status` will evaluate to the default value of 101 at the -time of this writing. But for a test which specifies a header command of, for -example, `// failure-status: 1`, `self.props.failure_status` will evaluate to -1, as `parse_failure_status()` will have overridden the `TestProps` default -value, for that test specifically. - -[`src/test`]: https://github.com/rust-lang/rust/tree/master/src/test -[`src/tools/compiletest/src/header.rs`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/header.rs -[`src/tools/compiletest/src/common.rs`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/common.rs -[`src/tools/compiletest/src/runtest.rs`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/runtest.rs diff --git a/src/profiling.md b/src/profiling.md index ae76d977fd82a..0d333f7be88d4 100644 --- a/src/profiling.md +++ b/src/profiling.md @@ -4,11 +4,8 @@ This section talks about how to profile the compiler and find out where it spend Depending on what you're trying to measure, there are several different approaches: -- If you want to see if a PR improves or regresses compiler performance: - - The [rustc-perf](https://github.com/rust-lang/rustc-perf) project makes this easy and can be triggered to run on a PR via the `@rust-timer` bot. - The `@bors try @rust-timer queue` command, in a comment on the PR, will queue a try build and a - benchmarking run. - Note: you need `try` privileges to be able to do this. More details are available in the [perf collector documentation](https://github.com/rust-lang/rustc-perf/blob/master/collector/README.md). +- If you want to see if a PR improves or regresses compiler performance, + see the [rustc-perf chapter](tests/perf.md) for requesting a benchmarking run. - If you want a medium-to-high level overview of where `rustc` is spending its time: - The `-Z self-profile` flag and [measureme](https://github.com/rust-lang/measureme) tools offer a query-based approach to profiling. diff --git a/src/tests/adding.md b/src/tests/adding.md index d7ad2871df49f..2fb1e0b65e0d0 100644 --- a/src/tests/adding.md +++ b/src/tests/adding.md @@ -7,574 +7,189 @@ accompanied by a regression test of some kind.** This test should fail in master but pass after the PR. These tests are really useful for preventing us from repeating the mistakes of the past. -To add a new test, the first thing you generally do is to create a -file, typically a Rust source file. Test files have a particular -structure: - -- They should have some kind of - [comment explaining what the test is about](#explanatory_comment); -- next, they can have one or more [header commands](#header_commands), which - are special comments that the test interpreter knows how to interpret. -- finally, they have the Rust source. This may have various [error - annotations](#error_annotations) which indicate expected compilation errors or - warnings. - -Depending on the test suite, there may be some other details to be aware of: - - For [the `ui` test suite](#ui), you need to generate reference output files. - -## What kind of test should I add? - -It can be difficult to know what kind of test to use. Here are some -rough heuristics: - -- Some tests have specialized needs: - - need to run gdb or lldb? use the `debuginfo` test suite - - need to inspect LLVM IR or MIR IR? use the `codegen` or `mir-opt` test - suites - - need to run rustdoc? Prefer a `rustdoc` or `rustdoc-ui` test. - Occasionally you'll need `rustdoc-js` as well. - - need to inspect the resulting binary in some way? Then use `run-make` -- Library tests should go in `library/${crate}/tests` (where `${crate}` is - usually `core`, `alloc`, or `std`). Library tests include: - - tests that an API behaves properly, including accepting various types or - having some runtime behavior - - tests where any compiler warnings are not relevant to the test - - tests that a use of an API gives a compile error, where the exact error - message is not relevant to the test. These should have an - [error number] (`E0XXX`) in the code block to make sure it's the correct error. -- For most other things, [a `ui` (or `ui-fulldeps`) test](#ui) is to be preferred: - - in the case of warnings or errors, `ui` tests capture the full output, - which makes it easier to review but also helps prevent "hidden" regressions - in the output - -[error number]: https://doc.rust-lang.org/rustdoc/unstable-features.html#error-numbers-for-compile-fail-doctests - -## Naming your test - -We have not traditionally had a lot of structure in the names of -tests. Moreover, for a long time, the rustc test runner did not -support subdirectories (it now does), so test suites like -[`src/test/ui`] have a huge mess of files in them. This is not -considered an ideal setup. - +The first thing to decide is which kind of test to add. +This will depend on the nature of the change and what you want to exercise. +Here are some rough guidelines: + +- The majority of compiler tests are done with [compiletest]. + - The majority of compiletest tests are [UI](ui.md) tests in the [`src/test/ui`] directory. +- Changes to the standard library are usually tested within the standard library itself. + - The majority of standard library tests are written as doctests, + which illustrate and exercise typical API behavior. + - Additional [unit tests](intro.md#package-tests) should go in + `library/${crate}/tests` (where `${crate}` is usually `core`, `alloc`, or `std`). +- If the code is part of an isolated system, and you are not testing compiler output, + consider using a [unit or integration test](intro.md#package-tests). +- Need to run rustdoc? Prefer a `rustdoc` or `rustdoc-ui` test. + Occasionally you'll need `rustdoc-js` as well. +- Other compiletest test suites are generally used for special purposes: + - Need to run gdb or lldb? Use the `debuginfo` test suite. + - Need to inspect LLVM IR or MIR IR? Use the `codegen` or `mir-opt` test suites. + - Need to inspect the resulting binary in some way? + Then use `run-make`. + - Check out the [compiletest] chapter for more specialized test suites. + +[compiletest]: compiletest.md [`src/test/ui`]: https://github.com/rust-lang/rust/tree/master/src/test/ui/ -For regression tests – basically, some random snippet of code that -came in from the internet – we often name the test after the issue -plus a short description. Ideally, the test should be added to a -directory that helps identify what piece of code is being tested here -(e.g., `src/test/ui/borrowck/issue-54597-reject-move-out-of-borrow-via-pat.rs`) -If you've tried and cannot find a more relevant place, -the test may be added to `src/test/ui/issues/`. -Still, **do include the issue number somewhere**. -But please avoid putting your test there as possible since that -directory has too many tests and it causes poor semantic organization. - -When writing a new feature, **create a subdirectory to store your -tests**. For example, if you are implementing RFC 1234 ("Widgets"), -then it might make sense to put the tests in a directory like -`src/test/ui/rfc1234-widgets/`. - -In other cases, there may already be a suitable directory. (The proper -directory structure to use is actually an area of active debate.) - - - -## Comment explaining what the test is about - -When you create a test file, **include a comment summarizing the point -of the test at the start of the file**. This should highlight which -parts of the test are more important, and what the bug was that the -test is fixing. Citing an issue number is often very helpful. - -This comment doesn't have to be super extensive. Just something like -"Regression test for #18060: match arms were matching in the wrong -order." might already be enough. +## UI test walkthrough -These comments are very useful to others later on when your test -breaks, since they often can highlight what the problem is. They are -also useful if for some reason the tests need to be refactored, since -they let others know which parts of the test were important (often a -test must be rewritten because it no longer tests what is was meant to -test, and then it's useful to know what it *was* meant to test -exactly). +The following is a basic guide for creating a [UI test](ui.md), which is one +of the most common compiler tests. +For this tutorial, we'll be adding a test for an async error message. - +### Step 1. Add a test file -## Header commands: configuring rustc +The first step is to create a Rust source file somewhere in the +[`src/test/ui`] tree. +When creating a test, do your best to find a good location and name (see [Test +organization](ui.md#test-organization) for more). +Since naming is the hardest part of development, everything should be downhill +from here! -Header commands are special comments that the test runner knows how to -interpret. They must appear before the Rust source in the test. They -are normally put after the short comment that explains the point of -this test. For example, this test uses the `// compile-flags` command -to specify a custom flag to give to rustc when the test is compiled: +Let's place our async test at `src/test/ui/async-await/await-without-async.rs`: ```rust,ignore -// Test the behavior of `0 - 1` when overflow checks are disabled. +// Check what happens when using await in a non-async fn. +// edition:2018 -// compile-flags: -C overflow-checks=off +async fn foo() {} -fn main() { - let x = 0 - 1; - ... +fn bar() { + foo().await } -``` -### Ignoring tests - -These are used to ignore the test in some situations, which means the test won't -be compiled or run. - -* `ignore-X` where `X` is a target detail or stage will ignore the - test accordingly (see below) -* `only-X` is like `ignore-X`, but will *only* run the test on that - target or stage -* `ignore-pretty` will not compile the pretty-printed test (this is - done to test the pretty-printer, but might not always work) -* `ignore-test` always ignores the test -* `ignore-lldb` and `ignore-gdb` will skip a debuginfo test on that - debugger. -* `ignore-gdb-version` can be used to ignore the test when certain gdb - versions are used - -Some examples of `X` in `ignore-X`: - -* Architecture: `aarch64`, `arm`, `asmjs`, `mips`, `wasm32`, `x86_64`, - `x86`, ... -* OS: `android`, `emscripten`, `freebsd`, `ios`, `linux`, `macos`, - `windows`, ... -* Environment (fourth word of the target triple): `gnu`, `msvc`, - `musl`. -* Pointer width: `32bit`, `64bit`. -* Stage: `stage0`, `stage1`, `stage2`. -* When cross compiling: `cross-compile` -* When remote testing is used: `remote` -* When debug-assertions are enabled: `debug` -* When particular debuggers are being tested: `cdb`, `gdb`, `lldb` -* Specific compare modes: `compare-mode-nll`, `compare-mode-polonius` - -### Other Header Commands - -Here is a list of other header commands. This list is not -exhaustive. Header commands can generally be found by browsing the -`TestProps` structure found in [`header.rs`] from the compiletest -source. - -* `run-rustfix` for UI tests, indicates that the test produces - structured suggestions. The test writer should create a `.fixed` - file, which contains the source with the suggestions applied. - When the test is run, compiletest first checks that the correct - lint/warning is generated. Then, it applies the suggestion and - compares against `.fixed` (they must match). Finally, the fixed - source is compiled, and this compilation is required to succeed. - The `.fixed` file can also be generated automatically with the - `--bless` option, described in [this section][bless]. -* `rustfix-only-machine-applicable` is equivalent to `run-rustfix` except it - will only apply [`MachineApplicable`](../diagnostics.md#suggestions) - suggestions. `run-rustfix` will apply *all* suggestions. This should be used - if there is a mixture of different suggestion levels, and some of the - non-machine-applicable ones do not apply cleanly. -* `min-gdb-version` specifies the minimum gdb version required for - this test; see also `ignore-gdb-version` -* `min-lldb-version` specifies the minimum lldb version required for - this test -* `rust-lldb` causes the lldb part of the test to only be run if the - lldb in use contains the Rust plugin -* `no-system-llvm` causes the test to be ignored if the system llvm is used -* `min-llvm-version` specifies the minimum llvm version required for - this test -* `min-system-llvm-version` specifies the minimum system llvm version - required for this test; the test is ignored if the system llvm is in - use and it doesn't meet the minimum version. This is useful when an - llvm feature has been backported to rust-llvm -* `ignore-llvm-version` can be used to skip the test when certain LLVM - versions are used. This takes one or two arguments; the first - argument is the first version to ignore. If no second argument is - given, all subsequent versions are ignored; otherwise, the second - argument is the last version to ignore. -* `build-pass` for UI tests, indicates that the test is supposed to - successfully compile and link, as opposed to the default where the test is - supposed to error out. -* `compile-flags` passes extra command-line args to the compiler, - e.g. `compile-flags -g` which forces debuginfo to be enabled. -* `edition` controls the edition the test should be compiled with - (defaults to 2015). Example usage: `// edition:2018`. -* `should-fail` indicates that the test should fail; used for "meta - testing", where we test the compiletest program itself to check that - it will generate errors in appropriate scenarios. This header is - ignored for pretty-printer tests. -* `gate-test-X` where `X` is a feature marks the test as "gate test" - for feature X. Such tests are supposed to ensure that the compiler - errors when usage of a gated feature is attempted without the proper - `#![feature(X)]` tag. Each unstable lang feature is required to - have a gate test. -* `needs-profiler-support` - a profiler runtime is required, i.e., - `profiler = true` in rustc's `config.toml`. -* `needs-sanitizer-support` - a sanitizer runtime is required, i.e., - `sanitizers = true` in rustc's `config.toml`. -* `needs-sanitizer-{address,hwaddress,leak,memory,thread}` - indicates that - test requires a target with a support for AddressSanitizer, hardware-assisted - AddressSanitizer, LeakSanitizer, MemorySanitizer or ThreadSanitizer - respectively. -* `error-pattern` checks the diagnostics just like the `ERROR` annotation - without specifying error line. This is useful when the error doesn't give - any span. -* `incremental` runs the test with the `-C incremental` flag and an empty - incremental directory. This should be avoided when possible; you should use - an *incremental mode* test instead. Incremental mode tests support running - the compiler multiple times and verifying that it can load the generated - incremental cache. This flag is for specialized circumstances, like checking - the interaction of codegen unit partitioning with generating an incremental - cache. -* `aux-build` is used to compile additional crates to link. Just pass it the - name of the source file. The source file should be in a directory called - `auxiliary` beside the test file. The aux crate will be built as a dylib if - possible (unless on a platform that does not support them, or - `no-prefer-dynamic` is specified in the aux file). The `-L` flag is used to - find the extern crates. -* `aux-crate` is very similar to `aux-build`; however, it uses the `--extern` - flag to link to the extern crate. That allows you to specify the additional - syntax of the `--extern` flag, such as renaming a dependency. For example, - `// aux-crate:foo=bar.rs` will compile `auxiliary/bar.rs` and make it - available under then name `foo` within the test. This is similar to how - Cargo does dependency renaming. -* `no-prefer-dynamic` will force an auxiliary crate to be built as an rlib - instead of a dylib. When specified in a test, it will remove the use of `-C - prefer-dynamic`. This can be useful in a variety of circumstances. For - example, it can prevent a proc-macro from being built with the wrong crate - type. Or if your test is specifically targeting behavior of other crate - types, it can be used to prevent building with the wrong crate type. -* `force-host` will force the test to build for the host platform instead of - the target. This is useful primarily for auxiliary proc-macros, which need - to be loaded by the host compiler. -* `pretty-mode` specifies the mode pretty-print tests should run in. - The default is `normal` if not specified. -* `pretty-compare-only` causes a pretty test to only compare the - pretty-printed output. It will not try to compile the expanded output to - typecheck it. This is needed for a pretty-mode that does not expand to valid - Rust, or for other situations where the expanded output cannot be compiled. -* `pretty-expanded` allows a pretty test to also run with - `-Zunpretty=expanded` as a final step. It will also try to compile the - resulting output (without codegen). This is needed because not all code can - be compiled after being expanded. Pretty tests should specify this if they - can. An example where this cannot be used is if the test includes - `println!`. That macro expands to reference private internal functions of - the standard library that cannot be called directly without the - `fmt_internals` feature gate. - - More history about this may be found in [#23616]. -* `pp-exact` is used to ensure a pretty-print test results in specific output. - If specified without a value, then it means the pretty-print output should - match the original source. If specified with a value, as in `// - pp-exact:foo.pp`, it will ensure that the pretty-printed output matches the - contents of the given file. Otherwise, if `pp-exact` is not specified, then - the pretty-printed output will be pretty-printed one more time, and the - output of the two pretty-printing rounds will be compared to ensure that the - pretty-printed output converges to a steady state. - -[`header.rs`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/header.rs -[bless]: ./running.md#editing-and-updating-the-reference-files -[#23616]: https://github.com/rust-lang/rust/issues/23616#issuecomment-484999901 - - - -## Error annotations - -Error annotations specify the errors that the compiler is expected to -emit. They are "attached" to the line in source where the error is -located. Error annotations are considered during tidy lints of line -length and should be formatted according to tidy requirements. You may -use an error message prefix sub-string if necessary to meet line length -requirements. Make sure that the text is long enough for the error -message to be self-documenting. - -The error annotation definition and source line definition association -is defined with the following set of idioms: - -* `~`: Associates the following error level and message with the - current line -* `~|`: Associates the following error level and message with the same - line as the previous comment -* `~^`: Associates the following error level and message with the - previous error annotation line. Each caret (`^`) that you add adds - a line to this, so `~^^^` is three lines above the error annotation - line. - -### Error annotation examples - -Here are examples of error annotations on different lines of UI test -source. - -#### Positioned on error line - -Use the `//~ ERROR` idiom: - -```rust,ignore -fn main() { - let x = (1, 2, 3); - match x { - (_a, _x @ ..) => {} //~ ERROR `_x @` is not allowed in a tuple - _ => {} - } -} +fn main() {} ``` -#### Positioned below error line - -Use the `//~^` idiom with number of carets in the string to indicate the -number of lines above. In the example below, the error line is four -lines above the error annotation line so four carets are included in -the annotation. - -```rust,ignore -fn main() { - let x = (1, 2, 3); - match x { - (_a, _x @ ..) => {} // <- the error is on this line - _ => {} - } -} -//~^^^^ ERROR `_x @` is not allowed in a tuple +A few things to notice about our test: + +* The top should start with a short comment that [explains what the test is + for](#explanatory_comment). +* The `// edition:2018` comment is called a [header](headers.md) which provides + instructions to compiletest on how to build the test. + Here we need to set the edition for `async` to work (the default is 2015). +* Following that is the source of the test. + Try to keep it succinct and to the point. + This may require some effort if you are trying to minimize an example from a + bug report. +* We end this test with an empty `fn main` function. + This is because the default for UI tests is a `bin` crate-type, + and we don't want the "main not found" error in our test. + Alternatively, you could add `#![crate_type="lib"]`. + +### Step 2. Generate the expected output + +The next step is to create the expected output from the compiler. +This can be done with the `--bless` option: + +```sh +./x.py test src/test/ui/async-await/await-without-async.rs --bless ``` -#### Use same error line as defined on error annotation line above +This will build the compiler (if it hasn't already been built), compile the +test, and place the output of the compiler in a file called +`src/test/ui/async-await/await-without-async.stderr`. -Use the `//~|` idiom to define the same error line as -the error annotation line above: +However, this step will fail! +You should see an error message, something like this: -```rust,ignore -struct Binder(i32, i32, i32); - -fn main() { - let x = Binder(1, 2, 3); - match x { - Binder(_a, _x @ ..) => {} // <- the error is on this line - _ => {} - } -} -//~^^^^ ERROR `_x @` is not allowed in a tuple struct -//~| ERROR this pattern has 1 field, but the corresponding tuple struct has 3 fields [E0023] -``` +> error: /rust/src/test/ui/async-await/await-without-async.rs:7: unexpected +> error: '7:10: 7:16: `await` is only allowed inside `async` functions and +> blocks [E0728]' -#### When error line cannot be specified +### Step 3. Add error annotations -Let's think about this test: +Every error needs to be annotated with a comment in the source with the text +of the error. +In this case, we can add the following comment to our test file: ```rust,ignore -fn main() { - let a: *const [_] = &[1, 2, 3]; - unsafe { - let _b = (*a)[3]; - } +fn bar() { + foo().await +//~^ ERROR `await` is only allowed inside `async` functions and blocks } ``` -We want to ensure this shows "index out of bounds" but we cannot use the `ERROR` annotation -since the error doesn't have any span. Then it's time to use the `error-pattern`: +The `//~^` squiggle caret comment tells compiletest that the error belongs to +the previous line (more on this in the [Error +annotations](ui.md#error-annotations) section). -```rust,ignore -// error-pattern: index out of bounds -fn main() { - let a: *const [_] = &[1, 2, 3]; - unsafe { - let _b = (*a)[3]; - } -} -``` +Save that, and run the test again: -But for strict testing, try to use the `ERROR` annotation as much as possible. +```sh +./x.py test src/test/ui/async-await/await-without-async.rs +``` -#### Error levels +It should now pass, yay! -The error levels that you can have are: +### Step 4. Review the output -1. `ERROR` -2. `WARNING` -3. `NOTE` -4. `HELP` and `SUGGESTION`[^sugg-placement] +Somewhat hand-in-hand with the previous step, you should inspect the `.stderr` +file that was created to see if it looks like how you expect. +If you are adding a new diagnostic message, now would be a good time to +also consider how readable the message looks overall, particularly for +people new to Rust. -[^sugg-placement]: **Note**: `SUGGESTION` must follow immediately after `HELP`. +Our example `src/test/ui/async-await/await-without-async.stderr` file should +look like this: -## Revisions +```text +error[E0728]: `await` is only allowed inside `async` functions and blocks + --> $DIR/await-without-async.rs:7:10 + | +LL | fn bar() { + | --- this is not `async` +LL | foo().await + | ^^^^^^ only allowed inside `async` functions and blocks -Certain classes of tests support "revisions" (as of February 2021, -this includes compile-fail, run-fail, and incremental, though -incremental tests are somewhat different). Revisions allow a single test file to -be used for multiple tests. This is done by adding a special header at the top -of the file: +error: aborting due to previous error -```rust -// revisions: foo bar baz +For more information about this error, try `rustc --explain E0728`. ``` -This will result in the test being compiled (and tested) three times, -once with `--cfg foo`, once with `--cfg bar`, and once with `--cfg -baz`. You can therefore use `#[cfg(foo)]` etc within the test to tweak -each of these results. - -You can also customize headers and expected error messages to a particular -revision. To do this, add `[foo]` (or `bar`, `baz`, etc) after the `//` -comment, like so: +You may notice some things look a little different than the regular +compiler output. +The `$DIR` removes the path information which will differ between systems. +The `LL` values replace the line numbers. +That helps avoid small changes in the source from triggering large diffs. +See the [Normalization](ui.md#normalization) section for more. -```rust -// A flag to pass in only for cfg `foo`: -//[foo]compile-flags: -Z verbose +Around this stage, you may need to iterate over the last few steps a few times +to tweak your test, re-bless the test, and re-review the output. -#[cfg(foo)] -fn test_foo() { - let x: usize = 32_u32; //[foo]~ ERROR mismatched types -} -``` - -Note that not all headers have meaning when customized to a revision. -For example, the `ignore-test` header (and all "ignore" headers) -currently only apply to the test as a whole, not to particular -revisions. The only headers that are intended to really work when -customized to a revision are error patterns and compiler flags. - - - -## Guide to the UI tests - -The UI tests are intended to capture the compiler's complete output, -so that we can test all aspects of the presentation. They work by -compiling a file (e.g., [`ui/hello_world/main.rs`][hw-main]), -capturing the output, and then applying some normalization (see -below). This normalized result is then compared against reference -files named `ui/hello_world/main.stderr` and -`ui/hello_world/main.stdout`. If either of those files doesn't exist, -the output must be empty (that is actually the case for -[this particular test][hw]). If the test run fails, we will print out -the current output, but it is also saved in -`build//test/ui/hello_world/main.stdout` (this path is -printed as part of the test failure message), so you can run `diff` -and so forth. - -[hw-main]: https://github.com/rust-lang/rust/blob/master/src/test/ui/hello_world/main.rs -[hw]: https://github.com/rust-lang/rust/blob/master/src/test/ui/hello_world/ - -We now have a ton of UI tests and some directories have too many entries. -This is a problem because it isn't editor/IDE friendly and GitHub UI won't -show more than 1000 entries. To resolve it and organize semantic structure, -we have a tidy check to ensure the number of entries is less than 1000. -However, since `src/test/ui` (UI test root directory) and -`src/test/ui/issues` directories have more than 1000 entries, -we set a different limit for each directories. So, please -avoid putting a new test there and try to find a more relevant place. -For example, if your test is related to closures, you should put it in -`src/test/ui/closures`. If you're not sure where is the best place, -it's still okay to add to `src/test/ui/issues/`. When you reach the limit, -you could increase it by tweaking [here][ui test tidy]. - -[ui test tidy]: https://github.com/rust-lang/rust/blob/master/src/tools/tidy/src/ui_tests.rs - -### Tests that do not result in compile errors - -By default, a UI test is expected **not to compile** (in which case, -it should contain at least one `//~ ERROR` annotation). However, you -can also make UI tests where compilation is expected to succeed, and -you can even run the resulting program. Just add one of the following -[header commands](#header_commands): - -- `// check-pass` - compilation should succeed but skip codegen - (which is expensive and isn't supposed to fail in most cases) -- `// build-pass` – compilation and linking should succeed but do - not run the resulting binary -- `// run-pass` – compilation should succeed and we should run the - resulting binary - -### Output Normalization - -The compiler output is normalized to eliminate output difference between -platforms, mainly about filenames. - -The following strings replace their corresponding values: - -- `$DIR`: The directory where the test is defined. - - Example: `/path/to/rust/src/test/ui/error-codes` -- `$SRC_DIR`: The root source directory. - - Example: `/path/to/rust/src` -- `$TEST_BUILD_DIR`: The base directory where the test's output goes. - - Example: `/path/to/rust/build/x86_64-unknown-linux-gnu/test/ui` - -Additionally, the following changes are made: - -- Line and column numbers for paths in `$SRC_DIR` are replaced with `LL:CC`. - For example, `/path/to/rust/library/core/src/clone.rs:122:8` is replaced with - `$SRC_DIR/core/src/clone.rs:LL:COL`. - - Note: The line and column numbers for `-->` lines pointing to the test are - *not* normalized, and left as-is. This ensures that the compiler continues - to point to the correct location, and keeps the stderr files readable. - Ideally all line/column information would be retained, but small changes to - the source causes large diffs, and more frequent merge conflicts and test - errors. See also `-Z ui-testing` below which applies additional line number - normalization. -- `\t` is replaced with an actual tab character. -- Error line annotations like `// ~ERROR some message` are removed. -- Backslashes (`\`) are converted to forward slashes (`/`) within paths (using - a heuristic). This helps normalize differences with Windows-style paths. -- CRLF newlines are converted to LF. - -Additionally, the compiler is run with the `-Z ui-testing` flag which causes -the compiler itself to apply some changes to the diagnostic output to make it -more suitable for UI testing. For example, it will anonymize line numbers in -the output (line numbers prefixing each source line are replaced with `LL`). -In extremely rare situations, this mode can be disabled with the header -command `// compile-flags: -Z ui-testing=no`. - -Sometimes these built-in normalizations are not enough. In such cases, you -may provide custom normalization rules using the header commands, e.g. - -```rust -// normalize-stdout-test: "foo" -> "bar" -// normalize-stderr-32bit: "fn\(\) \(32 bits\)" -> "fn\(\) \($$PTR bits\)" -// normalize-stderr-64bit: "fn\(\) \(64 bits\)" -> "fn\(\) \($$PTR bits\)" -``` +### Step 5. Check other tests -This tells the test, on 32-bit platforms, whenever the compiler writes -`fn() (32 bits)` to stderr, it should be normalized to read `fn() ($PTR bits)` -instead. Similar for 64-bit. The replacement is performed by regexes using -default regex flavor provided by `regex` crate. +Sometimes when adding or changing a diagnostic message, this will affect +other tests in the test suite. +The final step before posting a PR is to check if you have affected anything else. +Running the UI suite is usually a good start: -The corresponding reference file will use the normalized output to test both -32-bit and 64-bit platforms: - -```text -... - | - = note: source type: fn() ($PTR bits) - = note: target type: u16 (16 bits) -... +```sh +./x.py test src/test/ui ``` -Please see [`ui/transmute/main.rs`][mrs] and [`main.stderr`][] for a -concrete usage example. +If other tests start failing, you may need to investigate what has changed +and if the new output makes sense. +You may also need to re-bless the output with the `--bless` flag. -[mrs]: https://github.com/rust-lang/rust/blob/master/src/test/ui/transmute/main.rs -[`main.stderr`]: https://github.com/rust-lang/rust/blob/master/src/test/ui/transmute/main.stderr - -Besides `normalize-stderr-32bit` and `-64bit`, one may use any target -information or stage supported by [`ignore-X`](#ignoring-tests) here as well (e.g. -`normalize-stderr-windows` or simply `normalize-stderr-test` for unconditional -replacement). + -## Input Normalization +## Comment explaining what the test is about -Sometimes, you want to normalize the inputs to a test. For example, you may -want to pass `// compile-flags: --x=y.rs`, where y.rs is some file in the test -directory. In this case you can use input normalization. The following strings -are replaced in header inputs: +The first comment of a test file should **summarize the point +of the test**, and highlight what is important about it. +If there is an issue number associated with the test, include +the issue number. -- {{cwd}}: The directory where compiletest is run from. This may not be the - root of the checkout, so you should avoid using it where possible. - - Examples: `/path/to/rust`, `/path/to/build/root` -- {{src-base}}: The directory where the test is defined. This is equivalent to - `$DIR` for output normalization. - - Example: `/path/to/rust/src/test/ui/error-codes` -- {{build-base}}: The base directory where the test's output goes. This is - equivalent to `$TEST_BUILD_DIR` for output normalization. - - Example: `/path/to/rust/build/x86_64-unknown-linux-gnu/test/ui` +This comment doesn't have to be super extensive. Just something like +"Regression test for #18060: match arms were matching in the wrong +order." might already be enough. -See [`src/test/ui/commandline-argfile.rs`](https://github.com/rust-lang/rust/blob/a5029ac0ab372aec515db2e718da6d7787f3d122/src/test/ui/commandline-argfile.rs) -for an example of a test that uses input normalization. +These comments are very useful to others later on when your test +breaks, since they often can highlight what the problem is. They are +also useful if for some reason the tests need to be refactored, since +they let others know which parts of the test were important (often a +test must be rewritten because it no longer tests what is was meant to +test, and then it's useful to know what it *was* meant to test +exactly). diff --git a/src/tests/ci.md b/src/tests/ci.md new file mode 100644 index 0000000000000..21f8341306df6 --- /dev/null +++ b/src/tests/ci.md @@ -0,0 +1,73 @@ +# Testing with CI + +## Testing infrastructure + +When a Pull Request is opened on GitHub, [GitHub Actions] will automatically +launch a build that will run all tests on some configurations +(x86_64-gnu-llvm-12 linux. x86_64-gnu-tools linux, mingw-check linux). +In essence, each runs `./x.py test` with various different options. + +The integration bot [bors] is used for coordinating merges to the master branch. +When a PR is approved, it goes into a [queue] where merges are tested one at a +time on a wide set of platforms using GitHub Actions. Due to the limit on the +number of parallel jobs, we run CI under the [rust-lang-ci] organization except +for PRs. +Most platforms only run the build steps, some run a restricted set of tests, +only a subset run the full suite of tests (see Rust's [platform tiers]). + +If everything passes, then all of the distribution artifacts that were +generated during the CI run are published. + +[GitHub Actions]: https://github.com/rust-lang/rust/actions +[rust-lang-ci]: https://github.com/rust-lang-ci/rust/actions +[bors]: https://github.com/servo/homu +[queue]: https://bors.rust-lang.org/queue/rust +[platform tiers]: https://forge.rust-lang.org/release/platform-support.html#rust-platform-support + +## Using CI to test + +In some cases, a PR may run into problems with running tests on a particular +platform or configuration. +If you can't run those tests locally, don't hesitate to use CI resources to +try out a fix. + +As mentioned above, opening or updating a PR will only run on a small subset +of configurations. +Only when a PR is approved will it go through the full set of test configurations. +However, you can try one of those configurations in your PR before it is approved. +For example, if a Windows build fails, but you don't have access to a Windows +machine, you can try running the Windows job that failed on CI within your PR +after pushing a possible fix. + +To do this, you'll need to edit [`src/ci/github-actions/ci.yml`]. +The `jobs` section defines the jobs that will run. +The `jobs.pr` section defines everything that will run in a push to a PR. +The `jobs.auto` section defines the full set of tests that are run after a PR is approved. +You can copy one of the definitions from the `auto` section up to the `pr` section. + +For example, the `x86_64-msvc-1` and `x86_64-msvc-2` jobs are responsible for +running the 64-bit MSVC tests. +You can copy those up to the `jobs.pr.strategy.matrix.include` section with +the other jobs. + +The comment at the top of `ci.yml` will tell you to run this command: + +```sh +./x.py run src/tools/expand-yaml-anchors +```` + +This will generate the true [`.github/workflows/ci.yml`] which is what GitHub +Actions uses. + +Then, you can commit those two files and push to GitHub. +GitHub Actions should launch the tests. + +After you have finished, don't forget to remove any changes you have made to `ci.yml`. + +Although you are welcome to use CI, just be conscientious that this is a shared +resource with limited concurrency. +Try not to enable too many jobs at once (one or two should be sufficient in +most cases). + +[`src/ci/github-actions/ci.yml`]: https://github.com/rust-lang/rust/blob/master/src/ci/github-actions/ci.yml +[`.github/workflows/ci.yml`]: https://github.com/rust-lang/rust/blob/master/.github/workflows/ci.yml#L1 diff --git a/src/tests/compiletest.md b/src/tests/compiletest.md new file mode 100644 index 0000000000000..b54bdfcfd8566 --- /dev/null +++ b/src/tests/compiletest.md @@ -0,0 +1,526 @@ +# Compiletest + + + +## Introduction + +`compiletest` is the main test harness of the Rust test suite. +It allows test authors to organize large numbers of tests +(the Rust compiler has many thousands), +efficient test execution (parallel execution is supported), +and allows the test author to configure behavior and expected results of both +individual and groups of tests. + +`compiletest` may check test code for success, for runtime failure, +or for compile-time failure. +Tests are typically organized as a Rust source file with annotations in +comments before and/or within the test code. +These comments serve to direct `compiletest` on if or how to run the test, +what behavior to expect, and more. +See [header commands](headers.md) and the test suite documentation below +for more details on these annotations. + +See the [Adding new tests](adding.md) chapter for a tutorial on creating a new +test, and the [Running tests](running.md) chapter on how to run the test +suite. + +## Test suites + +All of the tests are in the [`src/test`] directory. +The tests are organized into "suites", with each suite in a separate subdirectory. +Each test suite behaves a little differently, with different compiler behavior +and different checks for correctness. +For example, the [`src/test/incremental`] directory contains tests for +incremental compilation. +The various suites are defined in [`src/tools/compiletest/src/common.rs`] in +the `pub enum Mode` declaration. + +The following test suites are available, with links for more information: + +- [`ui`](ui.md) — tests that check the stdout/stderr from the compilation + and/or running the resulting executable +- `ui-fulldeps` — `ui` tests which require a linkable build of `rustc` (such + as using `extern crate rustc_span;` or used as a plugin) +- [`pretty`](#pretty-printer-tests) — tests for pretty printing +- [`incremental`](#incremental-tests) — tests incremental compilation behavior +- [`debuginfo`](#debuginfo-tests) — tests for debuginfo generation running debuggers +- [`codegen`](#codegen-tests) — tests for code generation +- [`codegen-units`](#codegen-units-tests) — tests for codegen unit partitioning +- [`assembly`](#assembly-tests) — verifies assembly output +- [`mir-opt`](#mir-opt-tests) — tests for MIR generation +- [`run-make`](#run-make-tests) — general purpose tests using a Makefile +- `run-make-fulldeps` — `run-make` tests which require a linkable build of `rustc`, + or the rust demangler +- [`run-pass-valgrind`](#valgrind-tests) — tests run with Valgrind +- Rustdoc tests: + - `rustdoc` — tests for rustdoc, making sure that the generated files + contain the expected documentation. + - `rustdoc-gui` — TODO + - `rustdoc-js` — TODO + - `rustdoc-js-std` — TODO + - `rustdoc-json` — TODO + - `rustdoc-ui` — TODO + +[`src/test`]: https://github.com/rust-lang/rust/blob/master/src/test +[`src/tools/compiletest/src/common.rs`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/common.rs + +### Pretty-printer tests + +The tests in [`src/test/pretty`] exercise the "pretty-printing" functionality of `rustc`. +The `-Z unpretty` CLI option for `rustc` causes it to translate the input source +into various different formats, such as the Rust source after macro expansion. + +The pretty-printer tests have several [header commands](headers.md) described below. +These commands can significantly change the behavior of the test, but the +default behavior without any commands is to: + +1. Run `rustc -Zunpretty=normal` on the source file +2. Run `rustc -Zunpretty=normal` on the output of the previous step +3. The output of the previous two steps should be the same. +4. Run `rustc -Zno-codegen` on the output to make sure that it can type check + (this is similar to running `cargo check`) + +If any of the commands above fail, then the test fails. + +The header commands for pretty-printing tests are: + +* `pretty-mode` specifies the mode pretty-print tests should run in + (that is, the argument to `-Zunpretty`). + The default is `normal` if not specified. +* `pretty-compare-only` causes a pretty test to only compare the pretty-printed output + (stopping after step 3 from above). + It will not try to compile the expanded output to type check it. + This is needed for a pretty-mode that does not expand to valid + Rust, or for other situations where the expanded output cannot be compiled. +* `pretty-expanded` allows a pretty test to also check that the expanded + output can be type checked. + That is, after the steps above, it does two more steps: + + > 5. Run `rustc -Zunpretty=expanded` on the original source + > 6. Run `rustc -Zno-codegen` on the expanded output to make sure that it can type check + + This is needed because not all code can be compiled after being expanded. + Pretty tests should specify this if they can. + An example where this cannot be used is if the test includes `println!`. + That macro expands to reference private internal functions of the standard + library that cannot be called directly without the `fmt_internals` feature + gate. + + More history about this may be found in + [#23616](https://github.com/rust-lang/rust/issues/23616#issuecomment-484999901). +* `pp-exact` is used to ensure a pretty-print test results in specific output. + If specified without a value, then it means the pretty-print output should + match the original source. + If specified with a value, as in `// pp-exact:foo.pp`, + it will ensure that the pretty-printed output matches the contents of the given file. + Otherwise, if `pp-exact` is not specified, then the pretty-printed output + will be pretty-printed one more time, and the output of the two + pretty-printing rounds will be compared to ensure that the pretty-printed + output converges to a steady state. + +[`src/test/pretty`]: https://github.com/rust-lang/rust/tree/master/src/test/pretty + +### Incremental tests + +The tests in [`src/test/incremental`] exercise incremental compilation. +They use [revision headers](#revisions) to tell compiletest to run the +compiler in a series of steps. +Compiletest starts with an empty directory with the `-C incremental` flag, and +then runs the compiler for each revision, reusing the incremental results from +previous steps. +The revisions should start with: + +* `rpass` — the test should compile and run successfully +* `rfail` — the test should compile successfully, but the executable should fail to run +* `cfail` — the test should fail to compile + +To make the revisions unique, you should add a suffix like `rpass1` and `rpass2`. + +To simulate changing the source, compiletest also passes a `--cfg` flag with +the current revision name. +For example, this will run twice, simulating changing a function: + +```rust,ignore +// revisions: rpass1 rpass2 + +#[cfg(rpass1)] +fn foo() { + println!("one"); +} + +#[cfg(rpass2)] +fn foo() { + println!("two"); +} + +fn main() { foo(); } +``` + +`cfail` tests support the `forbid-output` header to specify that a certain +substring must not appear anywhere in the compiler output. +This can be useful to ensure certain errors do not appear, but this can be +fragile as error messages change over time, and a test may no longer be +checking the right thing but will still pass. + +`cfail` tests support the `should-ice` header to specify that a test should +cause an Internal Compiler Error (ICE). +This is a highly specialized header to check that the incremental cache +continues to work after an ICE. + +[`src/test/incremental`]: https://github.com/rust-lang/rust/tree/master/src/test/incremental + + +### Debuginfo tests + +The tests in [`src/test/debuginfo`] test debuginfo generation. +They build a program, launch a debugger, and issue commands to the debugger. +A single test can work with cdb, gdb, and lldb. + +Most tests should have the `// compile-flags: -g` header or something similar +to generate the appropriate debuginfo. + +To set a breakpoint on a line, add a `// #break` comment on the line. + +The debuginfo tests consist of a series of debugger commands along with +"check" lines which specify output that is expected from the debugger. + +The commands are comments of the form `// $DEBUGGER-command:$COMMAND` where +`$DEBUGGER` is the debugger being used and `$COMMAND` is the debugger command +to execute. +The debugger values can be: + +* `cdb` +* `gdb` +* `gdbg` — GDB without Rust support (versions older than 7.11) +* `gdbr` — GDB with Rust support +* `lldb` +* `lldbg` — LLDB without Rust support +* `lldbr` — LLDB with Rust support (this no longer exists) + +The command to check the output are of the form `// $DEBUGGER-check:$OUTPUT` +where `$OUTPUT` is the output to expect. + +For example, the following will build the test, start the debugger, set a +breakpoint, launch the program, inspect a value, and check what the debugger +prints: + +```rust,ignore +// compile-flags: -g + +// lldb-command: run +// lldb-command: print foo +// lldb-check: $0 = 123 + +fn main() { + let foo = 123; + b(); // #break +} + +fn b() {} +``` + +The following [header commands](headers.md) are available to disable a +test based on the debugger currently being used: + +* `min-cdb-version: 10.0.18317.1001` — ignores the test if the version of cdb + is below the given version +* `min-gdb-version: 8.2` — ignores the test if the version of gdb is below the + given version +* `ignore-gdb-version: 9.2` — ignores the test if the version of gdb is equal + to the given version +* `ignore-gdb-version: 7.11.90 - 8.0.9` — ignores the test if the version of + gdb is in a range (inclusive) +* `min-lldb-version: 310` — ignores the test if the version of lldb is below + the given version +* `rust-lldb` — ignores the test if lldb is not contain the Rust plugin. + NOTE: The "Rust" version of LLDB doesn't exist anymore, so this will always be ignored. + This should probably be removed. + +[`src/test/debuginfo`]: https://github.com/rust-lang/rust/tree/master/src/test/debuginfo + + +### Codegen tests + +The tests in [`src/test/codegen`] test LLVM code generation. +They compile the test with the `--emit=llvm-ir` flag to emit LLVM IR. +They then run the LLVM [FileCheck] tool. +The test is annotated with various `// CHECK` comments to check the generated code. +See the FileCheck documentation for a tutorial and more information. + +See also the [assembly tests](#assembly-tests) for a similar set of tests. + +[`src/test/codegen`]: https://github.com/rust-lang/rust/tree/master/src/test/codegen +[FileCheck]: https://llvm.org/docs/CommandGuide/FileCheck.html + + +### Assembly tests + +The tests in [`src/test/assembly`] test LLVM assembly output. +They compile the test with the `--emit=asm` flag to emit a `.s` file with the +assembly output. +They then run the LLVM [FileCheck] tool. + +Each test should be annotated with the `// assembly-output:` header +with a value of either `emit-asm` or `ptx-linker` to indicate +the type of assembly output. + +Then, they should be annotated with various `// CHECK` comments to check the +assembly output. +See the FileCheck documentation for a tutorial and more information. + +See also the [codegen tests](#codegen-tests) for a similar set of tests. + +[`src/test/assembly`]: https://github.com/rust-lang/rust/tree/master/src/test/assembly + + +### Codegen-units tests + +The tests in [`src/test/codegen-units`] test the +[monomorphization](../backend/monomorph.md) collector and CGU partitioning. + +These tests work by running `rustc` with a flag to print the result of the +monomorphization collection pass, and then special annotations in the file are +used to compare against that. + +Each test should be annotated with the `// compile-flags:-Zprint-mono-items=VAL` +header with the appropriate VAL to instruct `rustc` to print the +monomorphization information. + +Then, the test should be annotated with comments of the form `//~ MONO_ITEM name` +where `name` is the monomorphized string printed by rustc like `fn ::foo`. + +To check for CGU partitioning, a comment of the form `//~ MONO_ITEM name @@ cgu` +where `cgu` is a space separated list of the CGU names and the linkage +information in brackets. +For example: `//~ MONO_ITEM static function::FOO @@ statics[Internal]` + +[`src/test/codegen-units`]: https://github.com/rust-lang/rust/tree/master/src/test/codegen-units + + +### Mir-opt tests + +The tests in [`src/test/mir-opt`] check parts of the generated MIR to make +sure it is generated correctly and is doing the expected optimizations. +Check out the [MIR Optimizations](../mir/optimizations.md) chapter for more. + +Compiletest will build the test with several flags to dump the MIR output and +set a baseline for optimizations: + +* `-Copt-level=1` +* `-Zdump-mir=all` +* `-Zmir-opt-level=4` +* `-Zvalidate-mir` +* `-Zdump-mir-exclude-pass-number` + +The test should be annotated with `// EMIT_MIR` comments that specify files that +will contain the expected MIR output. +You can use `x.py test --bless` to create the initial expected files. + +There are several forms the `EMIT_MIR` comment can take: + +* `// EMIT_MIR $MIR_PATH.mir` — This will check that the given filename + matches the exact output from the MIR dump. + For example, `my_test.main.SimplifyCfg-elaborate-drops.after.mir` will load + that file from the test directory, and compare it against the dump from + rustc. + + Checking the "after" file (which is after optimization) is useful if you are + interested in the final state after an optimization. + Some rare cases may want to use the "before" file for completeness. + +* `// EMIT_MIR $MIR_PATH.diff` — where `$MIR_PATH` is the filename of the MIR + dump, such as `my_test_name.my_function.EarlyOtherwiseBranch`. + Compiletest will diff the `.before.mir` and `.after.mir` files, and compare + the diff output to the expected `.diff` file from the `EMIT_MIR` comment. + + This is useful if you want to see how an optimization changes the MIR. + +* `// EMIT_MIR $MIR_PATH.dot` or `$MIR_PATH.html` — These are special cases + for other MIR outputs (via `-Z dump-mir-graphviz` and `-Z dump-mir-spanview`) + that will check that the output matches the given file. + +By default 32 bit and 64 bit targets use the same dump files, which can be +problematic in the presence of pointers in constants or other bit width +dependent things. In that case you can add `// EMIT_MIR_FOR_EACH_BIT_WIDTH` to +your test, causing separate files to be generated for 32bit and 64bit systems. + +[`src/test/mir-opt`]: https://github.com/rust-lang/rust/tree/master/src/test/mir-opt + + +### Run-make tests + +The tests in [`src/test/run-make`] are general-purpose tests using Makefiles +which provide the ultimate in flexibility. +These should be used as a last resort. +If possible, you should use one of the other test suites. +If there is some minor feature missing which you need for your test, +consider extending compiletest to add a header command for what you need. +However, sometimes just running a bunch of commands is really what you +need, `run-make` is here to the rescue! + +Each test should be in a separate directory with a `Makefile` indicating the +commands to run. +There is a [`tools.mk`] Makefile which you can include which provides a bunch of +utilities to make it easier to run commands and compare outputs. +Take a look at some of the other tests for some examples on how to get started. + +[`tools.mk`]: https://github.com/rust-lang/rust/blob/master/src/test/run-make-fulldeps/tools.mk +[`src/test/run-make`]: https://github.com/rust-lang/rust/tree/master/src/test/run-make + + +### Valgrind tests + +The tests in [`src/test/run-pass-valgrind`] are for use with [Valgrind]. +These are currently vestigial, as Valgrind is no longer used in CI. +These may be removed in the future. + +[Valgrind]: https://valgrind.org/ +[`src/test/run-pass-valgrind`]: https://github.com/rust-lang/rust/tree/master/src/test/run-pass-valgrind + + +## Building auxiliary crates + +It is common that some tests require additional auxiliary crates to be compiled. +There are two [headers](headers.md) to assist with that: + +* `aux-build` +* `aux-crate` + +`aux-build` will build a separate crate from the named source file. +The source file should be in a directory called `auxiliary` beside the test file. + +```rust,ignore +// aux-build: my-helper.rs + +extern crate my_helper; +// ... You can use my_helper. +``` + +The aux crate will be built as a dylib if possible (unless on a platform that +does not support them, or the `no-prefer-dynamic` header is specified in the +aux file). +The `-L` flag is used to find the extern crates. + +`aux-crate` is very similar to `aux-build`; however, it uses the `--extern` +flag to link to the extern crate. +That allows you to specify the additional syntax of the `--extern` flag, such +as renaming a dependency. +For example, `// aux-crate:foo=bar.rs` will compile `auxiliary/bar.rs` and +make it available under then name `foo` within the test. +This is similar to how Cargo does dependency renaming. + +### Auxiliary proc-macro + +If you want a proc-macro dependency, then there currently is some ceremony +needed. +Place the proc-macro itself in a file like `auxiliary/my-proc-macro.rs` +with the following structure: + +```rust,ignore +// force-host +// no-prefer-dynamic + +#![crate_type = "proc-macro"] + +extern crate proc_macro; +use proc_macro::TokenStream; + +#[proc_macro] +pub fn foo(input: TokenStream) -> TokenStream { + "".parse().unwrap() +} +``` + +The `force-host` is needed because proc-macros are loaded in the host +compiler, and `no-prefer-dynamic` is needed to tell compiletest to not use +`prefer-dynamic` which is not compatible with proc-macros. +The `#![crate_type]` attribute is needed to specify the correct crate-type. + +Then in your test, you can build with with `aux-build`: + +```rust,ignore +// aux-build: my-proc-macro.rs + +extern crate my_proc_macro; + +fn main() { + my_proc_macro::foo!(); +} +``` + + +## Revisions + +Certain classes of tests support "revisions" (as of February 2021, +this includes compile-fail, run-fail, and incremental, though +incremental tests are somewhat different). +Revisions allow a single test file to be used for multiple tests. +This is done by adding a special header at the top of the file: + +```rust,ignore +// revisions: foo bar baz +``` + +This will result in the test being compiled (and tested) three times, +once with `--cfg foo`, once with `--cfg bar`, and once with `--cfg +baz`. +You can therefore use `#[cfg(foo)]` etc within the test to tweak +each of these results. + +You can also customize headers and expected error messages to a particular +revision. To do this, add `[foo]` (or `bar`, `baz`, etc) after the `//` +comment, like so: + +```rust,ignore +// A flag to pass in only for cfg `foo`: +//[foo]compile-flags: -Z verbose + +#[cfg(foo)] +fn test_foo() { + let x: usize = 32_u32; //[foo]~ ERROR mismatched types +} +``` + +Note that not all headers have meaning when customized to a revision. +For example, the `ignore-test` header (and all "ignore" headers) +currently only apply to the test as a whole, not to particular +revisions. The only headers that are intended to really work when +customized to a revision are error patterns and compiler flags. + + +## Compare modes + +Compiletest can be run in different modes, called *compare modes*, which can +be used to compare the behavior of all tests with different compiler flags +enabled. +This can help highlight what differences might appear with certain flags, and +check for any problems that might arise. + +To run the tests in a different mode, you need to pass the `--compare-mode` +CLI flag: + +```bash +./x.py test src/test/ui --compare-mode=nll +``` + +The possible compare modes are: + +* `nll` — Runs in the "true" NLL mode with `-Zborrowck=mir`. + See [UI compare modes](ui.md#compare-modes) for more. +* `polonius` — Runs with Polonius with `-Zpolonius -Zborrowck=mir`, and reuses + the `nll` stderr files. +* `chalk` — Runs with Chalk with `-Zchalk`. +* `split-dwarf` — Runs with unpacked split-DWARF with `-Csplit-debuginfo=unpacked`. +* `split-dwarf-single` — Runs with packed split-DWARF with `-Csplit-debuginfo=packed`. + +In CI, compare modes are only used in one Linux builder, and only with the +following settings: + +* `src/test/ui`: Uses `nll` mode. +* `src/test/debuginfo`: Uses `split-dwarf` mode. + This helps ensure that none of the debuginfo tests are affected when + enabling split-DWARF. + +Note that compare modes are separate to [revisions](#revisions). +All revisions are tested when running `./x.py test src/test/ui`, however +compare-modes must be manually run individually via the `--compare-mode` flag. diff --git a/src/tests/crater.md b/src/tests/crater.md new file mode 100644 index 0000000000000..9a7ff38715b5c --- /dev/null +++ b/src/tests/crater.md @@ -0,0 +1,45 @@ +# Crater + +[Crater](https://github.com/rust-lang/crater) is a tool for compiling +and running tests for _every_ crate on [crates.io](https://crates.io) (and a +few on GitHub). It is mainly used for checking the extent of breakage when +implementing potentially breaking changes and ensuring lack of breakage by +running beta vs stable compiler versions. + +## When to run Crater + +You should request a crater run if your PR makes large changes to the compiler +or could cause breakage. If you are unsure, feel free to ask your PR's reviewer. + +## Requesting Crater Runs + +The rust team maintains a few machines that can be used for running crater runs +on the changes introduced by a PR. If your PR needs a crater run, leave a +comment for the triage team in the PR thread. Please inform the team whether +you require a "check-only" crater run, a "build only" crater run, or a +"build-and-test" crater run. The difference is primarily in time; the +conservative (if you're not sure) option is to go for the build-and-test run. +If making changes that will only have an effect at compile-time (e.g., +implementing a new trait) then you only need a check run. + +Your PR will be enqueued by the triage team and the results will be posted when +they are ready. Check runs will take around ~3-4 days, with the other two +taking 5-6 days on average. + +While crater is really useful, it is also important to be aware of a few +caveats: + +- Not all code is on crates.io! There is a lot of code in repos on GitHub and + elsewhere. Also, companies may not wish to publish their code. Thus, a + successful crater run is not a magically green light that there will be no + breakage; you still need to be careful. + +- Crater only runs Linux builds on x86_64. Thus, other architectures and + platforms are not tested. Critically, this includes Windows. + +- Many crates are not tested. This could be for a lot of reasons, including + that the crate doesn't compile any more (e.g. used old nightly features), + has broken or flaky tests, requires network access, or other reasons. + +- Before crater can be run, `@bors try` needs to succeed in building artifacts. + This means that if your code doesn't compile, you cannot run crater. diff --git a/src/tests/docker.md b/src/tests/docker.md new file mode 100644 index 0000000000000..29837dfea4b8b --- /dev/null +++ b/src/tests/docker.md @@ -0,0 +1,52 @@ +# Testing with Docker + +The Rust tree includes [Docker] image definitions for the platforms used on +GitHub Actions in [`src/ci/docker`]. +The script [`src/ci/docker/run.sh`] is used to build the Docker image, run it, +build Rust within the image, and run the tests. + +You can run these images on your local development machine. This can be +helpful to test environments different from your local system. First you will +need to install Docker on a Linux, Windows, or macOS system (typically Linux +will be much faster than Windows or macOS because the later use virtual +machines to emulate a Linux environment). To enter interactive mode which will +start a bash shell in the container, run `src/ci/docker/run.sh --dev ` +where `` is one of the directory names in `src/ci/docker` (for example +`x86_64-gnu` is a fairly standard Ubuntu environment). + +The docker script will mount your local Rust source tree in read-only mode, +and an `obj` directory in read-write mode. All of the compiler artifacts will +be stored in the `obj` directory. The shell will start out in the `obj` +directory. From there, you can run `../src/ci/run.sh` which will run the build +as defined by the image. + +Alternatively, you can run individual commands to do specific tasks. For +example, you can run `python3 ../x.py test src/test/ui` to just run UI tests. +Note that there is some configuration in the [`src/ci/run.sh`] script that you +may need to recreate. Particularly, set `submodules = false` in your +`config.toml` so that it doesn't attempt to modify the read-only directory. + +Some additional notes about using the Docker images: + +- Some of the std tests require IPv6 support. Docker on Linux seems to have it + disabled by default. Run the commands in [`enable-docker-ipv6.sh`] to enable + IPv6 before creating the container. This only needs to be done once. +- The container will be deleted automatically when you exit the shell, however + the build artifacts persist in the `obj` directory. If you are switching + between different Docker images, the artifacts from previous environments + stored in the `obj` directory may confuse the build system. Sometimes you + will need to delete parts or all of the `obj` directory before building + inside the container. +- The container is bare-bones, with only a minimal set of packages. You may + want to install some things like `apt install less vim`. +- You can open multiple shells in the container. First you need the container + name (a short hash), which is displayed in the shell prompt, or you can run + `docker container ls` outside of the container to list the available + containers. With the container name, run `docker exec -it + /bin/bash` where `` is the container name like `4ba195e95cef`. + +[Docker]: https://www.docker.com/ +[`src/ci/docker`]: https://github.com/rust-lang/rust/tree/master/src/ci/docker +[`src/ci/docker/run.sh`]: https://github.com/rust-lang/rust/blob/master/src/ci/docker/run.sh +[`src/ci/run.sh`]: https://github.com/rust-lang/rust/blob/master/src/ci/run.sh +[`enable-docker-ipv6.sh`]: https://github.com/rust-lang/rust/blob/master/src/ci/scripts/enable-docker-ipv6.sh diff --git a/src/tests/headers.md b/src/tests/headers.md new file mode 100644 index 0000000000000..adc5fa6c9b68e --- /dev/null +++ b/src/tests/headers.md @@ -0,0 +1,403 @@ +# Test headers + + + +Header commands are special comments that tell compiletest how to build and +interpret a test. +They must appear before the Rust source in the test. +They may also appear in Makefiles for [run-make tests](compiletest.md#run-make-tests). + +They are normally put after the short comment that explains the point of this test. +For example, this test uses the `// compile-flags` command to specify a custom +flag to give to rustc when the test is compiled: + +```rust,ignore +// Test the behavior of `0 - 1` when overflow checks are disabled. + +// compile-flags: -C overflow-checks=off + +fn main() { + let x = 0 - 1; + ... +} +``` + +Header commands can be standalone (like `// run-pass`) or take a value (like +`// compile-flags: -C overflow-checks=off`). + +## Header commands + +The following is a list of header commands. +Commands are linked to sections the describe the command in more detail if available. +This list may not be exhaustive. +Header commands can generally be found by browsing the `TestProps` structure +found in [`header.rs`] from the compiletest source. + +[`header.rs`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/header.rs + +* [Controlling pass/fail expectations](ui.md#controlling-passfail-expectations) + * `check-pass` — building (no codegen) should pass + * `build-pass` — building should pass + * `run-pass` — running the test should pass + * `check-fail` — building (no codegen) should fail (the default if no header) + * `build-fail` — building should fail + * `run-fail` — running should fail + * `ignore-pass` — ignores the `--pass` flag + * `check-run-results` — checks run-pass/fail-pass output +* [UI](ui.md) headers + * [`normalize-X`](ui.md#normalization) — normalize compiler output + * [`run-rustfix`](ui.md#rustfix-tests) — checks diagnostic suggestions + * [`rustfix-only-machine-applicable`](ui.md#rustfix-tests) — checks only + machine applicable suggestions + * [`stderr-per-bitwidth`](ui.md#output-comparison) — separate output per bit width + * [`dont-check-compiler-stderr`](ui.md#output-comparison) — don't validate stderr + * [`dont-check-compiler-stdout`](ui.md#output-comparison) — don't validate stdout +* [Building auxiliary crates](compiletest.md#building-auxiliary-crates) + * `aux-build` + * `aux-crate` +* [Pretty-printer](compiletest.md#pretty-printer-tests) headers + * `pretty-compare-only` + * `pretty-expanded` + * `pretty-mode` + * `pp-exact` +* [Ignoring tests](#ignoring-tests) + * `ignore-X` + * `only-X` + * `needs-X` + * `no-system-llvm` + * `min-llvm-versionX` + * `min-system-llvm-version` + * `ignore-llvm-version` + * `ignore-llvm-version` +* [Environment variable headers](#environment-variable-headers) + * `rustc-env` + * `exec-env` + * `unset-rustc-env` +* [Miscellaneous headers](#miscellaneous-headers) + * `compile-flags` — adds compiler flags + * `run-flags` — adds flags to executable tests + * `edition` — sets the edition + * `failure-status` — expected exit code + * `should-fail` — testing compiletest itself + * `gate-test-X` — feature gate testing + * [`error-pattern`](ui.md#error-pattern) — errors not on a line + * `incremental` — incremental tests not in the incremental test-suite + * `no-prefer-dynamic` — don't use `-C prefer-dynamic`, don't build as a dylib + * `force-host` — build only for the host target + * [`revisions`](compiletest.md#revisions) — compile multiple times + * [`forbid-output`](compiletest.md#incremental-tests) — incremental cfail rejects output pattern + * [`should-ice`](compiletest.md#incremental-tests) — incremental cfail should ICE +* [Assembly](compiletest.md#assembly-tests) headers + * `assembly-output` — the type of assembly output to check + + +### Ignoring tests + +These header commands are used to ignore the test in some situations, +which means the test won't be compiled or run. + +* `ignore-X` where `X` is a target detail or stage will ignore the + test accordingly (see below) +* `only-X` is like `ignore-X`, but will *only* run the test on that + target or stage +* `ignore-test` always ignores the test. + This can be used to temporarily disable a test if it is currently not working, + but you want to keep it in tree to re-enable it later. + +Some examples of `X` in `ignore-X` or `only-X`: + +* A full target triple: `aarch64-apple-ios` +* Architecture: `aarch64`, `arm`, `asmjs`, `mips`, `wasm32`, `x86_64`, + `x86`, ... +* OS: `android`, `emscripten`, `freebsd`, `ios`, `linux`, `macos`, + `windows`, ... +* Environment (fourth word of the target triple): `gnu`, `msvc`, + `musl` +* WASM: `wasm32-bare` matches `wasm32-unknown-unknown`. + `emscripten` also matches that target as well as the emscripten targets. +* Pointer width: `32bit`, `64bit` +* Endianness: `endian-big` +* Stage: `stage0`, `stage1`, `stage2` +* Channel: `stable`, `beta` +* When cross compiling: `cross-compile` +* When [remote testing] is used: `remote` +* When debug-assertions are enabled: `debug` +* When particular debuggers are being tested: `cdb`, `gdb`, `lldb` +* Specific [compare modes]: `compare-mode-nll`, `compare-mode-polonius`, + `compare-mode-chalk`, `compare-mode-split-dwarf`, + `compare-mode-split-dwarf-single` + +The following header commands will check rustc build settings and target settings: + +* `needs-asm-support` — ignores if it is running on a target that doesn't have + stable support for `asm!` +* `needs-profiler-support` — ignores if profiler support was not enabled for + the target (`profiler = true` in rustc's `config.toml`) +* `needs-sanitizer-support` — ignores if the sanitizer support was not enabled + for the target (`sanitizers = true` in rustc's `config.toml`) +* `needs-sanitizer-{address,hwaddress,leak,memory,thread}` — ignores + if the corresponding sanitizer is not enabled for the target + (AddressSanitizer, hardware-assisted AddressSanitizer, LeakSanitizer, + MemorySanitizer or ThreadSanitizer respectively) +* `needs-run-enabled` — ignores if it is a test that gets executed, and + running has been disabled. Running tests can be disabled with the `x.py test + --run=never` flag, or running on fuchsia. +* `needs-unwind` — ignores if the target does not support unwinding +* `needs-rust-lld` — ignores if the rust lld support is not enabled + (`rust.lld = true` in `config.toml`) + +The following header commands will check LLVM support: + +* `no-system-llvm` — ignores if the system llvm is used +* `min-llvm-version: 13.0` — ignored if the LLVM version is less than the given value +* `min-system-llvm-version: 12.0` — ignored if using a system LLVM and its + version is less than the given value +* `ignore-llvm-version: 9.0` — ignores a specific LLVM version +* `ignore-llvm-version: 7.0 - 9.9.9` — ignores LLVM versions in a range (inclusive) +* `needs-llvm-components: powerpc` — ignores if the specific LLVM component was not built. + Note: The test will fail on CI if the component does not exist. +* `needs-matching-clang` — ignores if the version of clang does not match the + LLVM version of rustc. + These tests are always ignored unless a special environment variable is set + (which is only done in one CI job). + +See also [Debuginfo tests](compiletest.md#debuginfo-tests) for headers for +ignoring debuggers. + +[remote testing]: running.md#running-tests-on-a-remote-machine +[compare modes]: ui.md#compare-modes + +### Environment variable headers + +The following headers affect environment variables. + +* `rustc-env` is an environment variable to set when running `rustc` of the + form `KEY=VALUE`. +* `exec-env` is an environment variable to set when executing a test of the + form `KEY=VALUE`. +* `unset-rustc-env` specifies an environment variable to unset when running + `rustc`. + +### Miscellaneous headers + +The following headers are generally available, and not specific to particular +test suites. + +* `compile-flags` passes extra command-line args to the compiler, + e.g. `compile-flags -g` which forces debuginfo to be enabled. +* `run-flags` passes extra args to the test if the test is to be executed. +* `edition` controls the edition the test should be compiled with + (defaults to 2015). Example usage: `// edition:2018`. +* `failure-status` specifies the numeric exit code that should be expected for + tests that expect an error. + If this is not set, the default is 1. +* `should-fail` indicates that the test should fail; used for "meta + testing", where we test the compiletest program itself to check that + it will generate errors in appropriate scenarios. This header is + ignored for pretty-printer tests. +* `gate-test-X` where `X` is a feature marks the test as "gate test" + for feature X. + Such tests are supposed to ensure that the compiler errors when usage of a + gated feature is attempted without the proper `#![feature(X)]` tag. + Each unstable lang feature is required to have a gate test. + This header is actually checked by [tidy](intro.md#tidy), it is not checked + by compiletest. +* `error-pattern` checks the diagnostics just like the `ERROR` annotation + without specifying error line. This is useful when the error doesn't give + any span. See [`error-pattern`](ui.md#error-pattern). +* `incremental` runs the test with the `-C incremental` flag and an empty + incremental directory. This should be avoided when possible; you should use + an *incremental mode* test instead. Incremental mode tests support running + the compiler multiple times and verifying that it can load the generated + incremental cache. This flag is for specialized circumstances, like checking + the interaction of codegen unit partitioning with generating an incremental + cache. +* `no-prefer-dynamic` will force an auxiliary crate to be built as an rlib + instead of a dylib. When specified in a test, it will remove the use of `-C + prefer-dynamic`. This can be useful in a variety of circumstances. For + example, it can prevent a proc-macro from being built with the wrong crate + type. Or if your test is specifically targeting behavior of other crate + types, it can be used to prevent building with the wrong crate type. +* `force-host` will force the test to build for the host platform instead of + the target. This is useful primarily for auxiliary proc-macros, which need + to be loaded by the host compiler. + + +## Substitutions + +Headers values support substituting a few variables which will be replaced +with their corresponding value. +For example, if you need to pass a compiler flag with a path to a specific +file, something like the following could work: + +```rust,ignore +// compile-flags: --remap-path-prefix={{src-base}}=/the/src +``` + +Where the sentinel `{{src-base}}` will be replaced with the appropriate path +described below: + +- `{{cwd}}`: The directory where compiletest is run from. This may not be the + root of the checkout, so you should avoid using it where possible. + - Examples: `/path/to/rust`, `/path/to/build/root` +- `{{src-base}}`: The directory where the test is defined. This is equivalent to + `$DIR` for [output normalization]. + - Example: `/path/to/rust/src/test/ui/error-codes` +- `{{build-base}}`: The base directory where the test's output goes. This is + equivalent to `$TEST_BUILD_DIR` for [output normalization]. + - Example: `/path/to/rust/build/x86_64-unknown-linux-gnu/test/ui` + +See [`src/test/ui/commandline-argfile.rs`](https://github.com/rust-lang/rust/blob/a5029ac0ab372aec515db2e718da6d7787f3d122/src/test/ui/commandline-argfile.rs) +for an example of a test that uses this substitution. + +[output normalization]: ui.md#normalization + + +## Adding a new header command + +One would add a new header command if there is a need to define some test +property or behavior on an individual, test-by-test basis. +A header command property serves as the header command's backing store (holds +the command's current value) at runtime. + +To add a new header command property: + + 1. Look for the `pub struct TestProps` declaration in + [`src/tools/compiletest/src/header.rs`] and add the new public property to + the end of the declaration. + 2. Look for the `impl TestProps` implementation block immediately following + the struct declaration and initialize the new property to its default + value. + +### Adding a new header command parser + +When `compiletest` encounters a test file, it parses the file a line at a time +by calling every parser defined in the `Config` struct's implementation block, +also in [`src/tools/compiletest/src/header.rs`] (note that the `Config` +struct's declaration block is found in [`src/tools/compiletest/src/common.rs`]). +`TestProps`'s `load_from()` method will try passing the current line of text to +each parser, which, in turn typically checks to see if the line begins with a +particular commented (`//`) header command such as `// must-compile-successfully` +or `// failure-status`. Whitespace after the comment marker is optional. + +Parsers will override a given header command property's default value merely by +being specified in the test file as a header command or by having a parameter +value specified in the test file, depending on the header command. + +Parsers defined in `impl Config` are typically named `parse_` +(note kebab-case `` transformed to snake-case +``). `impl Config` also defines several 'low-level' parsers +which make it simple to parse common patterns like simple presence or not +(`parse_name_directive()`), header-command:parameter(s) +(`parse_name_value_directive()`), optional parsing only if a particular `cfg` +attribute is defined (`has_cfg_prefix()`) and many more. The low-level parsers +are found near the end of the `impl Config` block; be sure to look through them +and their associated parsers immediately above to see how they are used to +avoid writing additional parsing code unnecessarily. + +As a concrete example, here is the implementation for the +`parse_failure_status()` parser, in [`src/tools/compiletest/src/header.rs`]: + +```diff +@@ -232,6 +232,7 @@ pub struct TestProps { + // customized normalization rules + pub normalize_stdout: Vec<(String, String)>, + pub normalize_stderr: Vec<(String, String)>, ++ pub failure_status: i32, + } + + impl TestProps { +@@ -260,6 +261,7 @@ impl TestProps { + run_pass: false, + normalize_stdout: vec![], + normalize_stderr: vec![], ++ failure_status: 101, + } + } + +@@ -383,6 +385,10 @@ impl TestProps { + if let Some(rule) = config.parse_custom_normalization(ln, "normalize-stderr") { + self.normalize_stderr.push(rule); + } ++ ++ if let Some(code) = config.parse_failure_status(ln) { ++ self.failure_status = code; ++ } + }); + + for key in &["RUST_TEST_NOCAPTURE", "RUST_TEST_THREADS"] { +@@ -488,6 +494,13 @@ impl Config { + self.parse_name_directive(line, "pretty-compare-only") + } + ++ fn parse_failure_status(&self, line: &str) -> Option { ++ match self.parse_name_value_directive(line, "failure-status") { ++ Some(code) => code.trim().parse::().ok(), ++ _ => None, ++ } ++ } +``` + +### Implementing the behavior change + +When a test invokes a particular header command, it is expected that some +behavior will change as a result. What behavior, obviously, will depend on the +purpose of the header command. In the case of `failure-status`, the behavior +that changes is that `compiletest` expects the failure code defined by the +header command invoked in the test, rather than the default value. + +Although specific to `failure-status` (as every header command will have a +different implementation in order to invoke behavior change) perhaps it is +helpful to see the behavior change implementation of one case, simply as an +example. To implement `failure-status`, the `check_correct_failure_status()` +function found in the `TestCx` implementation block, located in +[`src/tools/compiletest/src/runtest.rs`], was modified as per below: + +```diff +@@ -295,11 +295,14 @@ impl<'test> TestCx<'test> { + } + + fn check_correct_failure_status(&self, proc_res: &ProcRes) { +- // The value the Rust runtime returns on failure +- const RUST_ERR: i32 = 101; +- if proc_res.status.code() != Some(RUST_ERR) { ++ let expected_status = Some(self.props.failure_status); ++ let received_status = proc_res.status.code(); ++ ++ if expected_status != received_status { + self.fatal_proc_rec( +- &format!("failure produced the wrong error: {}", proc_res.status), ++ &format!("Error: expected failure status ({:?}) but received status {:?}.", ++ expected_status, ++ received_status), + proc_res, + ); + } +@@ -320,7 +323,6 @@ impl<'test> TestCx<'test> { + ); + + let proc_res = self.exec_compiled_test(); +- + if !proc_res.status.success() { + self.fatal_proc_rec("test run failed!", &proc_res); + } +@@ -499,7 +501,6 @@ impl<'test> TestCx<'test> { + expected, + actual + ); +- panic!(); + } + } +``` + +Note the use of `self.props.failure_status` to access the header command +property. In tests which do not specify the failure status header command, +`self.props.failure_status` will evaluate to the default value of 101 at the +time of this writing. But for a test which specifies a header command of, for +example, `// failure-status: 1`, `self.props.failure_status` will evaluate to +1, as `parse_failure_status()` will have overridden the `TestProps` default +value, for that test specifically. + +[`src/tools/compiletest/src/header.rs`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/header.rs +[`src/tools/compiletest/src/common.rs`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/common.rs +[`src/tools/compiletest/src/runtest.rs`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/runtest.rs diff --git a/src/tests/intro.md b/src/tests/intro.md index efce3a6580e47..65facccbaad1c 100644 --- a/src/tests/intro.md +++ b/src/tests/intro.md @@ -1,350 +1,154 @@ -# The compiler testing framework +# Testing the compiler The Rust project runs a wide variety of different tests, orchestrated by -the build system (`./x.py test`). The main test harness for testing the -compiler itself is a tool called compiletest (located in the -[`src/tools/compiletest`] directory). This section gives a brief -overview of how the testing framework is setup, and then gets into some -of the details on [how to run tests](./running.html) as well as [how to -add new tests](./adding.html). +the build system (`./x.py test`). +This section gives a brief overview of the different testing tools. +Subsequent chapters dive into [running tests](running.md) and [adding new tests](adding.md). -[`src/tools/compiletest`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest +## Kinds of tests -## Compiletest test suites +There are several kinds of tests to exercise things in the Rust distribution. +Almost all of them are driven by `./x.py test`, with some exceptions noted below. -The compiletest tests are located in the tree in the [`src/test`] -directory. Immediately within you will see a series of subdirectories -(e.g. `ui`, `run-make`, and so forth). Each of those directories is -called a **test suite** – they house a group of tests that are run in -a distinct mode. +### Compiletest +The main test harness for testing the compiler itself is a tool called [compiletest]. +It supports running different styles of tests, called *test suites*. +The tests are all located in the [`src/test`] directory. +The [Compiletest chapter][compiletest] goes into detail on how to use this tool. + +> Example: `./x.py test src/test/ui` + +[compiletest]: compiletest.md [`src/test`]: https://github.com/rust-lang/rust/tree/master/src/test -Here is a brief summary of the test suites and what they mean. In some -cases, the test suites are linked to parts of the manual that give more -details. - -- [`ui`](./adding.html#ui) – tests that check the exact - stdout/stderr from compilation and/or running the test -- `run-pass-valgrind` – tests that ought to run with valgrind -- `pretty` – tests targeting the Rust "pretty printer", which - generates valid Rust code from the AST -- `debuginfo` – tests that run in gdb or lldb and query the debug info -- `codegen` – tests that compile and then test the generated LLVM - code to make sure that the optimizations we want are taking effect. - See [LLVM docs](https://llvm.org/docs/CommandGuide/FileCheck.html) for how to - write such tests. -- `codegen-units` – tests for the [monomorphization](../backend/monomorph.md) - collector and CGU partitioning -- `assembly` – similar to `codegen` tests, but verifies assembly output - to make sure LLVM target backend can handle provided code. -- `mir-opt` – tests that check parts of the generated MIR to make - sure we are building things correctly or doing the optimizations we - expect. -- `incremental` – tests for incremental compilation, checking that - when certain modifications are performed, we are able to reuse the - results from previous compilations. -- `run-make` – tests that basically just execute a `Makefile`; the - ultimate in flexibility but quite annoying to write. -- `rustdoc` – tests for rustdoc, making sure that the generated files - contain the expected documentation. -- `*-fulldeps` – same as above, but indicates that the test depends - on things other than `std` (and hence those things must be built) - -## Other Tests - -The Rust build system handles running tests for various other things, -including: - -- **Tidy** – This is a custom tool used for validating source code - style and formatting conventions, such as rejecting long lines. - There is more information in the - [section on coding conventions](../conventions.html#formatting). - - Example: `./x.py test tidy` - -- **Formatting** – Rustfmt is integrated with the build system to enforce - uniform style across the compiler. In the CI, we check that the formatting - is correct. The formatting check is also automatically run by the Tidy tool - mentioned above. - - Example: `./x.py fmt --check` checks formatting and exits with an error if - formatting is needed. - - Example: `./x.py fmt` runs rustfmt on the codebase. - - Example: `./x.py test tidy --bless` does formatting before doing - other tidy checks. - -- **Unit tests** – The Rust standard library and many of the Rust packages - include typical Rust `#[test]` unittests. Under the hood, `x.py` will run - `cargo test` on each package to run all the tests. - - Example: `./x.py test library/std` - -- **Doc tests** – Example code embedded within Rust documentation is executed - via `rustdoc --test`. Examples: - - `./x.py test src/doc` – Runs `rustdoc --test` for all documentation in - `src/doc`. - - `./x.py test --doc library/std` – Runs `rustdoc --test` on the standard - library. - -- **Link checker** – A small tool for verifying `href` links within - documentation. - - Example: `./x.py test src/tools/linkchecker` - -- **Dist check** – This verifies that the source distribution tarball created - by the build system will unpack, build, and run all tests. - - Example: `./x.py test distcheck` - -- **Tool tests** – Packages that are included with Rust have all of their - tests run as well (typically by running `cargo test` within their - directory). This includes things such as cargo, clippy, rustfmt, rls, miri, - bootstrap (testing the Rust build system itself), etc. - -- **Cargo test** – This is a small tool which runs `cargo test` on a few - significant projects (such as `servo`, `ripgrep`, `tokei`, etc.) just to - ensure there aren't any significant regressions. - - Example: `./x.py test src/tools/cargotest` - -## Testing infrastructure - -When a Pull Request is opened on Github, [GitHub Actions] will automatically -launch a build that will run all tests on some configurations -(x86_64-gnu-llvm-8 linux. x86_64-gnu-tools linux, mingw-check linux). In -essence, it runs `./x.py test` after building for each of them. - -The integration bot [bors] is used for coordinating merges to the master branch. -When a PR is approved, it goes into a [queue] where merges are tested one at a -time on a wide set of platforms using GitHub Actions. Due to the limit on the -number of parallel jobs, we run CI under the [rust-lang-ci] organization except -for PRs. Most platforms only run the build steps, some run a restricted set of -tests, only a subset run the full suite of tests (see Rust's [platform tiers]). - -[GitHub Actions]: https://github.com/rust-lang/rust/actions -[rust-lang-ci]: https://github.com/rust-lang-ci/rust/actions -[bors]: https://github.com/servo/homu -[queue]: https://bors.rust-lang.org/queue/rust -[platform tiers]: https://forge.rust-lang.org/release/platform-support.html#rust-platform-support - -## Testing with Docker images - -The Rust tree includes [Docker] image definitions for the platforms used on -GitHub Actions in [`src/ci/docker`]. The script [`src/ci/docker/run.sh`] is used to build -the Docker image, run it, build Rust within the image, and run the tests. - -You can run these images on your local development machine. This can be -helpful to test environments different from your local system. First you will -need to install Docker on a Linux, Windows, or macOS system (typically Linux -will be much faster than Windows or macOS because the later use virtual -machines to emulate a Linux environment). To enter interactive mode which will -start a bash shell in the container, run `src/ci/docker/run.sh --dev ` -where `` is one of the directory names in `src/ci/docker` (for example -`x86_64-gnu` is a fairly standard Ubuntu environment). - -The docker script will mount your local Rust source tree in read-only mode, -and an `obj` directory in read-write mode. All of the compiler artifacts will -be stored in the `obj` directory. The shell will start out in the `obj` -directory. From there, you can run `../src/ci/run.sh` which will run the build -as defined by the image. - -Alternatively, you can run individual commands to do specific tasks. For -example, you can run `python3 ../x.py test src/test/ui` to just run UI tests. -Note that there is some configuration in the [`src/ci/run.sh`] script that you -may need to recreate. Particularly, set `submodules = false` in your -`config.toml` so that it doesn't attempt to modify the read-only directory. - -Some additional notes about using the Docker images: - -- Some of the std tests require IPv6 support. Docker on Linux seems to have it - disabled by default. Run the commands in [`enable-docker-ipv6.sh`] to enable - IPv6 before creating the container. This only needs to be done once. -- The container will be deleted automatically when you exit the shell, however - the build artifacts persist in the `obj` directory. If you are switching - between different Docker images, the artifacts from previous environments - stored in the `obj` directory may confuse the build system. Sometimes you - will need to delete parts or all of the `obj` directory before building - inside the container. -- The container is bare-bones, with only a minimal set of packages. You may - want to install some things like `apt install less vim`. -- You can open multiple shells in the container. First you need the container - name (a short hash), which is displayed in the shell prompt, or you can run - `docker container ls` outside of the container to list the available - containers. With the container name, run `docker exec -it - /bin/bash` where `` is the container name like `4ba195e95cef`. - -[Docker]: https://www.docker.com/ -[`src/ci/docker`]: https://github.com/rust-lang/rust/tree/master/src/ci/docker -[`src/ci/docker/run.sh`]: https://github.com/rust-lang/rust/blob/master/src/ci/docker/run.sh -[`src/ci/run.sh`]: https://github.com/rust-lang/rust/blob/master/src/ci/run.sh -[`enable-docker-ipv6.sh`]: https://github.com/rust-lang/rust/blob/master/src/ci/scripts/enable-docker-ipv6.sh - -## Running tests on a remote machine - -Tests may be run on a remote machine (e.g. to test builds for a different -architecture). This is done using `remote-test-client` on the build machine -to send test programs to `remote-test-server` running on the remote machine. -`remote-test-server` executes the test programs and sends the results back to -the build machine. `remote-test-server` provides *unauthenticated remote code -execution* so be careful where it is used. - -To do this, first build `remote-test-server` for the remote -machine, e.g. for RISC-V -```sh -./x.py build src/tools/remote-test-server --target riscv64gc-unknown-linux-gnu -``` +### Package tests -The binary will be created at -`./build/$HOST_ARCH/stage2-tools/$TARGET_ARCH/release/remote-test-server`. Copy -this over to the remote machine. +The standard library and many of the compiler packages include typical Rust `#[test]` +unit tests, integration tests, and documentation tests. +You can pass a path to `x.py` to almost any package in the `library` or `compiler` directory, +and `x.py` will essentially run `cargo test` on that package. -On the remote machine, run the `remote-test-server` with the `remote` argument -(and optionally `-v` for verbose output). Output should look like this: -```sh -$ ./remote-test-server -v remote -starting test server -listening on 0.0.0.0:12345! -``` +Examples: -You can test if the `remote-test-server` is working by connecting to it and -sending `ping\n`. It should reply `pong`: -```sh -$ nc $REMOTE_IP 12345 -ping -pong -``` +| Command | Description | +|---------|-------------| +| `./x.py test library/std` | Runs tests on `std` | +| `./x.py test library/core` | Runs tests on `core` | +| `./x.py test compiler/rustc_data_structures` | Runs tests on `rustc_data_structures` | -To run tests using the remote runner, set the `TEST_DEVICE_ADDR` environment -variable then use `x.py` as usual. For example, to run `ui` tests for a RISC-V -machine with the IP address `1.2.3.4` use -```sh -export TEST_DEVICE_ADDR="1.2.3.4:12345" -./x.py test src/test/ui --target riscv64gc-unknown-linux-gnu -``` +The standard library relies very heavily on documentation tests to cover its functionality. +However, unit tests and integration tests can also be used as needed. +Almost all of the compiler packages have doctests disabled. -If `remote-test-server` was run with the verbose flag, output on the test machine -may look something like -``` -[...] -run "/tmp/work/test1007/a" -run "/tmp/work/test1008/a" -run "/tmp/work/test1009/a" -run "/tmp/work/test1010/a" -run "/tmp/work/test1011/a" -run "/tmp/work/test1012/a" -run "/tmp/work/test1013/a" -run "/tmp/work/test1014/a" -run "/tmp/work/test1015/a" -run "/tmp/work/test1016/a" -run "/tmp/work/test1017/a" -run "/tmp/work/test1018/a" -[...] +The standard library and compiler always place all unit tests in a separate `tests` file +(this is enforced in [tidy][tidy-unit-tests]). +This approach ensures that when the test file is changed, the crate does not need to be recompiled. +For example: + +```rust,ignore +#[cfg(test)] +mod tests; ``` -Tests are built on the machine running `x.py` not on the remote machine. Tests -which fail to build unexpectedly (or `ui` tests producing incorrect build -output) may fail without ever running on the remote machine. - -## Testing on emulators - -Some platforms are tested via an emulator for architectures that aren't -readily available. For architectures where the standard library is well -supported and the host operating system supports TCP/IP networking, see the -above instructions for testing on a remote machine (in this case the -remote machine is emulated). - -There is also a set of tools for orchestrating running the -tests within the emulator. Platforms such as `arm-android` and -`arm-unknown-linux-gnueabihf` are set up to automatically run the tests under -emulation on GitHub Actions. The following will take a look at how a target's tests -are run under emulation. - -The Docker image for [armhf-gnu] includes [QEMU] to emulate the ARM CPU -architecture. Included in the Rust tree are the tools [remote-test-client] -and [remote-test-server] which are programs for sending test programs and -libraries to the emulator, and running the tests within the emulator, and -reading the results. The Docker image is set up to launch -`remote-test-server` and the build tools use `remote-test-client` to -communicate with the server to coordinate running tests (see -[src/bootstrap/test.rs]). - -> TODO: -> Is there any support for using an iOS emulator? -> -> It's also unclear to me how the wasm or asm.js tests are run. - -[armhf-gnu]: https://github.com/rust-lang/rust/tree/master/src/ci/docker/host-x86_64/armhf-gnu/Dockerfile -[QEMU]: https://www.qemu.org/ -[remote-test-client]: https://github.com/rust-lang/rust/tree/master/src/tools/remote-test-client -[remote-test-server]: https://github.com/rust-lang/rust/tree/master/src/tools/remote-test-server -[src/bootstrap/test.rs]: https://github.com/rust-lang/rust/tree/master/src/bootstrap/test.rs - -## Crater - -[Crater](https://github.com/rust-lang/crater) is a tool for compiling -and running tests for _every_ crate on [crates.io](https://crates.io) (and a -few on GitHub). It is mainly used for checking for extent of breakage when -implementing potentially breaking changes and ensuring lack of breakage by -running beta vs stable compiler versions. - -### When to run Crater - -You should request a crater run if your PR makes large changes to the compiler -or could cause breakage. If you are unsure, feel free to ask your PR's reviewer. - -### Requesting Crater Runs - -The Rust team maintains a few machines that can be used for running crater runs -on the changes introduced by a PR. If your PR needs a crater run, leave a -comment for the triage team in the PR thread. Please inform the team whether -you require a "check-only" crater run, a "build only" crater run, or a -"build-and-test" crater run. The difference is primarily in time; the -conservative (if you're not sure) option is to go for the build-and-test run. -If making changes that will only have an effect at compile-time (e.g., -implementing a new trait) then you only need a check run. - -Your PR will be enqueued by the triage team and the results will be posted when -they are ready. Check runs will take around ~3-4 days, with the other two -taking 5-6 days on average. - -While crater is really useful, it is also important to be aware of a few -caveats: - -- Not all code is on crates.io! There is a lot of code in repos on GitHub and - elsewhere. Also, companies may not wish to publish their code. Thus, a - successful crater run is not a magically green light that there will be no - breakage; you still need to be careful. - -- Crater only runs Linux builds on x86_64. Thus, other architectures and - platforms are not tested. Critically, this includes Windows. - -- Many crates are not tested. This could be for a lot of reasons, including - that the crate doesn't compile any more (e.g. used old nightly features), - has broken or flaky tests, requires network access, or other reasons. - -- Before crater can be run, `@bors try` needs to succeed in building artifacts. - This means that if your code doesn't compile, you cannot run crater. - -## Perf runs - -A lot of work is put into improving the performance of the compiler and -preventing performance regressions. A "perf run" is used to compare the -performance of the compiler in different configurations for a large collection -of popular crates. Different configurations include "fresh builds", builds -with incremental compilation, etc. - -The result of a perf run is a comparison between two versions of the -compiler (by their commit hashes). - -You should request a perf run if your PR may affect performance, especially -if it can affect performance adversely. +If it wasn't done this way, and the tests were placed in the same file as the source, +then changing or adding a test would cause the crate you are working on to be recompiled. +If you were working on something like `core`, +then that would require recompiling the entire standard library, and the entirety of `rustc`. + +`./x.py test` includes some CLI options for controlling the behavior with these tests: + +* `--doc` — Only runs documentation tests in the package. +* `--no-doc` — Run all tests *except* documentation tests. + +[tidy-unit-tests]: https://github.com/rust-lang/rust/blob/master/src/tools/tidy/src/unit_tests.rs + +### Tidy + +Tidy is a custom tool used for validating source code style and formatting conventions, +such as rejecting long lines. +There is more information in the [section on coding conventions](../conventions.md#formatting). + +> Example: `./x.py test tidy` + +### Formatting + +Rustfmt is integrated with the build system to enforce uniform style across the compiler. +The formatting check is automatically run by the Tidy tool mentioned above. + +Examples: + +| Command | Description | +|---------|-------------| +| `./x.py fmt --check` | Checks formatting and exits with an error if formatting is needed. | +| `./x.py fmt` | Runs rustfmt across the entire codebase. | +| `./x.py test tidy --bless` | First runs rustfmt to format the codebase, then runs tidy checks. | + +### Book documentation tests + +All of the books that are published have their own tests, +primarily for validating that the Rust code examples pass. +Under the hood, these are essentially using `rustdoc --test` on the markdown files. +The tests can be run by passing a path to a book to `./x.py test`. + +> Example: `./x.py test src/doc/book` + +### Documentation link checker + +Links across all documentation is validated with a link checker tool. + +> Example: `./x.py test src/tools/linkchecker` + +This requires building all of the documentation, which might take a while. + +### Dist check + +`distcheck` verifies that the source distribution tarball created by the build system +will unpack, build, and run all tests. + +> Example: `./x.py test distcheck` + +### Tool tests + +Packages that are included with Rust have all of their tests run as well. +This includes things such as cargo, clippy, rustfmt, rls, miri, bootstrap +(testing the Rust build system itself), etc. + +Most of the tools are located in the [`src/tools`] directory. +To run the tool's tests, just pass its path to `./x.py test`. + +> Example: `./x.py test src/tools/cargo` + +Usually these tools involve running `cargo test` within the tool's directory. + +In CI, some tools are allowed to fail. +Failures send notifications to the corresponding teams, and is tracked on the [toolstate website]. +More information can be found in the [toolstate documentation]. + +[`src/tools`]: https://github.com/rust-lang/rust/tree/master/src/tools/ +[toolstate documentation]: https://forge.rust-lang.org/infra/toolstate.html +[toolstate website]: https://rust-lang-nursery.github.io/rust-toolstate/ + +### Cargo test + +`cargotest` is a small tool which runs `cargo test` on a few sample projects +(such as `servo`, `ripgrep`, `tokei`, etc.). +This ensures there aren't any significant regressions. + +> Example: `./x.py test src/tools/cargotest` + +### Crater + +Crater is a tool which runs tests on many thousands of public projects. +This tool has its own separate infrastructure for running. +See the [Crater chapter](crater.md) for more details. + +### Performance testing + +A separate infrastructure is used for testing and tracking performance of the compiler. +See the [Performance testing chapter](perf.md) for more details. ## Further reading diff --git a/src/tests/perf.md b/src/tests/perf.md new file mode 100644 index 0000000000000..ac65faff1fb6d --- /dev/null +++ b/src/tests/perf.md @@ -0,0 +1,50 @@ +# Performance testing + +## rustc-perf + +A lot of work is put into improving the performance of the compiler and +preventing performance regressions. +The [rustc-perf](https://github.com/rust-lang/rustc-perf) project provides +several services for testing and tracking performance. +It provides hosted infrastructure for running benchmarks as a service. +At this time, only `x86_64-unknown-linux-gnu` builds are tracked. + +A "perf run" is used to compare the performance of the compiler in different +configurations for a large collection of popular crates. +Different configurations include "fresh builds", builds with incremental compilation, etc. + +The result of a perf run is a comparison between two versions of the compiler +(by their commit hashes). + +### Automatic perf runs + +After every PR is merged, a suite of benchmarks are run against the compiler. +The results are tracked over time on the website. +Any changes are noted in a comment on the PR. + +### Manual perf runs + +Additionally, performance tests can be ran before a PR is merged on an as-needed basis. +You should request a perf run if your PR may affect performance, especially if +it can affect performance adversely. + +To evaluate the performance impact of a PR, write this comment on the PR: + +`@bors try @rust-timer queue` + +> **Note**: Only users authorized to do perf runs are allowed to post this comment. +> Teams that are allowed to use it are tracked in the [Teams +> repository](https://github.com/rust-lang/team) with the `perf = true` value +> in the `[permissions]` section (and bors permissions are also required). +> If you are not on one of those teams, feel free to ask for someone to post +> it for you (either on Zulip or ask the assigned reviewer). + +This will first tell bors to do a "try" build which do a full release build +for `x86_64-unknown-linux-gnu`. +After the build finishes, it will place it in the queue to run the performance +suite against it. +After the performance tests finish, the bot will post a comment on the PR with +a summary and a link to a full report. + +More details are available in the [perf collector +documentation](https://github.com/rust-lang/rustc-perf/blob/master/collector/README.md). diff --git a/src/tests/running.md b/src/tests/running.md index cd8f12ba19412..2e823c0edda5b 100644 --- a/src/tests/running.md +++ b/src/tests/running.md @@ -18,8 +18,9 @@ I think is done, but rarely otherwise. -nmatsakis) The test results are cached and previously successful tests are `ignored` during testing. The stdout/stderr contents as well as a timestamp file for every test can be found under `build/ARCH/test/`. -To force-rerun a test (e.g. in case the test runner fails to notice -a change) you can simply remove the timestamp file. +To force-rerun a test (e.g. in case the test runner fails to notice a change) +you can simply remove the timestamp file, or use the `--force-rerun` CLI +option. Note that some tests require a Python-enabled gdb. You can test if your gdb install supports Python by using the `python` command from @@ -143,6 +144,18 @@ to automatically adjust the `.stderr`, `.stdout` or `.fixed` files of all tests. Of course you can also target just specific tests with the `--test-args your_test_name` flag, just like when running the tests. +## Configuring test running + +There are a few options for running tests: + +* `config.toml` has the `rust.verbose-tests` option. + If `false`, each test will print a single dot (the default). + If `true`, the name of every test will be printed. + This is equivalent to the `--quiet` option in the [Rust test + harness](https://doc.rust-lang.org/rustc/tests/) +* The environment variable `RUST_TEST_THREADS` can be set to the number of + concurrent threads to use for testing. + ## Passing `--pass $mode` Pass UI tests now have three modes, `check-pass`, `build-pass` and @@ -156,9 +169,8 @@ exists in the test file. For example, you can run all the tests in ``` By passing `--pass $mode`, you can reduce the testing time. For each -mode, please see [here][mode]. - -[mode]: ./adding.md#tests-that-do-not-result-in-compile-errors +mode, please see [Controlling pass/fail +expectations](ui.md#controlling-passfail-expectations). ## Using incremental compilation @@ -193,17 +205,7 @@ To run the UI test suite in NLL mode, one would use the following: ./x.py test src/test/ui --compare-mode=nll ``` -The possible compare modes are: - -* nll - currently nll is implemented in migrate mode, this option runs with true nll. -* polonius -* chalk -* split-dwarf -* split-dwarf-single - -Note that compare modes are separate to [revisions](./adding.html#revisions). -All revisions are tested when running `./x.py test src/test/ui`, -however compare-modes must be manually run individually via the `--compare-mode` flag. +See [Compare modes](compiletest.md#compare-modes) for more details. ## Running tests manually @@ -219,3 +221,105 @@ rustc +stage1 src/test/ui/issue-1234.rs This is much faster, but doesn't always work. For example, some tests include directives that specify specific compiler flags, or which rely on other crates, and they may not run the same without those options. + + +## Running tests on a remote machine + +Tests may be run on a remote machine (e.g. to test builds for a different +architecture). This is done using `remote-test-client` on the build machine +to send test programs to `remote-test-server` running on the remote machine. +`remote-test-server` executes the test programs and sends the results back to +the build machine. `remote-test-server` provides *unauthenticated remote code +execution* so be careful where it is used. + +To do this, first build `remote-test-server` for the remote +machine, e.g. for RISC-V +```sh +./x.py build src/tools/remote-test-server --target riscv64gc-unknown-linux-gnu +``` + +The binary will be created at +`./build/$HOST_ARCH/stage2-tools/$TARGET_ARCH/release/remote-test-server`. Copy +this over to the remote machine. + +On the remote machine, run the `remote-test-server` with the `remote` argument +(and optionally `-v` for verbose output). Output should look like this: +```sh +$ ./remote-test-server -v remote +starting test server +listening on 0.0.0.0:12345! +``` + +You can test if the `remote-test-server` is working by connecting to it and +sending `ping\n`. It should reply `pong`: +```sh +$ nc $REMOTE_IP 12345 +ping +pong +``` + +To run tests using the remote runner, set the `TEST_DEVICE_ADDR` environment +variable then use `x.py` as usual. For example, to run `ui` tests for a RISC-V +machine with the IP address `1.2.3.4` use +```sh +export TEST_DEVICE_ADDR="1.2.3.4:12345" +./x.py test src/test/ui --target riscv64gc-unknown-linux-gnu +``` + +If `remote-test-server` was run with the verbose flag, output on the test machine +may look something like +``` +[...] +run "/tmp/work/test1007/a" +run "/tmp/work/test1008/a" +run "/tmp/work/test1009/a" +run "/tmp/work/test1010/a" +run "/tmp/work/test1011/a" +run "/tmp/work/test1012/a" +run "/tmp/work/test1013/a" +run "/tmp/work/test1014/a" +run "/tmp/work/test1015/a" +run "/tmp/work/test1016/a" +run "/tmp/work/test1017/a" +run "/tmp/work/test1018/a" +[...] +``` + +Tests are built on the machine running `x.py` not on the remote machine. Tests +which fail to build unexpectedly (or `ui` tests producing incorrect build +output) may fail without ever running on the remote machine. + +## Testing on emulators + +Some platforms are tested via an emulator for architectures that aren't +readily available. For architectures where the standard library is well +supported and the host operating system supports TCP/IP networking, see the +above instructions for testing on a remote machine (in this case the +remote machine is emulated). + +There is also a set of tools for orchestrating running the +tests within the emulator. Platforms such as `arm-android` and +`arm-unknown-linux-gnueabihf` are set up to automatically run the tests under +emulation on GitHub Actions. The following will take a look at how a target's tests +are run under emulation. + +The Docker image for [armhf-gnu] includes [QEMU] to emulate the ARM CPU +architecture. Included in the Rust tree are the tools [remote-test-client] +and [remote-test-server] which are programs for sending test programs and +libraries to the emulator, and running the tests within the emulator, and +reading the results. The Docker image is set up to launch +`remote-test-server` and the build tools use `remote-test-client` to +communicate with the server to coordinate running tests (see +[src/bootstrap/test.rs]). + +> TODO: +> Is there any support for using an iOS emulator? +> +> It's also unclear to me how the wasm or asm.js tests are run. + +[armhf-gnu]: https://github.com/rust-lang/rust/tree/master/src/ci/docker/host-x86_64/armhf-gnu/Dockerfile +[QEMU]: https://www.qemu.org/ +[remote-test-client]: https://github.com/rust-lang/rust/tree/master/src/tools/remote-test-client +[remote-test-server]: https://github.com/rust-lang/rust/tree/master/src/tools/remote-test-server +[src/bootstrap/test.rs]: https://github.com/rust-lang/rust/tree/master/src/bootstrap/test.rs + diff --git a/src/tests/ui.md b/src/tests/ui.md new file mode 100644 index 0000000000000..a8e8d3b7209a2 --- /dev/null +++ b/src/tests/ui.md @@ -0,0 +1,500 @@ +# UI tests + + + +UI tests are a particular [test suite](compiletest.md#test-suites) of compiletest. + +## Introduction + +The tests in [`src/test/ui`] are a collection of general-purpose tests which +primarily focus on validating the console output of the compiler, but can be +used for many other purposes. +For example, tests can also be configured to [run the resulting +program](#controlling-passfail-expectations) to verify its behavior. + +[`src/test/ui`]: https://github.com/rust-lang/rust/blob/master/src/test/ui + +## General structure of a test + +A test consists of a Rust source file located anywhere in the `src/test/ui` directory. +For example, [`src/test/ui/hello.rs`] is a basic hello-world test. + +Compiletest will use `rustc` to compile the test, and compare the output +against the expected output which is stored in a `.stdout` or `.stderr` file +located next to the test. +See [Output comparison](#output-comparison) for more. + +Additionally, errors and warnings should be annotated with comments within +the source file. +See [Error annotations](#error-annotations) for more. + +[Headers](headers.md) in the form of comments at the top of the file control +how the test is compiled and what the expected behavior is. + +Tests are expected to fail to compile, since most tests are testing compiler +errors. +You can change that behavior with a header, see [Controlling pass/fail +expectations](#controlling-passfail-expectations). + +By default, a test is built as an executable binary. +If you need a different crate type, you can use the `#![crate_type]` attribute +to set it as needed. + +[`src/test/ui/hello.rs`]: https://github.com/rust-lang/rust/blob/master/src/test/ui/hello.rs + +## Output comparison + +UI tests store the expected output from the compiler in `.stderr` and +`.stdout` files next to the test. +You normally generate these files with the `--bless` CLI option, and then +inspect them manually to verify they contain what you expect. + +The output is normalized to ignore unwanted differences, see the +[Normalization](#normalization) section. +If the file is missing, then compiletest expects the corresponding output to +be empty. + +There can be multiple stdout/stderr files. +The general form is: + +*test-name*`.`*revision*`.`*compare_mode*`.`*extension* + +* *revision* is the [revision](#cfg-revisions) name. + This is not included when not using revisions. +* *compare_mode* is the [compare mode](#compare-modes). + This will only be checked when the given compare mode is active. + If the file does not exist, then compiletest will check for a file without + the compare mode. +* *extension* is the kind of output being checked: + * `stderr` — compiler stderr + * `stdout` — compiler stdout + * `run.stderr` — stderr when running the test + * `run.stdout` — stdout when running the test + * `64bit.stderr` — compiler stderr with `stderr-per-bitwidth` header on a 64-bit target + * `32bit.stderr` — compiler stderr with `stderr-per-bitwidth` header on a 32-bit target + +A simple example would be `foo.stderr` next to a `foo.rs` test. +A more complex example would be `foo.my-revision.nll.stderr`. + +There are several [headers](headers.md) which will change how compiletest will +check for output files: + +* `stderr-per-bitwidth` — checks separate output files based on the target + pointer width. Consider using the `normalize-stderr` header instead (see + [Normalization](#normalization)). +* `dont-check-compiler-stderr` — Ignores stderr from the compiler. +* `dont-check-compiler-stdout` — Ignores stdout from the compiler. + +UI tests run with with `-Zdeduplicate-diagnostics=no` flag which disables +rustc's built-in diagnostic deduplication mechanism. +This means you may see some duplicate messages in the output. +This helps illuminate situations where duplicate diagnostics are being +generated. + +### Normalization + +The compiler output is normalized to eliminate output difference between +platforms, mainly about filenames. + +Compiletest makes the following replacements on the compiler output: + +- The directory where the test is defined is replaced with `$DIR`. + Example: `/path/to/rust/src/test/ui/error-codes` +- The directory to the standard library source is replaced with `$SRC_DIR`. + Example: `/path/to/rust/library` +- Line and column numbers for paths in `$SRC_DIR` are replaced with `LL:COL`. + This helps ensure that changes to the layout of the standard library do not + cause widespread changes to the `.stderr` files. + Example: `$SRC_DIR/alloc/src/sync.rs:53:46` +- The base directory where the test's output goes is replaced with `$TEST_BUILD_DIR`. + This only comes up in a few rare circumstances. + Example: `/path/to/rust/build/x86_64-unknown-linux-gnu/test/ui` +- Tabs are replaced with `\t`. +- Backslashes (`\`) are converted to forward slashes (`/`) within paths (using + a heuristic). This helps normalize differences with Windows-style paths. +- CRLF newlines are converted to LF. +- Error line annotations like `//~ ERROR some message` are removed. +- Various v0 and legacy symbol hashes are replaced with placeholders like + `[HASH]` or ``. + +Additionally, the compiler is run with the `-Z ui-testing` flag which causes +the compiler itself to apply some changes to the diagnostic output to make it +more suitable for UI testing. +For example, it will anonymize line numbers in the output (line numbers +prefixing each source line are replaced with `LL`). +In extremely rare situations, this mode can be disabled with the header +command `// compile-flags: -Z ui-testing=no`. + +Note: The line and column numbers for `-->` lines pointing to the test are +*not* normalized, and left as-is. This ensures that the compiler continues +to point to the correct location, and keeps the stderr files readable. +Ideally all line/column information would be retained, but small changes to +the source causes large diffs, and more frequent merge conflicts and test +errors. + +Sometimes these built-in normalizations are not enough. In such cases, you +may provide custom normalization rules using the header commands, e.g. + +```rust,ignore +// normalize-stdout-test: "foo" -> "bar" +// normalize-stderr-32bit: "fn\(\) \(32 bits\)" -> "fn\(\) \($$PTR bits\)" +// normalize-stderr-64bit: "fn\(\) \(64 bits\)" -> "fn\(\) \($$PTR bits\)" +``` + +This tells the test, on 32-bit platforms, whenever the compiler writes +`fn() (32 bits)` to stderr, it should be normalized to read `fn() ($PTR bits)` +instead. Similar for 64-bit. The replacement is performed by regexes using +default regex flavor provided by `regex` crate. + +The corresponding reference file will use the normalized output to test both +32-bit and 64-bit platforms: + +```text +... + | + = note: source type: fn() ($PTR bits) + = note: target type: u16 (16 bits) +... +``` + +Please see [`ui/transmute/main.rs`][mrs] and [`main.stderr`] for a +concrete usage example. + +[mrs]: https://github.com/rust-lang/rust/blob/master/src/test/ui/transmute/main.rs +[`main.stderr`]: https://github.com/rust-lang/rust/blob/master/src/test/ui/transmute/main.stderr + +Besides `normalize-stderr-32bit` and `-64bit`, one may use any target +information or stage supported by [`ignore-X`](headers.md#ignoring-tests) +here as well (e.g. `normalize-stderr-windows` or simply +`normalize-stderr-test` for unconditional replacement). + + +## Error annotations + +Error annotations specify the errors that the compiler is expected to emit. +They are "attached" to the line in source where the error is located. + +```rust,ignore +fn main() { + boom //~ ERROR cannot find value `boom` in this scope [E0425] +} +``` + +Although UI tests have a `.stderr` file which contains the entire compiler output, +UI tests require that errors are also annotated within the source. +This redundancy helps avoid mistakes since the `.stderr` files are usually +auto-generated. +It also helps to directly see where the error spans are expected to point to +by looking at one file instead of having to compare the `.stderr` file with +the source. +Finally, they ensure that no additional unexpected errors are generated. + +They have several forms, but generally are a comment with the diagnostic +level (such as `ERROR`) and a substring of the expected error output. +You don't have to write out the entire message, just make sure to include the +important part of the message to make it self-documenting. + +The error annotation needs to match with the line of the diagnostic. +There are several ways to match the message with the line (see the examples below): + +* `~`: Associates the error level and message with the current line +* `~^`: Associates the error level and message with the previous error + annotation line. + Each caret (`^`) that you add adds a line to this, so `~^^^` is three lines + above the error annotation line. +* `~|`: Associates the error level and message with the same line as the + previous comment. + This is more convenient than using multiple carets when there are multiple + messages associated with the same line. + +### Error annotation examples + +Here are examples of error annotations on different lines of UI test +source. + +#### Positioned on error line + +Use the `//~ ERROR` idiom: + +```rust,ignore +fn main() { + let x = (1, 2, 3); + match x { + (_a, _x @ ..) => {} //~ ERROR `_x @` is not allowed in a tuple + _ => {} + } +} +``` + +#### Positioned below error line + +Use the `//~^` idiom with number of carets in the string to indicate the +number of lines above. +In the example below, the error line is four lines above the error annotation +line so four carets are included in the annotation. + +```rust,ignore +fn main() { + let x = (1, 2, 3); + match x { + (_a, _x @ ..) => {} // <- the error is on this line + _ => {} + } +} +//~^^^^ ERROR `_x @` is not allowed in a tuple +``` + +#### Use same error line as defined on error annotation line above + +Use the `//~|` idiom to define the same error line as the error annotation +line above: + +```rust,ignore +struct Binder(i32, i32, i32); + +fn main() { + let x = Binder(1, 2, 3); + match x { + Binder(_a, _x @ ..) => {} // <- the error is on this line + _ => {} + } +} +//~^^^^ ERROR `_x @` is not allowed in a tuple struct +//~| ERROR this pattern has 1 field, but the corresponding tuple struct has 3 fields [E0023] +``` + +### `error-pattern` + +The `error-pattern` [header](headers.md) can be used for +messages that don't have a specific span. + +Let's think about this test: + +```rust,ignore +fn main() { + let a: *const [_] = &[1, 2, 3]; + unsafe { + let _b = (*a)[3]; + } +} +``` + +We want to ensure this shows "index out of bounds" but we cannot use the +`ERROR` annotation since the error doesn't have any span. +Then it's time to use the `error-pattern` header: + +```rust,ignore +// error-pattern: index out of bounds +fn main() { + let a: *const [_] = &[1, 2, 3]; + unsafe { + let _b = (*a)[3]; + } +} +``` + +But for strict testing, try to use the `ERROR` annotation as much as possible. + +### Error levels + +The error levels that you can have are: + +1. `ERROR` +2. `WARN` or `WARNING` +3. `NOTE` +4. `HELP` and `SUGGESTION` + +You are allowed to not include a level, but you should include it at least for +the primary message. + +The `SUGGESTION` level is used for specifying what the expected replacement +text should be for a diagnostic suggestion. + +UI tests use the `-A unused` flag by default to ignore all unused warnings, as +unused warnings are usually not the focus of a test. +However, simple code samples often have unused warnings. +If the test is specifically testing an unused warning, just add the +appropriate `#![warn(unused)]` attribute as needed. + +### cfg revisions + +When using [revisions](compiletest.md#revisions), different messages can be +conditionally checked based on the current revision. +This is done by placing the revision cfg name in brackets like this: + +```rust,ignore +// edition:2018 +// revisions: mir thir +// [thir]compile-flags: -Z thir-unsafeck + +async unsafe fn f() {} + +async fn g() { + f(); //~ ERROR call to unsafe function is unsafe +} + +fn main() { + f(); //[mir]~ ERROR call to unsafe function is unsafe +} +``` + +In this example, the second error message is only emitted in the `mir` revision. +The `thir` revision only emits the first error. + +If the cfg causes the compiler to emit different output, then a test can have +multiple `.stderr` files for the different outputs. +In the example above, there would be a `.mir.stderr` and `.thir.stderr` file +with the different outputs of the different revisions. + + +## Controlling pass/fail expectations + +By default, a UI test is expected to **generate a compile error** because most +of the tests are checking for invalid input and error diagnostics. +However, you can also make UI tests where compilation is expected to succeed, +and you can even run the resulting program. +Just add one of the following [header commands](headers.md): + +* Pass headers: + * `// check-pass` — compilation should succeed but skip codegen + (which is expensive and isn't supposed to fail in most cases). + * `// build-pass` — compilation and linking should succeed but do + not run the resulting binary. + * `// run-pass` — compilation should succeed and running the resulting + binary should also succeed. +* Fail headers: + * `// check-fail` — compilation should fail (the codegen phase is skipped). + This is the default for UI tests. + * `// build-fail` — compilation should fail during the codegen phase. + This will run `rustc` twice, once to verify that it compiles successfully + without the codegen phase, then a second time the full compile should + fail. + * `// run-fail` — compilation should succeed, but running the resulting + binary should fail. + +For `run-pass` and `run-fail` tests, by default the output of the program +itself is not checked. +If you want to check the output of running the program, include the +`check-run-results` header. +This will check for a `.run.stderr` and `.run.stdout` files to compare +against the actual output of the program. + +Tests with the `*-pass` headers can be overridden with the `--pass` +command-line option: + +```sh +./x.py test src/test/ui --pass check +``` + +The `--pass` option only affects UI tests. +Using `--pass check` can run the UI test suite much faster (roughly twice as +fast on my system), though obviously not exercising as much. + +The `ignore-pass` header can be used to ignore the `--pass` CLI flag if the +test won't work properly with that override. + + +## Test organization + +When deciding where to place a test file, please try to find a subdirectory +that best matches what you are trying to exercise. +Do your best to keep things organized. +Admittedly it can be difficult as some tests can overlap different categories, +and the existing layout may not fit well. + +For regression tests – basically, some random snippet of code that came in +from the internet – we often name the test after the issue plus a short +description. +Ideally, the test should be added to a directory that helps identify what +piece of code is being tested here (e.g., +`src/test/ui/borrowck/issue-54597-reject-move-out-of-borrow-via-pat.rs`) + +When writing a new feature, **create a subdirectory to store your tests**. +For example, if you are implementing RFC 1234 ("Widgets"), then it might make +sense to put the tests in a directory like `src/test/ui/rfc1234-widgets/`. + +In other cases, there may already be a suitable directory. (The proper +directory structure to use is actually an area of active debate.) + +Over time, the [`src/test/ui`] directory has grown very fast. +There is a check in [tidy](intro.md#tidy) that will ensure none of the +subdirectories has more than 1000 entries. +Having too many files causes problems because it isn't editor/IDE friendly and +the GitHub UI won't show more than 1000 entries. +However, since `src/test/ui` (UI test root directory) and `src/test/ui/issues` +directories have more than 1000 entries, we set a different limit for those +directories. +So, please avoid putting a new test there and try to find a more relevant +place. + +For example, if your test is related to closures, you should put it in +`src/test/ui/closures`. +If you're not sure where is the best place, it's still okay to add to +`src/test/ui/issues/`. +When you reach the limit, you could increase it by tweaking [here][ui test +tidy]. + +[ui test tidy]: https://github.com/rust-lang/rust/blob/master/src/tools/tidy/src/ui_tests.rs + + +## Rustfix tests + +UI tests can validate that diagnostic suggestions apply correctly +and that the resulting changes compile correctly. +This can be done with the `run-rustfix` header: + +```rust,ignore +// run-rustfix +// check-pass +#![crate_type = "lib"] + +pub struct not_camel_case {} +//~^ WARN `not_camel_case` should have an upper camel case name +//~| HELP convert the identifier to upper camel case +//~| SUGGESTION NotCamelCase +``` + +Rustfix tests should have a file with the `.fixed` extension which contains +the source file after the suggestion has been applied. + +When the test is run, compiletest first checks that the correct +lint/warning is generated. +Then, it applies the suggestion and compares against `.fixed` (they must match). +Finally, the fixed source is compiled, and this compilation is required to succeed. + +Usually when creating a rustfix test you will generate the `.fixed` file +automatically with the `x.py test --bless` option. + +The `run-rustfix` header will cause *all* suggestions to be applied, even +if they are not [`MachineApplicable`](../diagnostics.md#suggestions). +If this is a problem, then you can instead use the `rustfix-only-machine-applicable` +header. +This should be used if there is a mixture of different suggestion levels, and +some of the non-machine-applicable ones do not apply cleanly. + + +## Compare modes + +[Compare modes](compiletest.md#compare-modes) can be used to run all tests +with different flags from what they are normally compiled with. +In some cases, this might result in different output from the compiler. +To support this, different output files can be saved which contain the +output based on the compare mode. + +For example, when in "non-lexical lifetimes" (NLL) mode a test `foo.rs` will +first look for expected output in `foo.nll.stderr`, falling back to the usual +`foo.stderr` if not found. +This is useful as "true" NLL mode can sometimes result in different +diagnostics and behavior compared to the "migrate mode" NLL (which is the +current default). +This can help track which tests have differences between the modes, and to +visually inspect those diagnostic differences. + +If in the rare case you encounter a test that has different behavior, you can +run something like the following to generate the alternate stderr file: + +```sh +./x.py test src/test/ui --compare-mode=nll --bless +``` + +Currently, only `nll` mode is checked in CI for UI tests.