diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 8d9f94653..4d377048a 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -24,8 +24,10 @@ - [Compiletest](./tests/compiletest.md) - [UI tests](./tests/ui.md) - [Test headers](./tests/headers.md) + - [Integration testing](./tests/integration.md) + - [Crater](./tests/crater.md) + - [Fuchsia](./tests/fuchsia.md) - [Performance testing](./tests/perf.md) - - [Crater](./tests/crater.md) - [Suggest tests tool](./tests/suggest-tests.md) - [Debugging the compiler](./compiler-debugging.md) - [Using the tracing/logging instrumentation](./tracing.md) diff --git a/src/tests/fuchsia.md b/src/tests/fuchsia.md new file mode 100644 index 000000000..15fcacefe --- /dev/null +++ b/src/tests/fuchsia.md @@ -0,0 +1,167 @@ +# Fuchsia integration tests + +[Fuchsia](https://fuchsia.dev) is an open-source operating system with about 2 +million lines of Rust code.[^loc] It has caught a large number of [regressions] +in the past and was subsequently included in CI. + +## Building Fuchsia in CI + +Fuchsia builds as part of the suite of bors tests that run before a pull request +is merged. + +If you are worried that a pull request might break the Fuchsia builder and want +to test it out before submitting it to the bors queue, simply add this line to +your PR description: + +> try-job: x86_64-gnu-integration + +Then when you `@bors try` it will pick the job that builds Fuchsia. + +## Building Fuchsia locally + +Because Fuchsia uses languages other than Rust, it does not use Cargo as a build +system. It also requires the toolchain build to be configured in a [certain +way][build-toolchain]. + +The recommended way to build Fuchsia is to use the Docker scripts that check out +and run a Fuchsia build for you. If you've run Docker tests before, you can +simply run this command from your Rust checkout to download and build Fuchsia +using your local Rust toolchain. + +``` +src/ci/docker/run.sh x86_64-gnu-integration +``` + +See the [Testing with Docker](docker.md) chapter for more details on how to run +and debug jobs with Docker. + +Note that a Fuchsia checkout is *large* – as of this writing, a checkout and +build takes 46G of space – and as you might imagine, it takes awhile to +complete. + +### Modifying the Fuchsia checkout + +The main reason you would want to build Fuchsia locally is because you need to +investigate a regression. After running a Docker build, you'll find the Fuchsia +checkout inside the `obj/fuchsia` directory of your Rust checkout. If you +modify the `KEEP_CHECKOUT` line in the [build-fuchsia.sh] script to +`KEEP_CHECKOUT=1`, you can change the checkout as needed and rerun the build +command above. This will reuse all the build results from before. + +You can find more options to customize the Fuchsia checkout in the +[build-fuchsia.sh] script. + +### Customizing the Fuchsia build + +You can find more info about the options used to build Fuchsia in Rust CI in the +[build_fuchsia_from_rust_ci.sh] script invoked by [build-fuchsia.sh]. + +The Fuchsia build system uses [GN], a metabuild system that generates [Ninja] +files and then hands off the work of running the build to Ninja. + +Fuchsia developers use `fx` to run builds and perform other development tasks. +This tool is located in `.jiri_root/bin` of the Fuchsia checkout; you may need +to add this to your `$PATH` for some workflows. + +There are a few `fx` subcommands that are relevant, including: + +* `fx set` accepts build arguments, writes them to `out/default/args.gn`, and runs GN. +* `fx build` builds the Fuchsia project using Ninja. It will automatically pick + up changes to build arguments and rerun GN. By default it builds everything, + but it also accepts target paths to build specific targets (see below). +* `fx clippy` runs Clippy on specific Rust targets (or all of them). We use this + in the Rust CI build to avoid running codegen on most Rust targets. Underneath + it invokes Ninja, just like `fx build`. The clippy results are saved in json + files inside the build output directory before being printed. + +#### Target paths + +GN uses paths like the following to identify build targets: + +``` +//src/starnix/kernel:starnix_core +``` + +The initial `//` means the root of the checkout, and the remaining slashes are +directory names. The string after `:` is the _target name_ of a target defined +in the `BUILD.gn` file of that directory. + +The target name can be omitted if it is the same as the directory name. In other +words, `//src/starnix/kernel` is the same as `//src/starnix/kernel:kernel`. + +These target paths are used inside `BUILD.gn` files to reference dependencies, +and can also be used in `fx build`. + +#### Modifying compiler flags + +You can put custom compiler flags inside a GN `config` that is added to a target. +As a simple example: + +``` +config("everybody_loops") { + rustflags = [ "-Zeverybody-loops" ] +} + +rustc_binary("example") { + crate_root = "src/bin.rs" + # ...existing keys here... + configs += [ ":everybody_loops" ] +} +``` + +This will add the flag `-Zeverybody-loops` to rustc when building the `example` +target. Note that you can also use [`public_configs`] for a config to be added +to every target that depends on that target. + +If you want to add a flag to every Rust target in the build, you can add +rustflags to the [`//build/config:compiler`] config or to the OS-specific +configs referenced in that file. Note that `cflags` and `ldflags` are ignored on +Rust targets. + +#### Running ninja and rustc commands directly + +Going down one layer, `fx build` invokes `ninja`, which in turn eventually +invokes `rustc`. All build actions are run inside the out directory, which is +usually `out/default` inside the Fuchsia checkout. + +You can get ninja to print the actual command it invokes by forcing that command +to fail, e.g. by adding a syntax error to one of the source files of the target. +Once you have the command, you can run it from inside the output directory. + +After changing the toolchain itself, the build setting `rustc_version_string` in +`out/default/args.gn` needs to be changed so that `fx build` or `ninja` will +rebuild all the Rust targets. This can be done in a text editor and the contents +of the string do not matter, as long as it changes from one build to the next. +[build_fuchsia_from_rust_ci.sh] does this for you by hashing the toolchain +directory. + +The Fuchsia website has more detailed documentation of the [build system]. + +#### Other tips and tricks + +When using `build_fuchsia_from_rust_ci.sh` you can comment out the `fx set` +command after the initial run so it won't rerun GN each time. If you do this you +can also comment out the version_string line to save a couple seconds. + +`export NINJA_PERSISTENT_MODE=1` to get faster ninja startup times after the +initial build. + +## Fuchsia target support + +To learn more about Fuchsia target support, see the Fuchsia chapter in [the +rustc book][platform-support]. + +[regressions]: https://gist.github.com/tmandry/7103eba4bd6a6fb0c439b5a90ae355fa +[build-toolchain]: https://fuchsia.dev/fuchsia-src/development/build/rust_toolchain +[build-fuchsia.sh]: https://github.com/rust-lang/rust/blob/99f77a2eda555b50b518f74823ab636a20efb87f/src/ci/docker/host-x86_64/x86_64-gnu-integration/build-fuchsia.sh +[build_fuchsia_from_rust_ci.sh]: https://cs.opensource.google/fuchsia/fuchsia/+/main:scripts/rust/build_fuchsia_from_rust_ci.sh?q=build_fuchsia_from_rust_ci&ss=fuchsia +[platform-support]: https://doc.rust-lang.org/nightly/rustc/platform-support/fuchsia.html +[GN]: https://gn.googlesource.com/gn/+/main#gn +[Ninja]: https://ninja-build.org/ +[`public_configs`]: https://gn.googlesource.com/gn/+/main/docs/reference.md#var_public_configs +[`//build/config:compiler`]: https://cs.opensource.google/fuchsia/fuchsia/+/main:build/config/BUILD.gn;l=121;drc=c26c473bef93b33117ae417893118907a026fec7 +[build system]: https://fuchsia.dev/fuchsia-src/development/build/build_system + +[^loc]: As of June 2024, Fuchsia had about 2 million lines of first-party Rust code +and a roughly equal amount of third-party code, as counted by tokei (excluding +comments and blanks). diff --git a/src/tests/integration.md b/src/tests/integration.md new file mode 100644 index 000000000..ff2d44742 --- /dev/null +++ b/src/tests/integration.md @@ -0,0 +1,49 @@ +# Integration testing + +Rust tests integration with real-world code to catch regressions and make +informed decisions about the evolution of the language. + +## Testing methods + +### Crater + +Crater is a tool which runs tests on many thousands of public projects. This +tool has its own separate infrastructure for running, and is not run as part of +CI. See the [Crater chapter](crater.md) for more details. + +### Cargo test + +`cargotest` is a small tool which runs `cargo test` on a few sample projects +(such as `servo`, `ripgrep`, `tokei`, etc.). +This runs as part of CI and ensures there aren't any significant regressions. + +> Example: `./x test src/tools/cargotest` + +### Integration builders + +Integration jobs build large open-source Rust projects that are used as +regression tests in CI. Our integration jobs build the following projects: + +- [Fuchsia](fuchsia.md) +- Linux kernel + +## A note about terminology + +The term "integration testing" can be used to mean many things. Many of the +compiletest tests within the Rust repo could be justifiably called integration +tests, because they test the integration of many parts of the compiler, or test +the integration of the compiler with other external tools. Calling all of them +integration tests would not be very helpful, especially since those kinds of +tests already have their own specialized names. + +We use the term "integration" here to mean integrating the Rust compiler and +toolchain with the ecosystem of Rust projects that depend on it. This is partly +for lack of a better term, but it also reflects a difference in testing approach +from other projects and the comparative advantage it implies. + +The Rust compiler is part of the ecosystem, and the ecosystem is in many cases +part of Rust, both in terms of libraries it uses and in terms of the efforts of many +contributors who come to "scratch their own itch". Finally, because Rust has the +ability to do integration testing at such a broad scale, it shortens development +cycles by finding defects earlier. + diff --git a/src/tests/intro.md b/src/tests/intro.md index 647d1fbea..491135a58 100644 --- a/src/tests/intro.md +++ b/src/tests/intro.md @@ -132,19 +132,12 @@ More information can be found in the [toolstate documentation]. [toolstate documentation]: https://forge.rust-lang.org/infra/toolstate.html [toolstate website]: https://rust-lang-nursery.github.io/rust-toolstate/ -### Cargo test +### Integration testing -`cargotest` is a small tool which runs `cargo test` on a few sample projects -(such as `servo`, `ripgrep`, `tokei`, etc.). -This ensures there aren't any significant regressions. - -> Example: `./x test src/tools/cargotest` - -### Crater - -Crater is a tool which runs tests on many thousands of public projects. -This tool has its own separate infrastructure for running. -See the [Crater chapter](crater.md) for more details. +Rust tests integration with real-world code to catch regressions and make +informed decisions about the evolution of the language. There are several kinds +of integration tests, including Crater. See the [Integration testing +chapter](integration.md) for more details. ### Performance testing