diff --git a/docs/rbe/luci.png b/docs/rbe/luci.png new file mode 100644 index 0000000000000..5eecd722f9b43 Binary files /dev/null and b/docs/rbe/luci.png differ diff --git a/docs/rbe/rbe.md b/docs/rbe/rbe.md new file mode 100644 index 0000000000000..b13660ed4b6cd --- /dev/null +++ b/docs/rbe/rbe.md @@ -0,0 +1,161 @@ +# RBE for Flutter Engine Developers + +## Overview + +This is documentation on setting up RBE for building the Flutter engine. It is +expected to work for Googlers only on corp Linux, macOS, and Windows hosts, +including cloudtop instances. + +## Getting started + +The first step is to add an entry to the `.gclient` file. The entry to add is +`"use_rbe": True` in the `custom_vars` section. It should look like this: + +After making this edit, you must be authenticated as a Googler by CIPD so that +the RBE configurations can be downloaded from the `flutter_internal` +[CIPD bucket](https://chrome-infra-packages.appspot.com/p/flutter_internal): + +After authentication successfully, run `gclient sync -D`. + +## Running an RBE build + +In the engine repo, all RBE builds must be initiated through the `et` tool whose +entrypoint is the script `//flutter/bin/et`. This is so that the local RBE +proxy is correctly initialized and shut down around invocations of `ninja`. + +### gcloud + +Before running an RBE build, you must be authenticated with the Google cloud +project that owns the RBE worker pool. A gcloud SDK is checked out with the +engine repo under e.g. `//flutter/buildtools/mac-arm64/gcloud`. +(Replace `mac-arm64` with the directory name that is correct for your host +platform.) + +The `gcloud` tool in this SDK must be on your path. The tool lives under +`//flutter/buildtools/mac-arm64/gcloud/bin`, which is the path to add to your +`PATH` environment variable. Alternatively, you can get the gcloud SDK on your +path by installing it on your system by following the instructions at +[https://cloud.google.com/sdk/docs/install] +(https://cloud.google.com/sdk/docs/install). + +On macOS, before running the `gcloud` command ensure that `python3` is on your +path, and does not come from e.g. homebrew. The command `which python3` should +return `/usr/bin/python3`. + +### Listing builds + +The builds available to the `et` tool are those specified by the build +configuration json files under `//flutter/ci/builders`. A list of builds +suitable for local development can also be printed by running `et help build`. +The list of all builds can be printed by running `et help build --verbose`. +Builds in the verbose list prefixed with `ci/` or `ci\` are exactly the builds +run on CI with the same GN flags and Ninja targets. + +### Running builds + +To run a build, pass the name of the configuration to the `et build` command: + +If RBE is working correctly, you should see logs like the following: + +To disable RBE in a build, pass the flag `--no-rbe` to the `build` command. + +Since LTO is slow and rarely useful in local builds, `et` disables it by default +in all builds, even when it is specified by a build configuration. To enable +it, pass the `--lto` flag to the `build` command. + +### Customizing builds + +Beyond enabling/disabling RBE and LTO, the `build` command does not currently +support customizing builds at the command line. + +If you need custom GN flags or Ninja targets, then this can be achieved by +making local edits to the build configuration json files in your checkout. In +particular, the file `//flutter/ci/builders/local_engine.json` is intended to +contain builds that are commonly used for local engine development. + +If a configuration does not exist there, and will be needed by multiple people +over a long period of time, consider checking in the new configuration so that +it will be built on CI, which will keep the RBE cache warm for fast local +builds. + +On the other hand, if a build is intended to produce artifacts or run tests on +CI, then it should _not_ go in `local_engine.json`. Instead, it should go in +the json file appropriate for its purpose. There are many examples available, +so following suit is likely your best bet, but if you are unsure, ask in the +`hackers-engine` Discord. + +## Maintaining Compatibility with RBE + +RBE is sensitive to what paths in compile commands look like. In particular, all +paths in compile commands must be relative paths, and those relative paths must +all resolve to files within the engine source tree. In practice, this means +that the GN function `rebase_path()` should only lack a second parameter when +its result won’t be used in a compile or link command. There are unfortunately +many examples in the tree where `rebase_path()` lacks a second parameter. +However to be on the safe side a good rule of thumb is: do not add new usages +of the single parameter `rebase_path()`. + +## Troubleshooting + +### Slow builds + +RBE builds can be slow for a few different reasons. The most common reason is +likely to be that the remote caches are cold. When the caches are warm, a +compile step consists only of downloading the compiled TU from the cache. When +the caches are cold, the remote workers must run the compilation commands, +which takes more time. + +RBE builds can also be slow if your network connection is bandwidth constrained. +Anecdotally, even with a warm cache, I have noticed slow builds from home due +to RBE saturating my low-tier Comcast Business connection. + +When RBE builds are slow, non-RBE builds may be faster, especially incremental +builds. + +### Proxy status and debug logs + +The status of the local RBE proxy can be queried with the following command + +It will give output describing the number of actions completed and in progress, +and the number of remote executions, local executions, and remote cache hits. + +The logs for RBE live under the system `/tmp` folder in the files `/tmp/reproxy. +{INFO,WARNING,ERROR}` and `/tmp/bootstrap.{INFO,WARNING,ERROR}`. + +In CI runs, the RBE logs are also available by following the links as in the +screenshot below under `teardown remote execution > collect rbe logs`: + +![LUCI logs links](luci.png "LUCI logs links") + +### Dependency analysis failures + +These logs can be useful to highlight failures in RBE’s dependency analysis. The +dependency analysis can fail for a variety of reasons, but a common one during +development is likely to be that the source file is really malformed somehow. +This can be debugged by doing a local build with RBE turned off. + + +## References + +* Code for RBE (i.e. reproxy, rewrapper, bootstrap, etc.) lives in [this GitHub + repository](https://github.com/bazelbuild/reclient). The tools are not + well-documented, so the source code is the source of truth for the command + line flags that they accept, for example. +* Internal-facing RBE migration guide is [here](go/reclient-migration-guide). + (Mostly focused on Chrome and Android, so not all parts are relevant to + Flutter.) +* The version of RBE for local development is set in the DEPS file [here] + (https://github.com/flutter/engine/blob/8578edf9c9393471ca9eab18e9154f0e6066dcb6/DEPS#L53). + It needs to be manually rolled occasionally. +* The version of RBE used by CI is set in a LUCI recipe [here] + (https://flutter.googlesource.com/recipes/+/be12675150183af68223f5fbc6e0f888a1139e79/recipe_modules/rbe/api.py#16). + It also needs to be manually rolled occasionally. +* Googler-only RBE configuration files live in the CIPD bucket [here] + (https://chrome-infra-packages.appspot.com/p/flutter_internal/rbe/reclient_cfgs). + They need to be updated when we roll clang to a new version as described + [here] + (https://github.com/flutter/engine/pull/52062#issuecomment-2050902282). +* Flutter’s RBE worker pool is defined [here] + (https://source.corp.google.com/piper///depot/google3/configs/cloud/gong/services/flutter_rbe/modules/infra/prod/main.tf). +* Using RBE for Engine clang-tidy is blocked on [b/326591374] + (http://b/326591374).