diff --git a/docs/index_while_building.md b/docs/index_while_building.md new file mode 100644 index 00000000..bf87ff64 --- /dev/null +++ b/docs/index_while_building.md @@ -0,0 +1,138 @@ +# Index While Building in Bazel + +This document summarizes the status of index while building in Bazel and the +plan to improve it. "Index while building" encompasses the writing of index data +from swift and clang and the usage of that data in Xcode. +For a high level design of the feature in LLVM, check the paper [Adding Index-While-Building support to Clang](https://docs.google.com/document/d/1cH2sTpgSnJZCkZtJl1aY-rzy4uGPcrI-6RrUpdATO2Q/edit) + +## Current status + +At the time of writing there's prior art that adds `-index-store-path` to +`rules_swift` and `rules_ios` but enabling these code paths slows down build +time in testing. + +`swift` and `clang` write index data based on the status of what resides in +index-store-path. When using a transient per-swift-library index, it writes O( +M imports * N libs) indexer data and slows down compilation significantly: to +the tune of 300% slow down and 6GB+ index data in benchmarks. `rules_swift` +likes to use per `swift_library` data in order to remote cache indexes. +`rules_ios` uses a per `apple_library` index to reduce `index-import` wallclock +time. Adding "Index while building" to Bazel needs special consideration to +both preserve performance characteristics of the original architecture of a +global index and interoperate with Bazel's remote execution and caching. + +## Index while building V2 design + +Like the current implementation, index while building V2 piggy backs on the +`-index-store-path` feature in clang and swift. However, in V2 of index while +building, `swift` and `clang` compilers use a global index cache internally to +preserve performance. Finally, to integrate with remote caching, actions copy a +small subset of data into `bazel-out`. In other words, Bazel actions import +relevant units and records to a tree artifact output so Bazel can write a small +subset of the index data to caches. + +### Per compilation index management + +As Bazel remote caching is oriented around action outputs, the index store data +is also updated to fit into this paradigm and retaining the global cache. + +A program is implemented to clone a minimal per-action subset of an index store. +The program operates at the compilation level rather than caching the entire +global index by mapping compiler outputs to unit files. The program uses the +LLVM index store library to interface with index store primitives e.g. units and +records. _[This program](https://github.com/lyft/index-import/pull/53) is +orthogonal to and may or may not end up as part of Lyft's index-import_. + +### rules_swift workers + +Workers are extended to use a global index internally. Then, it writes records +and units into the remote cache by copying material from the worker's global +index into `bazel-out`. This logic lives in the action binary to optimize +performance and interface per compilation records with Bazel. _note: it's also +not currently possible or gainful to express the global index store in a Bazel +action output. Multiple actions cannot specify the same indexstore directory as an output._ + +### clang compilation + +The native rules currently don't have an specify indexstore as a clang +compilation action output. It may use rules_cc compilation actions to write to +a global index internally and then copy the updated units+records to an output +directory to ensure remote caching will work. Here, it uses rules_cc to add an +extra compilation output for the translation unit / source file. + +### Incrementally importing remotely compiled indexes + +In order to import indexes into a local, global index incrementally, a program +is invoked to "import" remotely compiled indexes based on action outputs. This +may be an aspect that runs during the Xcode build in order to pass an output +file map to index import, or an ad-hoc program picking up output maps from build +events. + +By default clang and swift seem to write absolute paths into the Index. If this +is invariant continues to hold, it needs to be accounted for in Bazel. e.g. +Unless all source code and compilation resided at the same directory remotely +and locally there will likely need to be a `remap`ing of paths. The previously +mentioned importing program will be smart enough to govern this - and import +only the files that were recompiled. + +### Local optimization - sans index-import + +One major shortcut exists locally to remove index-import: Xcode is fixed to read +directly from Bazel's global index store. Then, instead of writing to global +index and copying to `bazel-out` for remote caching, Xcode should can read right +from Bazels' global index. A "out of band" sentenial value can noop the code +path for `bazel-out`. _(out of band means abstracted from Bazel)_ + +In order to remove the requirement to "index-import" bazel indexes, there are +special considerations in the way that Xcode and Bazel invoke compilation and +index-while-building abilities. Specifically, the file paths that Xcode and +Bazel use to address index records must be aligned. This may be achievable by +just aligning outputs between Xcode and Bazel: for example in the build system +xcbuild ( e.g. XCBuildKit ) protocol messages, in xcconfig by means of the +`.xcspec` features, or by plugin. + + +## Task roadmap + +### Preliminary + +- [x] Profile and determine impact of current state of IWB. +- [x] Swift compilation https://github.com/bazelbuild/rules_swift/issues/561 +- [x] Consumption of index in Xcode https://github.com/bazel-ios/rules_ios/issues/203 + +### Determine direction forward with prototypes - spiking + +- [x] Output design doc and map out tasks + +1. Consider possibilities of improving performance of "Index while building" +There are a couple avenues here: + +- solution that works with remote caching + +- A solution that only runs locally. This is the status quo of objc right +now: it doesn't work with remote caching + +2. Disable "Index while building" and correspondingly index-import +Need to consider the user facing impact: e.g. faster build times, can't index out-of-focus impl files. + +### Index while building V2 - first pass + +- [x] Propose changes to `rules_swift` for compilation https://github.com/bazelbuild/rules_swift/pull/567 +- [x] Propose changes to `index-import` for Bazel action helper binary https://github.com/lyft/index-import/pull/53 +- [x] Evaluate possibilities to remove `index-import` for local execution +- [x] Fork `rules_swift` to `rules_ios` to iterate until upstream PR lands +- [ ] Fork `index-import` to `rules_ios` +- [ ] CI script to compile `index-import` +- [ ] Patch `rules_ios` to use first pass + +### Sans index-import for local execution + +- [ ] Remove the need to use `index-import` for local execution +- [ ] Implement an aspect to quickly pull remotely built indexes + +### objc remote cache support + +- [ ] determine a pattern to add Index While Building to objc with remote caching +- [ ] consider adding an extra action output for `rules_ios` for translation + unit index +