Move installation instructions to new docs (TraceMachina#1127)

Author: aaronmondal

.vale.ini

@@ -14,6 +14,9 @@ mdx = md
 [*.{md,mdx}]
 BasedOnStyles = alex, Vale, Microsoft, write-good
 
+# Ignore code blocks in Starlight's TabItems.
+BlockIgnores = (?s)(<TabItem.*?>.*?```.*?```.*?</TabItem>)
+
 # Too harsh. The `write-good.Passive` check already covers many cases.
 write-good.E-Prime = NO
 

README.md

@@ -19,215 +19,78 @@
 NativeLink is an extremely (blazingly?) fast and efficient build cache and
 remote executor for systems that communicate using the [Remote execution
 protocol](https://github.com/bazelbuild/remote-apis/blob/main/build/bazel/remote/execution/v2/remote_execution.proto) such as [Bazel](https://bazel.build), [Buck2](https://buck2.build), [Goma](https://chromium.googlesource.com/infra/goma/client/) and
-[Reclient](https://github.com/bazelbuild/reclient). NativeLink powers over one billion requests per month for customers using the system for their  production workloads.
+[Reclient](https://github.com/bazelbuild/reclient). NativeLink powers over one
+billion requests per month in production workloads.
 
 Supports Unix-based operating systems and Windows.
 
-## Getting Started with NativeLink
+## 🚀 Quickstart
 
-Below, you will find a few different options for getting started with NativeLink.
+The setups below are **production-grade** installations. See the
+[contribution docs](https://docs.nativelink.com/contribute/nix/) for
+instructions on how to build from source with [Bazel](https://docs.nativelink.com/contribute/bazel/),
+[Cargo](https://docs.nativelink.com/contribute/cargo/), and [Nix](https://docs.nativelink.com/contribute/nix/).
 
-## 🚀 Example Deployments
+### 📦 Prebuilt images
 
-You can find a few example deployments in the [deployment-examples directory](./deployment-examples).
+Fast to spin up, but currently limited to `x86_64` systems. See the [container
+registry](https://github.com/TraceMachina/nativelink/pkgs/container/nativelink)
+for all image tags and the [contribution docs](https://docs.nativelink.com/contribute/nix)
+for how to build the images yourself.
 
-### 📝 Clone the NativeLink repository
-1. Go to the [NativeLink](https://github.com/TraceMachina/nativelink) repository on GitHub. Clone the repository via SSH or HTTPS. In this example the repository is cloned via SSH:
-```bash
-git clone [email protected]:TraceMachina/nativelink.git
-```
-
-### 📦 Installing with Cargo
-
-1. First install Rust, but skip to step 2 if you have it already.
-```bash
-curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
-```
-2. Install NativeLink with Cargo.
-```bash
-cargo install --git https://github.com/TraceMachina/nativelink --tag v0.4.0
-```
-
-### ⚙️ Configure and 🦾 Start NativeLink
-
-The `nativelink` executable reads a JSON file as it's only parameter,
-`--config`. See [`nativelink-config`](./nativelink-config/examples/basic_cas.json)
-for more details and examples.
+**Linux x86_64**
 
-To grab the example in your current working directory, run:
-
-```bash
-curl -O https://raw.githubusercontent.com/TraceMachina/nativelink/main/nativelink-config/examples/basic_cas.json
-
-### you can modify the example above to replace the filesystem store with the memory store if you favor speed over data durability.
-nativelink basic_cas.json
-```
-
-## 🧪 Evaluating NativeLink
-
-1. Once you've built NativeLink and have an instance running with the `basic_cas.json` configuration, launch a separate terminal session.
-2. Navigate to where you cloned the NativeLink repository:
 ```bash
-cd $HOME/nativelink
-```
-3. In the new terminal, run the following command to connect the running server launched above to Bazel or another RBE client:
-
-```sh
-bazel test //... \
-  --remote_instance_name=main \
-  --remote_cache=grpc://127.0.0.1:50051 \
-  --remote_executor=grpc://127.0.0.1:50051 \
-  --remote_default_exec_properties=cpu_count=1
+curl -O \
+    https://raw.githubusercontent.com/TraceMachina/nativelink/main/nativelink-config/examples/basic_cas.json
+
+# See https://github.com/TraceMachina/nativelink/pkgs/container/nativelink
+# to find the latest tag
+docker run \
+    -v $(pwd)/basic_cas.json:/config \
+    -p 50051 \
+    ghcr.io/tracemachina/nativelink:v0.4.0
 ```
 
-For Windows PowerShell;
+**Windows x86_64**
 
 ```powershell
-bazel test //... `
-  --remote_instance_name=main `
-  --remote_cache=grpc://127.0.0.1:50051 `
-  --remote_executor=grpc://127.0.0.1:50051 `
-  --remote_default_exec_properties=cpu_count=1
-```
-This causes Bazel to run the commands through an all-in-one `CAS`, `scheduler`
-and `worker`.
-
-> [!WARNING]
-> If you're using MacOS, encountering errors is anticipated at this stage. Our team is actively working on enhancing support for executing remoteable Bazel builds with MacOS. For now, you can run with [Docker](https://github.com/blakehatch/nativelink/tree/main/deployment-examples/docker-compose) or a Linux virtual machine. If you have any questions, reach out on the [NativeLink](https://join.slack.com/t/nativelink/shared_invite/zt-2forhp5n9-L7dTD21nCSY9_IRteQvZmw) slack.
-
-## How it Works
-
-This diagram is a high-level overview of the data flow in the NativeLink system. It refers to NativeLink concepts like Scheduler pool, Worker pool, and CAS rather than the cloud concepts like functions, compute nodes, and object storage to which they correspond.
-
-```mermaid
-sequenceDiagram
-    participant build server (client)
-    participant scheduler pool
-    participant worker pool
-    participant cas
-    build server (client)->>scheduler pool: queue jobs
-    scheduler pool->>worker pool: route jobs
-    worker pool->>cas: upload artifacts
-    worker pool->>scheduler pool: result download instructions
-    scheduler pool->>build server (client): result download instructions
-    cas->>build server (client): service queries
-    build server (client)->>cas: service queries
+# Download the configuration file
+Invoke-WebRequest `
+    -Uri "https://raw.githubusercontent.com/TraceMachina/nativelink/main/nativelink-config/examples/basic_cas.json" `
+    -OutFile "basic_cas.json"
+
+# Run the Docker container
+# Note: Adjust the path if the script is not run from the directory containing basic_cas.json
+docker run `
+    -v ${PWD}/basic_cas.json:/config `
+    -p 50051 `
+    ghcr.io/tracemachina/nativelink:v0.4.0
 ```
-## ❄️ Installing with Nix
 
-**Installation requirements:**
+### ❄️ Raw executable with Nix
 
-* Nix with [flakes](https://nixos.wiki/wiki/Flakes) enabled
+Slower, since it's built from source, but more flexible and supports MacOS.
+Doesn't support native Windows, but works in WSL2.
 
-This build doesn't require cloning the repository, but you need to provide a
-configuration file, for instance the one at [`nativelink-config/examples/basic_cas.json`](./nativelink-config/examples/basic_cas.json).
+Make sure your Nix version is recent and supports flakes. For instance, install
+it via the [next-gen nix installer](https://github.com/NixOS/experimental-nix-installer).
 
-The following command builds and runs NativeLink in release (optimized) mode:
+> [!CAUTION]
+> Executables built for MacOS are dynamically linked against libraries from Nix
+> and won't work on systems that don't have these libraries present.
 
-```sh
-nix run github:TraceMachina/nativelink ./basic_cas.json
-```
+**Linux, MacOS, WSL2**
 
-For use in production pin the executable to a specific revision:
-
-```sh
-nix run github:TraceMachina/nativelink/<revision> ./basic_cas.json
 ```
+curl -O \
+    https://raw.githubusercontent.com/TraceMachina/nativelink/main/nativelink-config/examples/basic_cas.json
 
-## 🌱 Building with Bazel
-
-**Build requirements:**
-
-* Bazel `7.0.2`
-* A recent C++ toolchain with LLD as linker
-
-> [!TIP]
-> This build supports Nix/direnv which provides Bazel but no C++ toolchain
-> (yet).
-
-The following commands places an executable in `./bazel-bin/nativelink` and
-starts the service:
-
-```sh
-# Unoptimized development build on Unix
-bazel run nativelink -- $(pwd)/nativelink-config/examples/basic_cas.json
-
-# Optimized release build on Unix
-bazel run -c opt nativelink -- $(pwd)/nativelink-config/examples/basic_cas.json
-
-# Unoptimized development build on Windows
-bazel run --config=windows nativelink -- $(pwd)/nativelink-config/examples/basic_cas.json
-
-# Optimized release build on Windows
-bazel run --config=windows -c opt nativelink -- $(pwd)/nativelink-config/examples/basic_cas.json
-```
-
-> [!WARNING]
-> The Rust compiler `rustc` generates numerous artifacts during compilation,
-> including dependencies, macros, and intermediate files.
-> When compiling programs from source, be mindful of the associated files'
-> impact on your disk usage in the `bazel-bin/` directory.
-> This directory can grow substantially in size.
->
-> If the facing issues due to this, run the following command
-> to clear cache files:
-> ```sh
-> bazel clean --expunge
-> ```
-
-## 📦 Building with Cargo
-
-**Build requirements:**
-
-* Cargo 1.74.0+
-* A recent C++ toolchain with LLD as linker
-
-> [!TIP]
-> This build supports Nix/direnv which provides Cargo but no C++
-> toolchain/stdenv (yet).
-
-```bash
-# Unoptimized development build
-cargo run --bin nativelink -- ./nativelink-config/examples/basic_cas.json
-
-# Optimized release build
-cargo run --release --bin nativelink -- ./nativelink-config/examples/basic_cas.json
+nix run github:TraceMachina/nativelink ./basic_cas.json
 ```
 
-> [!WARNING]
-> The Rust compiler `rustc` generates numerous artifacts during compilation,
-> including dependencies, macros, and intermediate files.
-> When compiling programs from source, be mindful of the associated files'
-> impact on your disk usage in the target/ directory.
-> This directory can grow substantially in size.
->
-> If the facing issues due to this, run the following command
-> to clear cache files:
-> ```sh
-> cargo clean
-> ```
-
-## Project Health
-
-* Powering more than 1 billion devices at the edge.
-
-## 🏺 History
-
-This project was first created due to frustration with similar projects not
-working or being extremely inefficient. Rust was chosen as the language to write
-it in because at the time Rust was going through a revolution in the new-ish
-feature `async-await`. This made making multi-threading simpler when
-paired with a runtime like [Tokio](https://github.com/tokio-rs/tokio) while
-still giving all the lifetime and other protections that Rust gives. This pretty
-much guarantees that we will never have crashes due to race conditions. This
-kind of project seemed perfect, since there is so much asynchronous activity
-happening and running them on different threads is most preferable. Other
-languages like `Go` are good candidates, but other similar projects rely heavily
-on channels and mutex locks which are cumbersome and have to be carefully
-designed by the developer. Rust doesn't have these issues, since the compiler
-will always tell you when the code you are writing might introduce undefined
-behavior. The last major reason is because Rust is extremely fast and has no
-garbage collection (like C++, but unlike `Java`, `Go`, or `Typescript`).
+See the [contribution docs](https://docs.nativelink.com/contribute/nix) for
+further information.
 
 ## 📜 License
 

docs/.gitignore

@@ -27,12 +27,13 @@ pnpm-debug.log*
 stats.html
 
 # Generated files
-src/content/docs/guides/contributing.mdx
+src/content/docs/contribute/docs.mdx
+src/content/docs/contribute/guidelines.mdx
 src/content/docs/explanations/lre.mdx
-src/content/docs/reference/changelog.mdx
-src/content/docs/reference/nativelink-config.mdx
-src/content/docs/tutorials/setup.mdx
 src/content/docs/guides/chromium.mdx
 src/content/docs/guides/configuration.mdx
 src/content/docs/guides/kubernetes.mdx
 src/content/docs/guides/setup.md
+src/content/docs/reference/changelog.mdx
+src/content/docs/reference/nativelink-config.mdx
+src/content/docs/tutorials/setup.mdx

docs/README.md

@@ -1,17 +1,17 @@
 # The NativeLink documentation
 
-This is the NativeLink documentation hosted at <https://docs.nativelink.com>.
+The NativeLink documentation gets deployed to <https://docs.nativelink.com>.
 
-> [!WARNING]
-> Setup for working on these docs differs substantially between Linux and Mac.
+> [!CAUTION]
+> Setup for working on these docs differs between Linux and Mac.
 >
 > For Linux: Use the Nix flake and run `pnpm i`.
 >
 > For Mac: If you're in the Nix flake, exit it, for instance with `direnv
 > revoke`. Then manually install `pnpm`, run `pnpm i` and run `pnpm exec
 > playwright install`.
 >
-> Long term we'll add the automated setup to Mac.
+> It's a long term goal to add the automated setup to Mac.
 
 ## 📚 Stack
 
@@ -32,7 +32,8 @@ challenging. Feel free to copy-paste it into your own projects.
 
 ## 🚀 Common workflows
 
-See `package.json` for build scripts.
+See [`docs/package.json`](https://github.com/TraceMachina/nativelink/blob/main/docs/package.json)
+for build scripts.
 
 This project requires `pnpm`. The nix flake ships a compatible version.
 
@@ -65,12 +66,12 @@ pnpm preview
 
 When deploying to Cloudflare, make sure to set the `PNPM_VERSION` to `8.15.5` to
 stay in sync with the flake. Also, use `pnpm exec playwright install && pnpm
-build` on the Cloudflare worker. This sets up headless Chromium which is used to
+build` on the Cloudflare worker. This sets up headless Chromium which to
 generate mermaid diagrams during the build. You don't need to set playwright up
 locally as it's already configured in the flake.
 
 ## 🐛 Known issues
 
-- We use Bun as internal TypeScript processor, but can't use it as bundler yet.
-- `"@playform/compress": "=0.0.12"` because `0.0.13` doesn't properly compress
-  CSS.
+- The build process uses Bun as internal TypeScript processor, but can't use it
+  as bundler yet.
+- `"@playform/compress": "=0.0.12"` because `0.0.13` doesn't compress CSS.

docs/astro.config.mjs

@@ -117,10 +117,6 @@ export default defineConfig({
               label: "Kubernetes example",
               link: "/guides/kubernetes",
             },
-            {
-              label: "Contributing",
-              link: "/guides/contributing",
-            },
           ],
         },
         {
@@ -129,6 +125,10 @@ export default defineConfig({
           // explain design decisions, constraints, etc.
           label: "Understanding NativeLink",
           items: [
+            {
+              label: "Architecture",
+              link: "/explanations/architecture/",
+            },
             {
               label: "Local Remote Execution",
               link: "/explanations/lre/",
@@ -139,6 +139,31 @@ export default defineConfig({
             },
           ],
         },
+        {
+          label: "For Contributors",
+          items: [
+            {
+              label: "Contribution Guidelines",
+              link: "contribute/guidelines/",
+            },
+            {
+              label: "Working on documentation",
+              link: "contribute/docs/",
+            },
+            {
+              label: "Develop with Nix",
+              link: "contribute/nix/",
+            },
+            {
+              label: "Develop with Bazel",
+              link: "contribute/bazel/",
+            },
+            {
+              label: "Developing with Cargo",
+              link: "contribute/cargo/",
+            },
+          ],
+        },
         {
           // Corresponds to https://diataxis.fr/reference/. Technical
           // descriptions with the intent to be used as consulting material.