Skip to content

Commit a610ea3

Browse files
aaronmondalaaronmondal
committed
Introduce the NativeLink Kubernetes operator
A single `kubectl apply -k` now deploys NativeLink in a self-configuring, self-healing and self-updating fashion. To achieve this we implement a two-stage depoyment to asynchronously reconciliate various parts of NativeLink Kustomizations. First, we deploy Flux Alerts that trigger Tekton Pipelines on GitRepository updates to bring required images into the cluster. Second, and technically at the same time, we start a Flux Kustomization to deploy a NativeLink Kustomization. This is similar to the previous 01_operations and 02_applicaion scripts, but now happens fully automated in the cluster and no longer requires a local Nix installation as all tag evaluations have become implementation details of the Tekton Pipelines. This commit also changes the K8s resource layout to a "best-practice" Kustomize directory layout. This further reduces code duplication and gives third parties greater flexibility and more useful reference points to build custom NativeLink setups. Includes an overhaul of the Kubernetes documentation.
1 parent b2386fd commit a610ea3

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

45 files changed

+806
-585
lines changed

‎.github/styles/config/vocabularies/TraceMachina/accept.txt

-1
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,6 @@ Cloudflare
55
ELB
66
GPUs
77
Goma
8-
Kustomization
98
[Hh]ermeticity
109
Kustomization
1110
LLD

‎.github/workflows/lre.yaml

+90-5
Original file line numberDiff line numberDiff line change
@@ -83,19 +83,104 @@ jobs:
8383
uses: >- # v4
8484
DeterminateSystems/magic-nix-cache-action@fc6aaceb40b9845a02b91e059ec147e78d1b4e41
8585
86-
- name: Start Kubernetes cluster (Infra)
86+
- name: Start Kubernetes cluster
8787
run: >
8888
nix run .#native up
8989
90-
- name: Start Kubernetes cluster (Operations)
90+
# - name: Start NativeLink operator
91+
# run: >
92+
# nix develop --impure --command
93+
# bash -c "kubectl apply -k deploy/dev"
94+
95+
- name: Start NativeLink operator
96+
env:
97+
GITHUB_REF: ${{ github.ref }}
98+
run: >
99+
nix develop --impure --command bash -c '
100+
WORK_DIR=$(pwd) &&
101+
TEMP_DIR=$(mktemp -d) &&
102+
cat <<EOF > $TEMP_DIR/kustomization.yaml
103+
apiVersion: kustomize.config.k8s.io/v1beta1
104+
kind: Kustomization
105+
components:
106+
- ${WORK_DIR}/kubernetes/components/operator
107+
patches:
108+
- patch: |-
109+
- op: replace
110+
path: /spec/path
111+
value: ./kubernetes/overlays/lre
112+
target:
113+
kind: Kustomization
114+
name: nativelink
115+
- patch: |-
116+
- op: replace
117+
path: /spec/ref/name
118+
value: ${GITHUB_REF}
119+
target:
120+
kind: GitRepository
121+
name: nativelink
122+
- patch: |-
123+
- op: replace
124+
path: /spec/eventMetadata/flakeOutput
125+
value: ./src_root#image
126+
target:
127+
kind: Alert
128+
name: nativelink-image-alert
129+
- patch: |-
130+
- op: replace
131+
path: /spec/eventMetadata/flakeOutput
132+
value: ./src_root#nativelink-worker-init
133+
target:
134+
kind: Alert
135+
name: nativelink-worker-init-alert
136+
- patch: |-
137+
- op: replace
138+
path: /spec/eventMetadata/flakeOutput
139+
value: ./src_root#nativelink-worker-lre-cc
140+
target:
141+
kind: Alert
142+
name: nativelink-worker-alert
143+
EOF
144+
kubectl apply -k $TEMP_DIR &&
145+
rm -rf $TEMP_DIR'
146+
147+
- name: Wait for Tekton pipelines
148+
run: >
149+
nix develop --impure --command
150+
bash -c "kubectl wait \
151+
--for=condition=Succeeded \
152+
--timeout=45m \
153+
pipelinerun \
154+
-l tekton.dev/pipeline=rebuild-nativelink"
155+
156+
- name: Wait for Configmaps
157+
run: >
158+
nix develop --impure --command
159+
bash -c "flux reconcile kustomization -n default \
160+
--timeout=15m \
161+
nativelink-configmaps"
162+
163+
- name: Wait for NativeLink Kustomization
164+
run: >
165+
nix develop --impure --command
166+
bash -c "flux reconcile kustomization -n default \
167+
--timeout=15m \
168+
nativelink"
169+
170+
- name: Wait for CAS
171+
run: >
172+
nix develop --impure --command
173+
bash -c "kubectl rollout status deploy/nativelink-cas"
174+
175+
- name: Wait for scheduler
91176
run: >
92177
nix develop --impure --command
93-
bash -c "./deployment-examples/kubernetes/01_operations.sh"
178+
bash -c "kubectl rollout status deploy/nativelink-scheduler"
94179
95-
- name: Start Kubernetes cluster (Application)
180+
- name: Wait for worker
96181
run: >
97182
nix develop --impure --command
98-
bash -c "./deployment-examples/kubernetes/02_application.sh"
183+
bash -c "kubectl rollout status deploy/nativelink-worker"
99184
100185
- name: Get gateway IPs
101186
id: gateway-ips

‎README.md

+1
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,7 @@ To start, you can deploy NativeLink as a Docker image (as shown below) or by usi
5959

6060
The setups below are **production-grade** installations. See the [contribution docs](https://nativelink.com/docs/contribute/nix/) for instructions on how to build from source with [Bazel](https://nativelink.com/docs/contribute/bazel/), [Cargo](https://nativelink.com/docs/contribute/cargo/), and [Nix](https://nativelink.com/docs/contribute/nix/).
6161

62+
You can find a few example deployments in the [Docs](https://docs.nativelink.com/guides/kubernetes).
6263

6364
### 📦 Prebuilt images
6465

‎deployment-examples/chromium/03_build_chrome_tests.sh renamed to ‎deploy/chromium-example/build_chromium_tests.sh

File renamed without changes.

‎deploy/chromium-example/kustomization.yaml

+35
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
apiVersion: kustomize.config.k8s.io/v1beta1
2+
kind: Kustomization
3+
4+
components:
5+
- ../../kubernetes/components/operator
6+
7+
patches:
8+
- patch: |-
9+
- op: replace
10+
path: /spec/path
11+
value: ./kubernetes/overlays/chromium
12+
target:
13+
kind: Kustomization
14+
name: nativelink
15+
- patch: |-
16+
- op: replace
17+
path: /spec/url
18+
value: https://github.com/aaronmondal/nativelink
19+
target:
20+
kind: GitRepository
21+
name: nativelink
22+
- patch: |-
23+
- op: replace
24+
path: /spec/ref/branch
25+
value: flux
26+
target:
27+
kind: GitRepository
28+
name: nativelink
29+
- patch: |-
30+
- op: replace
31+
path: /spec/eventMetadata/flakeOutput
32+
value: github:TraceMachina/nativelink#nativelink-worker-siso-chromium
33+
target:
34+
kind: Alert
35+
name: nativelink-worker-alert

‎deploy/dev/kustomization.yaml

+69
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
apiVersion: kustomize.config.k8s.io/v1beta1
2+
kind: Kustomization
3+
4+
components:
5+
- ../../kubernetes/components/operator
6+
7+
# Change this value to deploy custom overlays.
8+
patches:
9+
- patch: |-
10+
- op: replace
11+
path: /spec/path
12+
value: ./kubernetes/overlays/lre
13+
target:
14+
kind: Kustomization
15+
name: nativelink
16+
17+
# Modify this value to change the URL of the repository with deployment files.
18+
#
19+
# This is usually only necessary if you change deployment YAML files or
20+
# NativeLink config files. If you only intend to change the Rust sources you can
21+
# leave this as is and need to ensure that the Alerts below are patched to build
22+
# your local sources.
23+
- patch: |-
24+
- op: replace
25+
path: /spec/url
26+
value: https://github.com/aaronmondal/nativelink
27+
target:
28+
kind: GitRepository
29+
name: nativelink
30+
31+
# Patch this to change to track a custom branch.
32+
- patch: |-
33+
- op: replace
34+
path: /spec/ref/branch
35+
value: flux
36+
target:
37+
kind: GitRepository
38+
name: nativelink
39+
40+
# Setting the flake outputs to `./src_root#xxx` causes the Tekton pipelines to
41+
# build nativelink from your local sources.
42+
#
43+
# During development, the following formats might be useful as well:
44+
#
45+
# `github:user/repo#outname` to build an image from an arbitrary flake output.
46+
#
47+
# `github:TraceMachina/nativelink?ref=pull/<PR_NUMBER>/head#<OUT>` to deploy a
48+
# outputs from a Pull request.
49+
- patch: |-
50+
- op: replace
51+
path: /spec/eventMetadata/flakeOutput
52+
value: ./src_root#image
53+
target:
54+
kind: Alert
55+
name: nativelink-image-alert
56+
- patch: |-
57+
- op: replace
58+
path: /spec/eventMetadata/flakeOutput
59+
value: ./src_root#nativelink-worker-init
60+
target:
61+
kind: Alert
62+
name: nativelink-worker-init-alert
63+
- patch: |-
64+
- op: replace
65+
path: /spec/eventMetadata/flakeOutput
66+
value: ./src_root#nativelink-worker-lre-cc
67+
target:
68+
kind: Alert
69+
name: nativelink-worker-alert

‎deploy/kubernetes-example/kustomization.yaml

+49
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
apiVersion: kustomize.config.k8s.io/v1beta1
2+
kind: Kustomization
3+
4+
components:
5+
- ../../kubernetes/components/operator
6+
7+
patches:
8+
- patch: |-
9+
- op: replace
10+
path: /spec/path
11+
value: ./kubernetes/overlays/lre
12+
target:
13+
kind: Kustomization
14+
name: nativelink
15+
- patch: |-
16+
- op: replace
17+
path: /spec/url
18+
value: https://github.com/aaronmondal/nativelink
19+
target:
20+
kind: GitRepository
21+
name: nativelink
22+
- patch: |-
23+
- op: replace
24+
path: /spec/ref/branch
25+
value: flux
26+
target:
27+
kind: GitRepository
28+
name: nativelink
29+
- patch: |-
30+
- op: replace
31+
path: /spec/eventMetadata/flakeOutput
32+
value: github:aaronmondal/nativelink/flux#nativelink-worker-lre-cc
33+
target:
34+
kind: Alert
35+
name: nativelink-worker-alert
36+
- patch: |-
37+
- op: replace
38+
path: /spec/eventMetadata/flakeOutput
39+
value: github:aaronmondal/nativelink/flux#image
40+
target:
41+
kind: Alert
42+
name: nativelink-image-alert
43+
- patch: |-
44+
- op: replace
45+
path: /spec/eventMetadata/flakeOutput
46+
value: github:aaronmondal/nativelink/flux#nativelink-worker-init
47+
target:
48+
kind: Alert
49+
name: nativelink-worker-init-alert

‎deployment-examples/chromium/.gitignore

-2
This file was deleted.

‎deployment-examples/chromium/01_operations.sh

-39
This file was deleted.

‎deployment-examples/chromium/02_application.sh

-30
This file was deleted.

‎deployment-examples/chromium/04_delete_application.sh

-6
This file was deleted.

0 commit comments

Comments
 (0)