Repository navigation
ci: configure shared fleet Rust cache #18369
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 3 commits
Commits
Show all changes
44 commits
Select commit
Hold shift + click to select a range
5ad8aad
ci: configure shared fleet Rust cache
teamleaderleo 3d9ea5f
ci: resolve stable fleet Rust target directories
teamleaderleo d571386
Merge remote-tracking branch 'origin/feat-cmux-next' into fix/fleet-r…
teamleaderleo 53cf8fe
cx-m0p7 adapters: red tests for the VS Code family and Zed sources an…
lawrencecchen 166eb3a
cx-m0p7 adapters: VS Code family and Zed project sources on project.s…
lawrencecchen 30204c6
cx-m0p7 adapters: review fixes (no partial complete lists from a busy…
lawrencecchen 365fbc2
Merge remote-tracking branch 'origin/feat-cmux-next-m0p7-projects-4' …
lawrencecchen f7eeb55
cx-m0p7 adapters: socket-level daemon test for the editor import repl…
lawrencecchen 6aecd65
cx-m0p7 adapters: the socket test picks a non-temporary folder
lawrencecchen 723bac5
Merge remote-tracking branch 'origin/feat-cmux-next-m0p7-projects-4' …
lawrencecchen 966e9a2
Merge remote-tracking branch 'origin/feat-cmux-next-m0p7-projects-4' …
lawrencecchen f4abab4
Merge remote-tracking branch 'origin/feat-cmux-next' into feat-cmux-n…
lawrencecchen 63eeb64
cx-m0p7 adapters: daemon-capabilities.json lists project-list-v1
lawrencecchen 6683150
Merge remote-tracking branch 'origin/feat-cmux-next' into feat-cmux-n…
lawrencecchen c1bf51e
cmux-next: credential.session answers the backend team of the install…
lawrencecchen 4ca47b6
test(cmux-vm): delete the cmux VM unit tests (cx-034r)
lawrencecchen e5dc62d
test(browser-parity): markdown, searchText, extract, storageState and…
lawrencecchen 4d9d631
feat(browser-host): page.markdown, searchText, extract, dropdownOptio…
lawrencecchen 6459968
release-notes: markdown and agent tool reads stay within the page-rea…
lawrencecchen 596734a
test(browser-parity): a click in a child frame works beside 20,000 ne…
lawrencecchen 909f9b1
feat(browser-host): a child frame's <iframe> is found by its place in…
lawrencecchen d3fc10d
test(browser-parity): tabs.content HTML masks sensitive values; frame…
lawrencecchen 08c42a5
fix(browser-host): tabs.content HTML is an observe read; storageState…
lawrencecchen be01df2
test(bindings): delete the SDK unit tests (cx-034r)
lawrencecchen e247337
Merge remote-tracking branch 'origin/feat-cmux-next' into feat-cmux-n…
lawrencecchen 2a37d8f
Merge remote-tracking branch 'origin/feat-cmux-next' into HEAD
lawrencecchen 768cc47
Merge remote-tracking branch 'origin/feat-cmux-next' into feat-cmux-n…
lawrencecchen 4f8c5ab
Merge remote-tracking branch 'origin/feat-cmux-next' into feat-cmux-n…
lawrencecchen 0d94be2
fix(messageslab): a Chief subagent link check is a URL property, not …
lawrencecchen ef7c3d8
Merge remote-tracking branch 'origin/feat-cmux-next' into feat-cmux-n…
lawrencecchen 521b219
Merge remote-tracking branch 'origin/feat-cmux-next' into feat-cmux-n…
lawrencecchen 20cd2f6
ci: a base autofix bot commits generated files, tree inputs and fmt (…
teamleaderleo 7af5d8b
fix(release-rails): resolve capabilities re-exported from another cmu…
teamleaderleo 1e0c0bb
ci: format staged files before commit (#18808)
teamleaderleo 08cd108
ci: check rustfmt on the cmux-tui packages a PR touches (#18800)
teamleaderleo 7a52911
ci: add mini-first Swift canary for cmux-next (#18781)
teamleaderleo c980cc5
cmux-next: repin app FFI to current base release (#18776)
teamleaderleo 8da37f7
cmux-next: same-tree cmux-tui is a hard check only for PRs that chang…
teamleaderleo 1beff16
fix(ci): keep cmux-next minis first through bot retries (#18615)
teamleaderleo 5c0a0fc
docs(cmux-next): map UI parity reference sources (#18561)
teamleaderleo c08fa90
fix(nightly): bound the publish notes step at 10 minutes (#18404)
teamleaderleo 0e4c30a
ci(cmux-next): retry bad R2 binary reads and let the tree gate pin it…
teamleaderleo d808aa1
ci(gallery): capture narrow responsive states (#18901)
teamleaderleo dd68964
Merge feat-cmux-next into fix/fleet-rust-sccache
teamleaderleo File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,82 @@ | ||
| #!/usr/bin/env bash | ||
| # Resolve a stable Cargo target directory for a managed fleet build. | ||
| # | ||
| # This helper is intentionally credential-free. HQ injects sccache and the | ||
| # worker-only R2 configuration in the managed worker recipe. Source checkouts | ||
| # use this file only to derive a target path from the toolchain selected by the | ||
| # manifest and the requested target triple. | ||
| # | ||
| # Source this file, then call: | ||
| # fleet_rust_target_dir PROJECT [TARGET_TRIPLE] [MANIFEST_DIR] | ||
| # | ||
| # PROJECT is a short workspace name (for example acpmux, chief, or cmux-tui). | ||
| # MANIFEST_DIR is the directory Cargo/rustup should inspect for its manifest; | ||
| # it defaults to the caller's current directory. | ||
|
|
||
| fleet_rust_cache_slug() { | ||
| local value="${1:-}" | ||
| [[ "$value" =~ ^[A-Za-z0-9][A-Za-z0-9._-]*$ ]] || return 2 | ||
| value="$(printf '%s' "$value" | tr '[:upper:]' '[:lower:]')" | ||
| printf '%s\n' "$value" | ||
| } | ||
|
|
||
| fleet_rust_target_dir() { | ||
| local project="${1:-}" | ||
| local requested_target="${2:-}" | ||
| local manifest_dir="${3:-$PWD}" | ||
| local toolchain rust_release rust_host target root project_key toolchain_key target_key | ||
|
|
||
| project_key="$(fleet_rust_cache_slug "$project")" || { | ||
| echo "fleet-rust-cache: invalid project name" >&2 | ||
| return 2 | ||
| } | ||
| [[ -d "$manifest_dir" ]] || { | ||
| echo "fleet-rust-cache: manifest directory does not exist: $manifest_dir" >&2 | ||
| return 2 | ||
| } | ||
|
|
||
| # Resolve both values from the manifest's working directory. `rustc` and | ||
| # `rustup` proxies honor rust-toolchain.toml only after changing into it. | ||
| toolchain="$(cd "$manifest_dir" && rustup show active-toolchain 2>/dev/null | awk 'NR == 1 {print $1}')" || true | ||
| rust_release="$(cd "$manifest_dir" && rustc -Vv 2>/dev/null | awk -F': ' '$1 == "release" {print $2; exit}')" || true | ||
| rust_host="$(cd "$manifest_dir" && rustc -Vv 2>/dev/null | awk -F': ' '$1 == "host" {print $2; exit}')" || true | ||
| [[ -n "$toolchain" && -n "$rust_release" && -n "$rust_host" ]] || { | ||
| echo "fleet-rust-cache: cannot identify the manifest Rust toolchain in $manifest_dir" >&2 | ||
| return 78 | ||
| } | ||
| target="${requested_target:-${CARGO_BUILD_TARGET:-$rust_host}}" | ||
| target_key="$(fleet_rust_cache_slug "$target")" || { | ||
| echo "fleet-rust-cache: invalid target triple" >&2 | ||
| return 2 | ||
| } | ||
| toolchain_key="$(fleet_rust_cache_slug "${toolchain}-${rust_release}-${rust_host}")" || { | ||
| echo "fleet-rust-cache: invalid toolchain identity" >&2 | ||
| return 2 | ||
| } | ||
| root="${CMUX_FLEET_RUST_TARGET_ROOT:-${CI_SHARED_CACHE_DIR:-$HOME/.cache/cmux-build-fleet}/rust-targets}" | ||
| [[ -n "$root" && "$root" != */.. && "$root" != *$'\n'* ]] || { | ||
| echo "fleet-rust-cache: invalid target root" >&2 | ||
| return 2 | ||
| } | ||
| root="${root%/}" | ||
| printf '%s/%s/%s/%s\n' "$root" "$project_key" "$toolchain_key" "$target_key" | ||
| } | ||
|
|
||
| if [[ "${BASH_SOURCE[0]}" == "$0" ]]; then | ||
| case "${1:-}" in | ||
| --target-dir) | ||
| [[ $# -ge 2 && $# -le 4 ]] || { | ||
| echo "usage: $0 --target-dir PROJECT [TARGET_TRIPLE] [MANIFEST_DIR]" >&2 | ||
| exit 2 | ||
| } | ||
| fleet_rust_target_dir "$2" "${3:-}" "${4:-$PWD}" | ||
| ;; | ||
| -h|--help) | ||
| sed -n '2,17p' "$0" | sed 's/^# //' | ||
| ;; | ||
| *) | ||
| echo "usage: $0 --target-dir PROJECT [TARGET_TRIPLE] [MANIFEST_DIR]" >&2 | ||
| exit 2 | ||
| ;; | ||
| esac | ||
| fi | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,66 @@ | ||
| #!/usr/bin/env bash | ||
| set -euo pipefail | ||
|
|
||
| root="$(cd "$(dirname "$0")/.." && pwd)" | ||
| tmp="$(mktemp -d)" | ||
| trap 'rm -rf "$tmp"' EXIT | ||
| bin="$tmp/bin" | ||
| manifest="$tmp/cmux-tui" | ||
| mkdir -p "$bin" "$manifest" | ||
|
|
||
| cat > "$bin/rustc" <<'STUB' | ||
| #!/usr/bin/env bash | ||
| cat <<'OUT' | ||
| rustc 1.88.0 (stable) | ||
| binary: rustc | ||
| commit-hash: test | ||
| commit-date: test | ||
| host: aarch64-apple-darwin | ||
| release: 1.88.0 | ||
| LLVM version: 20.1.8 | ||
| OUT | ||
| STUB | ||
| cat > "$bin/rustup" <<'STUB' | ||
| #!/usr/bin/env bash | ||
| [[ "$*" == "show active-toolchain" ]] && echo '1.88.0-aarch64-apple-darwin (directory override)' || exit 0 | ||
| STUB | ||
| chmod +x "$bin"/* | ||
|
|
||
| # Resolver uses the manifest-selected toolchain, compiler identity, project and | ||
| # target triple, and never reads a worker secret or mutates cache credentials. | ||
| out="$(env -i PATH="$bin:/usr/bin:/bin" HOME="$tmp/home" \ | ||
| CMUX_FLEET_RUST_TARGET_ROOT="$tmp/rust-targets" \ | ||
| bash -c 'source "$1/scripts/ci/fleet-rust-cache.sh"; \ | ||
| [[ -z "${RUSTC_WRAPPER:-}" && -z "${AWS_SECRET_ACCESS_KEY:-}" ]]; \ | ||
| fleet_rust_target_dir acpmux aarch64-apple-darwin "$2"; \ | ||
| fleet_rust_target_dir acpmux x86_64-apple-darwin "$2"; \ | ||
| fleet_rust_target_dir chief aarch64-apple-darwin "$2"' _ "$root" "$manifest" 2>/dev/null)" | ||
|
cursor[bot] marked this conversation as resolved.
Resolved
|
||
| expected="$tmp/rust-targets/acpmux/1.88.0-aarch64-apple-darwin-1.88.0-aarch64-apple-darwin/aarch64-apple-darwin" | ||
| expected_x86="$tmp/rust-targets/acpmux/1.88.0-aarch64-apple-darwin-1.88.0-aarch64-apple-darwin/x86_64-apple-darwin" | ||
| expected_chief="$tmp/rust-targets/chief/1.88.0-aarch64-apple-darwin-1.88.0-aarch64-apple-darwin/aarch64-apple-darwin" | ||
| [[ "$(printf '%s\n' "$out" | sed -n '1p')" == "$expected" ]] || { echo "unexpected acpmux target: $out" >&2; exit 1; } | ||
| [[ "$(printf '%s\n' "$out" | sed -n '2p')" == "$expected_x86" ]] || { echo "unexpected cross target: $out" >&2; exit 1; } | ||
| [[ "$(printf '%s\n' "$out" | sed -n '3p')" == "$expected_chief" ]] || { echo "unexpected chief target: $out" >&2; exit 1; } | ||
| [[ ! -e "$tmp/rust-targets" ]] || { echo "resolver created target directories" >&2; exit 1; } | ||
|
|
||
| # Slugs cannot escape the configured root, and missing manifest toolchains fail. | ||
| if env -i PATH="$bin:/usr/bin:/bin" HOME="$tmp/home" CMUX_FLEET_RUST_TARGET_ROOT="$tmp/rust-targets" \ | ||
| bash -c 'source "$1/scripts/ci/fleet-rust-cache.sh"; fleet_rust_target_dir ../escape aarch64-apple-darwin "$2"' _ "$root" "$manifest" >/dev/null 2>&1; then | ||
| echo "accepted a path-like project name" >&2 | ||
| exit 1 | ||
| fi | ||
| if env -i PATH="$bin:/usr/bin:/bin" HOME="$tmp/home" CMUX_FLEET_RUST_TARGET_ROOT="$tmp/rust-targets" \ | ||
| bash -c 'source "$1/scripts/ci/fleet-rust-cache.sh"; fleet_rust_target_dir acpmux "$2" "$3"' _ "$root" 'x/y' "$manifest" >/dev/null 2>&1; then | ||
| echo "accepted a path-like target" >&2 | ||
| exit 1 | ||
| fi | ||
|
|
||
| # Managed integration is explicit; no worker marker leaves existing defaults in | ||
| # place. These checks protect against restoring global CARGO_TARGET_DIR exports. | ||
| grep -q 'CMUX_FLEET_WORKER_TRUSTED' "$root/scripts/cmux-next/build-acpmux.sh" | ||
| grep -q 'CMUX_FLEET_WORKER_TRUSTED' "$root/scripts/cmux-next/build-optchat-chief.sh" | ||
| grep -q 'CMUX_FLEET_WORKER_TRUSTED' "$root/scripts/ci/cmux-tui-rust-check.sh" | ||
| ! grep -q 'CMUX_FLEET_RUST_CACHE_SECRET_FILE' "$root/scripts/ci/fleet-rust-cache.sh" | ||
| ! grep -q 'SCCACHE_NAMESPACE' "$root/scripts/ci/fleet-rust-cache.sh" | ||
|
|
||
| echo "fleet Rust target resolver tests passed" | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.