Skip to content
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 29 additions & 3 deletions .agents/skills/manage-ci/references/current-inventory.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ removable after this branch's runner contract is active on protected main.
| `ci-linux-sdk-slice.yml`, `ci-macos-sdk-slice.yml` | Platform-local Rust/Kotlin/Swift smoke consumers; SDK producers are independent top-level calls |
| `ci-runner-contract-slice.yml` | Provider/cache/plan trust and main runner-image checks |
| `native-sdk-artifact.yml` | Typed native SDK producer |
| `swift-sdk-artifact.yml` | Fixed `macos-15` host-only/full XCFramework producer |
| `swift-sdk-artifact.yml` | Host-only/full XCFramework producer; trusted main remains `macos-15`, while eligible same-repository PRs follow the protected Depot macOS 15 gate |
| `smoke.yml` | Artifact-based inference/OpenAI/split smoke |
| `scripted-binary-smoke.yml` | Artifact-based scripted product smoke |
| `sdk-smoke.yml` | Artifact-based SDK consumers |
Expand Down Expand Up @@ -109,6 +109,15 @@ from that same catalog.
- `configure-sccache-gha`: event/provider-derived compiler-cache setup.
- `capture-sccache-stats`: machine-readable cache evidence.

`scripts/collect-ci-metrics.py` is the read-only timing evidence collector. Its
schema-v3 report keeps workflow wall/queue, job runner queue, measured
dependency wait, job execution, runner-minutes, cancelled runner-minutes and
peak workers separate. It groups observations by provider, operating system,
architecture, semantic runner role and Depot size, and emits deterministic
queue/capacity heuristics plus an optional provider-cohort comparison. Raw
inputs and dated reports belong under `/tmp` or a tracking issue/artifact, not
under `ci/` or this inventory.

Artifacts are correctness boundaries; caches only accelerate regeneration.
PR artifacts generally retain for one day. Protected same-repository and fork
lanes cannot publish shared trusted-main caches, and Depot cache access is
Expand All @@ -129,15 +138,32 @@ non-fail-fast, failed producers suppress only declared consumers through
GitHub-hosted labels are `ubuntu-24.04`, `ubuntu-24.04-arm`, `macos-15`, and
`windows-2022`. Depot labels are selected only by `select-ci-runners` for
trusted main Linux when `DEPOT_RUNNERS_ENABLED` is exactly `true`; no workflow
accepts a raw provider label. The documented `gpu-nvidia` ephemeral scale set
is the sole uncredentialed, hardware-qualified same-repository PR exception.
accepts a raw provider label. Current ordinary PR execution remains hosted
until the protected PR gate is separately activated. The intended final gate
may cover eligible build/test rows across Linux, Depot macOS 15 and Windows
2022 when equivalent images/architectures exist; planning/required summaries,
credential-bearing smokes, `gpu-nvidia` hardware and uncertified Intel macOS
rows remain exceptions. The documented `gpu-nvidia` ephemeral scale set is
the sole current uncredentialed, hardware-qualified same-repository PR
exception.

The future Depot PR gate is documented in `ci/DEPOT_MIGRATION.md`. It requires
cache isolation, no PR cache/registry tokens, exact protected workflow refs,
ephemeral runners, a sentinel canary, and a tested GitHub rollback. No Depot
settings or runner groups are changed by this workflow refactor.

External administrative posture is now verified as follows: automatic Depot
Cache connectivity is disabled, automatic Registry Actions authentication is
disabled, and the Depot runner group is restricted to `Mesh-LLM/mesh-llm` and
the exact protected workflow refs. The repository token cannot independently
inspect organization runner-group settings through the API (403), so these
remain external facts rather than checked-in evidence. The protected
branch/main canary, same-repository PR canary, fork PR canary, provider-parity
comparison, and rollback run are still pending; `DEPOT_RUNNERS_ENABLED` or a
successful manual canary does not prove PR isolation.

Relevant repository variable names include `DEPOT_RUNNERS_ENABLED`,
`DEPOT_PR_RUNNERS_ENABLED` (absent/false until protected PR activation),
`CUDA_VERSION`, `VULKAN_SDK_VERSION`, smoke configuration variables, and
release/deployment variables. Secret values never belong in this inventory;
known names include `HF_TOKEN`, release-attestation keys, `CARGO_REGISTRY_TOKEN`
Expand Down
63 changes: 63 additions & 0 deletions .github/actions/audit-depot-pr-isolation/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
name: Audit Depot PR isolation
description: Fail closed when a PR build receives Depot cache, registry, or compiler-cache authority.

inputs:
original_event_name:
description: Original event used to derive the trust boundary.
required: false
default: ''

runs:
using: composite
steps:
- name: Verify PR runner has no privileged Depot credentials
shell: bash
env:
INPUT_ORIGINAL_EVENT_NAME: ${{ inputs.original_event_name }}
run: |
set -euo pipefail
original_event="${INPUT_ORIGINAL_EVENT_NAME:-${GITHUB_EVENT_NAME:-}}"
if [[ "$original_event" != "pull_request" &&
"$original_event" != "pull_request_target" ]]; then
exit 0
fi

forbidden_names=(
DEPOT_CACHE_TOKEN
DEPOT_TOKEN
DEPOT_REGISTRY_PULL_TOKEN
DEPOT_REGISTRY_TOKEN
DEPOT_CACHE_URL
DEPOT_CACHE_API_URL
DEPOT_REGISTRY_URL
DEPOT_REGISTRY_HOST
SCCACHE_WEBDAV_ENDPOINT
SCCACHE_WEBDAV_TOKEN
SCCACHE_WEBDAV_USERNAME
SCCACHE_WEBDAV_PASSWORD
TURBO_TOKEN
TURBO_API
TURBO_TEAM
GOCACHEPROG
)
for name in "${forbidden_names[@]}"; do
if [[ -n "${!name:-}" ]]; then
echo "PR runner received forbidden credential/cache authority: $name" >&2
exit 1
fi
done
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved

for endpoint_name in ACTIONS_CACHE_URL ACTIONS_RESULTS_URL; do
endpoint="${!endpoint_name:-}"
endpoint_lower="$(printf '%s' "$endpoint" | tr '[:upper:]' '[:lower:]')"
if [[ ! "$endpoint_lower" =~ ^https://([a-z0-9-]+\.)*actions\.githubusercontent\.com([/:]|$) ]]; then
echo "PR runner received a non-GitHub Actions endpoint: $endpoint_name" >&2
exit 1
fi
done
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved

docker_config="${DOCKER_CONFIG:-${HOME:-}/.docker}/config.json"
if [[ -f "$docker_config" ]] && grep -Eiq 'depot\.dev' "$docker_config"; then
echo "PR runner has Depot registry authentication configured" >&2
exit 1
fi
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
126 changes: 104 additions & 22 deletions .github/actions/select-ci-runners/action.yml
Original file line number Diff line number Diff line change
@@ -1,17 +1,35 @@
name: Select trusted CI runners
description: Select Depot or GitHub-hosted Linux runners without routing pull-request events to Depot.
description: Select Depot or GitHub-hosted CI runners from the real event and trust context.

inputs:
event_name:
description: GitHub event name.
required: true
original_event_name:
description: Source event declared by a workflow-dispatch controller, if any.
required: false
default: ""
repository:
description: Exact owner/name from the GitHub repository context.
required: true
head_repository:
description: Exact owner/name of the pull-request head repository, or repository outside PR events.
required: true
ref:
description: Fully qualified Git ref used to enforce the trusted-main boundary.
required: true
depot_main_enabled:
description: Repository-wide Depot gate for trusted push and dispatch events.
required: false
default: "false"
depot_pr_enabled:
description: Exact-string repository gate for direct pull-request Depot placement.
required: false
default: "false"
force_hosted:
description: Fail-closed planner signal for CI-control or runner-policy changes.
required: false
default: "false"
manual_use_depot:
description: Typed workflow-dispatch canary selection.
required: false
Expand Down Expand Up @@ -48,6 +66,12 @@ outputs:
runner_arm_16:
description: Sixteen-vCPU Linux ARM64 runner label for measured high-parallelism builds.
value: ${{ steps.select.outputs.runner_arm_16 }}
runner_macos:
description: macOS runner label selected by the centralized provider policy.
value: ${{ steps.select.outputs.runner_macos }}
runner_windows:
description: Windows runner label selected by the centralized provider policy.
value: ${{ steps.select.outputs.runner_windows }}

runs:
using: composite
Expand All @@ -57,15 +81,22 @@ runs:
shell: bash
env:
INPUT_EVENT_NAME: ${{ inputs.event_name }}
INPUT_ORIGINAL_EVENT_NAME: ${{ inputs.original_event_name }}
DISPATCH_ORIGINAL_EVENT_NAME: ${{ github.event.inputs.original_event_name || '' }}
INPUT_REPOSITORY: ${{ inputs.repository }}
INPUT_HEAD_REPOSITORY: ${{ inputs.head_repository }}
INPUT_REF: ${{ inputs.ref }}
INPUT_DEPOT_MAIN_ENABLED: ${{ inputs.depot_main_enabled }}
INPUT_DEPOT_PR_ENABLED: ${{ inputs.depot_pr_enabled }}
INPUT_FORCE_HOSTED: ${{ inputs.force_hosted }}
INPUT_MANUAL_USE_DEPOT: ${{ inputs.manual_use_depot }}
run: |
set -euo pipefail

for value_name in \
INPUT_DEPOT_MAIN_ENABLED \
INPUT_DEPOT_PR_ENABLED \
INPUT_FORCE_HOSTED \
INPUT_MANUAL_USE_DEPOT
do
value="${!value_name}"
Expand All @@ -75,42 +106,87 @@ runs:
fi
done

# A direct PR may opt into Depot only when this protected repository
# explicitly enables the feature. Both the base and PR-head repository
# comparisons remain exact so a fork, mirror, or similarly named
# repository cannot opt in.
trusted_repository="Mesh-LLM/mesh-llm"
depot_enabled=false
effective_event_name="$INPUT_EVENT_NAME"
if [[ "$INPUT_EVENT_NAME" == "workflow_dispatch" ]]; then
case "${DISPATCH_ORIGINAL_EVENT_NAME:-}" in
pull_request|pull_request_target) effective_event_name=pull_request ;;
push) effective_event_name=push ;;
esac
allow_depot_remote_cache=false
is_direct_pull_request=false
is_dispatched_pull_request=false
dispatch_original_event="${INPUT_ORIGINAL_EVENT_NAME:-}"
if [[ -z "$dispatch_original_event" ]]; then
dispatch_original_event="${DISPATCH_ORIGINAL_EVENT_NAME:-}"
fi
case "$effective_event_name" in
pull_request|pull_request_target)
# Current PR workflows and local actions are evaluated from
# PR-controlled code. A repository variable or same-repo check
# therefore cannot establish a runner trust boundary.

case "$INPUT_EVENT_NAME" in
pull_request)
# The caller must use the protected default-branch reusable lane;
# only the canonical PR merge ref is eligible for this path.
if [[ "$INPUT_REPOSITORY" == "$trusted_repository" && \
"$INPUT_HEAD_REPOSITORY" == "$trusted_repository" && \
"$INPUT_REF" =~ ^refs/pull/[0-9]+/merge$ && \
"$INPUT_FORCE_HOSTED" == "false" && \
"$INPUT_DEPOT_PR_ENABLED" == "true" ]]; then
is_direct_pull_request=true
depot_enabled=true
fi
;;
pull_request_target)
# Never execute pull_request_target content on Depot, even when a
# caller declares a matching repository, ref, or feature gate.
depot_enabled=false
;;
workflow_dispatch)
if [[ "$INPUT_REF" == "refs/heads/main" && ( \
"$INPUT_DEPOT_MAIN_ENABLED" == "true" || \
"$INPUT_MANUAL_USE_DEPOT" == "true" \
) ]]; then
depot_enabled=true
fi
# A dispatch controller can label its source event for cache and
# cancellation policy. A declared PR source is still untrusted and
# must remain on GitHub; it must not become a Depot opt-in.
case "$dispatch_original_event" in
pull_request|pull_request_target)
is_dispatched_pull_request=true
;;
push)
if [[ "$INPUT_REPOSITORY" == "$trusted_repository" && \
"$INPUT_REF" == "refs/heads/main" && \
"$INPUT_DEPOT_MAIN_ENABLED" == "true" ]]; then
depot_enabled=true
fi
;;
workflow_dispatch|"")
if [[ "$INPUT_REPOSITORY" == "$trusted_repository" && \
"$INPUT_REF" == "refs/heads/main" && ( \
"$INPUT_DEPOT_MAIN_ENABLED" == "true" || \
"$INPUT_MANUAL_USE_DEPOT" == "true" \
) ]]; then
depot_enabled=true
fi
;;
esac
;;
push)
if [[ \
if [[ "$INPUT_REPOSITORY" == "$trusted_repository" && \
"$INPUT_REF" == "refs/heads/main" && \
"$INPUT_DEPOT_MAIN_ENABLED" == "true" \
]]; then
"$INPUT_DEPOT_MAIN_ENABLED" == "true" ]]; then
depot_enabled=true
allow_depot_remote_cache=true
fi
;;
*)
depot_enabled=false
;;
esac

# Depot's repository-scoped cache is never authorized by a PR gate.
# Trusted main push/dispatch paths are the only paths that may retain
# the existing remote-cache behavior.
if [[ "$depot_enabled" == "true" && \
"$is_direct_pull_request" != "true" && \
"$is_dispatched_pull_request" != "true" && \
"$INPUT_EVENT_NAME" == "workflow_dispatch" ]]; then
allow_depot_remote_cache=true
fi

if [[ "$depot_enabled" == "true" ]]; then
runner=depot-ubuntu-24.04
runner_4=depot-ubuntu-24.04-4
Expand All @@ -120,6 +196,8 @@ runs:
runner_arm_4=depot-ubuntu-24.04-arm-4
runner_arm_8=depot-ubuntu-24.04-arm-8
runner_arm_16=depot-ubuntu-24.04-arm-16
runner_macos=depot-macos-15
runner_windows=depot-windows-2022
else
runner=ubuntu-24.04
runner_4=ubuntu-24.04
Expand All @@ -129,11 +207,13 @@ runs:
runner_arm_4=ubuntu-24.04-arm
runner_arm_8=ubuntu-24.04-arm
runner_arm_16=ubuntu-24.04-arm
runner_macos=macos-15
runner_windows=windows-2022
fi

{
echo "depot_enabled=$depot_enabled"
echo "allow_depot_remote_cache=$depot_enabled"
echo "allow_depot_remote_cache=$allow_depot_remote_cache"
echo "runner=$runner"
echo "runner_4=$runner_4"
echo "runner_8=$runner_8"
Expand All @@ -142,4 +222,6 @@ runs:
echo "runner_arm_4=$runner_arm_4"
echo "runner_arm_8=$runner_arm_8"
echo "runner_arm_16=$runner_arm_16"
echo "runner_macos=$runner_macos"
echo "runner_windows=$runner_windows"
} >> "$GITHUB_OUTPUT"
21 changes: 18 additions & 3 deletions .github/workflows/ci-linux-host-slice.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,16 @@ on:
description: Immutable product source revision.
required: true
type: string
original_event_name:
description: Original event used for protected runner and cache policy.
required: false
default: ''
type: string
force_hosted:
description: Force GitHub-hosted placement for runner-contract changes.
required: false
default: false
type: boolean
hosts_matrix:
description: JSON Linux host rows from the versioned CI plan.
required: true
Expand Down Expand Up @@ -63,8 +73,13 @@ jobs:
uses: ./.github/actions/select-ci-runners
with:
event_name: ${{ github.event_name }}
original_event_name: ${{ inputs.original_event_name }}
repository: ${{ github.repository }}
head_repository: ${{ github.event.pull_request.head.repo.full_name }}
ref: ${{ github.ref }}
depot_main_enabled: ${{ vars.DEPOT_RUNNERS_ENABLED == 'true' }}
depot_pr_enabled: ${{ vars.DEPOT_PR_RUNNERS_ENABLED == 'true' }}
force_hosted: ${{ inputs.force_hosted }}
manual_use_depot: ${{ inputs.use_depot }}

linux_host:
Expand All @@ -80,9 +95,6 @@ jobs:
host: ${{ fromJson(inputs.hosts_matrix) }}
container:
image: ghcr.io/mesh-llm/mesh-llm-cuda-runner@sha256:8d93de6ba30173e825a16fdecf011f9c632edc6e1259df7289e491b0a05f829d
credentials:
username: ${{ github.actor }}
password: ${{ github.token }}
defaults:
run:
shell: bash
Expand All @@ -92,6 +104,9 @@ jobs:
RUSTC_WRAPPER: sccache
RUSTFLAGS: "-C link-arg=-fuse-ld=lld"
steps:
- uses: Mesh-LLM/mesh-llm/.github/actions/audit-depot-pr-isolation@e0dd4586b585d2a20ee8e6d2c786424521dfdcfe
with:
original_event_name: ${{ inputs.original_event_name }}
- uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5.1.0
with:
ref: ${{ inputs.source_sha || github.sha }}
Expand Down
Loading
Loading