Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 8 additions & 23 deletions .github/workflows/depot-registry-canary.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,6 @@ on:
permissions:
contents: read

env:
DEPOT_PROJECT_ID: mzm95zcv7p

concurrency:
group: depot-registry-canary
cancel-in-progress: false
Expand Down Expand Up @@ -74,33 +71,16 @@ jobs:
needs: policy
permissions:
contents: read
id-token: write
strategy:
fail-fast: false
matrix:
source: [upstream, depot]
sample: [1, 2, 3, 4, 5]
# Use GitHub's OIDC issuer for the Depot project trust relationship while
# retaining a fresh hosted VM for every timing sample.
runs-on: ubuntu-24.04
# Depot provisions a fresh runner for every sample and pre-authenticates it
# to the organization Registry with a short-lived job credential.
runs-on: depot-ubuntu-24.04
timeout-minutes: 15
steps:
- name: Set up Depot CLI
if: ${{ matrix.source == 'depot' }}
uses: depot/setup-action@15c09a5f77a0840ad4bce955686522a257853461 # v1
with:
version: 2.101.77
- name: Authenticate to Depot Registry
if: ${{ matrix.source == 'depot' }}
shell: bash
env:
DEPOT_REGISTRY_HOST: ${{ vars.DEPOT_REGISTRY_HOST }}
run: |
set -euo pipefail

depot pull-token --project "$DEPOT_PROJECT_ID" |
docker login "$DEPOT_REGISTRY_HOST" --username x-token --password-stdin

- name: Pull exact image on a fresh runner
shell: bash
env:
Expand All @@ -112,6 +92,11 @@ jobs:
run: |
set -euo pipefail

if [[ "${DEPOT_ORG_ID:-}" != "1ntz5vlngn" ]]; then
echo "registry canaries require a pre-authenticated Mesh-LLM Depot runner" >&2
exit 1
fi

image="$UPSTREAM_IMAGE"
if [[ "$SOURCE" == "depot" ]]; then
image="$DEPOT_IMAGE"
Expand Down
16 changes: 9 additions & 7 deletions ci/DEPOT_MIGRATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -444,13 +444,15 @@ token to PR code.
The checked-in `depot-registry-canary.yml` implements that measurement gate for
any digest-pinned public base or runner image. Configure each upstream
repository as a distinct Depot pull-through repository, set the nonsecret
`DEPOT_REGISTRY_HOST` repository variable, and permit the job's GitHub OIDC
identity to mint a short-lived read-only `depot pull-token`. No stored registry
secret is required. Run the workflow from `main` with the exact upstream digest
and relative Depot repository name. It allocates five fresh ephemeral runners
per source, verifies digest identity, retains raw timing observations for 14
days, and reports whether both thresholds pass. Do not enable a mirror in
normal builds until its own retained cohort passes.
`DEPOT_REGISTRY_HOST` repository variable, and enable Depot's native Actions-job
Registry access for the organization. Depot pre-authenticates each trusted
ephemeral runner with a short-lived job credential, so no stored registry secret
or workflow-minted pull token is required. Run the workflow from `main` with the
exact upstream digest and relative Depot repository name. It allocates five
fresh ephemeral runners per source, verifies the injected Depot organization
identity and digest identity, retains raw timing observations for 14 days, and
reports whether both thresholds pass. Do not enable a mirror in normal builds
until its own retained cohort passes.

### `Mesh-LLM/mesh-packaging`

Expand Down
8 changes: 4 additions & 4 deletions ci/METRICS.md
Original file line number Diff line number Diff line change
Expand Up @@ -197,10 +197,10 @@ cold-pull, and publication-time thresholds in
Use the manual `depot-registry-canary.yml` workflow for registry comparisons.
The upstream input must be digest-pinned, and the Depot repository must mirror
that exact upstream repository. Each source receives five fresh ephemeral
GitHub-hosted runners so local layer reuse cannot turn a warm local pull into a
false registry result while Depot authenticates through the project's GitHub
Actions OIDC trust relationship. Downloaded observation artifacts can be
reevaluated with:
Depot-managed runners so local layer reuse cannot turn a warm local pull into a
false registry result. Depot pre-authenticates each ephemeral runner with a
short-lived organization Registry job credential. Downloaded observations can
be reevaluated with:

```bash
python3 scripts/summarize-depot-registry-pulls.py \
Expand Down
4 changes: 2 additions & 2 deletions ci/ci.md
Original file line number Diff line number Diff line change
Expand Up @@ -616,8 +616,8 @@ public reference with a configured Depot mirror using five fresh ephemeral
runner samples per source. The workflow verifies that every pull resolves to
the same manifest digest and requires both 20% and 10 seconds of median pull
improvement before a mirror is eligible for broader use. Its read-only pull
token is minted through GitHub OIDC only in the cached pull step; no stored
registry secret is used, and the OIDC permission is never available to PR code.
access comes from Depot's short-lived job credential on each trusted ephemeral
runner; no stored registry secret or workflow-minted pull token is used.
This measures registry transfer only; it does not measure package-manager,
Cargo, npm/pnpm, native compilation, or Docker export work.

Expand Down
18 changes: 12 additions & 6 deletions scripts/tests/test_depot_registry_canary_workflow.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,19 +16,25 @@ def test_canary_is_manual_and_main_only(self) -> None:
self.assertNotIn("push:", self.workflow)
self.assertIn('"refs/heads/main"', self.workflow)

def test_pull_token_is_short_lived_and_oidc_scoped(self) -> None:
self.assertIn("id-token: write", self.workflow)
self.assertIn("depot/setup-action@15c09a5f77a0840ad4bce955686522a257853461", self.workflow)
self.assertIn('depot pull-token --project "$DEPOT_PROJECT_ID"', self.workflow)
self.assertIn("docker login", self.workflow)
def test_registry_auth_is_native_and_job_scoped(self) -> None:
self.assertIn("runs-on: depot-ubuntu-24.04", self.workflow)
self.assertIn('"${DEPOT_ORG_ID:-}" != "1ntz5vlngn"', self.workflow)
self.assertIn(
"registry canaries require a pre-authenticated Mesh-LLM Depot runner",
self.workflow,
)
self.assertNotIn("id-token: write", self.workflow)
self.assertNotIn("depot pull-token", self.workflow)
self.assertNotIn("docker login", self.workflow)
self.assertNotIn("DEPOT_PROJECT_ID", self.workflow)
self.assertNotIn("secrets.", self.workflow)
self.assertNotIn("DEPOT_REGISTRY_PULL_TOKEN", self.workflow)
self.assertNotIn("printenv", self.workflow)

def test_canary_uses_fresh_runner_samples_and_exact_digest(self) -> None:
self.assertIn("source: [upstream, depot]", self.workflow)
self.assertIn("sample: [1, 2, 3, 4, 5]", self.workflow)
self.assertIn("runs-on: ubuntu-24.04", self.workflow)
self.assertEqual(self.workflow.count("runs-on: depot-ubuntu-24.04"), 1)
self.assertIn("upstream_image must be pinned by sha256 digest", self.workflow)
self.assertIn("digest mismatch", self.workflow)

Expand Down
Loading