Skip to content

ci: build runner images once before digest promotion - #9

Merged
ndizazzo merged 3 commits into
mainfrom
codex/build-once-digest-promotion
Jul 30, 2026
Merged

ndizazzo merged 3 commits into
mainfrom
codex/build-once-digest-promotion

Conversation

@ndizazzo

@ndizazzo ndizazzo commented Jul 29, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • build every selected public/self-hosted backend platform once through one reusable family workflow, then reuse the exact staged digest for verification, index assembly, and promotion
  • make pull requests lightweight by selecting affected image families plus one mandatory public CPU AMD64 contract row; Dockerfile/common/unknown inputs fail open to the exhaustive matrix
  • keep pull-request caches read-only: PR jobs restore the trusted scope but export no BuildKit cache; trusted main staging owns the maximal shared cache
  • separate execution into explicit validate, stage, and promote modes: manual dispatch defaults to validation, main pushes stage candidates only, and the weekly schedule or an explicit main dispatch performs promotion
  • derive runner labels inside the reusable workflow from exact repository/ref/caller-workflow/event/variable checks; caller-controlled runner JSON is no longer accepted
  • keep PRs and feature refs on GitHub-hosted runners; allow Depot only for trusted default-branch execution when DEPOT_RUNNERS_ENABLED=true
  • replace source-pair immutable tags with collision-proof *-digest-sha256-<manifest-digest> tags while retaining both full source revisions in OCI labels and candidate descriptors
  • record the complete target/previous latest map in a retained cohort manifest and reconcile it serially/idempotently after all versioned promotions and attestations succeed
  • verify both the MeshLLM revision and runner-images revision inside each exact staged image
  • remove obsolete provider references and retain only the GitHub/Depot provider model

Execution contract

Event Mode Image scope Registry mutation Cache export Runner provider
Pull request validate affected families + public CPU AMD64 contract none none GitHub-hosted
Feature-ref manual dispatch validate only exhaustive none none GitHub-hosted
Main manual validation validate exhaustive none stable canary mode=min gated Depot, GitHub fallback
Main push stage exhaustive run-scoped candidates and verified indexes; no production aliases trusted mode=max gated Depot, GitHub fallback
Main scheduled/manual promotion promote exhaustive same-run staging, versioned/content tags, then latest-cohort reconciliation trusted mode=max gated Depot, GitHub fallback

Performance evidence

Pull-request run Wall Allocated jobs Aggregate job time Slowest platform
30501276174, maximal PR cache export 22m57s 22 2h52m59s self-hosted ROCm 7.2 AMD64, 22m20s
30504335079, read-only PR cache 6m22s 22 1h13m07s self-hosted ROCm 7.2 AMD64, 5m48s

The corrected run passed all 22 allocated jobs and all 20 platform builds. It reduced wall time by 72.3%, aggregate job time by 57.7%, and the critical platform row by 74.0%. Queueing was at most 9 seconds. All allocated jobs used the GitHub Actions group with only ubuntu-24.04 and ubuntu-24.04-arm; zero Depot jobs were assigned to the pull request. The completed logs contain no actual sending cache export or exporting cache phase.

Both measured runs are exhaustive because this PR changes the Dockerfile. Once merged, workflow/docs/policy-only pull requests will select only the mandatory public CPU AMD64 contract row; image-input changes select their affected environment/backend families, and common or unknown image inputs continue to fail open to the complete matrix.

Historical context: a representative pre-restructure PR took 17m12s; representative publication runs took 39m15s and 48m01s.

Rollout boundary

This PR is merged. Depot access still requires the organization runner group to be restricted to this repository and both default-branch workflows documented in docs/OPERATIONS.md. The live repository currently has no main-branch ruleset or branch protection, so public Depot access remains blocked until main protection, required review for workflow changes, and selected-workflow runner-group restrictions are verified. This PR does not change repository settings.

Candidate cleanup also remains deliberately conservative: GHCR package-version deletion can remove promoted references sharing a digest. The operations guide records the 14-day target and the proof required before automating deletion.

Validation

  • actionlint 1.7.12
  • ShellCheck and Bash syntax
  • candidate descriptor and exact-digest promotion tests
  • content-derived immutable/mutable tag contract tests
  • latest-cohort generation, preflight, reconciliation, and duplicate-tag tests
  • workflow mode selector and feature/external trust rejection tests
  • runner selector, generated matrix, and affected-family planner tests
  • JSON/config validation
  • full-SHA action pins and checkout credential contract
  • git diff --check
  • case-insensitive obsolete-provider sweep

Summary by CodeRabbit

  • New Features

    • Added staged validation, image staging, digest-based promotion, and latest-image reconciliation workflows.
    • Added support for manual workflow operations: validate, stage, and promote.
    • Added runner image family configuration for public and self-hosted environments.
    • Added image verification with architecture, backend, version, and revision checks.
    • Added immutable digest-based image tags and expanded publication metadata.
  • Documentation

    • Updated build, maintenance, benchmarking, operations, and tag-contract documentation.
  • Tests

    • Added coverage for descriptor validation, promotion, reconciliation, tag generation, workflow matrices, and runner selection.

Post-merge evidence

Post-merge main run 30522118156 completed successfully at merge commit 4e79e68e22a5ea9bb1eedf9a2a7e7ccfc20b2bca: all 20 platform rows and 13 assembly/index jobs passed, with no failed, cancelled, or timed-out jobs. The local untracked .omo/ directory was not added or modified.

@coderabbitai

coderabbitai Bot commented Jul 29, 2026 •

Copy link
Copy Markdown

Review Change Stack

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: e4b9557b-a0b8-488a-bd65-af1f607605bc

📥 Commits

Reviewing files that changed from the base of the PR and between 890cdc6 and b96b5fb.

📒 Files selected for processing (29)
  • .dockerignore
  • .github/actionlint.yaml
  • .github/workflows/build-and-push.yml
  • .github/workflows/stage-image-family.yml
  • Dockerfile
  • Dockerfile.verify
  • README.md
  • config/runner-image-families.json
  • docs/CI_BENCHMARKS.md
  • docs/OPERATIONS.md
  • scripts/generate-latest-cohort.sh
  • scripts/generate-promotion-tags.sh
  • scripts/generate-workflow-matrices.sh
  • scripts/install-actionlint.sh
  • scripts/plan-pr-families.sh
  • scripts/promote-image-digest.sh
  • scripts/reconcile-image-cohort.sh
  • scripts/resolve-backend.sh
  • scripts/select-runner-provider.sh
  • scripts/select-workflow-mode.sh
  • scripts/validate-candidate-descriptor.sh
  • scripts/verify-runner-image.sh
  • tests/candidate-promotion.test.sh
  • tests/cohort-promotion.test.sh
  • tests/fixtures/candidate-index-manifest.json
  • tests/fixtures/candidate-index-valid.json
  • tests/fixtures/mock-docker.sh
  • tests/promotion-tags.test.sh
  • tests/workflow-matrix.test.sh

📝 Walkthrough

Walkthrough

The pull request replaces the monolithic image publication workflow with policy-driven validation, matrix-based family staging, immutable descriptor verification, digest promotion, and latest-cohort reconciliation. It also adds runner family configuration, revision metadata checks, operational documentation, and integration tests.

Changes

Runner image pipeline

Layer / File(s) Summary
Image metadata and descriptor contracts
.dockerignore, config/runner-image-families.json, Dockerfile*, scripts/validate-candidate-descriptor.sh, scripts/verify-runner-image.sh, tests/fixtures/*
Runner-image revisions, accelerator metadata, architecture children, and strict candidate descriptor schemas are added to image builds and verification.
Workflow policy and matrix planning
.github/actionlint.yaml, .github/workflows/build-and-push.yml, scripts/generate-workflow-matrices.sh, scripts/plan-pr-families.sh, scripts/select-*.sh, scripts/install-actionlint.sh, tests/workflow-matrix.test.sh
Workflow inputs, execution modes, runner-provider selection, family matrices, PR family planning, workflow validation, and contract tests are added.
Platform staging and index assembly
.github/workflows/stage-image-family.yml, .github/workflows/build-and-push.yml
Reusable jobs build platform images, verify candidate descriptors, and assemble multi-architecture family indexes for validation and staging.
Digest promotion and cohort reconciliation
.github/workflows/build-and-push.yml, scripts/generate-*.sh, scripts/promote-image-digest.sh, scripts/reconcile-image-cohort.sh, tests/*promotion.test.sh, tests/fixtures/mock-docker.sh
Versioned and immutable tags are generated and promoted by digest; latest tags are reconciled from a validated cohort manifest.
Operational contracts and rollout documentation
README.md, docs/CI_BENCHMARKS.md, docs/OPERATIONS.md
Documentation covers execution modes, Depot gating, tag semantics, retention rules, benchmarks, and revision-aware local builds.

Estimated code review effort: 5 (Critical) | ~120 minutes

Possibly related PRs

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/build-once-digest-promotion

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@ndizazzo
ndizazzo marked this pull request as ready for review July 30, 2026 07:11
@ndizazzo
ndizazzo merged commit 4e79e68 into main Jul 30, 2026
39 of 40 checks passed
@ndizazzo
ndizazzo deleted the codex/build-once-digest-promotion branch July 30, 2026 07:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant