ci(OMN-18291): route one container-backed job class to the cloud CI fleet

.github/workflows/application-acl-postgres16-proof.yml

@@ -26,22 +26,143 @@ concurrency:
   cancel-in-progress: true
 
 jobs:
+  # OMN-18291. THE CAPACITY REQUEST, and it exists because the AWS cloud runner
+  # fleet is scale-to-zero: at rest it holds no instances, so a job routed there
+  # with nothing to start it simply queues. Scale-DOWN has been automatic since
+  # OMN-18224 (an exit trap, bounded by two guard alarms); this is the other
+  # half.
+  #
+  # WHY IT IS A JOB HERE RATHER THAN A WEBHOOK IN AWS. GitHub's queued-job
+  # webhook is gated on the fleet's App holding read access to Actions, and that
+  # App deliberately holds one permission and no active webhook. Widening an
+  # organisation-wide App to solve a capacity problem is the wrong trade, so the
+  # trust runs the other way: the workflow that wants a cloud runner asks for
+  # one over OIDC, before the routed job is scheduled. No credential is stored
+  # anywhere.
+  #
+  # SKIPPED FOR A FORK PULL REQUEST, and the routed job below is skipped with
+  # it -- this repository is public, and GitHub refuses a fork pull request the
+  # `id-token: write` permission, so no token is minted for untrusted code at
+  # all. The fork guard in the routed job's own expression sends it to hosted
+  # runners in that case, which is the behaviour it already had.
+  request-cloud-capacity:
+    name: Ask the cloud fleet for a runner
+    if: >-
+      github.event_name != 'pull_request' ||
+      github.event.pull_request.head.repo.full_name == github.repository
+    # Hosted by literal, necessarily and deliberately. This job is the thing
+    # that causes the fleet to exist for this run, so it cannot resolve through
+    # the routing variable it is serving -- and pinning it by literal means no
+    # future flip of any seam can relocate the requester onto the capacity it is
+    # trying to request. Allowlisted in config/runner_routing_policy.yaml for
+    # exactly that reason.
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    permissions:
+      id-token: write
+      contents: read
+    steps:
+      - name: Assume the capacity-scaler invoker role
+        uses: aws-actions/configure-aws-credentials@d45db47bd94e3a7d50b9c36da4cfe07ceca89d74 # v6
+        with:
+          aws-region: us-east-1
+          role-to-assume: arn:aws:iam::${{ vars.AWS_ACCOUNT_ID }}:role/${{ vars.AWS_CI_RUNNER_SCALER_ROLE_NAME }}
+          role-session-name: gha-acl-proof-capacity-${{ github.run_id }}
+
+      - name: Ask for one runner
+        run: |
+          set -euo pipefail
+
+          # This workflow has exactly one fleet-bound job, so it wants exactly
+          # one runner. The FUNCTION owns the decision: it reads the group's
+          # live maximum, clamps to it, and refuses to lower the dial, which is
+          # why this job holds no autoscaling permission at all.
+          aws lambda invoke \
+            --region us-east-1 \
+            --function-name omninode-ci-runner-scaler \
+            --cli-binary-format raw-in-base64-out \
+            --payload '{"action":"scale_up","requested_runners":1}' \
+            --query '{status:StatusCode,error:FunctionError}' \
+            --output json \
+            scaler-response.json
+
+          echo "scaler said:"
+          cat scaler-response.json
+          echo
+
+          # A function that RAISED and a function that THREW both return HTTP
+          # 200. Reading only the status code is how a failed scale-up reports
+          # success and the job below then sits queued with no explanation.
+          if grep -q '"errorMessage"' scaler-response.json; then
+            echo "::error::the capacity scaler refused the request"
+            exit 1
+          fi
+
   rebuilt-postgres16-proof:
     name: Rebuilt PostgreSQL 16 ACL and rollback proof
+    # Ordered behind the request, not alongside it. Without `needs:` both jobs
+    # start together, this one is offered to a fleet that has not been asked for
+    # capacity, and it queues.
+    needs: request-cloud-capacity
+    # `always()` rather than a bare dependency, because a fork pull request
+    # skips the requester and this job must still run -- on hosted runners,
+    # which is where its own fork guard already sends it.
+    if: >-
+      always() &&
+      needs.request-cloud-capacity.result != 'failure' &&
+      needs.request-cloud-capacity.result != 'cancelled'
     env:
       COMPOSE_PROJECT_NAME: application-acl-proof-${{ github.run_id }}-${{ github.run_attempt }}
     permissions:
       contents: read
+    # OMN-18291: the cloud canary variable sits AHEAD of the docker seam, which
+    # sits ahead of the trusted seam. Precedence is the whole point -- the
+    # narrower variable governs this ONE job class and leaves every other
+    # docker-executing job in this repository exactly where the docker seam puts
+    # it. Flipping the docker seam instead would have moved fifteen job
+    # definitions, several of them on every pull request, onto a fleet whose
+    # maximum is four ephemeral single-job instances.
+    #
+    # The fork branch is untouched and still comes first: a fork pull request
+    # never reaches a self-hosted runner of any kind.
     runs-on: >-
       ${{
         (github.event_name == 'pull_request' &&
          github.event.pull_request.head.repo.full_name != github.repository)
         && fromJSON(vars.OMNI_PUBLIC_PR_RUNS_ON_JSON || '["ubuntu-latest"]')
-        || fromJSON(vars.OMNI_DOCKER_CI_RUNS_ON_JSON || vars.OMNI_TRUSTED_CI_RUNS_ON_JSON || '["self-hosted","omnibase-ci"]')
+        || fromJSON(vars.OMNI_CLOUD_CANARY_RUNS_ON_JSON || vars.OMNI_DOCKER_CI_RUNS_ON_JSON || vars.OMNI_TRUSTED_CI_RUNS_ON_JSON || '["self-hosted","omnibase-ci"]')
       }}
     timeout-minutes: 30
 
     steps:
+      - name: Prove which runner this job landed on
+        # OMN-18291. A green check does not say WHERE a job ran, and the whole
+        # claim of this routing change is about where. The runner name is
+        # assigned by the fleet's bootstrap; the instance metadata service is a
+        # second, independent discriminator that a lab runner -- a container on
+        # a machine with no metadata service -- does not have.
+        #
+        # Deliberately NOT fatal. If this job is on a hosted or lab runner the
+        # routing did not take effect, and that is a finding to read in the log
+        # rather than a reason to fail a proof that is otherwise valid.
+        run: |
+          set -uo pipefail
+          echo "runner name: ${RUNNER_NAME:-<unset>}"
+          case "${RUNNER_NAME:-}" in
+            omni-cloud-i-*)
+              TOKEN="$(curl -sS -X PUT --max-time 5 \
+                -H 'X-aws-ec2-metadata-token-ttl-seconds: 60' \
+                http://169.254.169.254/latest/api/token)"
+              echo "instance: $(curl -sS --max-time 5 \
+                -H "X-aws-ec2-metadata-token: ${TOKEN}" \
+                http://169.254.169.254/latest/meta-data/instance-id)"
+              echo "PLACEMENT: cloud fleet"
+              ;;
+            *)
+              echo "PLACEMENT: not the cloud fleet"
+              ;;
+          esac
+
       - name: Checkout code
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
         with:

config/runner_routing_policy.yaml

@@ -379,6 +379,102 @@ docker_ci_runner_variable:
   org_scope: absent
   revert_when: >-
     This value may go to '["self-hosted","omnibase-ci"]' only after a fleet run of "Customer Path Boundary (OMN-18012)" with the post-#3286 harness reports success on a named omninode-runner-*. Revert it if any docker-executing job reports a container it cannot reach from the test process, which is the 2026-09-07 signature, or if a registry push appears among the jobs it governs -- registry pushes are pinned hosted, not routed.
+cloud_canary_runner_variable:
+  name: OMNI_CLOUD_CANARY_RUNS_ON_JSON
+  # OMN-18291, 2026-09-13. THE FIRST ROUTING VARIABLE IN THIS ESTATE THAT POINTS
+  # AT THE AWS CLOUD FLEET, and the narrowest one: it governs exactly ONE job
+  # class, `rebuilt-postgres16-proof` in
+  # .github/workflows/application-acl-postgres16-proof.yml, and it is read
+  # AHEAD of the docker seam so that flipping it moves that job and nothing
+  # else.
+  #
+  # WHY NOT THE DOCKER SEAM, which is the surface a reader would reach for
+  # first. That variable governs FIFTEEN job definitions in this repository,
+  # measured from each job's parsed runs-on on dev rather than from a file
+  # grep. Several of them run on every pull request and one, "Customer Path
+  # Boundary (OMN-18012)", feeds the required CI Summary umbrella. The cloud
+  # fleet's maximum is FOUR ephemeral instances, each of which runs exactly one
+  # job and then terminates, and a boot costs minutes. Pointing fifteen job
+  # definitions at four single-job instances is not a canary; it is the
+  # saturation failure recorded in the trusted-seam entry above, repeated on a
+  # twentieth of the capacity. So the docker seam is left exactly where it is
+  # and the canary gets a variable of its own.
+  #
+  # WHY THIS JOB CLASS. It is a real job on a normal trigger, not a synthetic
+  # probe: it runs `docker compose up --build` against PostgreSQL 16 and proves
+  # an ACL matrix and its rollback. It is genuinely container-backed, which is
+  # the one capability this fleet has that the lab fleet does not -- a lab
+  # runner is itself a container reaching a shared daemon through a mounted
+  # socket, so a published port lands on the Docker host rather than the
+  # runner's own localhost. Its trigger is path-filtered rather than
+  # every-pull-request, it appears in no required-context list and in no
+  # EXPECTED_EXTERNAL_CONTEXTS tuple, and its fork guard already sends fork
+  # pull requests to hosted runners.
+  #
+  # THE HONEST COST, stated rather than left for someone to discover. This
+  # repository is PUBLIC, so its GitHub-hosted minutes are free and a cloud
+  # instance is not. This flip therefore costs money where the omnibase_core
+  # move recorded above saved it. That is accepted deliberately: the purpose is
+  # to prove the scale-up trigger and the routing path end to end on a real job
+  # with a small blast radius, not to save minutes on this particular job. The
+  # repositories where the fleet would pay for itself are the PRIVATE ones the
+  # runner group grants, and moving one of those is a separate claim with its
+  # own capacity measurement.
+  #
+  # WHAT THE FIRST REAL ROUTED RUN FOUND, recorded here rather than only in the
+  # ticket, because it is the second instance of a class that has now forced one
+  # routing revert and nearly forced a second. Run 34750171624, instance
+  # i-03f7450c38317c468: the job landed on the fleet correctly, proved its own
+  # placement from the instance metadata service, ran `docker info` green
+  # against a healthy LOCAL daemon -- and then exited 125 on `docker compose`,
+  # because Amazon Linux 2023's `docker` package ships buildx as its only CLI
+  # plugin and compose was simply not a command. Fifty milliseconds separate the
+  # healthy daemon from the missing plugin in that log, and the CLI answers a
+  # missing plugin by printing its usage text, so the failure reads as neither
+  # "not found" nor a daemon error.
+  #
+  # THE GENERAL RULE, which is the part worth carrying forward: a routing change
+  # is not finished when the variable is set and the job lands. It is finished
+  # when a real job OF THAT CLASS has run to completion there. Placement and
+  # capability are different claims, and a green placement proof next to a red
+  # job is exactly what the difference looks like. Fixed by installing the
+  # plugin in the fleet's own bootstrap, pinned by version and verified by
+  # digest (omninode_infra, OMN-18291), not by narrowing this job.
+  #
+  # AND THEN THE SAME CLASS AGAIN, one line further on. Run 34750597454,
+  # instance i-0b7b99b0fb129eb15: compose now existed and teardown passed, and
+  # the build stopped with "compose build requires buildx 0.17.0 or later". The
+  # distribution DOES ship a buildx plugin, so every presence check passes and
+  # `docker info` lists one -- it is 0.12.1. A tool that is present and too old
+  # is invisible to a check that asks whether the tool is there, which is why
+  # the only check that found either gap was running the real job. Both plugins
+  # are now installed from pinned, digest-verified release binaries and the boot
+  # asserts both before the instance registers.
+  #
+  # AND A THIRD, which is the one worth reading if you only read one. Run
+  # 34751054047, instance i-0298a739880307693: the image built, both containers
+  # started, PostgreSQL 16 initialised cleanly -- and the seed failed with
+  # "Permission denied" on a file the build had just copied in. Cause: the boot
+  # script sets a restrictive umask for credential hygiene, a umask is
+  # inherited, and it reached the runner process, so every checked-out file
+  # landed mode 0600. `COPY` preserves that faithfully, and any image that drops
+  # to a non-root user then cannot read what it was given. The official
+  # PostgreSQL image does exactly that.
+  #
+  # Blast radius: every container-backed job that copies a checked-out file into
+  # an image running as anything but root, which is most of them. Hosted runners
+  # use the ordinary mask, so a job that has only ever run hosted cannot have
+  # seen it. Fixed by running the job under the ordinary mask while the boot
+  # script's own credential handling keeps the restrictive one.
+  #
+  # Three capability gaps, none visible offline, all found by one real job of
+  # one real class. That is the argument for a canary of exactly this shape.
+  expected_json: '["self-hosted","omni-cloud-ci"]'
+  repositories:
+    - omnibase_infra
+  org_scope: absent
+  revert_when: >-
+    Revert to '["ubuntu-latest"]' if the routed job lands on the fleet and fails for a fleet-caused reason -- a container it cannot reach from the test process, a missing tool on the runner image, or a boot that never produces a runner -- or if the job is ever observed QUEUED rather than running, which means the capacity trigger did not fire. Revert also if this job class is ever added to a required-context list or to EXPECTED_EXTERNAL_CONTEXTS, because its blast radius is then no longer small. Widening this variable to a second job class is NOT a revert condition, it is a new claim: the group's maximum is four ephemeral single-job instances and a second class needs its own capacity measurement first.
 security_scan_runner_variable:
   name: OMNI_SECURITY_SCAN_RUNS_ON_JSON
   expected_json: '["ubuntu-latest"]'
@@ -460,6 +556,8 @@ repositories:
   # so the scoped-variable pass now asserts their ABSENCE here too.
   - omninode_infra
 hosted_runner_allowlist:
+  - path: .github/workflows/application-acl-postgres16-proof.yml
+    reason: "OMN-18291: the `request-cloud-capacity` job is bare hosted BY NECESSITY -- it is the job that asks the scale-to-zero cloud fleet for the runner the routed job below it needs, so it cannot itself resolve through the routing variable it is serving. Pinning it by literal also means no future flip of any seam can relocate the requester onto the capacity it is trying to request. The workflow's other job, `rebuilt-postgres16-proof`, is NOT hosted -- it is the canary routed to the cloud fleet through cloud_canary_runner_variable."
   - path: .github/workflows/ci.yml
     reason: "OMN-14127: CI Summary is a lightweight hosted fail-closed aggregator so required status always posts during self-hosted saturation."
   - path: .github/workflows/build-and-push-runtime.yml

scripts/audit-runner-routing.py

@@ -307,6 +307,13 @@ def runner_variable_for_event(
 SCOPED_VARIABLE_KEYS = (
     "docker_ci_runner_variable",
     "security_scan_runner_variable",
+    # OMN-18291. The narrowest routing surface in the estate: one variable
+    # governing ONE job class, read ahead of the docker seam. It is audited on
+    # the same terms as the two above, and the ABSENCE half matters more here
+    # than anywhere else -- this variable points at a cloud fleet that BILLS
+    # per instance, so an undeclared shadow appearing on another repository is
+    # a shadow that spends money, not merely one that moves a job.
+    "cloud_canary_runner_variable",
 )