ci: enforce egress-block on the three issue-triggered AI jobs

.github/CLAUDE.md

@@ -73,9 +73,13 @@ regardless of how convenient it is.
   `claude-pr-loop` (`fix`/`verify`), `claude-review`, `claude` and `bestaxbot-reply` all check
   out a branch and run repository code in the same job as that token, and they must — fixing and
   reviewing code is the job. What holds them is a different, weaker set: the tool allowlist, the
-  protected-path deny rules, the trusted-labeler gates, and (once #578 lands) an enforced egress
-  policy. Saying I1 covers those jobs would be the exact overstatement the checklist at the
-  bottom of this file ends on. It covers the pipeline that was designed around it.
+  protected-path deny rules, the trusted-labeler gates, and an enforced egress policy — which as
+  of #578 covers `claude`, `claude-implement` and `bestaxbot-reply`, while `claude-review` and
+  `claude-pr-loop`'s `fix`/`verify` are still at `audit` and so have that last leg only in name.
+  Check the rule 10 inventory before citing it for a specific job rather than reading this
+  sentence as a group claim. Saying I1 covers those jobs would be the exact overstatement the
+  checklist at the bottom of this file ends on. It covers the pipeline that was designed around
+  it.
 
   harden-runner's egress block enforces again as of #487 — `block` used to degrade silently to
   `audit`, and now enforces and is asserted (rule 10). **Do not promote it to a third leg of
@@ -337,13 +341,13 @@ Copy the full version from any block job — the three shapes above are each loa
 
 - **The `-e` check comes first** because harden-runner has deliberate paths that install nothing
   and still exit 0 (a StepSecurity outage, the `skip-harden-runner` repo property, a container or
-  slim runner). Without it a vendor outage fails every block job at once — thirteen of them as of
+  slim runner). Without it a vendor outage fails every block job at once — sixteen of them as of
   the inventory below, not the seven this line said before the `ai-scan`/`ai-triage` job splits —
   with a bare `jq: could not open file`. It still fails, because a job holding a credential must
   not run unprotected, but it says why. Do not hand-maintain that number: it is the count of
   block-mode jobs, and it has been wrong once already. Derive it, and grep for the **key** rather
   than the string — `grep -rn "^ *egress-policy: block$"`. A plain `grep egress-policy: block`
-  returns fourteen: `deploy-worker.yml` names the policy in a comment, which is the same way a
+  returns seventeen: `deploy-worker.yml` names the policy in a comment, which is the same way a
   `node -e` in a comment misclassifies a job below.
 - **The status check polls** rather than testing once. The pre-step waits only for the file to
   _exist_ and gives up after ~9s, while the agent resolves every allow-listed host before writing
@@ -474,11 +478,14 @@ jobs (`ci.yml`, `deploy.yml`, `test-deploy.yml`, `visual-regression.yml`, `story
 - **Enforcing and asserted** — all three `ai-scan` jobs (`gate`, `scan`, `label`), all four
   `ai-triage` jobs (`gate`, `triage`, `publish`, `cleanup`), `claude-repro` (`author` **only**),
   `deploy-worker` (`deploy`), `supply-chain` (`consumer-sbom` and `sign-sbom`),
-  `security-txt-expiry` (`check`), `auto-close-duplicates` (`auto-close`). Thirteen jobs; the
-  command below is the check.
-- **Audit, deliberately, pending a measured allowlist** — `claude`, `claude-implement`,
-  `claude-pr-loop` (`fix` and `verify`), `claude-review`, `bestaxbot-reply`. These run repo code
-  with a model token; their block flip is the follow-up this rule owes, tracked in #578.
+  `security-txt-expiry` (`check`), `auto-close-duplicates` (`auto-close`), `claude` (`claude`),
+  `claude-implement` (`implement`), `bestaxbot-reply` (`respond`). Sixteen jobs; the command
+  below is the check.
+- **Audit, deliberately, pending a measured allowlist** — `claude-pr-loop` (`fix` and `verify`)
+  and `claude-review`. These run repo code with a model token; their block flip is the remainder
+  of the follow-up this rule owes, tracked in #578. `fix` and `review` are measured and waiting
+  only on their own PR; `verify` is the one job with no measurement yet, because it runs only
+  when a deep review files a blocking inline finding (#593).
 - **No harden-runner at all**, and these are two different groups — do not merge them into one
   "API-only" line, which understates the second:
   - _Genuinely API-only_ — no checkout, no repository code: `on-slop`, `auto-label-claude-prs`,
@@ -536,6 +543,48 @@ allow-listed host (see I1). Widening an allowlist remains a security change unde
 Verify rather than assume, on any run: the assertion step passes, and harden-runner's post-step
 prints the effective `EgressPolicy:`.
 
+#### Measuring an allowlist
+
+An audit-mode run's endpoints are in the **run log**, not only the StepSecurity web UI — the
+`Post Harden runner` step dumps the agent log, which records every DNS resolution and connection.
+So this is a command rather than a transcription exercise, and it is the answer to "where does
+the list come from" wherever this rule says a list must be measured:
+
+```bash
+gh run view <run-id> --log \
+  | grep -oE 'domain: [a-z0-9.-]+\., pid' \
+  | sed 's/domain: //; s/\., pid//' | sort -u
+```
+
+Two things about reading its output, both learned assembling the #578 lists:
+
+- **Runner infrastructure appears in that log and must never be allow-listed.**
+  `results-receiver.actions.githubusercontent.com`,
+  `productionresultssa<N>.blob.core.windows.net`,
+  `run-actions-<N>-azure-*.actions.githubusercontent.com` and `hosted-compute-*.githubapp.com`
+  are the runner talking to its own control plane. The blob host cannot be pinned even in
+  principle — its name rotates per run (sa3, sa6, sa7, sa9, sa11 and sa13 across six runs). Run
+  33221210633 is the evidence that omitting them is right: `auto-close-duplicates` at `block`
+  with the five-host list observed only `api.github.com` and `github.com`, and completed all
+  fifteen of its API calls under the firewall.
+
+  **Leaving them out costs nothing, and that is measured rather than assumed.** harden-runner
+  does not gate the runner's own control-plane traffic, so the Actions cache keeps working:
+  run 33286967625 ran `claude-review` at `block` with neither the blob host nor
+  `results-receiver` allow-listed, and its `setup-node` step restored the pnpm cache
+  successfully before `pnpm install --frozen-lockfile` completed. Worth stating plainly
+  because the first revision of this bullet predicted a cache miss and called it an accepted
+  cost. That was a guess, it was wrong, and the run is what showed it. If you catch yourself
+  writing down what a policy will cost, go and read a run instead.
+
+- **One run is not a measurement.** Vendor telemetry samples, so a host can be absent from four
+  runs and present in the fifth — `telemetry.vercel.com` (turbo) showed up exactly once across
+  the six #578 runs. Prefer denying telemetry at the source (`CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC`,
+  `TURBO_TELEMETRY_DISABLED`) over allow-listing a host you would rather not talk to; that is the
+  standing call here, and it also keeps the next audit report clean for whoever reads it.
+
+Then resolve every host before it goes in the list, per the rule above.
+
 ## Review checklist for a workflow change
 
 - [ ] Does any allowlist grow? If yes, name the credential in that job and justify each entry.

.github/workflows/bestaxbot-reply.yml

@@ -77,18 +77,100 @@ jobs:
       issues: write # issue-comment replies
       id-token: write # OIDC exchange for the subscription token
     steps:
-      # AUDIT, not block, and rule 10's "existing live job" clause is why: this
-      # job checks out code, installs dependencies and runs a Claude session
-      # holding both bestaxbot's PAT and the OAuth token, and none of that egress
-      # has ever been measured. A guessed allowlist would break replies on the
-      # first miss; audit mode reports every endpoint the run touches without
-      # refusing any, which is what produces the list a block flip needs. That
-      # flip is the follow-up rule 10 owes (#578) — audit is a starting point
-      # here, not a resting place.
+      # BLOCK, with the allowlist MEASURED rather than guessed (#578). This job's
+      # audit run was 33231488110; all six runs the flip is built on are recorded
+      # on that issue, each read out of its own `Post Harden runner` log rather
+      # than transcribed from the StepSecurity UI.
+      #
+      # The eight hosts are the set already enforcing on ai-scan/scan,
+      # ai-triage/triage and claude-repro/author, and the measurement came out a
+      # strict subset of it: objects.githubusercontent.com,
+      # release-assets.githubusercontent.com and nodejs.org never appeared, because
+      # the Node 24 toolcache hit on every run. All three stay for the
+      # toolcache-miss path, and they are not interchangeable: the first two carry
+      # the actions/node-versions manifest and its release assets, while nodejs.org
+      # is setup-node v7 own fallback when that lookup misses.
+      #
+      # nodejs.org is the one host NOT in the Claude-session set copied above, and
+      # the reason is worth stating so it does not get trimmed back out: none of
+      # ai-scan/scan, ai-triage/triage or claude-repro/author runs setup-node at
+      # all, so that set never needed it. This job does. Every block job in this
+      # repo that runs setup-node allows it (auto-close-duplicates,
+      # security-txt-expiry, deploy-worker, supply-chain/consumer-sbom).
+      #
+      # Two observed hosts are deliberately absent. telemetry.vercel.com is turbo's
+      # telemetry, denied via TURBO_TELEMETRY_DISABLED below rather than
+      # allow-listed — the same call this repo already makes for the CLI's Datadog
+      # host. The Actions cache blob host cannot be pinned at all: its name rotates
+      # per run (productionresultssa3/6/7/9/11/13 across six runs). It does not need
+      # to be. harden-runner does not gate the runner own control-plane traffic, and
+      # run 33286967625 is the proof rather than the assumption: claude-review at
+      # block, with neither that host nor results-receiver.actions.githubusercontent.com
+      # allow-listed, restored its pnpm cache and completed pnpm install
+      # --frozen-lockfile. An earlier revision of this comment predicted a cache miss
+      # here; it was wrong, and measuring it is what showed that.
+      #
+      # Resolve any host before adding it here. An unresolvable entry is not inert:
+      # the agent reverts the firewall while the pre-step still exits 0, which is
+      # how statsig.anthropic.com silently disabled the policy on #577 (rule 10).
       - name: Harden runner
         uses: step-security/harden-runner@05e31511f85b41b11d1cf0ef85d0992719546e2c # v2.21.0
         with:
-          egress-policy: audit
+          egress-policy: block
+          allowed-endpoints: >
+            api.anthropic.com:443
+            claude.ai:443
+            downloads.claude.ai:443
+            api.github.com:443
+            github.com:443
+            objects.githubusercontent.com:443
+            release-assets.githubusercontent.com:443
+            registry.npmjs.org:443
+            nodejs.org:443
+
+      # Asserts the policy the pre-step decided (agent.json) AND that the agent
+      # actually came up (agent.status, written only once the firewall rules are
+      # installed). Either alone is a false pass: #487 was the first kind, and an
+      # unresolvable host in an allowlist is the second.
+      #
+      # Scope it honestly: this proves enforcement was armed AT THIS STEP. It does
+      # NOT prove the policy stays armed — the agent writes `Initialized` and then
+      # serves, and a later runtime error makes it revert the firewall with both
+      # files still reading exactly as they do here. Rule 10 in .github/CLAUDE.md
+      # carries the mechanism and what to check when a session looks wrong.
+      #
+      # Placement is rule 10's default (immediately after harden-runner): nothing
+      # `always()`-guarded keys off this job's outputs, and the job is not itself
+      # a fail-closed path, so neither placement exception applies.
+      #
+      # Linux-only paths (macOS uses /opt/step-security); all runners are ubuntu-latest.
+      - name: Assert egress policy is enforced
+        run: |
+          set -uo pipefail
+          # harden-runner has several deliberate "install nothing, exit 0" paths
+          # (StepSecurity unavailable, the skip-harden-runner repo property, a
+          # container or slim runner). None of them writes agent.json, so name
+          # that case rather than leaving a bare jq "could not open file".
+          if [ ! -e /home/agent/agent.json ]; then
+            echo "::error::harden-runner installed no agent, so nothing is enforcing egress. Usual causes: a StepSecurity outage (its pre-step returns green without installing), the skip-harden-runner repo property, or a container/slim runner. This job holds bestaxbot's PAT and a model token, and will not run unprotected (#487)."
+            exit 1
+          fi
+          if ! jq -e '.egress_policy == "block"' /home/agent/agent.json; then
+            echo "::error::effective egress policy is not block — harden-runner downgraded it (#487)."
+            exit 1
+          fi
+          # Poll rather than test once: the pre-step waits only for the file to
+          # EXIST and gives up after ~9s, while the agent resolves every
+          # allow-listed host before writing its status, so a cold resolver or a
+          # long list can leave the file absent or still empty here.
+          for _ in $(seq 1 30); do
+            if grep -q '^Initialized' /home/agent/agent.status 2>/dev/null; then
+              exit 0
+            fi
+            sleep 1
+          done
+          echo "::error::agent.json reads block but the agent never reported Initialized — the firewall is not up. Check the agent log for 'Reverted changes' (usually an unresolvable allow-listed host)."
+          exit 1
 
       # Layer 2 (authoritative): author_association can be COLLABORATOR with
       # only read access. Re-check the LIVE role and require write/triage+.
@@ -175,10 +257,17 @@ jobs:
         # it (#487). It also covers feature-flag evaluation, which is why
         # statsig.anthropic.com is gone.
         #
-        # On this `audit` job the point is the MEASUREMENT: audit mode exists to
-        # produce the endpoint list a block flip will use (#578), and leaving
-        # telemetry on would put the Datadog host in that report, inviting whoever
-        # reads it to allow-list a host we have just decided to deny.
+        # This job now enforces `block` (#578), so the denial is doing two jobs at
+        # once and both matter. The allowlist above refuses the Datadog host, and
+        # this setting stops the CLI trying to reach it, which keeps a denied call
+        # out of the run rather than merely blocked. Leaving telemetry on and
+        # relying on the allowlist alone would fill the logs with refused traffic
+        # and invite the next reader to "fix" it by allow-listing the host.
+        #
+        # It also earned the allowlist: this was set during the audit phase
+        # precisely so the endpoint list the flip was built from reflected the
+        # intended end state, rather than recording a host we had already decided
+        # to deny.
         #
         # A step `env:` rather than the action's `settings:` input, and here that
         # is load-bearing rather than tidiness: `settings:` is written to the
@@ -195,14 +284,20 @@ jobs:
         # `with:`.
         env:
           CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC: '1'
+          # Same call, second vendor: turbo phones telemetry.vercel.com from the
+          # pnpm commands this session runs. It appeared in only one of the six
+          # audit runs because turbo samples, which is precisely the intermittent
+          # that reddens a block job weeks after a flip. Denied, not allow-listed
+          # (#578).
+          TURBO_TELEMETRY_DISABLED: '1'
         uses: anthropics/claude-code-action@51c47629174b5dbca955cf93690fd39c8a41dadf # v1
         with:
           # Respond as the bestaxbot machine account so replies/commits come
           # from a real user (CodeRabbit engages, commits are SSH-Verified).
           github_token: ${{ secrets.AI_LOOP_PAT }}
           ssh_signing_key: ${{ secrets.AI_LOOP_SSH_SIGNING_KEY }}
           bot_name: bestaxbot
-          bot_id: "300268469"
+          bot_id: '300268469'
           claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
           use_commit_signing: true
           # Per-turn tool calls + denials in the log (the artifact blob host is