Skip to content

Add an on-demand Claude security review and a model-free flag on risky paths - #895

Closed
seathatflowsinourveins wants to merge 12 commits into
mainfrom
claude/api-actions-w4c-security-review-20261008
Closed

seathatflowsinourveins wants to merge 12 commits into
mainfrom
claude/api-actions-w4c-security-review-20261008

Conversation

@seathatflowsinourveins

@seathatflowsinourveins seathatflowsinourveins commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

Scope

  • What this PR changes, in one or two sentences: adds security-review-flag.yml, which on pull requests touching workflows, scripts/hooks/, tools/credentials/, the credential inventory or the paper-trading blueprints only writes a notice with the exact dispatch command (no model, no checkout, no token scope), and claude-security-review.yml, an on-demand, read-only security review of one same-repository pull request head by anthropics/claude-code-action v1.0.247, bounded at 30 assistant turns and a $5 client budget. The review runs only while CLAUDE_SECURITY_REVIEW_ENABLED is true, so merging this starts no model run.

  • Effort (added 2026-10-08 at head 7a131b5): --effort max in claude_args, because judgment work (designated reads, adjudication, pull request and security reviews, audits) runs at max under the command center's effort mapping, and every job records its level and reason. The level has to be in claude_args: --restricted ignores the settings files that would otherwise carry it, and Opus 5.5 runs at medium on the API when a request leaves effort unset. The bounds test now requires the flag; removing it fails the test.

  • Fixes from the pre-cue read of the sibling Bound the harness audit and move it to claude-code-action v1.0.247 #892 (2026-10-09, head 459777d): the record names every behaviour of both new workflows, including the slug, owner, first-attempt and variable guards, the timeout and grants, the guard and binding exits, the explicit action inputs display_report and track_progress (defaults in the pinned action.yml) and the 14-day usage artifact. The bounds step now uses an allow-list, fails on a start record without lists or on an empty result, and names each unmet bound. The review cut is character-safe and says when it cut. The step tests no longer skip without PyYAML. The evidence line states the fence receipt's flags. The 2026-10-03 cooldown waiver is cited.

  • Fixes at df440f4 (not described here until now): a final step, "Require the run's execution file", fails a success without an execution file; the optional paths expand safely under set -u on bash before 4.4; the record gains a Visibility section (a public run's summary is public); ci-least-privilege.md is edited in place (the write-grant and per-workflow table rows, and the exemption sentence) as well as given an addendum.

  • Pre-cue toolkit read of this PR (J8 on 459777d, 2026-10-09; pr-test-analyzer and silent-failure-hunter): every finding at confidence 80 or above, and its status at head fbe30ad.

    Finding Status at fbe30ad Fixing commit and evidence
    pr-test-analyzer (90): the record links Add an on-demand, read-only pull request review by the Claude action #894's record, absent from this tree fixed fbe30ad: the link says the file comes with Add an on-demand, read-only pull request review by the Claude action #894, which lands first, and resolves once Add an on-demand, read-only pull request review by the Claude action #894 has landed
    silent-failure-hunter (90): the same dangling reference fixed fbe30ad, as above
    pr-test-analyzer (85): the docs/github-automation.md bullet describes an absent workflow fixed fbe30ad: the bullet names Add an on-demand, read-only pull request review by the Claude action #894 as where the shared pin, federation, fence and bounds are defined, and says Add an on-demand, read-only pull request review by the Claude action #894 lands first
    pr-test-analyzer (90): the cited fence receipt is absent declared; resolves at landing receipt.json is added by Bound the harness audit and move it to claude-code-action v1.0.247 #892 and Add an on-demand, read-only pull request review by the Claude action #894, which land first; this description already said so, and fbe30ad says it in the record too
    silent-failure-hunter (90): the same receipt declared; resolves at landing as above
    pr-test-analyzer (90): the binding step's id: bind is unused fixed fbe30ad: removed (no step or test referred to it)
    silent-failure-hunter (90): the description does not match the head fixed this description: the base and check lines at fbe30ad, 34 tests, df440f4's changes (the bullet above), and the in-place ci-least-privilege.md edits
  • R4, debug logging set as a repository secret or variable (2026-10-09; the cc-native-practice lane's L3 read against anthropics/claude-code-action v1.0.247 practice): the guard binds (secrets.ACTIONS_STEP_DEBUG || vars.ACTIONS_STEP_DEBUG) == 'true' and (secrets.ACTIONS_RUNNER_DEBUG || vars.ACTIONS_RUNNER_DEBUG) == 'true' as booleans and refuses either before any token is requested; a test pins both bindings, and six weakened copies each fail at least one test. This is hardening: the action step already pins ACTIONS_STEP_DEBUG: 'false', which is what the action's full-output switch reads. zizmor's auditor persona reports secrets-outside-env on the two reads; CI runs the regular persona with --no-ignores, which reports nothing. The R4 lines are byte-identical in Bound the harness audit and move it to claude-code-action v1.0.247 #892, Add an on-demand, read-only pull request review by the Claude action #894, Add an on-demand Claude security review and a model-free flag on risky paths #895, Add a weekly lane triage: the model proposes, a model-free job applies #896 and Add a pre-cue read by Anthropic's pr-review-toolkit agents in Actions #909, and R5 leaves them unchanged.

  • The report notice at the cap (2026-10-09; the J8 micro read of Bound the harness audit and move it to claude-code-action v1.0.247 #892): the publish step wrote with jq -r, which adds a newline, so a 60,000-byte report was announced as cut. It writes with jq -j now; a test publishes exactly 60,000 and 60,001 bytes, and a copy that writes with jq -r fails it.

  • From the J8 micro read of fbe30ad (2026-10-09; id: bind fixed, the cross-PR items partly fixed because they rested on Add an on-demand, read-only pull request review by the Claude action #894 being present): the record and the docs bullet now stand alone. They state the action pin, the federation variables, the fence flags and the bounds themselves; name the pull request review's record by plain path instead of a link; and cite receipt.json from this PR's own tree, a byte-identical copy of the one Bound the harness audit and move it to claude-code-action v1.0.247 #892 and Add an on-demand, read-only pull request review by the Claude action #894 add.

  • R5 (2026-10-09, at 5903cee): the symbolic-link step, exact pins, non-string tool entries, the caps from measurement and the corrected "what has run"; see the R5 section below.

  • Base commit: aba02ec3 (merge base at 5903cee, as at f5f1fd2; at the last local fetch main is at 861c475e, twelve commits on, through Scope research Hindsight skills and make wiring checks fail closed #835; only manifests/evidence.json overlaps, and the one rebase happens at landing)

  • Lane: lane:foundation

  • Owned paths touched: .github/workflows/claude-security-review.yml (new), .github/workflows/security-review-flag.yml (new), tests/test_claude_security_review_workflow.py (new), tests/test_workflow_policy.py (one inventory entry, one exemption), tests/test_workflow_security_coverage.py (coverage set and two zizmor tests), docs/decisions/2026-10-08-claude-actions-security-review.md (new), docs/decisions/2026-10-04-ci-least-privilege.md (two table rows and the exemption sentence edited in place, and a dated addendum), docs/github-automation.md (current-practice bullet), manifests/evidence.json (registration, last commit). R5 touches the review workflow, its test module, the record, the docs/github-automation.md bullet and manifests/evidence.json.

R5 (2026-10-09), at 5903cee

R5 makes the changes asked for by the command center's security read and the GPT designated read of f5f1fd2, and by the command center's caps from measurement, all on 2026-10-09. It is limited to quality and correctness.

  • claude-security-review.yml:
    • Symbolic links (the security read's P2): a new model-free step, "Remove symbolic links from the pull request head", runs right after the head checkout: find pr-head -path pr-head/.git -prune -o -type l -exec rm -f {} + under set -euo pipefail. A link in the head to /proc/self/environ or ../.git/config could otherwise carry what it points at into the review and the public job summary. --restricted confines the file tools to the working directories, and claude --help on 2.1.295 does not say whether a link's target is resolved before that check or the deny rules apply. The step's comment says why it is there.
    • Non-string tool entries (the GPT read's P2): the numbers step no longer filters entries that are not strings out of the session's tool list, which let {"name":"Bash"}, null or 17 pass every bound in a synthetic transcript. Each such entry is now a forbidden tool, named non-string tool entry in usage.json, in the summary's tools column and in the bounds message; string entries are checked against Read, Glob and Grep as before, and the step's comment names the new case.
    • Caps from measurement: --max-turns goes from 12 to 30 and --max-budget-usd from 3 to 5. The numbers step's bounds become 1 to 30 assistant turns and at most $5, the prompt says "at most 30 turns", and the opening comment carries the new values. A cap bounds a runaway; it never trims a normal run. This review has no runs of its own yet and takes the caps of the pull request review (Add an on-demand, read-only pull request review by the Claude action #894), which has the same fence, model and effort with a different prompt. Add an on-demand, read-only pull request review by the Claude action #894's prompt and claude_args reading one pull request in full used 12 of 12 assistant turns and cost $2.23, so 12 is a floor: 2.5 times it gives 30 turns, and twice $2.23 is about $4.5, rounded up to $5. Four delta re-reads took 5 to 7 turns and $0.73 to $1.75. Expected cost per run: about $2, by that shape, not yet measured.
  • tests/test_claude_security_review_workflow.py (36 to 42 tests):
    • New tests:
      • test_symbolic_links_in_the_head_are_removed_and_nothing_else
      • test_the_symbolic_link_step_has_no_condition_and_a_fixed_script
      • test_claude_args_are_exactly_the_reviewed_list
      • test_settings_are_exactly_the_reviewed_json
      • test_a_tool_entry_that_is_not_a_string_is_a_forbidden_tool, which refuses {"name":"Bash"}, null and 17
      • test_the_caps_hold_at_the_bound_and_fail_just_above_it, where 30 turns and $5 pass and 31 turns and $5.01 fail
    • Changed tests:
      • test_steps_run_in_the_order_the_binding_depends_on requires the new step right after the head checkout.
      • test_no_step_executes_anything_from_the_pull_request_head exempts that step by name, pins its script, and checks working-directory on every step.
      • test_claude_has_three_read_tools_and_fixed_bounds checks 30 and 5, and the prompt's turn count.
      • test_an_unmet_bound_fails_after_the_numbers_were_kept, test_the_step_names_every_unmet_bound and test_the_turn_bound_counts_assistant_turns_not_transcript_messages now go just past the caps (31 turns, $5.01 or $5.50).
    • New module helpers: STRIP, STRIP_RUN, RUNNER_TEMP_STAND_IN, SETTINGS, CLAUDE_ARGS and claude_args().
  • docs/decisions/2026-10-08-claude-actions-security-review.md:
    • By-name bullets for the new step and for the numbers step's non-string rule and new bounds; the new caps in the claude_args bullet.
    • The last sentence of "Effort" now says the figures come from runs at max.
    • "Cost of one run" restates the caps' derivation with its measured figures and the expected cost, and withdraws the earlier $0.70 to $1.30 dry estimate.
    • "Evidence class" is corrected. The fence smoke did run --setting-sources user, in two of its three runs. The workflow's model-free steps have run locally under the unit tests. The action step's own configuration has not run, locally or hosted.
    • The test count goes from 36 to 42.
    • A new section, "Symbolic links, exact pins, non-string tool entries, caps and what has run (2026-10-09, R5)", names every change and the weakened copies, and holds the two sentences below.
  • docs/github-automation.md: the "Security review on demand" bullet now says 30 turns and a $5 client budget.
  • manifests/evidence.json: the four changed rows are rehashed.
  • security-review-flag.yml: unchanged.
  • Weakened copies. 22 weakened copies of the review workflow were run on 2026-10-09 by a script kept outside the repository, with PyYAML 6.0.3 present, as the Linux validate job has it. All 22 fail at least one test. Six weaken the new step, six the arguments, two the tool check and eight the caps. Five of the six argument copies pass f5f1fd2's tests.
  • Without PyYAML, as on the macOS validate job, the module's shape tests skip. Of the new tests, three run there: the symbolic-link behaviour, non-string entry and caps tests, together with the three changed bounds tests. The step pin and both exact pins skip, and 10 of the 22 copies still fail a test.
  • Two record sentences shared with the other W4 records, byte for byte:
    • "In agent mode the action rewrites the checkout's origin URL with the job token (src/github/operations/git-config.ts:129-134, called from src/modes/agent/index.ts:52-61 at 2dca132f), so persist-credentials: false does not keep the token out of .git/config; the deny rules Read(./.git/**) and Read(./**/.git/**) are what keep the model from reading it, as two 2026-10-09 probes on the installed 2.1.295 measured."
    • "Security hardening from the 2026-10-09 read (flag-path additions, a fork notice, debug-value widening, tool-list shape, extra deny rules, token-source and guard-step assertions) is filed as follow-ups before any enabling variable is set."
  • Not part of R5:
    • The hardening in the second sentence above is filed as follow-ups. For the tool-list shape, only the non-string part landed here; requiring a non-empty list and checking tool_use names remain.
    • The macOS coverage gap is in the same follow-ups register: without PyYAML, 12 of the 22 weakened copies pass, all of them changes that only the skipped shape tests read.
    • A planted-symbolic-link case in the fence smoke needs a model run and is not done.
    • --setting-sources user beside --restricted is pinned as it stands, pending Add a weekly lane triage: the model proposes, a model-free job applies #896's ruling on that pair.

R6 (2026-10-09), at e57d8eb

  • Cost bound: the budget × 1.10 = $5.50, answering the command center's read of 5903cee (P2), from four measured J8 runs on a $5 budget, re-derived after the first three hosted runs. A budget stop at or under it publishes and is named. Publishing is gated on !cancelled() && steps.numbers.outcome == 'success'.
  • --max-turns 30 removed: the pinned action (base-action/src/run-claude-sdk.ts:241-250 at 2dca132f, since 6ef6450f) fails a success whose num_turns exceeds it. On Claude Code 2.1.295 num_turns counts transcript messages: 57 for 12 requests under --max-turns 12. That is a code reading plus a local measurement, not a hosted run. The numbers step's 1-to-30 assistant-turn check and the prompt's "at most 30 turns" stay.
  • Exact pins: both compare canonical JSON.
  • Record: the shared cost paragraph, and an upstream-issue draft (not filed).
  • Local checks at e57d8eb:
    • validate.py and the manifest check pass (10,906 files);
    • six modules, 263 tests, OK;
    • this module, 44 tests (19 skip without PyYAML);
    • actionlint and zizmor are clean;
    • 35 of 35 weakened copies fail a test with PyYAML.
  • Hosted checks at e57d8eb: not read yet.

R7 (2026-10-09), at 02b80c6

SOTA sources

  • anthropics/claude-code-action, v1.0.247 at 2dca132ff0e0c4094ce6048b422c6915a071210b (https://github.com/anthropics/claude-code-action/tree/2dca132ff0e0c4094ce6048b422c6915a071210b): docs/solutions.md ("Security Review" example, the source of the general checklist), docs/security.md ("Do not check out an untrusted ref into the workspace root before this action"; base ref at the root and head ref in a subdirectory), docs/setup.md (federation inputs; "a static credential takes precedence and federation will not be used"), action.yml (inputs and the execution_file output), base-action/src/parse-sdk-options.ts (argument parsing with shell-quote, default setting sources, debug forcing full output), src/github/operations/git-config.ts (git authentication in the root checkout; lines 129-134 rewrite the origin URL with the job token), src/modes/agent/index.ts (lines 52-61: agent mode calls that git configuration). R5 read the last two at v1.0.246 (38c80c1); per the command center's compare, v1.0.247 changes only the bundled Claude Code version, so neither file differs at the pin.
  • Claude Code 2.1.295, the version the pin installs, claude --help: --restricted ("removes the built-in tools that run commands or code ... and ignores user, project and local settings files ... Also confines the file tools to the working directories (--add-dir included)"; the help says nothing about symbolic links, read 2026-10-09), --permission-prompts none ("anything that would prompt is denied automatically"), --tools, --settings, --strict-mcp-config, --max-budget-usd.
  • GNU findutils find 4.10.0, as installed where the R5 tests ran: -path ... -prune, -type l and -exec ... {} +, the semantics the symbolic-link step relies on.
  • anthropics/claude-code-security-review at 0c6a49f1fa56a1d472575da86a94dbc1edb78eda, its only pin (no release or tag; last commit 2026-02-11), read and not adopted: action.yml (API key only), claudecode/constants.py (default claude-opus-4-1-20250805), scripts/comment-pr-findings.js (submits a PR review), README.md ("not hardened against prompt injection attacks"); Anthropic, Model deprecations (https://platform.claude.com/docs/en/about-claude/model-deprecations): that model retired 2026-08-05.
  • GitHub, OpenID Connect reference, "Filtering for pull_request events" and "Filtering for a specific branch" (https://docs.github.com/en/actions/reference/security/oidc); GitHub Security Lab, Preventing pwn requests (https://securitylab.github.com/research/github-actions-preventing-pwn-requests/).
  • Anthropic documentation read 2026-10-08: Workload identity federation (https://platform.claude.com/docs/en/manage-claude/workload-identity-federation); How Claude Code uses prompt caching (https://code.claude.com/docs/en/prompt-caching); Pricing (https://platform.claude.com/docs/en/about-claude/pricing).

Evidence-class table

Claim Evidence class Command / receipt
The fence flags (--restricted, --tools/--allowedTools Read,Glob,Grep, --strict-mcp-config, --permission-prompts none, --settings with hooks off and three deny rules; Haiku 5.5, 10 turns, $0.50, as the receipt lists; two of its three runs also passed --setting-sources, run2-user with user) give a read-only session (tools Glob, Grep, Read; reads outside the tree and of .git/config denied; planted hooks and instruction files without effect) native_proven on the installed 2.1.295 client (not through the action, not on a GitHub runner) evidence/artifacts/claude-actions-fence-smoke-20261008/receipt.json, carried byte-identical by this PR, #892 and #894
This workflow's own configuration (that fence plus --add-dir, --effort max, blockReadsOutsideWorkingDirectories and four more deny rules; Opus 5.5, 30 turns, $5; through the action) holds not established it has not run, locally or hosted
The review's guard, binding, symbolic-link, diff, numbers and report steps, and the flag's notice, behave as described synthetic tests/test_claude_security_review_workflow.py, 42 tests; each step's shell is taken from the workflow file and executed. Without PyYAML the 19 shape and flag tests skip and the 23 step tests run
Symbolic links under pr-head/ are gone before the model starts, while regular files and pr-head/.git stay synthetic test_symbolic_links_in_the_head_are_removed_and_nothing_else (links to /proc/self/environ, to ../.git/config and from a subdirectory); test_the_symbolic_link_step_has_no_condition_and_a_fixed_script
A tool-list entry that is not a string fails the bounds and is named synthetic test_a_tool_entry_that_is_not_a_string_is_a_forbidden_tool: {"name":"Bash"}, null and 17 each exit 1 with non-string tool entry
The caps: 30 assistant turns and $5 pass; 31 turns and $5.01 fail and are named synthetic test_the_caps_hold_at_the_bound_and_fail_just_above_it
The caps are a multiple of the measured need (#894's prompt reading one pull request in full: 12 of 12 turns, $2.23; four delta re-reads: 5 to 7 turns, $0.73 to $1.75) measured for #894's prompt and claude_args on the installed 2.1.295 client, not through the action; applied to this workflow by task shape, not measured on it the command center's caps from measurement, 2026-10-09; the run receipts are kept outside the repository
The tests detect a weakened workflow synthetic 11 mutated copies on 2026-10-08 (that day's bounds), 11 failed at least one test; on 2026-10-09 three turn-bound mutants each failed; at R5, 22 weakened copies run by a script outside the repository: 22 of 22 fail a test with PyYAML 6.0.3 and 10 of 22 without it, and five of the six argument copies pass f5f1fd2's tests
Both workflows are well formed and have no audit finding local_integration actionlint 1.17.0 exit 0; zizmor 1.30.1 offline, regular (--no-ignores) and pedantic, no findings (2 suppressed), at R5
A hosted run authenticates, stays in bounds and reads the cache not established no hosted run is part of this PR

Local commands run

$ python3 scripts/validate.py            # on the R5 tree (5903ceed)
{"components": 70, "hashed_files": 10906, "profiles": 4, "receipts": 239, "status": "passed"}   exit 0
$ python3 scripts/evidence_manifest.py --check
{"files": 10906, "status": "passed"}   exit 0
$ UV_OFFLINE=1 uv run --quiet --no-project --python 3.12 --with pyyaml==6.0.3 python -m unittest tests.test_claude_security_review_workflow tests.test_workflow_policy tests.test_workflow_security_coverage tests.test_workflow_hardening tests.test_github_automation_practice tests.test_adoption_docs_consistency
Ran 261 tests ... OK (skipped=2)   exit 0
$ UV_OFFLINE=1 uv run --quiet --no-project --python 3.12 --with pyyaml==6.0.3 python -m unittest tests.test_claude_security_review_workflow
Ran 42 tests ... OK   exit 0
$ UV_OFFLINE=1 uv run --quiet --no-project --python 3.12 python -m unittest tests.test_claude_security_review_workflow   # without PyYAML
Ran 42 tests ... OK (skipped=19)   exit 0
$ actionlint .github/workflows/claude-security-review.yml .github/workflows/security-review-flag.yml
exit 0
$ zizmor --offline --no-config --persona=pedantic .github/workflows/claude-security-review.yml .github/workflows/security-review-flag.yml
No findings to report. Good job! (2 suppressed)   exit 0
$ zizmor --offline --no-config --no-ignores --persona regular .github/workflows/claude-security-review.yml .github/workflows/security-review-flag.yml
No findings to report. Good job! (2 suppressed)   exit 0

Hosted checks at 5903cee: not read yet.

Decision record

docs/decisions/2026-10-08-claude-actions-security-review.md (why not anthropics/claude-code-security-review, why the flag and the review are separate, the prompt, the caps and their derivation under "Cost of one run", changed test contracts, alternatives, what would overturn it, and the dated R4 and R5 sections). docs/decisions/2026-10-04-ci-least-privilege.md lists the review job in its write-grant table and both workflows in its per-workflow table, names each exempt job in its federation-exemption sentence, and gets a dated addendum naming this job on the federation rule.

Host evidence

Not applicable: no file under evidence/hosts/ changes.

Checklist

  • New/changed GitHub Actions are pinned to a full commit SHA with a
    version comment (no floating tags).
  • New/changed workflows declare top-level permissions: {} and grant each job
    only what it needs (contents: read, or an explicitly justified addition).
  • No secrets are printed, logged or committed; no new required secret was
    added without a documented owner.
  • No new paid hosting, subscription or billing surface was introduced.
  • Peer-owned untracked files and worktrees were preserved (not deleted,
    moved or overwritten).

Activation is a separate step after landing: an owner sets CLAUDE_SECURITY_REVIEW_ENABLED to true; the flag then names the dispatch command on each matching pull request. Runs authenticate by workload identity federation and spend from the Console organization the four ANTHROPIC_* repository variables name; no Anthropic key is stored in GitHub. Each run is bounded by the $5 client budget and 30 assistant turns, the owner's per-run spend limit once the variable is set. The review workflow and its tests share their guard and accounting steps with #894 (on-demand pull request review); this PR, #892 and #894 all edit the write-grant inventory and the evidence manifest, so each later one rebases once.

🤖 Generated with Claude Code

seathatflowsinourveins and others added 2 commits October 8, 2026 18:31
…y paths

security-review-flag.yml runs on pull requests that touch workflows, hooks,
credential tooling, the credential inventory or the paper-trading blueprints
and only writes a notice with the exact dispatch command; it runs no model,
checks out nothing and holds no scope. claude-security-review.yml is that
review: a manual dispatch from main for one pull request head, with the same
read-only fence, bounds and accounting as the on-demand pull request review
and a security-review prompt built from the action's own example and this
repository's risk classes. anthropics/claude-code-security-review is not used
(API key only, retired default model, writes a PR review, unpinned installs).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@seathatflowsinourveins seathatflowsinourveins added the lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers label Oct 8, 2026
@socket-security

socket-security Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Dependency limit exceeded — report not shown.

This pull request scan exceeded the 10,000-dependency limit applied to this scan, so the results are incomplete and may be inaccurate. To avoid reporting false positives, Socket has not posted a report.

Upgrade your plan to raise the dependency limit and get complete reports, or view the partial scan in the dashboard.

Socket is always free for open source. If this is a non-commercial open source project, contact us to request a free Team account.

seathatflowsinourveins and others added 7 commits October 8, 2026 19:55
The review job now passes `--effort max` in claude_args. Under the command
center's effort mapping of 2026-10-08, judgment work (designated reads,
adjudication, pull request and security reviews, audits) runs at `max`, and
every job records its level and the reason. The level has to be in
claude_args because `--restricted` ignores the settings files that would
otherwise carry it, and Opus 5.5 runs at `medium` on the API when a request
leaves effort unset. The bounds test now requires the flag (removing it
fails the test).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The bounds step failed the job unless the result's num_turns was at most
the --max-turns value. On Claude Code 2.1.295 num_turns counts transcript
messages, tool results included: a local run with #894's prompt and
claude_args (api-actions LR, 2026-10-08) made 12 API requests under
--max-turns 12 and reported num_turns 57. The step now counts distinct
assistant message ids in the execution file and checks that count against
12; num_turns stays in the record and the job summary as "messages". The
test fixture emits two messages per turn id, as the client does, and a new
test passes a 12-turn run that reports 57 messages and fails a 13-turn one.
Three mutants (bound on num_turns, count forced to 0, count without
de-duplicating ids) each fail the tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
main moved to aba02ec (#889 and others), and manifests/evidence.json
conflicted, so no pull_request workflow ran on the branch. The manifest is
resolved by the hot-file protocol: main's copy, then this branch's 8
changed files re-registered with host_receipts.register_file, followed by
component_matrix.py --write and new_host_grand_list.py --write;
validate.py exits 0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…kflows

The pre-cue read of the sibling harness-audit PR found gaps that the
review workflow shares; this commit carries the same fixes here.

Bounds step: an allow-list tool check (Read, Glob and Grep), a session
start record without its tool or MCP lists fails, a run without result
text fails, and the step names every bound it finds unmet. The review
step takes the last non-empty result, cuts on a character boundary and
says when the review was cut.

Tests: the step tests read the workflow with the policy test's own
loader when PyYAML is absent, so they no longer skip; new tests cover
the new bounds, the named failures, the character-safe cap and the last
non-empty result (32 tests).

Record: a section names every behaviour of both new workflows; the
evidence-class line states the fence receipt's flags instead of "the
shared flag set" and says this workflow has not run; the 2026-10-03
cooldown waiver is cited; the workflow header says the job holds no
write scope beyond the OIDC token.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The sweep of the W4 drafts for the classes #894's pre-cue toolkit read
found, applied here: the empty paths array is expanded safely on bash
before 4.4; a review step that succeeds without an execution file fails
the job; display_report and track_progress are asserted; the record
states that a public run's summary is public and when a finding belongs
on the local route instead; ci-least-privilege lists the review job and
the flag workflow in its tables and names each exempt job (in place).
34 tests; actionlint and zizmor report nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ding findings

The pre-cue toolkit read of 459777d left findings standing at df440f4:

- The record's link to the pull request review's record, and its receipt
  citation, now say that #894 (and #892) add those files and land first.
- The docs bullet names #894 as where the shared pin, federation, fence and
  bounds are defined.
- The binding step's unused id `bind` is removed; nothing referred to it.
- The record's by-name list now names what df440f4 added: the final
  execution-file step, the bash 3.2 expansion and the Visibility section.
  It also names this PR's edits outside the record.
- The test count, stale at 32, is 34.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the record stand alone

- R4: the guard binds the ACTIONS_STEP_DEBUG and ACTIONS_RUNNER_DEBUG
  secrets and variables as booleans, the secret first, and refuses either.
  A test pins both bindings, and six weakened copies each fail it.
- The publish step writes the review with `jq -j`, so a 60,000-byte review
  is no longer announced as cut. A boundary test covers it.
- From the J8 micro read of fbe30ad: the record and the docs bullet no
  longer rest on #894 being present.
  - They state the action pin, the federation variables, the fence flags
    and the bounds themselves.
  - They name the pull request review's record by plain path, not by link.
  - They cite receipt.json from this PR's own tree, a byte-identical copy of
    the one #892 and #894 add.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
seathatflowsinourveins added a commit that referenced this pull request Oct 9, 2026
…e the audit budget to $5

R5, quality and correctness only. The numbers step counts every tool-list
entry that is not a string as a forbidden tool, named "non-string tool
entry", instead of dropping it (GPT designated read of #895, P2). A new
test pins the whole claude_args list and the whole --settings JSON. The
client budget moves from $3 to $5 with the turn limit at 20, from the
command center's measurement-based caps: the one measured run used 10
turns and $2.03. The shape tests now read the workflow with the policy
test's strict loader, so none skips without PyYAML. The record gains a
dated R5 section, the cap derivation, the shared agent-mode origin-URL
sentence and the follow-ups line, and withdraws the earlier dry estimate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tries and raise the caps to 30 turns and $5

R5, quality and correctness only. A model-free step removes every symbolic
link from the pull request head right after its checkout, so a link to a
process environment file or to ../.git/config cannot carry credentials into
the job summary (GPT designated read of #909, P2; the command center's
security read). The numbers step counts every tool-list entry that is not a
string as a forbidden tool, named "non-string tool entry", instead of
dropping it (GPT designated read of this PR at f5f1fd2, P2). New tests pin
the whole claude_args list and the whole --settings JSON. The caps follow
the command center's measurement-based derivation: 30 turns and a $5 client
budget. The record gains a dated R5 section, corrects its evidence class,
withdraws the earlier dry cost estimate and states which tests run without
PyYAML.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
seathatflowsinourveins added a commit that referenced this pull request Oct 9, 2026
…tries and raise the caps to 30 turns and $5

R5, quality and correctness only. A model-free step removes every symbolic
link from the pull request head right after its checkout, with tests on
planted links and on a link-free head. The numbers step counts every
tool-list entry that is not a string as a forbidden tool, named "non-string
tool entry", instead of dropping it (GPT designated read of #895, P2). New
tests pin the whole claude_args list, the whole --settings JSON and the
exact if: conditions of the publish and numbers steps. The caps follow the
command center's measurement-based derivation: 30 turns and a $5 client
budget, with the prompt and the numbers step's bounds following. The record
gains a dated R5 section, withdraws the earlier dry cost estimate and states
which tests run without PyYAML; the R4 guard lines are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
seathatflowsinourveins and others added 2 commits October 9, 2026 04:33
…thin-bound budget stops, and drop --max-turns

R6. The cost bound becomes the budget times the measured overrun factor 1.10
($5.50) (command center read of 5903cee, P2); a budget stop at or under
it publishes and is named, and publishing is gated on the numbers step's
own outcome. --max-turns 30 leaves claude_args: the pinned action fails a
success whose num_turns exceeds it, and on Claude Code 2.1.295 num_turns
counts transcript messages (57 for 12 requests); the numbers step's check
of 1 to 30 assistant turns stays. Both exact pins compare canonical
JSON. The record gains the shared cost paragraph and an upstream-issue
draft (not filed).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…o 30 minutes

R7. With --max-turns removed in R6, the numbers step's bound of 30 assistant
turns is the turn limit. At the one measured pace of this shape (LR: 12 turns
in 512,670 ms, Opus 5.5 at max) 30 turns take about 21.4 minutes before
checkout and setup, so a 20-minute job timeout would cancel a run inside the
approved bounds before execution_file exists. A new test pins 30, and a
weakened copy setting it back to 20 fails it; #892 and #894 use 30.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
seathatflowsinourveins added a commit that referenced this pull request Oct 9, 2026
…894)

### Scope

- What this PR changes, in one or two sentences: adds `claude-pr-review.yml`, an on-demand, read-only review of one same-repository pull request head by `anthropics/claude-code-action` v1.0.247. A maintainer dispatches it from `main` with the pull request number and the exact head commit; the review goes to the job summary and nothing is posted to the pull request. It runs only while the repository variable `CLAUDE_PR_REVIEW_ENABLED` is `true`, so merging this starts nothing.
- Effort (added 2026-10-08 at head a6d2379): `--effort max` in `claude_args`, because judgment work (designated reads, adjudication, pull request and security reviews, audits) runs at `max` under the command center's effort mapping, and every job records its level and reason. The level has to be in `claude_args`: `--restricted` ignores the settings files that would otherwise carry it, and Opus 5.5 runs at `medium` on the API when a request leaves effort unset. The bounds test now requires the flag; removing it fails the test. A loopback dry run of the installed 2.1.295 client with this workflow's `claude_args` sent `output_config.effort: "max"` on every request.
- Fixes from the pre-cue read of the sibling #892 (2026-10-09, head d6f8909): the record names every behaviour of this new workflow: the trigger; the slug, owner, first-attempt and variable guards; the timeout and grants; the guard and binding exits; the head-as-data checkout and diff limits; the explicit action inputs `display_report` and `track_progress` (defaults in the pinned `action.yml`); and the 14-day usage artifact. The bounds step now uses an allow-list, fails on a start record without lists or on an empty result, and names each unmet bound. The review cut is character-safe and says when it cut. The step tests no longer skip without PyYAML. The 2026-10-03 cooldown waiver is cited.
- Fixes from this PR's own pre-cue toolkit read (J8 on d6f8909, head 8e64120):
  - the empty paths array is now expanded safely on bash before 4.4;
  - a success without an execution file now fails the job;
  - `display_report` and `track_progress` are asserted;
  - the artifact name and the action-pin sentence are corrected, and the parity receipt's dependency on #892 is stated;
  - `claude-pr-review.yml:review` is now in both grant tables of ci-least-privilege;
  - the record states that a public run's summary is public.
- Pre-cue toolkit read of this PR (J8 on d6f8909, 2026-10-09; `pr-test-analyzer` and `silent-failure-hunter`): every finding at confidence 80 or above, and its status at head 41c9c15.

  | Finding | Status at 41c9c15 | Fixing commit and evidence |
  | --- | --- | --- |
  | `pr-test-analyzer` 1 (90): the record names the wrong artifact | fixed | 8e64120: the record names `claude-pr-review-usage-<run id>-<attempt>`, the upload's name; 41c9c15 adds `test_the_usage_artifact_is_named_for_the_run_and_its_attempt` |
  | `silent-failure-hunter` 2 (88): the record and the workflow disagree on the artifact | fixed | the same two commits |
  | `pr-test-analyzer` 2 (85): `docs/github-automation.md` claims the harness audit's action pin | fixed | 8e64120: the bullet says the two pins differ until #892 lands |
  | `silent-failure-hunter` 1 (92): the same false claim (pin and federation variables) | fixed | 8e64120, as above; the four federation variables are the same |
  | `pr-test-analyzer` 3 (85): the headline evidence file is not in this PR | fixed | 8e64120 states that #892 adds it and lands first; 41c9c15 cites its `LR` entry field by field |
  | `silent-failure-hunter` 3 (80): "#897" and "two blocking findings" have no source in that receipt | fixed | #892 adds `target_pull_request`, `target_head`, `report_blocking_findings` and `report_sha256` to the `LR` entry (a marked 2026-10-09 amendment from the run's harness receipt); 41c9c15 cites them, calls the verdict the model's own, and states that the prompt and `claude_args` are unchanged from the run's workflow commit a6d2379 to that head (R5 later raises the two caps and the prompt's turn sentence; see R5 below) |
- R4, debug logging set as a repository secret or variable (2026-10-09; the cc-native-practice lane's L3 read against `anthropics/claude-code-action` v1.0.247 practice): the guard binds `(secrets.ACTIONS_STEP_DEBUG || vars.ACTIONS_STEP_DEBUG) == 'true'` and `(secrets.ACTIONS_RUNNER_DEBUG || vars.ACTIONS_RUNNER_DEBUG) == 'true'` as booleans and refuses either before any token is requested; a test pins both bindings, and six weakened copies each fail at least one test. This is hardening: the action step already pins `ACTIONS_STEP_DEBUG: 'false'`, which is what the action's full-output switch reads. zizmor's auditor persona reports `secrets-outside-env` on the two reads; CI runs the regular persona with `--no-ignores`, which reports nothing. The R4 lines are byte-identical in #892, #894, #895, #896 and #909; R5 leaves them unchanged here.
- The report notice at the cap (2026-10-09; the J8 micro read of #892): the publish step wrote with `jq -r`, which adds a newline, so a 60,000-byte report was announced as cut. It writes with `jq -j` now; a test publishes exactly 60,000 and 60,001 bytes, and a copy that writes with `jq -r` fails it. R5 adds the 59,999-byte case.
- From the J8 micro read of 41c9c15 (2026-10-09; 4 of 6 listed findings fixed, PTA-3 and SFH-3 partly because the receipt the record cites was not in this tree): this PR now carries a byte-identical copy of `local-parity-receipt.json` (sha256 `ba498b82…`, the file #892 adds), so the citation no longer depends on which PR lands first, and the docs bullet states the action pin and the federation variables instead of comparing with the harness audit on `main`.
- R5 (2026-10-09, head d1bcb6c, limited by the command center to correctness and test quality) adds a model-free step, `Remove symbolic links from the pull request head`, right after the head checkout: `set -euo pipefail; find pr-head -path pr-head/.git -prune -o -type l -exec rm -f {} +`. It raises the approved caps from measurement: `--max-turns` 12→30 and `--max-budget-usd` 3→5. The prompt's "You have at most 30 turns", the numbers step's bounds (1 to 30 assistant turns, at most $5, with matching messages), the header comment and `docs/github-automation.md` follow. The numbers step now counts each non-string tool-list entry as a forbidden tool named `non-string tool entry` instead of dropping it. The R4 guard lines, the deny list and the `github_token` input are kept exactly as at a1bc3bc. The test module goes from 33 to 41 tests. Eight are new: the symlink step's position and exact text; its behaviour on planted links to `/proc/self/environ`, `../.git/config` and a subdirectory, plus a re-run on a link-free head; a 59,999-byte report published whole; object, null and number tool entries refused and named; the prompt stating the `--max-turns` value; and the caps at 30 turns and $5 and just above. Changed tests pin the full `claude_args` list and `--settings` JSON exactly, assert the exact `if:` strings of the publish and numbers steps, run the guard with production's debug-off values (false/false/empty), and check that the 59,999-byte prefix before a cut character survives. All 28 weakened copies of the workflow fail at least one test with PyYAML present, as the hosted validate job has it; without PyYAML 15 of 28 do, because the pins sit in the PyYAML-only shape tests. The decision record gains a dated R5 section and names the R4 guard change in its what-changed paragraph. It corrects "Read denies for every `.git` directory" to "for the files under every `.git` directory". It restates the cap derivation and the expected cost per run, and withdraws the dry $0.70–$1.30 estimate, which was below the measured first-read cost. It adds the shared follow-up line, the shared `origin` URL sentence and two action sources. `manifests/evidence.json` is rehashed for the four changed files.
  - Sources of the R5 items: the command center's security read of a1bc3bc (the symbolic links, the closed flag list, the exact `if:` strings, the guard's debug-off values, the 59,999-byte prefix, and the stale guard sentence in the record); the command center's caps decision of 2026-10-09; the GPT designated read of #895, P2 (non-string tool entries, applied to all five W4 workflows); the cc-native-practice lane's third audit, P3-1 (the shared `origin` URL sentence).
  - Caps derivation: `LR`, a first read of #897 with this workflow's prompt and `claude_args` on the installed client, used all 12 of its 12 assistant turns and cost $2.23 ($1.87 by the client's estimate). So the need is above 12, and 2.5 times that floor gives 30 turns; twice $2.23 is about $4.5, rounded up to $5. Four delta re-reads (LR2 to LR5) used 5, 6, 7 and 6 turns and cost $1.75, $1.35, $1.54 and $0.73. A run is expected to cost about $1.9 to $2.3 for a first read and $0.7 to $1.8 for a delta re-read.
  - Filed as follow-ups before any enabling variable is set, not in R5: the security hardening from the 2026-10-09 read (debug-value widening, the tool-list shape apart from the non-string entry above, extra deny rules, token-source and guard-step assertions); the rest of F-02 (a non-empty tool list, checking the `tool_use` names); F-16 (the pins that run only with PyYAML).
  - Not run in R5: a fence smoke with a planted symbolic link, which needs a model run; any hosted run. On the first run, watch whether the action appends an `--mcp-config` that `--strict-mcp-config` still honours (the read's G3); the numbers step fails the run on any MCP server.
- Base commit: `aba02ec3` (merge base at a1bc3bc and at d1bcb6c, which builds on a1bc3bc; as read at a1bc3bc, main had moved to `ebe6c223` with #890, #900, #897 and #907; only `manifests/evidence.json` overlaps, and the one rebase happens at landing)
- Lane: `lane:foundation`
- Owned paths touched: `.github/workflows/claude-pr-review.yml` (new), `tests/test_claude_pr_review_workflow.py` (new), `tests/test_workflow_policy.py` (one inventory entry, one exemption), `tests/test_workflow_security_coverage.py` (coverage set and one zizmor test), `docs/decisions/2026-10-08-claude-actions-pr-review.md` (new), `docs/decisions/2026-10-04-ci-least-privilege.md` (dated addendum only), `docs/github-automation.md` (current-practice bullet), `evidence/artifacts/claude-actions-fence-smoke-20261008/receipt.json` (new), `evidence/artifacts/claude-actions-fence-smoke-20261008/local-parity-receipt.json` (new; byte-identical to the copy #892 adds), `manifests/evidence.json` (registration, last commit).

### R6 (2026-10-09), at bd10ba3

- **Cost bound:** the budget × 1.10 = $5.50, from four measured J8 runs on a $5 budget ($5.46, $5.33, $5.16, $5.0007),
  re-derived after the first three hosted runs.
  - A budget stop at or under it publishes the text written before the stop, with a line naming the stop.
  - Publishing is gated on `!cancelled() && steps.numbers.outcome == 'success'`, because the action fails its own step on
    any budget stop.
- **`--max-turns 30` removed:** the pinned action (`base-action/src/run-claude-sdk.ts:241-250` at 2dca132f, since
  6ef6450f) fails a success whose `num_turns` exceeds it.
  - On Claude Code 2.1.295, `num_turns` counts transcript messages: 57 for 12 requests under `--max-turns 12`. That is a
    code reading plus a local measurement, not a hosted run.
  - The numbers step's 30-assistant-turn bound and the prompt's "at most 30 turns" stay.
- **Job timeout:** 20 → 30 minutes. 30 turns at the measured pace (12 turns in 512,670 ms) take about 21.4 minutes before
  setup. #892 uses 30.
- **Settings pins:** both compare canonical JSON, so `1` no longer passes for `true` (J8 R5 micro, N1, confidence 70).
- **Record:**
  - the shared cost paragraph;
  - an upstream-issue draft (not filed);
  - the seven changed test expectations the J8 R5 micro found undeclared (U1–U7).
- **Local checks at bd10ba3:**
  - `validate.py` and the manifest check pass (10,906 files);
  - six modules: 262 tests, OK;
  - this module: 44 tests (16 skip without PyYAML);
  - actionlint and zizmor are clean;
  - 39 of 39 weakened copies fail a test with PyYAML.
- **Hosted checks at bd10ba3:** not read yet.

### SOTA sources

- anthropics/claude-code-action, v1.0.247 at `2dca132ff0e0c4094ce6048b422c6915a071210b` (<https://github.com/anthropics/claude-code-action/tree/2dca132ff0e0c4094ce6048b422c6915a071210b>): `docs/security.md` ("Do not check out an untrusted ref into the workspace root before this action"; base ref at the root and head ref in a subdirectory), `docs/setup.md` (federation inputs; "a static credential takes precedence and federation will not be used"), `action.yml` (inputs and the `execution_file` output), `base-action/src/parse-sdk-options.ts` (argument parsing with `shell-quote`, default setting sources, debug forcing full output), `src/github/operations/git-config.ts` (git authentication in the root checkout; lines 129-134 write the job token into the checkout's `origin` URL when `allowed_non_write_users` is unset) and, new in R5, `src/modes/agent/index.ts` (lines 52-61, where agent mode calls that configuration). R5 read both files at v1.0.246 (`38c80c1`); the command center's compare of v1.0.246 and v1.0.247 changes only the bundled Claude Code version.
- Claude Code 2.1.295, the version the pin installs, `claude --help`: `--restricted` ("removes the built-in tools that run commands or code ... and ignores user, project and local settings files ... Also confines the file tools to the working directories"), `--permission-prompts none` ("anything that would prompt is denied automatically"), `--tools`, `--settings`, `--strict-mcp-config`, `--max-budget-usd` and, for the R5 caps, `--max-turns`. The help does not say whether a symbolic link's target is resolved, which is why R5 removes the links before the model runs.
- GNU findutils `find` 4.10.0 as installed (`find --version`), new in R5: `find pr-head -path pr-head/.git -prune -o -type l -exec rm -f {} +`, measured by the step test on planted links and on a link-free head (exit 0, no `rm` started). The runner image's `find` was not measured.
- GitHub, Contexts reference, `runner.debug` (set only while debug logging is on, so the guard's `RUNNER_DEBUG_SIGNAL` is empty on a normal run), new in R5 as the basis of the guard tests' debug-off values; taken from the command center's read of a1bc3bc, not re-read in R5 (no network).
- GitHub, OpenID Connect reference, "Filtering for pull_request events" and "Filtering for a specific branch" (<https://docs.github.com/en/actions/reference/security/oidc>); GitHub Security Lab, Preventing pwn requests (<https://securitylab.github.com/research/github-actions-preventing-pwn-requests/>).
- Anthropic documentation read 2026-10-08: Workload identity federation (<https://platform.claude.com/docs/en/manage-claude/workload-identity-federation>); How Claude Code uses prompt caching (<https://code.claude.com/docs/en/prompt-caching>); Pricing (<https://platform.claude.com/docs/en/about-claude/pricing>).

### Evidence-class table

| Claim | Evidence class | Command / receipt |
| --- | --- | --- |
| With the fence flags (`--restricted`, `--tools`/`--allowedTools` Read,Glob,Grep, `--strict-mcp-config`, `--permission-prompts none`, `--settings` with hooks off, `claudeMdExcludes` and three deny rules; Haiku 5.5, 10 turns, $0.50, as the receipt lists; this workflow adds `--setting-sources user`, `--add-dir`, `--effort max`, four deny rules and `blockReadsOutsideWorkingDirectories`, a key in the 2.1.295 build that the receipt does not exercise) the session's tools are Glob, Grep and Read; reads outside the tree and of `.git/config` are denied; planted hooks, instruction files and a skill under `pr-head` have no effect | `native_proven` on the installed 2.1.295 client (not through the action, not on a GitHub runner) | `evidence/artifacts/claude-actions-fence-smoke-20261008/receipt.json`, three runs |
| The action's own parser turns this workflow's `claude_args` into those flags and that JSON | `local_integration` (at a1bc3bc only) | the parser's three helpers replayed with `shell-quote` 1.10.0 on the workflow text at a1bc3bc; not re-run at d1bcb6c (no network to install `shell-quote`). R5 changes two plain values (`12` to `30`, `3` to `5`), and `test_claude_has_three_read_tools_and_fixed_bounds` asserts the whole list, split per line with `shlex`, and the `--settings` JSON |
| The guard, binding, symbolic-link, diff, numbers and review steps behave as described | `synthetic` | `tests/test_claude_pr_review_workflow.py`, 41 tests at d1bcb6c (the 26 step tests run without PyYAML too; the 15 shape tests need it); each step's shell is taken from the workflow file and executed against a local stand-in for `gh`, a local git repository or a planted `pr-head` tree |
| Symbolic links under `pr-head` are removed before the model runs, `pr-head/.git` is left as the checkout wrote it, and a link-free head passes | `synthetic` | `test_symbolic_links_in_the_head_are_removed_and_nothing_else` (links to `/proc/self/environ`, `../.git/config` and `../..`; GNU find 4.10.0); no fence smoke with a planted link has run on the client |
| The tests detect a weakened workflow | `synthetic` | 20 mutated copies on 2026-10-08 (that day's bounds), 20 failed at least one test; on 2026-10-09 three turn-bound mutants each failed, and the new bounds have their own tests; at 41c9c15 three artifact-name mutants each pass the 30 earlier tests and fail the new name test; at d1bcb6c 28 mutated copies (the symbolic-link step, the flag and settings pins, the `if:` strings, the guard's debug-off values, the report cap, the non-string tool entries, the caps and the prompt) each fail at least one test with PyYAML 6.0.3 present, as the hosted validate job has it; without PyYAML 15 of 28 do (the other 13 are pins in the shape tests; follow-up F-16) |
| This workflow's own prompt and `claude_args` review a real pull request within bounds at `--effort max` on Opus 5.5 | `local_integration` (installed client, billed to a second key; not through the action) | `evidence/artifacts/claude-actions-fence-smoke-20261008/local-parity-receipt.json`, carried byte-identical in this PR and in #892 (sha256 `ba498b82…`), its `LR` entry: workflow commit a6d2379, whose prompt and `claude_args` are unchanged at d1bcb6c apart from the two caps R5 raised (`--max-turns` 12 to 30, `--max-budget-usd` 3 to 5) and the prompt's turn sentence; #897 at `target_head` 24821c5; 12 of 12 assistant turns; $1.868667 of the then $3 budget; tools Glob/Grep/Read; no MCP; `success`; `report_blocking_findings: 2`, the report's own verdict line, unchecked (`report_sha256` c49bee04, a 2026-10-09 amendment) |
| The caps, 30 assistant turns and $5, bound a runaway at about twice the measured need; a run is expected to cost about $1.9 to $2.3 for a first read and $0.7 to $1.8 for a delta re-read | `local_integration` (installed 2.1.295 client, billed to the second key; not through the action) | the `LR` entry above (12 of 12 turns, client estimate $1.868667); the $2.23 figure and the delta re-reads LR2 to LR5 (5, 6, 7 and 6 turns; $1.75, $1.35, $1.54 and $0.73) come from the command center's caps decision of 2026-10-09 and are not in this repository |
| In agent mode the action writes the job token into the root checkout's `origin` URL, so `persist-credentials: false` does not keep it out of `.git/config`; the deny rules keep the model from reading it | `source_review` for the code path; the denial of `.git/config` reads is the fence receipt's (first row) | `src/github/operations/git-config.ts:129-134` and `src/modes/agent/index.ts:52-61`, read at v1.0.246 (`38c80c1`; the command center's compare shows only the bundled client version changing at v1.0.247); the record's shared sentence also cites two 2026-10-09 probes on the installed 2.1.295, not reproduced here |
| The workflow is well formed and has no audit finding | `local_integration` | at d1bcb6c: `actionlint` 1.17.0 exit 0; `zizmor` 1.30.1 offline, regular (`--no-ignores`) and pedantic, no findings (2 suppressed in each: `secrets-outside-env` on the R4 secret reads, as at a1bc3bc) |
| A hosted run authenticates, stays in bounds and reads the cache | not established | no hosted run is part of this PR |

### Local commands run

```
$ python3 scripts/validate.py            # on the R5 tree, committed as d1bcb6c
{"components": 70, "hashed_files": 10906, "profiles": 4, "receipts": 239, "status": "passed"}   exit 0
$ python3 scripts/evidence_manifest.py --check
{"files": 10906, "status": "passed"}   exit 0
$ uv run --quiet --no-project --python 3.12 --with pyyaml==6.0.3 python -m unittest tests.test_claude_pr_review_workflow tests.test_workflow_policy tests.test_workflow_security_coverage tests.test_workflow_hardening tests.test_github_automation_practice tests.test_adoption_docs_consistency
Ran 259 tests ... OK (skipped=2)   exit 0
$ uv run --quiet --no-project --python 3.12 python -m unittest tests.test_claude_pr_review_workflow    # without PyYAML
Ran 41 tests ... OK (skipped=15)   exit 0
$ actionlint .github/workflows/claude-pr-review.yml
exit 0
$ zizmor --offline --no-config --persona=pedantic .github/workflows/claude-pr-review.yml
No findings to report. Good job! (2 suppressed)   exit 0
$ zizmor --offline --no-config --no-ignores --persona regular .github/workflows/claude-pr-review.yml
No findings to report. Good job! (2 suppressed)   exit 0
$ uv run --quiet --no-project --python 3.12 --with pyyaml==6.0.3 python -B <scratchpad>/r5_pr894_mutants.py . a1bc3bc    # outside the repository
28 mutants, 28 caught, survivors: []   exit 0
$ uv run --quiet --no-project --python 3.12 python -B <scratchpad>/r5_pr894_mutants.py . a1bc3bc    # without PyYAML
28 mutants, 15 caught   exit 1 (expected: the 13 survivors are pins in the PyYAML-only shape tests; follow-up F-16)
```

Hosted checks at d1bcb6c: not read yet.

### Decision record

`docs/decisions/2026-10-08-claude-actions-pr-review.md` (why a manual dispatch and no pull request trigger, how the head is handled, the model's fence and what was checked at runtime, cost of one run with the measured runs and the cap derivation, changed test contracts, alternatives, what would overturn it), with a dated R5 section: the symbolic-link step, the caps from measurement, the non-string tool entries, the closed flag list and the boundary tests, the mutants, and the follow-ups. `docs/decisions/2026-10-04-ci-least-privilege.md` gets a dated addendum naming the second job on the federation rule.

### Host evidence

Not applicable: no file under `evidence/hosts/` changes.

### Checklist

- [x] New/changed GitHub Actions are pinned to a full commit SHA with a
      version comment (no floating tags).
- [x] New/changed workflows declare top-level `permissions: {}` and grant each job
      only what it needs (`contents: read`, or an explicitly justified addition).
- [x] No secrets are printed, logged or committed; no new required secret was
      added without a documented owner.
- [x] No new paid hosting, subscription or billing surface was introduced.
- [x] Peer-owned untracked files and worktrees were preserved (not deleted,
      moved or overwritten).

Activation is a separate step after landing: an owner sets `CLAUDE_PR_REVIEW_ENABLED` to `true`, then dispatches the workflow with a pull request number and head commit. Runs authenticate by workload identity federation and spend from the Console organization the four `ANTHROPIC_*` repository variables name; no Anthropic key is stored in GitHub. Each run is capped at 30 assistant turns and a $5 client budget, the owner's per-run spend limit on that organization. This PR and #892 both edit the write-grant inventory and the evidence manifest, so the second to land rebases once.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
seathatflowsinourveins added a commit that referenced this pull request Oct 9, 2026
…e the audit budget to $5

R5, quality and correctness only. The numbers step counts every tool-list
entry that is not a string as a forbidden tool, named "non-string tool
entry", instead of dropping it (GPT designated read of #895, P2). A new
test pins the whole claude_args list and the whole --settings JSON. The
client budget moves from $3 to $5 with the turn limit at 20, from the
command center's measurement-based caps: the one measured run used 10
turns and $2.03. The shape tests now read the workflow with the policy
test's strict loader, so none skips without PyYAML. The record gains a
dated R5 section, the cap derivation, the shared agent-mode origin-URL
sentence and the follow-ups line, and withdraws the earlier dry estimate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
seathatflowsinourveins added a commit that referenced this pull request Oct 10, 2026
)

* Bound the harness audit and move it to claude-code-action v1.0.247

The audit job ran with 60 turns, no budget, a `gh issue create` shell tool and
`issues: write`. It now gets Read, Glob and Grep, 20 turns and a $3 client
budget, holds only `contents: read` and `id-token: write`, and runs only while
CLAUDE_HARNESS_AUDIT_ENABLED is 'true'. A model-free step keeps the run's token
and cost numbers and fails the job when a bound or the cache-read check is
unmet; a second copies the report to the job summary, escaped and capped.

The pin moves from v1.0.245 to v1.0.247 (two one-commit releases that bump the
bundled Claude Code to 2.1.295). Federation stays the only authentication.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Harden the bounded audit after an independent workflow audit

Claude Code now runs under --restricted with --permission-prompts none, and
the deny rules travel in --settings, because --restricted ignores the settings
file the action's `settings` input writes. The job runs on a first attempt
only (a re-run would spend again), for the owner as actor and triggering
actor. The numbers step also reads the session's own tool list and fails on a
shell, write, web, subagent or MCP tool, writes its record atomically and
replaces names that are not plain identifiers. Step debugging is pinned off in
the action step's environment.

The flag set was run on the installed 2.1.295 client against a planted tree;
the receipt is under evidence/artifacts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Set the harness audit's effort level explicitly

The audit job now passes `--effort max` in claude_args. Under the command
center's effort mapping of 2026-10-08, judgment work (designated reads,
adjudication, pull request and security reviews, audits) runs at `max`, and
every job records its level and the reason. The level has to be in
claude_args because `--restricted` ignores the settings files that would
otherwise carry it, and Opus 5.5 runs at `medium` on the API when a request
leaves effort unset. A loopback dry run of the installed 2.1.295 client with
this workflow's claude_args sent `output_config.effort: "max"` on every
request. The bounds test now requires the flag (removing it fails the test).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Bound the harness audit's turns by assistant turns, not num_turns

The bounds step failed the job unless the result's num_turns was at most
the --max-turns value. On Claude Code 2.1.295 num_turns counts transcript
messages, tool results included: a local run with #894's prompt and
claude_args (api-actions LR, 2026-10-08) made 12 API requests under
--max-turns 12 and reported num_turns 57. The step now counts distinct
assistant message ids in the execution file and checks that count against
20; num_turns stays in the record and the job summary as "messages". The
test fixture emits two messages per turn id, as the client does, and a new
test passes a 20-turn run that reports 57 messages and fails a 21-turn one.
Three mutants (bound on num_turns, count forced to 0, count without
de-duplicating ids) each fail the tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Fix the harness audit's bounds and declare every change it makes

A pre-cue read by Anthropic's pr-review-toolkit agents (J8, on 4d0a14d)
found gaps in this PR; this commit fixes them.

Bounds step: the tool check is an allow-list (any tool outside Read,
Glob and Grep fails, not only the nine named ones); a session start
record without its tool or MCP lists fails instead of passing as empty;
a run without result text fails; and the step names every bound it finds
unmet instead of a bare exit 1. The report step takes the last non-empty
result, cuts on a character boundary (iconv -c, whose exit 1 for a cut
character no longer fails the step) and says when the report was cut.

Tests: the step tests read the workflow with the policy test's own
loader when PyYAML is absent, so they no longer skip; new tests cover an
unknown tool, a start record without lists, an empty result, the named
failures, the character-safe cap and its notice, and the last non-empty
result (21 tests).

Record: every behaviour change against main is named (timeout 45 to
30, the slug, first-attempt, variable and owner-only guards, the guard
step's exit 2, the display name and prompt, no issue, the two explicit
action inputs verified in the pinned action.yml, the usage artifact);
the fence receipt's flags are stated instead of "the same flag set"; the
cooldown waiver of 2026-10-03 is cited; and a numbers-only receipt of
local runs of this workflow's own prompt and claude_args at max effort
(10 of 20 turns, about $2.03 of $3) and of the num_turns measurement is
added. ci-least-privilege's stale issues-write lines are fixed in place.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Apply the pre-cue read's fix classes to the harness audit

The sweep of the W4 drafts for the classes #894's pre-cue toolkit read
found, applied here: an audit step that succeeds without an execution
file now fails the job instead of leaving it green with nothing checked;
display_report and track_progress are asserted; the record states that
a public run's summary is public. The workflow has no paths input, so
the bash 3.2 empty-array class does not apply. 22 tests; actionlint and
zizmor report nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Close the harness audit's partly fixed pre-cue findings

The pre-cue toolkit read of 4d0a14d left six findings only partly fixed at
89d05dd, and the test count was stale again.

- The record now lists every flag difference between the fence receipt and
  this workflow's claude_args. The receipt's runs set claudeMdExcludes, so
  they do not show that --restricted alone keeps instruction files out, and
  native_proven is limited to the receipt's own flag set.
- The .git deny rules: a 2026-10-09 local subscription probe measured Read
  and Grep into .git as refused. Whether the empty Glob results come from the
  rules is pending a control run. Whether
  blockReadsOutsideWorkingDirectories binds Glob and Grep stays not
  established.
- The numbers step records the result's subtype through its name filter, in
  the usage record, the summary table and the failure message, so a
  turn-limit stop and a budget stop read apart. Two new tests; seven weakened
  copies are each caught.
- The by-name list now names the execution-file step 89d05dd added, and the
  other inputs and messages it had left out.
- The docs bullet adds the slug, first-attempt and owner-only guards. The
  record says why the triggering-actor clause stays.
- The source list adds action.yml's lines for track_progress,
  display_report and show_full_output.
- The test count is now 24.
- local-parity-receipt.json's LR entry gains its target pull request #897
  at 24821c5, its report's two blocking findings and the report's sha256.
  This is a marked 2026-10-09 amendment from the run's harness receipt, and
  #894's record cites it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Record that the deny rules bind Glob and Grep, measured against a control

A control run of the 2026-10-09 probe, with the same fence flags and the
deny rules removed, lists the .git files, finds both markers, returns the
.env marker through Grep, and reads every marker. With the rules, the same
calls returned nothing or were refused. So the empty Glob and Grep results
come from the deny rules, not from the tools skipping hidden paths. The
record now shows probe against control step by step, and the bounds table
and evidence classes say so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Bind repository debug settings in the guard, fix the cap notice, keep the probe receipt

- R4, from the cc-native-practice lane's L3 read against claude-code-action
  v1.0.247 practice: the guard binds
  (secrets.ACTIONS_STEP_DEBUG || vars.ACTIONS_STEP_DEBUG) == 'true' and the
  same for ACTIONS_RUNNER_DEBUG. Only a boolean reaches the shell, and the
  guard refuses either setting. A test pins both bindings, and six weakened
  copies each fail it.
- From the J8 micro read of 61494a3: the publish step wrote the report with
  `jq -r`, which adds a newline, so a 60,000-byte report was announced as
  cut. It now uses `jq -j`, and a boundary test covers exactly 60,000 and
  60,001 bytes. The step comment now says it publishes the last non-empty
  result.
- The deny-rule probe and its control are kept as deny-probe-receipt.json:
  flags, client, model and authentication, every call and its outcome, and
  each stream's sha256.
- The record states that the workflow is unchanged from e48de2e to the
  merge base aba02ec, and no longer says the documentation exceeds the runs
  for the deny-rule question.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Refuse non-string tool entries, pin the client flags exactly and raise the audit budget to $5

R5, quality and correctness only. The numbers step counts every tool-list
entry that is not a string as a forbidden tool, named "non-string tool
entry", instead of dropping it (GPT designated read of #895, P2). A new
test pins the whole claude_args list and the whole --settings JSON. The
client budget moves from $3 to $5 with the turn limit at 20, from the
command center's measurement-based caps: the one measured run used 10
turns and $2.03. The shape tests now read the workflow with the policy
test's strict loader, so none skips without PyYAML. The record gains a
dated R5 section, the cap derivation, the shared agent-mode origin-URL
sentence and the follow-ups line, and withdraws the earlier dry estimate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Bound the audit's cost at the budget times 1.10, publish within-bound budget stops, and drop --max-turns

R6. The cost bound becomes the budget times the measured overrun factor 1.10
($5.50); a budget stop at or under it publishes and is named, and the
publish step runs on the bounds step's own outcome because the action
fails its step on any budget stop. --max-turns 20 leaves claude_args:
the pinned action fails a success whose num_turns exceeds it, and on
Claude Code 2.1.295 num_turns counts transcript messages (57 for 12
requests); the bounds step's check of 1 to 20 assistant turns stays. The
exact --settings pin compares canonical JSON, so 1 no longer passes for
true. The record gains the shared cost paragraph and an upstream-issue
draft (not filed).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Fix the harness audit's J8 R5-R6 findings: exemption scope, budget-stop note, stale turn text

The budget-stop exemption is tested to waive only the success and result-text checks (a stop that breaks any other
bound fails, named). The budget-stop note moves to the publish step, so it is written only once every bound passed.
Record and workflow comments no longer describe a client turn limit or an error_max_turns stop. N5-N9 are
dispositioned in the record's R7 section.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Pin the harness audit's action inputs and budget-stop floor that R7 claimed were held

The GPT read of 0cf13fc found R7's N8/N9 disposition overstated the tests. The action step's complete input set is
now compared, and budget stops at $0.01, $1 and $4.99 must pass, so an added input or a cost floor fails. The record
corrects the disposition.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Add the R8 source resolution for the harness audit's input-set and budget-floor pins

The record names the pinned upstream sources the two R8 tests rely on (the action's input declarations at 2dca132f,
the Agent SDK 0.3.295 result type, Python 3.13 unittest assertEqual and subTest), states that the 10-input set and the
absent cost floor are this repository's acceptance policy over an unchanged workflow, and labels the mutation results
synthetic.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Publish a budget-stopped audit's written text and keep instruction files out of the audit session

R10, the two Codex P2 threads at b41b969:
- harness-audit.yml:231: after a budget stop without result text, the bounds and publish steps use the text the
  model wrote until then, as claude-pr-review.yml does; every other subtype still gets result text only.
- harness-audit.yml:125: the audit step sets CLAUDE_CODE_DISABLE_CLAUDE_MDS=1 (documented); a measured 2.1.295
  probe shows project, imported, rules and nested instruction files load only without both --restricted and
  --setting-sources user, and not at all with the switch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: seathatflowsinourveins <234074349+seathatflowsinourveins@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@seathatflowsinourveins

Copy link
Copy Markdown
Collaborator Author

Parked: Claude review inside GitHub is retired (owner decision 2026-10-10, ~17:15Z); see #981 and docs/decisions/2026-10-10-retire-in-github-claude-review.md. No further pushes.

@seathatflowsinourveins

Copy link
Copy Markdown
Collaborator Author

Closing as superseded, with the owner's OK (CC, 2026-10-11).

It adds .github/workflows/claude-security-review.yml, which runs the Claude action. main's tests.test_workflow_hardening.InGitHubClaudeReviewRetiredTests now forbids any workflow that runs it.

Claude review inside GitHub was retired by #981 (8bffa2c, docs/decisions/2026-10-10-retire-in-github-claude-review.md). The Anthropic API refuses the action's entry point on Console keys (CC correction #141; the action never had a successful run). AI reviews now run locally, and GitHub enforces them as required checks. The local route is #953 (the review worker).

Nothing is lost: this PR can be reopened, and any model-free part of it can come back as a new PR under the local-review route. Evidence: the CC's PR triage (research-cc-role-20261008/pr-triage-20261011.md, class CLOSE).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant