Repository navigation
ci: add loop observability + mirror inspection tools to implement agent - #241
Conversation
Two hardening changes ahead of stress-testing the loop on a real bug: 1. Mirror the read-only inspection utilities (rg/grep/cat/sed/ls/head/tail/ wc/find) that #239 added to the fix agent into the implement agent's allowlist. The implement agent has the same latent gap that thrashed the sonnet fixer (42 denials); it has coped on sonnet-5 + 150 turns + native Read/Grep tools, but a heavy issue could hit the same wall. Cheap insurance, all read-only. 2. Upload the action's execution-output.json as an artifact (if: always()) from both the implement and fix jobs. That file carries the full turn-by-turn stream — every tool call AND every permission denial — whereas the Actions log only streams init + final result. When the sonnet fixer thrashed we could see '42 denials' but not WHICH tools; this makes the next failure diagnosable instead of guesswork. Reuses the repo's existing pinned actions/upload-artifact and never fails the job (if-no-files-found: ignore). Models are deliberately left as-is: sonnet-5 implement, opus deep review, opus fix — the heterogeneity decorrelates implementer and reviewer blind spots. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NVR5yWevceZEFbTJmpviiM
WalkthroughThis PR updates two GitHub Actions workflow files. It expands the Bash tool allowlist in claude-implement.yml to include additional shell utilities, and adds steps to both claude-implement.yml and claude-pr-loop.yml that upload Claude execution log JSON as a build artifact with 14-day retention. ChangesClaude Workflow Logging and Tool Allowlist
Estimated code review effort: 1 (Trivial) | ~5 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Preview DeploymentPreview URL: https://90ffc42d.bestax.pages.dev |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/claude-implement.yml:
- Around line 177-190: The Upload Claude execution log step is publishing a
potentially sensitive execution trace from the Claude workflow, which can expose
secrets. Update the logging flow around the Claude Code action output and the
artifact upload step so sensitive tool output is redacted or suppressed before
writing claude-execution-output.json, or conditionally skip uploading it when
secret-bearing commands may run. Keep the fix localized to the Upload Claude
execution log job step and the related Claude action output handling.
- Line 109: The read-only allowedTools list is too permissive because it
includes Bash(sed:*) and Bash(find:*), which can be used to modify files or
launch subprocesses. Remove those entries from the allowedTools string in the
workflow configuration, and make the same change in claude-pr-loop.yml if both
agent workflows should share the same restricted tool set. Use the allowedTools
definition in the workflow job as the place to update.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 65f91a9b-764d-428f-81b7-9d39de7debc9
📒 Files selected for processing (2)
.github/workflows/claude-implement.yml.github/workflows/claude-pr-loop.yml
| --max-turns 150 | ||
| --model claude-sonnet-5 | ||
| --allowedTools "Bash(pnpm install:*),Bash(pnpm run:*),Bash(pnpm test:*),Bash(pnpm exec:*),Bash(pnpm --filter:*),Bash(pnpm ls:*),Bash(pnpm why:*),Bash(node:*),Bash(git status:*),Bash(git diff:*),Bash(git log:*),Bash(git show:*),Bash(git checkout:*),Bash(git switch:*),Bash(git branch:*),Bash(git add:*),Bash(git commit:*),Bash(git push:*),Bash(git config:*),Bash(mkdir:*),Bash(gh issue view:*),Bash(gh issue comment:*),Bash(gh pr create:*),Bash(gh pr view:*),Bash(gh pr edit:*),Bash(gh pr diff:*),Bash(gh pr checks:*)" | ||
| --allowedTools "Bash(pnpm install:*),Bash(pnpm run:*),Bash(pnpm test:*),Bash(pnpm exec:*),Bash(pnpm --filter:*),Bash(pnpm ls:*),Bash(pnpm why:*),Bash(node:*),Bash(rg:*),Bash(grep:*),Bash(cat:*),Bash(sed:*),Bash(ls:*),Bash(head:*),Bash(tail:*),Bash(wc:*),Bash(find:*),Bash(git status:*),Bash(git diff:*),Bash(git log:*),Bash(git show:*),Bash(git checkout:*),Bash(git switch:*),Bash(git branch:*),Bash(git add:*),Bash(git commit:*),Bash(git push:*),Bash(git config:*),Bash(mkdir:*),Bash(gh issue view:*),Bash(gh issue comment:*),Bash(gh pr create:*),Bash(gh pr view:*),Bash(gh pr edit:*),Bash(gh pr diff:*),Bash(gh pr checks:*)" |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
claude code allowedTools Bash prefix matching how permission patterns match commands
💡 Result:
In Claude Code, the Bash tool uses permission rules based on shell-style glob pattern matching [1][2]. These rules govern whether commands are allowed or denied and rely on specific syntax to determine how command strings are matched [3][4]. Key mechanisms for Bash permission patterns include: General Syntax and Glob Matching Rules use the format Bash(pattern), where the pattern is a glob applied to the full command string [4][2]. Wildcards () can be placed at the beginning, middle, or end of the command string to allow or deny groups of commands [1][3]. Prefix Matching and Word Boundaries The presence or absence of a space before an asterisk is critical for how patterns match commands [1][3]: - Bash(ls ) (with a space): This enforces a word boundary, meaning it matches commands that start exactly with the prefix followed by a space or end-of-string [1]. For instance, Bash(ls ) matches ls -la but does not match lsof [1][3]. - Bash(ls) (no space): This performs a standard prefix match and does not enforce a word boundary [1]. Consequently, Bash(ls) matches both ls -la and lsof [1][3]. Special Suffixes The: suffix is functionally equivalent to a trailing space plus an asterisk (e.g., Bash(ls:*) is equivalent to Bash(ls *)) [1]. It is specifically recognized as a shorthand when placed at the end of a pattern [1][5]. Precedence and Behavior - Evaluation Order: Rules are evaluated in the order: deny, then ask, then allow [3]. A match in the deny list takes precedence over everything else and will block a command regardless of other settings [3][4]. - Command Parsing: Claude Code parses compound commands [3]. Each subcommand in a chain (e.g., cmd1 && cmd2) must match the permission rules independently; a rule that permits one command will not automatically permit subcommands within the same line [3]. - Fixed Wrappers: Claude Code automatically strips certain command wrappers (such as timeout, time, nice, nohup, and stdbuf) before evaluating permission matches [3]. This ensures that adding a wrapper does not circumvent established security rules [3]. Permission rules can be configured in settings files (e.g., ~/.claude.json), via CLI flags like --allowedTools or --disallowedTools, or through the Agent SDK [6][7][8]. Bare tool names (e.g., Bash) without a pattern match all calls to that tool [4][2].
Citations:
- 1: https://code.claude.com/docs/en/permissions.md
- 2: https://www.mintlify.com/VineeTagarwaL-code/claude-code/reference/sdk/permissions-api
- 3: https://getclaudekit.com/blog/guide/mechanics/permission-model
- 4: https://engineering-playbook.vercel.app/claude-code/permission-rules-and-allowlists
- 5: https://www.iamraghuveer.com/posts/claude-code-project-scoped-permissions/
- 6: https://code.claude.com/docs/en/tools-reference.md
- 7: https://code.claude.com/docs/en/tools-reference
- 8: https://code.claude.com/docs/en/settings?_rsc=crtel
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== claude-implement.yml relevant lines =="
nl -ba .github/workflows/claude-implement.yml | sed -n '95,205p'
echo
echo "== claude-pr-loop.yml relevant lines =="
if [ -f .github/workflows/claude-pr-loop.yml ]; then
nl -ba .github/workflows/claude-pr-loop.yml | sed -n '330,370p'
fi
echo
echo "== search for execution_file / artifact upload references =="
rg -n "execution_file|upload-artifact|claude-execution-output|RUNNER_TEMP|if-no-files-found|retention-days|allowedTools" .github/workflows . -g '!**/node_modules/**'Repository: allxsmith/bestax
Length of output: 236
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
def show(path, start, end):
print(f"== {path} {start}-{end} ==")
p = Path(path)
if not p.exists():
print("MISSING")
return
lines = p.read_text().splitlines()
for i in range(start-1, min(end, len(lines))):
print(f"{i+1:4d}: {lines[i]}")
print()
show(".github/workflows/claude-implement.yml", 100, 205)
show(".github/workflows/claude-pr-loop.yml", 335, 360)
PY
echo "== search hits =="
rg -n "allowedTools|upload-artifact|claude-execution-output|RUNNER_TEMP|if-no-files-found|retention-days|execution_file" .github/workflows . -g '!**/node_modules/**'Repository: allxsmith/bestax
Length of output: 13573
Bash(sed:*) and Bash(find:*) should stay out of the read-only allowlist. sed -i can edit files and find -exec can launch subprocesses, so these entries widen the agent’s capabilities on untrusted issue text. Drop them here, and in .github/workflows/claude-pr-loop.yml too if you want both agents constrained the same way.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/claude-implement.yml at line 109, The read-only
allowedTools list is too permissive because it includes Bash(sed:*) and
Bash(find:*), which can be used to modify files or launch subprocesses. Remove
those entries from the allowedTools string in the workflow configuration, and
make the same change in claude-pr-loop.yml if both agent workflows should share
the same restricted tool set. Use the allowedTools definition in the workflow
job as the place to update.
| # Observability: the action writes a full turn-by-turn stream (every | ||
| # tool call, and every permission denial) to this file. Upload it so a | ||
| # failed/thrashing run is diagnosable after the fact — the Actions log | ||
| # itself only carries the init + final result. `always()` so it is kept | ||
| # even when the Claude step fails (exactly when it is needed). | ||
| - name: Upload Claude execution log | ||
| if: always() && steps.perm.outputs.allowed == 'true' | ||
| uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 | ||
| with: | ||
| name: claude-implement-log-${{ github.event.issue.number }} | ||
| path: ${{ runner.temp }}/claude-execution-output.json | ||
| if-no-files-found: ignore | ||
| retention-days: 14 | ||
|
|
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
claude-code-action execution output file redact secrets masking sensitive data
💡 Result:
Claude Code does not have a native, built-in feature to automatically redact secrets or mask sensitive data from tool execution outputs [1][2][3]. By default, raw output from tools like Bash, Read, or Grep is stored verbatim in the conversation context and remains accessible for the duration of the session, including in any shared transcripts [1][2]. Because Claude Code relies on the harness to manage tool execution, sensitive data that enters the context window is permanent and cannot be removed or redacted by the model itself [2]. To address this, the community has developed several third-party hooks and filtering tools that operate by intercepting tool calls or outputs [4][5][6][7][8]. Key Approaches for Redaction and Security: 1. Permission Rules (Official): The primary built-in mechanism is to use strict permission settings to prevent Claude Code from accessing sensitive files or running risky commands altogether [9][10][11]. You can use the /permissions command or configure permissions.deny in your settings to block access to files like.env or specific directories containing credentials [10][11]. 2. Third-Party Redaction Hooks: Several open-source projects provide hooks to intercept tool usage and redact data [4][5][6][8]. These typically function by: - Scanning tool inputs or outputs for common secret patterns (API keys, tokens, PII) using regex or entropy analysis [4][6][7][8]. - Replacing detected sensitive strings with placeholders (e.g., [REDACTED]) before they enter the model's context [4][5][7][8]. - Blocking tool execution entirely if sensitive patterns are detected in files that the model attempts to read [5][6][8]. 3. Security Patterns: You can define custom security rules via .claude/security-patterns.yaml to monitor and warn against specific patterns in code edits, which helps catch hardcoded secrets before they are committed [12]. While these tools provide an essential layer of protection, they operate as community-developed extensions and not as part of the core Claude Code functionality [4][5][2][6][7][8]. Users working with highly sensitive information should prioritize strict file-system permissions and avoid granting Claude Code access to directories containing credentials [9][10][11].
Citations:
- 1: Feature: Auto-redact secrets in tool outputs before they enter context / transcripts anthropics/claude-code#46201
- 2: [FEATURE] Mechanism to redact secrets/PII from the context window anthropics/claude-code#29434
- 3: Title: Claude Code displays PII from tool outputs without masking anthropics/claude-code#51509
- 4: https://github.com/l-mb/claude-code-redaction-hooks
- 5: https://github.com/ShindouMihou/cc-redact/
- 6: https://github.com/coo-quack/sensitive-canary
- 7: https://github.com/paroque28/claude-code-redact
- 8: https://github.com/wissem/cc-filter
- 9: https://code.claude.com/docs/en/security
- 10: https://code.claude.com/docs/en/permissions
- 11: https://code.claude.com/docs/en/settings.md
- 12: https://code.claude.com/docs/en/security-guidance
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Locate the workflow and any references to the execution log artifact or the Claude action.
git ls-files | rg -n '(^|/)(claude-implement\.yml|.*claude.*|.*execution.*|README\.md|docs/|\.github/workflows/)'
printf '\n--- matches for execution-output ---\n'
rg -n "claude-execution-output|execution output|turn-by-turn|permission denial|Upload Claude execution log|always\(\).*allowed == 'true'" .github/workflows README.md docs . 2>/dev/null || true
printf '\n--- workflow excerpt around artifact upload ---\n'
sed -n '150,210p' .github/workflows/claude-implement.ymlRepository: allxsmith/bestax
Length of output: 16538
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n--- claude-implement.yml top section ---\n'
sed -n '1,170p' .github/workflows/claude-implement.yml
printf '\n--- exact action references in claude workflows ---\n'
rg -n "uses: .*claude|claude-execution-output|execution-output|mask|redact|secret" .github/workflows/claude-implement.yml .github/workflows/claude-pr-loop.yml .github/workflows/claude-review.yml .github/workflows/claude.yml
printf '\n--- docs references for the action or log file ---\n'
rg -n "claude-execution-output|redact|mask|secret|artifact|upload-artifact|claude-code-action|anthropics/claude" README.md docs .github 2>/dev/null || trueRepository: allxsmith/bestax
Length of output: 18045
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the workflow and any adjacent evidence of redaction/masking.
sed -n '1,170p' .github/workflows/claude-implement.yml
printf '\n---\n'
rg -n "uses: .*claude|claude-execution-output|execution-output|mask|redact|secret" .github/workflows/claude-implement.yml .github/workflows/claude-pr-loop.yml .github/workflows/claude-review.yml .github/workflows/claude.yml
printf '\n---\n'
rg -n "claude-execution-output|redact|mask|secret|artifact|upload-artifact|claude-code-action|anthropics/claude" README.md docs .github 2>/dev/null || trueRepository: allxsmith/bestax
Length of output: 17908
🌐 Web query:
anthropics claude code action execution output json sensitive information redaction
💡 Result:
As of July 2026, Claude Code does not have a native, built-in, or automatic system to redact sensitive information (such as API keys, secrets, or credentials) from tool execution outputs, conversation transcripts, or logs [1][2][3][4]. When Claude Code executes tools—particularly the Bash tool—it captures and stores the raw, verbatim output in the conversation history and persists it to disk (e.g., in ~/.claude/projects/ as JSONL files) without a sanitization pass [2][3]. This means that if a shell command inadvertently outputs a sensitive value, that value is permanently recorded in the session transcript [2][4]. Key security considerations regarding this behavior include: 1. Absence of Native Redaction: Users frequently report that secret values leak into the model's context and persistent session files because there is no harness-level redaction layer to filter these values before they are stored or processed [1][2][3]. 2. Limitations of Existing Hooks: While users can attempt to use hooks (such as PostToolUse) to redact or modify tool outputs, there have been documented technical issues—such as the Bash tool ignoring modified updatedToolOutput—that can render these custom workarounds ineffective at preventing secrets from reaching the model's context [5]. 3. Risk of Exposure in GitHub Actions: When using the claude-code-action in GitHub, enabling show_full_output (which is disabled by default) will expose all Claude Code messages, including tool execution results that may contain secrets, in publicly visible GitHub Action logs [6][7][8]. 4. Community Workarounds: To mitigate these risks, users have developed third-party tools and custom hooks aimed at intercepting tool calls to redact secrets [2][9][10][11]. These typically involve filtering output based on regex patterns, masking values, or denying access to specific files (e.g., .env, .pem), but these are not official features provided by Anthropic [9][11][4]. Because of these limitations, it is strongly recommended that you never commit API keys to your repository, rely strictly on secure secret management (e.g., GitHub Secrets), and treat all session transcripts and logs as potentially containing sensitive information [8][3].
Citations:
- 1: Feature: Auto-redact secrets in tool outputs before they enter context / transcripts anthropics/claude-code#46201
- 2: [security] Built-in secret redaction at jsonl write-time in ~/.claude/projects/ anthropics/claude-code#63593
- 3: SECURITY: Claude Code reveals secret keys and credentials in terminal output despite explicit prohibitions anthropics/claude-code#32523
- 4: Feature request: harness-level Bash-output secret redaction (3 leak incidents in 6 days, willing to contribute regex set + test suite) anthropics/claude-code#65122
- 5: [BUG] PostToolUse updatedToolOutput still not honored for Bash tool — secrets leak into model context anthropics/claude-code#65403
- 6: https://github.com/anthropics/claude-code-action/blob/refs/heads/main/docs/security.md
- 7: https://github.com/anthropics/claude-code-action/blob/9a3c761f/base-action/action.yml
- 8: https://github.com/anthropics/claude-code-action/blob/eb99fb38/base-action/README.md
- 9: https://github.com/ShindouMihou/cc-redact/
- 10: [FEATURE] Environment Variable Redaction for Hook Security anthropics/claude-code#21528
- 11: https://github.com/paroque28/claude-code-redact
Execution log artifact can expose secrets .github/workflows/claude-implement.yml:182-188 uploads claude-execution-output.json with 14-day retention, and the Claude Code action does not redact tool output by default. Redact or suppress sensitive output before writing the file, or skip the artifact when secret-bearing commands can run.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/claude-implement.yml around lines 177 - 190, The Upload
Claude execution log step is publishing a potentially sensitive execution trace
from the Claude workflow, which can expose secrets. Update the logging flow
around the Claude Code action output and the artifact upload step so sensitive
tool output is redacted or suppressed before writing
claude-execution-output.json, or conditionally skip uploading it when
secret-bearing commands may run. Keep the fix localized to the Upload Claude
execution log job step and the related Claude action output handling.
|
🎉 This PR is included in version 5.2.1 🎉 The release is available on: Your semantic-release bot 📦🚀 |
|
🎉 This PR is included in version 3.2.0 🎉 The release is available on: Your semantic-release bot 📦🚀 |
|
🎉 This PR is included in version 1.0.0 🎉 The release is available on: Your semantic-release bot 📦🚀 |
|
🎉 This PR is included in version 1.0.0 🎉 The release is available on: Your semantic-release bot 📦🚀 |
2 participants
What
Two hardening changes to the AI loop, ahead of stress-testing it on a real bug (#189):
claude-implement.yml's allowlist gainsrg/grep/cat/sed/ls/head/tail/wc/find— the same read-only set ci: give the fix agent Opus 4.8 (1M), more turns, and inspection tools #239 added to the fix agent.if: always()) from both the implement and fix jobs.Why
(1) The sonnet fixer thrashed on #238 with 42 permission denials because it lacked read-only inspection utilities. The implement agent has the same allowlist gap — it's coped so far on sonnet-5 + 150 turns + the native Read/Grep tools, but a heavy issue could hit the same wall. Cheap insurance; all read-only.
(2) When the fixer thrashed, the Actions log told us "42 denials" but not which tools — the action streams only
init+ finalresult, and uploads no artifact. Theclaude-execution-output.jsonit writes to${RUNNER_TEMP}carries the full turn-by-turn stream, including every tool call and every denial. Uploading it (withalways(), so a failed run is captured) turns the next failure from guesswork into a downloadable trace. Reuses the repo's existing pinnedactions/upload-artifact@043fb46d…, and never fails the job (if-no-files-found: ignore).Deliberately unchanged: the models
Left as sonnet-5 implement / opus deep review / opus fix. The heterogeneity is intentional — the opus reviewer decorrelates from the sonnet implementer's blind spots, so unifying on one model (either direction) would weaken review. See
claude-review.yml's header.Scope
Two allowlist entries + two
upload-artifactsteps. No prompt or logic changes. YAML validated.Testing
claude-*-log-*artifact.Generated by Claude Code
Summary by CodeRabbit