Skip to content

ci: add loop observability + mirror inspection tools to implement agent #241

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
allxsmith merged 1 commit into from
Jul 7, 2026
+29 −1
Merged

ci: add loop observability + mirror inspection tools to implement agent #241

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 15 additions & 1 deletion .github/workflows/claude-implement.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ jobs:
claude_args: |
--max-turns 150
--model claude-sonnet-5
--allowedTools "Bash(pnpm install:*),Bash(pnpm run:*),Bash(pnpm test:*),Bash(pnpm exec:*),Bash(pnpm --filter:*),Bash(pnpm ls:*),Bash(pnpm why:*),Bash(node:*),Bash(git status:*),Bash(git diff:*),Bash(git log:*),Bash(git show:*),Bash(git checkout:*),Bash(git switch:*),Bash(git branch:*),Bash(git add:*),Bash(git commit:*),Bash(git push:*),Bash(git config:*),Bash(mkdir:*),Bash(gh issue view:*),Bash(gh issue comment:*),Bash(gh pr create:*),Bash(gh pr view:*),Bash(gh pr edit:*),Bash(gh pr diff:*),Bash(gh pr checks:*)"
--allowedTools "Bash(pnpm install:*),Bash(pnpm run:*),Bash(pnpm test:*),Bash(pnpm exec:*),Bash(pnpm --filter:*),Bash(pnpm ls:*),Bash(pnpm why:*),Bash(node:*),Bash(rg:*),Bash(grep:*),Bash(cat:*),Bash(sed:*),Bash(ls:*),Bash(head:*),Bash(tail:*),Bash(wc:*),Bash(find:*),Bash(git status:*),Bash(git diff:*),Bash(git log:*),Bash(git show:*),Bash(git checkout:*),Bash(git switch:*),Bash(git branch:*),Bash(git add:*),Bash(git commit:*),Bash(git push:*),Bash(git config:*),Bash(mkdir:*),Bash(gh issue view:*),Bash(gh issue comment:*),Bash(gh pr create:*),Bash(gh pr view:*),Bash(gh pr edit:*),Bash(gh pr diff:*),Bash(gh pr checks:*)"

@coderabbitai coderabbitai Bot Jul 7, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🌐 Web query:

claude code allowedTools Bash prefix matching how permission patterns match commands

💡 Result:

In Claude Code, the Bash tool uses permission rules based on shell-style glob pattern matching [1][2]. These rules govern whether commands are allowed or denied and rely on specific syntax to determine how command strings are matched [3][4]. Key mechanisms for Bash permission patterns include: General Syntax and Glob Matching Rules use the format Bash(pattern), where the pattern is a glob applied to the full command string [4][2]. Wildcards () can be placed at the beginning, middle, or end of the command string to allow or deny groups of commands [1][3]. Prefix Matching and Word Boundaries The presence or absence of a space before an asterisk is critical for how patterns match commands [1][3]: - Bash(ls ) (with a space): This enforces a word boundary, meaning it matches commands that start exactly with the prefix followed by a space or end-of-string [1]. For instance, Bash(ls ) matches ls -la but does not match lsof [1][3]. - Bash(ls) (no space): This performs a standard prefix match and does not enforce a word boundary [1]. Consequently, Bash(ls) matches both ls -la and lsof [1][3]. Special Suffixes The: suffix is functionally equivalent to a trailing space plus an asterisk (e.g., Bash(ls:*) is equivalent to Bash(ls *)) [1]. It is specifically recognized as a shorthand when placed at the end of a pattern [1][5]. Precedence and Behavior - Evaluation Order: Rules are evaluated in the order: deny, then ask, then allow [3]. A match in the deny list takes precedence over everything else and will block a command regardless of other settings [3][4]. - Command Parsing: Claude Code parses compound commands [3]. Each subcommand in a chain (e.g., cmd1 && cmd2) must match the permission rules independently; a rule that permits one command will not automatically permit subcommands within the same line [3]. - Fixed Wrappers: Claude Code automatically strips certain command wrappers (such as timeout, time, nice, nohup, and stdbuf) before evaluating permission matches [3]. This ensures that adding a wrapper does not circumvent established security rules [3]. Permission rules can be configured in settings files (e.g., ~/.claude.json), via CLI flags like --allowedTools or --disallowedTools, or through the Agent SDK [6][7][8]. Bare tool names (e.g., Bash) without a pattern match all calls to that tool [4][2].

Citations:

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== claude-implement.yml relevant lines =="
nl -ba .github/workflows/claude-implement.yml | sed -n '95,205p'

echo
echo "== claude-pr-loop.yml relevant lines =="
if [ -f .github/workflows/claude-pr-loop.yml ]; then
  nl -ba .github/workflows/claude-pr-loop.yml | sed -n '330,370p'
fi

echo
echo "== search for execution_file / artifact upload references =="
rg -n "execution_file|upload-artifact|claude-execution-output|RUNNER_TEMP|if-no-files-found|retention-days|allowedTools" .github/workflows . -g '!**/node_modules/**'

Repository: allxsmith/bestax

Length of output: 236

🏁 Script executed:

#!/bin/bash
set -euo pipefail

python3 - <<'PY'
from pathlib import Path

def show(path, start, end):
    print(f"== {path} {start}-{end} ==")
    p = Path(path)
    if not p.exists():
        print("MISSING")
        return
    lines = p.read_text().splitlines()
    for i in range(start-1, min(end, len(lines))):
        print(f"{i+1:4d}: {lines[i]}")
    print()

show(".github/workflows/claude-implement.yml", 100, 205)
show(".github/workflows/claude-pr-loop.yml", 335, 360)
PY

echo "== search hits =="
rg -n "allowedTools|upload-artifact|claude-execution-output|RUNNER_TEMP|if-no-files-found|retention-days|execution_file" .github/workflows . -g '!**/node_modules/**'

Repository: allxsmith/bestax

Length of output: 13573

Bash(sed:*) and Bash(find:*) should stay out of the read-only allowlist. sed -i can edit files and find -exec can launch subprocesses, so these entries widen the agent’s capabilities on untrusted issue text. Drop them here, and in .github/workflows/claude-pr-loop.yml too if you want both agents constrained the same way.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/claude-implement.yml at line 109, The read-only
allowedTools list is too permissive because it includes Bash(sed:*) and
Bash(find:*), which can be used to modify files or launch subprocesses. Remove
those entries from the allowedTools string in the workflow configuration, and
make the same change in claude-pr-loop.yml if both agent workflows should share
the same restricted tool set. Use the allowedTools definition in the workflow
job as the place to update.

prompt: |
REPO: ${{ github.repository }}
ISSUE NUMBER: ${{ github.event.issue.number }}
Expand Down Expand Up @@ -174,6 +174,20 @@ jobs:
to main, and never write "@claude" or "@coderabbitai" in any
comment, commit message, or PR text.

# Observability: the action writes a full turn-by-turn stream (every
# tool call, and every permission denial) to this file. Upload it so a
# failed/thrashing run is diagnosable after the fact — the Actions log
# itself only carries the init + final result. `always()` so it is kept
# even when the Claude step fails (exactly when it is needed).
- name: Upload Claude execution log
if: always() && steps.perm.outputs.allowed == 'true'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: claude-implement-log-${{ github.event.issue.number }}
path: ${{ runner.temp }}/claude-execution-output.json
if-no-files-found: ignore
retention-days: 14

Comment on lines +177 to +190

@coderabbitai coderabbitai Bot Jul 7, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🌐 Web query:

claude-code-action execution output file redact secrets masking sensitive data

💡 Result:

Claude Code does not have a native, built-in feature to automatically redact secrets or mask sensitive data from tool execution outputs [1][2][3]. By default, raw output from tools like Bash, Read, or Grep is stored verbatim in the conversation context and remains accessible for the duration of the session, including in any shared transcripts [1][2]. Because Claude Code relies on the harness to manage tool execution, sensitive data that enters the context window is permanent and cannot be removed or redacted by the model itself [2]. To address this, the community has developed several third-party hooks and filtering tools that operate by intercepting tool calls or outputs [4][5][6][7][8]. Key Approaches for Redaction and Security: 1. Permission Rules (Official): The primary built-in mechanism is to use strict permission settings to prevent Claude Code from accessing sensitive files or running risky commands altogether [9][10][11]. You can use the /permissions command or configure permissions.deny in your settings to block access to files like.env or specific directories containing credentials [10][11]. 2. Third-Party Redaction Hooks: Several open-source projects provide hooks to intercept tool usage and redact data [4][5][6][8]. These typically function by: - Scanning tool inputs or outputs for common secret patterns (API keys, tokens, PII) using regex or entropy analysis [4][6][7][8]. - Replacing detected sensitive strings with placeholders (e.g., [REDACTED]) before they enter the model's context [4][5][7][8]. - Blocking tool execution entirely if sensitive patterns are detected in files that the model attempts to read [5][6][8]. 3. Security Patterns: You can define custom security rules via .claude/security-patterns.yaml to monitor and warn against specific patterns in code edits, which helps catch hardcoded secrets before they are committed [12]. While these tools provide an essential layer of protection, they operate as community-developed extensions and not as part of the core Claude Code functionality [4][5][2][6][7][8]. Users working with highly sensitive information should prioritize strict file-system permissions and avoid granting Claude Code access to directories containing credentials [9][10][11].

Citations:

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Locate the workflow and any references to the execution log artifact or the Claude action.
git ls-files | rg -n '(^|/)(claude-implement\.yml|.*claude.*|.*execution.*|README\.md|docs/|\.github/workflows/)'
printf '\n--- matches for execution-output ---\n'
rg -n "claude-execution-output|execution output|turn-by-turn|permission denial|Upload Claude execution log|always\(\).*allowed == 'true'" .github/workflows README.md docs . 2>/dev/null || true

printf '\n--- workflow excerpt around artifact upload ---\n'
sed -n '150,210p' .github/workflows/claude-implement.yml

Repository: allxsmith/bestax

Length of output: 16538

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '\n--- claude-implement.yml top section ---\n'
sed -n '1,170p' .github/workflows/claude-implement.yml

printf '\n--- exact action references in claude workflows ---\n'
rg -n "uses: .*claude|claude-execution-output|execution-output|mask|redact|secret" .github/workflows/claude-implement.yml .github/workflows/claude-pr-loop.yml .github/workflows/claude-review.yml .github/workflows/claude.yml

printf '\n--- docs references for the action or log file ---\n'
rg -n "claude-execution-output|redact|mask|secret|artifact|upload-artifact|claude-code-action|anthropics/claude" README.md docs .github 2>/dev/null || true

Repository: allxsmith/bestax

Length of output: 18045

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Inspect the workflow and any adjacent evidence of redaction/masking.
sed -n '1,170p' .github/workflows/claude-implement.yml
printf '\n---\n'
rg -n "uses: .*claude|claude-execution-output|execution-output|mask|redact|secret" .github/workflows/claude-implement.yml .github/workflows/claude-pr-loop.yml .github/workflows/claude-review.yml .github/workflows/claude.yml
printf '\n---\n'
rg -n "claude-execution-output|redact|mask|secret|artifact|upload-artifact|claude-code-action|anthropics/claude" README.md docs .github 2>/dev/null || true

Repository: allxsmith/bestax

Length of output: 17908

🌐 Web query:

anthropics claude code action execution output json sensitive information redaction

💡 Result:

As of July 2026, Claude Code does not have a native, built-in, or automatic system to redact sensitive information (such as API keys, secrets, or credentials) from tool execution outputs, conversation transcripts, or logs [1][2][3][4]. When Claude Code executes tools—particularly the Bash tool—it captures and stores the raw, verbatim output in the conversation history and persists it to disk (e.g., in ~/.claude/projects/ as JSONL files) without a sanitization pass [2][3]. This means that if a shell command inadvertently outputs a sensitive value, that value is permanently recorded in the session transcript [2][4]. Key security considerations regarding this behavior include: 1. Absence of Native Redaction: Users frequently report that secret values leak into the model's context and persistent session files because there is no harness-level redaction layer to filter these values before they are stored or processed [1][2][3]. 2. Limitations of Existing Hooks: While users can attempt to use hooks (such as PostToolUse) to redact or modify tool outputs, there have been documented technical issues—such as the Bash tool ignoring modified updatedToolOutput—that can render these custom workarounds ineffective at preventing secrets from reaching the model's context [5]. 3. Risk of Exposure in GitHub Actions: When using the claude-code-action in GitHub, enabling show_full_output (which is disabled by default) will expose all Claude Code messages, including tool execution results that may contain secrets, in publicly visible GitHub Action logs [6][7][8]. 4. Community Workarounds: To mitigate these risks, users have developed third-party tools and custom hooks aimed at intercepting tool calls to redact secrets [2][9][10][11]. These typically involve filtering output based on regex patterns, masking values, or denying access to specific files (e.g., .env, .pem), but these are not official features provided by Anthropic [9][11][4]. Because of these limitations, it is strongly recommended that you never commit API keys to your repository, rely strictly on secure secret management (e.g., GitHub Secrets), and treat all session transcripts and logs as potentially containing sensitive information [8][3].

Citations:

Execution log artifact can expose secrets .github/workflows/claude-implement.yml:182-188 uploads claude-execution-output.json with 14-day retention, and the Claude Code action does not redact tool output by default. Redact or suppress sensitive output before writing the file, or skip the artifact when secret-bearing commands can run.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/claude-implement.yml around lines 177 - 190, The Upload
Claude execution log step is publishing a potentially sensitive execution trace
from the Claude workflow, which can expose secrets. Update the logging flow
around the Claude Code action output and the artifact upload step so sensitive
tool output is redacted or suppressed before writing
claude-execution-output.json, or conditionally skip uploading it when
secret-bearing commands may run. Keep the fix localized to the Upload Claude
execution log job step and the related Claude action output handling.

# Optional second reviewer (repo variable AI_LOOP_COPILOT=true).
# Copilot's automatic-review ruleset skips bot-authored PRs on personal
# repos, so the review must be requested explicitly.
Expand Down
14 changes: 14 additions & 0 deletions .github/workflows/claude-pr-loop.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -421,6 +421,20 @@ jobs:
(every remaining thread is refuted), say so in the summary
comment and stop — do not make cosmetic commits.

# Observability: the action writes a full turn-by-turn stream (every
# tool call, and every permission denial) to this file. Upload it so a
# failed/thrashing fix pass is diagnosable after the fact — the Actions
# log itself only carries the init + final result. `always()` so it is
# kept even when the fix step fails (exactly when it is needed).
- name: Upload Claude execution log
if: always() && steps.claude.conclusion != 'skipped'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: claude-fix-log-pr${{ needs.gate.outputs.pr }}-iter${{ needs.gate.outputs.iteration }}
path: ${{ runner.temp }}/claude-execution-output.json
if-no-files-found: ignore
retention-days: 14

# No push + threads still open would stall the loop forever (no event
# fires). End in a named state instead.
- name: Terminal state check
Expand Down
Loading