Repository navigation
ci: surface silent Claude session failures (ai-triage + deep review); triage prompt fix - #319
Conversation
…l session output The first ai-triage session to ever execute (run 29380908689) ended 'success' after 11 turns with 3 permission denials and posted nothing. The prompt said to 'run the /triage-dedupe command', steering the session toward the SlashCommand tool, which the allowlist denies. - Reword the prompt: open the .claude/commands/triage-*.md files with the Read tool and follow them; slash invocation explicitly forbidden. - show_full_output: true on the Claude step so the public job log shows the session transcript — the session only sees public issue/PR text and GET-only gh output, and silent early exits were invisible without it. Refs #317 (close after a live run posts a triage comment with zero permission denials).
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (2)
WalkthroughThe AI triage workflow now reads documented procedure files, restricts Claude tools, exposes full session output, and detects silent errors. The deep-review workflow explicitly allows the Claude bot and validates successful session output. ChangesClaude workflow guardrails
Estimated code review effort: 2 (Simple) | ~15 minutes Possibly related PRs
Suggested labels: 🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Preview DeploymentPreview URL: https://f2d89b31.bestax.pages.dev |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
.github/workflows/ai-triage.yml (1)
292-301: 🔒 Security & Privacy | 🟠 Major | ⚡ Quick winExplicitly block the forbidden tools here.
--allowedToolsonly auto-approves the listed tools; it doesn’t stop other built-ins from being available. Add--disallowedTools(or switch to a hard--toolsallowlist) so the triage agent cannot useEdit,Write, orSlashCommandon untrusted issue/PR text.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/ai-triage.yml around lines 292 - 301, Update the claude_args configuration to explicitly block Edit, Write, and SlashCommand using --disallowedTools, or replace the current tool configuration with a hard --tools allowlist. Preserve the existing approved read-only and GitHub command tools while ensuring the triage agent cannot invoke the forbidden tools.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/ai-triage.yml:
- Around line 286-291: Disable show_full_output in the workflow configuration to
prevent raw session and tool output from being exposed in public job logs.
Preserve full-output behavior only through an explicitly isolated private or
debug path, if one already exists.
---
Outside diff comments:
In @.github/workflows/ai-triage.yml:
- Around line 292-301: Update the claude_args configuration to explicitly block
Edit, Write, and SlashCommand using --disallowedTools, or replace the current
tool configuration with a hard --tools allowlist. Preserve the existing approved
read-only and GitHub command tools while ensuring the triage agent cannot invoke
the forbidden tools.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: e2f96e83-db96-490a-b957-5dc8ec675297
📒 Files selected for processing (1)
.github/workflows/ai-triage.yml
CodeRabbit review: --allowedTools only auto-approves; other built-ins remain available and fail per-call. --disallowedTools removes Edit/ Write/MultiEdit/NotebookEdit/SlashCommand/WebFetch/WebSearch outright — defense-in-depth for a session that ingests untrusted issue/PR text, and it enforces the #317 fix at the tool layer too.
|
Addressed the outside-diff-range finding in 14e8a8f: added The inline |
Preview DeploymentPreview URL: https://1c9c4911.bestax.pages.dev |
…n is_error The deep review on PR #319 died on its first API call (result is_error: true, 1 turn, $0) but the action step and job stayed green, so no review posted and nothing flagged it — the same silent-failure class as ai-triage #317. Debug-mode re-runs do not reveal the error on the pinned action version. - show_full_output: true on the deep-review step (same public-data rationale as ai-triage.yml: the session reads only the public PR and repo). - New step fails the job when the session result has is_error: true, so an API auth/limit failure shows red instead of a green no-op the AI loop's deep-review gate would wait on forever.
Preview DeploymentPreview URL: https://6f38da5e.bestax.pages.dev |
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/claude-review.yml:
- Around line 268-277: Update the execution-file validation and result check
around EXEC_FILE so a missing or invalid file exits with failure instead of
success. Require jq to find a final result record whose is_error field is
boolean; fail on missing records, malformed JSON, or is_error=true, and only
print the successful completion message when is_error is validly false.
- Around line 164-170: Disable show_full_output in the relevant job
configuration, removing or setting the show_full_output option to false. Keep
the existing review workflow behavior unchanged while preventing full session,
tool, API, and command output from being exposed in the public job log.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: d5d6167e-8eb3-4742-bbc4-c519d40e8cd9
📒 Files selected for processing (1)
.github/workflows/claude-review.yml
Same is_error gate as the deep-review step: a session that dies on its first API call otherwise leaves the run green with no comment posted. Empty execution file (the action's workflow-validation skip, seen when a PR modifies the workflow file itself) is tolerated — no session ran.
allxsmith
changed the title
Preview DeploymentPreview URL: https://a19c36b5.bestax.pages.dev |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/ai-triage.yml:
- Around line 346-369: Make both session-result guards fail closed: in
.github/workflows/ai-triage.yml lines 346-369 and
.github/workflows/claude-review.yml lines 257-280, continue skipping only when
EXEC_FILE is missing or empty, then validate that the execution data contains a
final result record with boolean is_error set to false. Treat malformed files,
missing result records, invalid result shapes, or any other jq validation
failure as errors, and only report session completion after strict validation
succeeds.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 43bdbd28-d5ec-43e3-9c98-53c87a3a10de
📒 Files selected for processing (2)
.github/workflows/ai-triage.yml.github/workflows/claude-review.yml
…re-runs CodeRabbit round 2: - Both is_error gates now require a final result record with is_error exactly false (array or NDJSON); malformed JSON, missing result, or wrong shape fails the job instead of passing silently. Only a missing execution file is tolerated — that is the action's no-session workflow-validation skip, observed on run 29462306574. - claude-review.yml: show_full_output now only on debug re-runs (runner.debug). Unlike ai-triage's GET-only public surface, the deep-review session runs pnpm against untrusted PR-head code whose output enters the transcript; transformed env values would evade GitHub's secret masking. The red is_error gate signals THAT it failed; a debug re-run reveals WHY. Predicate verified against 8 shapes: array/NDJSON x ok/error, missing result, malformed, empty file, result without is_error flag.
Preview DeploymentPreview URL: https://467c6906.bestax.pages.dev |
|
🎉 This PR is included in version 3.3.3 🎉 The release is available on: Your semantic-release bot 📦🚀 |
|
🎉 This PR is included in version 5.7.0 🎉 The release is available on: Your semantic-release bot 📦🚀 |
|
🎉 This PR is included in version 1.0.0 🎉 The release is available on: Your semantic-release bot 📦🚀 |
|
🎉 This PR is included in version 1.0.0 🎉 The release is available on: Your semantic-release bot 📦🚀 |
1 participant
Summary
Started as the fix for the ai-triage silent no-op (#317 — referenced, not auto-closed; needs a live verification run post-merge). While it was under review, the deep review on this very PR failed the same way (session
is_error: trueon its first API call, 1 turn, $0 — yet the job stayed green and no review posted), so the scope grew to making every silent Claude-session failure loud.Changes
.github/workflows/ai-triage.yml.claude/commands/triage-*.mdfiles with the Read tool and follow them; slash invocation andSlashCommandexplicitly forbidden. The old prompt ("run the /triage-dedupe command") steered the session intoSlashCommand, which the allowlist denies → 3 denials → session gave up, action reported success.--disallowedTools "Edit,Write,MultiEdit,NotebookEdit,SlashCommand,WebFetch,WebSearch"(CodeRabbit finding):--allowedToolsonly auto-approves; this hard-blocks the mutating tools and SlashCommand outright.show_full_output: true: public job log shows the session transcript. Safe: the session's entire tool surface is public data (GET-onlyghallowlist, read-only file tools on a public checkout, no env access); GitHub secret-masking as backstop.Fail on silent session errorstep: red run when the session result hasis_error: true..github/workflows/claude-review.yml(deep review)show_full_output: true+is_errorgate on the deep-review step.Deep-review failure observed on this PR (2026-07-16, runs 29461943976 + debug re-run)
is_error: true,total_cost_usd: 0,permission_denials_count: 0→ the first API call itself failed (auth/limit-shaped, not a tool problem). Same model/args worked on PR test(bulma-ui): Storybook smoke gate — every story, light + dark, no console errors #315 yesterday ($1.47, 22 turns), so config is not the cause. Error text is hidden by the action's log sanitizer (debug re-run does not reveal it on the pinned version).claude-review.ymlmodified in the PR, the OIDC exchange refuses ("workflow file must match the default branch") and skips — also reported as success. The new gate tolerates that case (no session ran → no execution file).Verification (post-merge — both workflows validate/run from main)
deep-reviewon any open PR: either a real review posts, or the run is now red with the actual API error visible in the full session output.ai-triageon an open issue (e.g. [Refactor] Wire eslint-plugin-storybook (deferred from #267) #288): expect one marker-tagged triage comment andpermission_denials_count: 0; a session failure now turns the run red.Summary by CodeRabbit
.claude/commands/triage-*.mdfiles and the correct PR/issue flow.