chore(codebase-audit): tighten skill to prevent 2026-05-15 FP classes#1923
Merged
Conversation
Retires agent 14 (unused-python-exports, 100% FP rate from regex limits), rewrites agent 30 with operator-tunable criteria + 15-finding cap (was 70% FP rate), adds PEP-758 hard-stop directive to agent 88 (3 misflags as CRITICAL last run) and the validator prompt, requires multi-source grep evidence for agent 78 (--encrypt-secrets misflag), adds R-F (pre-write self-review) and R-G (validator code-quote) to FP-prevention rules, mandates 7-step verdict-application procedure for Phase 4 INDEX builder (synthesizer ignored FALSE_POSITIVE verdicts), requires path-existence verification in Top-20 (3 hallucinated paths last run), extends Phase 7 scratch-script sweep with 25+ documented leaks plus investigation.py never-touch rule, captures full 2026-05-15 lessons subsection.
Line 679 claimed agent 14 was 'Replaced by vulture in CI', but vulture is
not in pyproject.toml, .pre-commit-config.yaml, or any workflow. Line 3715
already framed the replacement as planned ('add to dev deps + pre-push
gate'). Aligning line 679 to match.
Pre-reviewed by 3 agents (tool-parity-checker, docs-consistency,
comment-quality-rot); 1 MAJOR finding addressed; 1 INFO observation
about working-tree investigation.py state surfaced separately
(file is untracked, no committed version to git-restore from).
Contributor
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (1)
📜 Recent review details⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
🧰 Additional context used🪛 LanguageTool.claude/skills/codebase-audit/SKILL.md[style] ~351-~351: To elevate your writing, try using an alternative expression here. (MATTERS_RELEVANT) 🔇 Additional comments (3)
WalkthroughThis PR updates the Full Codebase Audit CLAUDE skill prompt with comprehensive false-positive prevention and stricter validation mechanics. Changes include establishing PEP 758 Python 3.14 syntax awareness (where 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request significantly updates the codebase-audit skill documentation to incorporate lessons from a recent audit run, introducing a "HARD STOP" for false-positive Python 2 syntax findings, mandatory source code quoting for validators, and the retirement of Agent 14. It also adds a pre-write self-review process for agents and protects the investigation.py file from being overwritten. Review feedback suggested broadening the scratch script ban to include shell scripts, correcting over-escaped regex in a documentation example, removing invalid git restore instructions for untracked files, and replacing hardcoded user paths with environment-agnostic placeholders.
| @@ -169,15 +171,32 @@ Rules: | |||
| - Do NOT fix anything -- audit only | |||
| - Do NOT use Bash to write files -- use the Write tool | |||
| - **DO NOT write helper / analysis Python scripts to disk anywhere** (no `*.py` in | |||
Contributor
There was a problem hiding this comment.
The rule currently only explicitly bans Python scripts (*.py). However, the cleanup sweep on line 4183 includes audit_via_grep.sh, indicating that shell scripts are also being leaked by agents. Broadening the rule to include .sh files would help prevent all types of scratch scripts from polluting the workspace.
Suggested change
| - **DO NOT write helper / analysis Python scripts to disk anywhere** (no `*.py` in | |
| - **DO NOT write helper / analysis Python or Bash scripts to disk anywhere** (no `*.py` or `*.sh` in |
| 4. When composing INDEX entries: if a finding's key is in the FALSE_POSITIVE list, SKIP it (do not include at any severity). If in the INTENTIONAL list, include with `[INTENTIONAL]` prefix and do not count toward severity totals. | ||
| 5. **Final self-check**: before saving INDEX.md, scan the draft for the substrings "Python 2 syntax", "missing parens", "missing parentheses around exception", "PEP 2". If any appear, delete those rows -- they slipped past the validation purge. | ||
| 6. **Path verification**: for every Top-20 critical+high entry, run Read on the cited file. If the file does not exist, drop the entry; if the cited line is out of range, drop the entry. The 2026-05-15 synthesizer hallucinated `audit/chain_coordinator.py`, `docs/design/permission-model.md`, `docs/design/approval-flow.md` -- none existed in the repo. | ||
| 7. **Zero-finding agent list verification**: do NOT guess which agents had zero findings. Run `Bash: grep -L '\\*\\*Findings\\*\\*: [1-9]' _audit/latest/findings/*.md` to enumerate files where the `**Findings**` header is 0 or missing. Cross-reference with files that have substantive content -- some agents wrote prose summaries instead of strict-format headers, so a missing header does not always mean zero findings. |
Contributor
There was a problem hiding this comment.
The grep command uses double backslashes (\\*\\*) inside a single-quoted string. In a shell command, this will pass literal backslashes to grep, which will then look for literal backslashes in the file instead of literal asterisks. Since this is inside a markdown code span (backticks), single backslashes are sufficient and correct for grep (BRE) to match the literal ** characters.
Suggested change
| 7. **Zero-finding agent list verification**: do NOT guess which agents had zero findings. Run `Bash: grep -L '\\*\\*Findings\\*\\*: [1-9]' _audit/latest/findings/*.md` to enumerate files where the `**Findings**` header is 0 or missing. Cross-reference with files that have substantive content -- some agents wrote prose summaries instead of strict-format headers, so a missing header does not always mean zero findings. | |
| 7. **Zero-finding agent list verification**: do NOT guess which agents had zero findings. Run `Bash: grep -L '\*\*Findings\*\*: [1-9]' _audit/latest/findings/*.md` to enumerate files where the `**Findings**` header is 0 or missing. Cross-reference with files that have substantive content -- some agents wrote prose summaries instead of strict-format headers, so a missing header does not always mean zero findings. |
| rm -f find_missing_logging.py find_missing_logging_filtered.py parse_audit.py validate_config_examples.py audit_diff.py audit_parity.py check_docs.py check_rate_limits.py circular_dep_analyzer.py check_protocols.py debug_scanner.py detailed_check.py final_audit.py find_unwired.py test_regex.py validate_configs.py verify_final.py verify_protocols.py audit_exports.py audit_final.py audit_orphans.py audit_refined.py audit_via_grep.sh comprehensive_consumed_check.py extract_consumed_settings.py final_unwired_audit.py find_orphans.py investigate_unwired_settings.py run_orphan_check.py verify_all_unwired.py verify_settings_consumption.py audit_models.py audit_settings.py audit_api_docs.py check_immutability.py check_model_mutations.py pattern_finder.py || true | ||
| ``` | ||
|
|
||
| **NEVER include `investigation.py` in this sweep.** That file belongs to the user as their standing debug script (per memory rule `feedback_investigation_script.md`). If `investigation.py` is unexpectedly modified, prompt the user to `git restore investigation.py` -- do NOT delete it. |
Contributor
There was a problem hiding this comment.
The instruction to git restore investigation.py is currently invalid because the file is not tracked in git (as noted in the PR description's consistency check). git restore will fail for untracked files. The documentation should be updated to suggest a manual recovery path or the file should be committed as a template.
Suggested change
| **NEVER include `investigation.py` in this sweep.** That file belongs to the user as their standing debug script (per memory rule `feedback_investigation_script.md`). If `investigation.py` is unexpectedly modified, prompt the user to `git restore investigation.py` -- do NOT delete it. | |
| **NEVER include `investigation.py` in this sweep.** That file belongs to the user as their standing debug script (per memory rule `feedback_investigation_script.md`). If `investigation.py` is unexpectedly modified, prompt the user to restore investigation.py -- do NOT delete it. |
|
|
||
| - **Synthesizer hallucinated file paths** in Top-20: `audit/chain_coordinator.py`, `docs/design/permission-model.md`, `docs/design/approval-flow.md` did not exist in the repo. Fixed via path-verification step in Phase 4. | ||
|
|
||
| - **~25 scratch scripts leaked** to project root + `scripts/` + `c:\tmp\` + `C:\Users\Aurelio\tmp\` + `C:\Users\Aurelio\.claude\` despite Rule #10. Documented inventory added to Rule #10 + Phase 7 cleanup sweep extended. Recommendation: add a PreToolUse hook blocking Write of `*.py` files at project root + `scripts/` during audit runs (couldn't be added retroactively for this run). |
Contributor
There was a problem hiding this comment.
The lessons section hardcodes a specific user's home directory path (C:\Users\Aurelio\...). This should be generalized using the <name> placeholder established in the rules section (line 174) to maintain environment-agnostic documentation.
Suggested change
| - **~25 scratch scripts leaked** to project root + `scripts/` + `c:\tmp\` + `C:\Users\Aurelio\tmp\` + `C:\Users\Aurelio\.claude\` despite Rule #10. Documented inventory added to Rule #10 + Phase 7 cleanup sweep extended. Recommendation: add a PreToolUse hook blocking Write of `*.py` files at project root + `scripts/` during audit runs (couldn't be added retroactively for this run). | |
| - **~25 scratch scripts leaked** to project root + `scripts/` + `c:\tmp\` + `C:\Users\<name>\tmp\` + `C:\Users\<name>\.claude\` despite Rule #10. Documented inventory added to Rule #10 + Phase 7 cleanup sweep extended. Recommendation: add a PreToolUse hook blocking Write of `*.py` files at project root + `scripts/` during audit runs (couldn't be added retroactively for this run). |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.claude/skills/codebase-audit/SKILL.md:
- Around line 3811-3814: The fenced code block in SKILL.md is missing a
fence-adjacent blank line and a language tag (MD031/MD040); update the example
by adding a blank line before the opening ``` fence and a blank line after the
closing ``` fence and add a language identifier (e.g., use ```text) so the
snippet becomes markdownlint-compliant, and include the suggested “Verdict” line
after the fenced block as shown in the proposed fix to preserve intent.
- Line 3851: The grep pattern in the SKILL.md text is over-escaped: change the
doubled backslashes in the command string "grep -L '\\*\\*Findings\\*\\*: [1-9]'
_audit/latest/findings/*.md" to single backslashes so the pattern reads "grep -L
'\*\*Findings\*\*: [1-9]' _audit/latest/findings/*.md"; update the literal
command text in the file (the line containing that grep example) so grep can
correctly match the "**Findings**:" header.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 117194a1-e556-4677-936d-82dbb157aeaa
📒 Files selected for processing (1)
.claude/skills/codebase-audit/SKILL.md
📜 Review details
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
- GitHub Check: Analyze (javascript-typescript)
- GitHub Check: Analyze (python)
🧰 Additional context used
🪛 markdownlint-cli2 (0.22.1)
.claude/skills/codebase-audit/SKILL.md
[warning] 3811-3811: Fenced code blocks should be surrounded by blank lines
(MD031, blanks-around-fences)
[warning] 3814-3814: Fenced code blocks should be surrounded by blank lines
(MD031, blanks-around-fences)
[warning] 3814-3814: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
- Broaden scratch-script ban to include .sh (gemini, line 173) - Fix MD031/MD040 on inner fenced block in validator prompt (coderabbit, lines 3811-3814) - De-escape grep pattern (gemini + coderabbit, line 3851) - Replace git-restore reference for investigation.py with manual-restore prompt; file is untracked (gemini, line 4186) - Generalise hardcoded user paths to use the name placeholder (gemini, line 4257) - Update consistency reference in lessons section (line 4260) for the new investigation.py recovery instructions
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (2)
.claude/skills/codebase-audit/SKILL.md (2)
3851-3851:⚠️ Potential issue | 🟠 Major | ⚡ Quick winApply the grep pattern fix from past review comments.
The grep pattern uses
\\*\\*(double-escaped) which in grep BRE results in zero matches. A past review comment proposed changing to\*\*(single-escaped) to correctly match**Findings**:headers. This fix hasn't been applied yet.✅ Proposed fix (from past review)
-7. **Zero-finding agent list verification**: do NOT guess which agents had zero findings. Run `Bash: grep -L '\\*\\*Findings\\*\\*: [1-9]' _audit/latest/findings/*.md` to enumerate files where the `**Findings**` header is 0 or missing. Cross-reference with files that have substantive content -- some agents wrote prose summaries instead of strict-format headers, so a missing header does not always mean zero findings. +7. **Zero-finding agent list verification**: do NOT guess which agents had zero findings. Run `Bash: grep -L '\*\*Findings\*\*: [1-9]' _audit/latest/findings/*.md` to enumerate files where the `**Findings**` header is 0 or missing. Cross-reference with files that have substantive content -- some agents wrote prose summaries instead of strict-format headers, so a missing header does not always mean zero findings.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.claude/skills/codebase-audit/SKILL.md at line 3851, The grep pattern in .claude/skills/codebase-audit/SKILL.md is double-escaped as "\\*\\*" which fails to match the "**Findings**:" headers; update the pattern to use a single-escape version (i.e., change the "\\*\\*" pattern reference to the correctly escaped "\*\*" pattern) so grep/regex will find "**Findings**:" entries and the path verification logic will correctly detect nonexistent files cited by the audit.
3811-3816:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winApply the markdownlint fix from past review comments.
The fenced code block example is missing fence-adjacent blank lines and should use a language tag per MD031/MD040. A past review comment proposed this fix but it hasn't been applied yet.
✅ Proposed fix (from past review)
**Actual code**: -``` + +```text <file:line> <2-5 line quote>
Verdict: [reason tying quote to verdict]
</details> <details> <summary>🤖 Prompt for AI Agents</summary>Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.In @.claude/skills/codebase-audit/SKILL.md around lines 3811 - 3816, Update the
fenced code block in SKILL.md to comply with MD031/MD040 by ensuring there is a
blank line immediately before and after the triple-backtick fence and that the
fence includes a language tag (use "text" as shown in the proposed fix); locate
the example fenced block (the snippet containing "file:line" and "<2-5 line
quote>") and add the blank lines surrounding the ```text fence so the block is
fence-adjacent blank-line compliant.</details> </blockquote></details> </blockquote></details> <details> <summary>🤖 Prompt for all review comments with AI agents</summary>Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.Inline comments:
In @.claude/skills/codebase-audit/SKILL.md:
- Around line 901-902: Update the "Cap at 15 findings total per run" rule to
explicitly state the agent behavior: when the agent detects more than 15
candidate findings (i.e., the sentence "Cap at 15 findings total per run"), it
must stop emitting results, automatically narrow the search scope (define
narrowing as applying stricter filters or reducing target files/regex by X% or
by domain-specific filters) and re-run the scan until <=15 findings or a minimum
scope threshold is reached; if repeated narrowing still yields >15, emit the top
15 by confidence and include a note recommending manual scope refinement.
Include the explicit steps and stopping conditions in the SKILL.md text so the
rule is unambiguous.
Duplicate comments:
In @.claude/skills/codebase-audit/SKILL.md:
- Line 3851: The grep pattern in .claude/skills/codebase-audit/SKILL.md is
double-escaped as "\\" which fails to match the "Findings:" headers;
update the pattern to use a single-escape version (i.e., change the "\\"
pattern reference to the correctly escaped "**" pattern) so grep/regex will
find "Findings:" entries and the path verification logic will correctly
detect nonexistent files cited by the audit.- Around line 3811-3816: Update the fenced code block in SKILL.md to comply with
MD031/MD040 by ensuring there is a blank line immediately before and after the
triple-backtick fence and that the fence includes a language tag (use "text" as
shown in the proposed fix); locate the example fenced block (the snippet
containing "file:line" and "<2-5 line quote>") and add the blank lines
surrounding the ```text fence so the block is fence-adjacent blank-line
compliant.</details> <details> <summary>🪄 Autofix (Beta)</summary> Fix all unresolved CodeRabbit comments on this PR: - [ ] <!-- {"checkboxId": "4b0d0e0a-96d7-4f10-b296-3a18ea78f0b9"} --> Push a commit to this branch (recommended) - [ ] <!-- {"checkboxId": "ff5b1114-7d8c-49e6-8ac1-43f82af23a33"} --> Create a new PR with the fixes </details> --- <details> <summary>ℹ️ Review info</summary> <details> <summary>⚙️ Run configuration</summary> **Configuration used**: Repository UI **Review profile**: ASSERTIVE **Plan**: Pro **Run ID**: `8cb51820-0233-47a6-81c1-8a67e50b0d78` </details> <details> <summary>📥 Commits</summary> Reviewing files that changed from the base of the PR and between 2d0e9ab6ff27efcec1620905e1c29b709bc17d84 and daaaea953f1009c0e5ff7591ac2a2339849abc9a. </details> <details> <summary>📒 Files selected for processing (1)</summary> * `.claude/skills/codebase-audit/SKILL.md` </details> </details> <details> <summary>📜 Review details</summary> <details> <summary>⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)</summary> * GitHub Check: Analyze (python) * GitHub Check: Analyze (javascript-typescript) * GitHub Check: Analyze (go) </details> <details> <summary>🧰 Additional context used</summary> <details> <summary>🪛 markdownlint-cli2 (0.22.1)</summary> <details> <summary>.claude/skills/codebase-audit/SKILL.md</summary> [warning] 3816-3816: Heading style Expected: atx; Actual: setext (MD003, heading-style) </details> </details> </details> <details> <summary>🔇 Additional comments (8)</summary><blockquote> <details> <summary>.claude/skills/codebase-audit/SKILL.md (8)</summary><blockquote> `99-100`: LGTM! --- `173-199`: LGTM! --- `293-345`: LGTM! --- `677-693`: LGTM! --- `1674-1698`: LGTM! --- `1849-1866`: LGTM! --- `3912-3914`: LGTM! --- `4184-4187`: LGTM! </blockquote></details> </blockquote></details> </details> <!-- This is an auto-generated comment by CodeRabbit for review status -->
- Clarify findings-cap enforcement: narrow by adding exclusion criteria and re-evaluate from scratch (coderabbit, line 901-902) - Complete markdownlint MD031/MD003 fix on validator-prompt example: add missing blank line before opening fence and before horizontal rule (coderabbit, lines 3811/3819) Skipped: stale duplicate at line 3851 (grep pattern). Already fixed in round 1; current line 3852 reads single-escaped \*\* correctly.
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.claude/skills/codebase-audit/SKILL.md:
- Around line 4269-4270: The Phase 4 INDEX builder instruction currently
mandates a strict grep ("grep -cE \"^### (critical|high|medium|low|info)\"")
which conflicts with the lesson noting varied "finding-file" formats; update the
Phase 4 instruction (the text that references the grep at line 3845 / the "Each
entry's finding count..." sentence) to require a tolerant extraction: first
attempt the grep as a fast check, but if it returns 0 or seems inconsistent with
the file's "Finding" or "Summary" sections then parse the file's summary blocks
for findings (e.g., scan for prose headings like "Finding", "Severity", or
numbered/paragraph entries) and use that count; explicitly note the grep is a
fallback and callers must read each summary section when grep yields zero.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 4f94466a-17e8-444c-9f14-2816b527a83c
📒 Files selected for processing (1)
.claude/skills/codebase-audit/SKILL.md
📜 Review details
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
- GitHub Check: Analyze (python)
- GitHub Check: Analyze (javascript-typescript)
- GitHub Check: Analyze (go)
🔇 Additional comments (13)
.claude/skills/codebase-audit/SKILL.md (13)
99-100: LGTM!
173-199: LGTM!
293-345: LGTM!
677-693: LGTM!
3716-3716: LGTM!
855-903: LGTM!
1675-1699: LGTM!
1850-1865: LGTM!
3775-3827: LGTM!
3847-3856: LGTM!
3915-3917: LGTM!
4187-4190: LGTM!
4247-4268: LGTM!
- Reconcile Phase 4 INDEX-counting instruction with the lessons-section caveat about finding-file format variation: turn the strict grep into a two-step extraction (fast-path grep, tolerant fallback when grep returns 0) per the 2026-05-15 lesson (coderabbit, line 3845)
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.claude/skills/codebase-audit/SKILL.md:
- Around line 677-680: Update all references that assume 159 active agents after
marking "Agent 14" (slot 14) retired: adjust the active-agent totals,
orchestration text, progress targets, and any launch-order or scheduling
language in the "Wave 3: Dead Code & Unused" section and the "Retired Agents"
table so counts and targets reflect one fewer agent; specifically search for the
string "159 active agents", "Agent 14", "slot 14", and the Wave 3 paragraph
around lines shown in the diff and change totals/ordering semantics and any
example schedules that rely on the old count so that progress targets and launch
order logic remain consistent with the new active-agent total.
- Line 3858: The grep command used to enumerate zero-finding agents is including
non-agent artifacts (e.g., files like validate-batch-*.md) under the
_audit/latest/findings/*.md glob, producing false positives; update the
enumeration to exclude non-agent markdown files by filtering the glob or using a
find/grep pipeline that omits known artifact patterns (for example exclude
validate-batch-*.md and any other non-agent filenames) so that only true agent
files are checked for the "**Findings**: [1-9]" header; reference the existing
grep usage on `_audit/latest/findings/*.md` and ensure the exclusion applies to
that pattern before cross-referencing with agent content.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 2e071f24-3b3e-4326-b296-f3356c02d9bc
📒 Files selected for processing (1)
.claude/skills/codebase-audit/SKILL.md
📜 Review details
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
- GitHub Check: Analyze (python)
- GitHub Check: Analyze (javascript-typescript)
- GitHub Check: Analyze (go)
🔇 Additional comments (1)
.claude/skills/codebase-audit/SKILL.md (1)
99-100: LGTM!Also applies to: 173-199, 293-345, 855-904, 1675-1696, 1850-1864, 3775-3827, 3919-3921, 4190-4194, 4250-4273
- Restrict zero-finding grep glob to numbered-slot agent files so validate-batch-*.md and other synthesis artifacts no longer inflate the zero-finding list (coderabbit, line 3858) - Update active-agent total from 159 to 158 in description, key principles, scope table, streaming-pool text, progress reporting, validation phase, JSON schema example, and golden-test bootstrap text to reflect that slot 14 is retired (coderabbit, outside-diff-range; lines 2, 9, 28, 349, 355, 357, 3741, 4090, 4177)
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.claude/skills/codebase-audit/SKILL.md:
- Line 29: The table row containing the scope for `src/` still includes the
range "09-15" which reintroduces retired slot 14; update that cell so the agent
range excludes slot 14 (e.g., change "09-15" to two ranges "09-13, 15" or
explicitly remove "14") in the entry that reads "`src/` | `src/synthorg/`,
`tests/`, `web/src/types/`, `docs/design/` | 01-06, 09-15, 16-34, ...", ensuring
the contract "158 active agents / slot 14 retired" is honored.
- Line 3858: The grep glob `[0-9][0-9]-*.md` only matches exactly two-digit
agent files and omits 100–159; update the pattern to include multi-digit IDs
(e.g., replace `[0-9][0-9]-*.md` with `[0-9][0-9]*-*.md` in the command `grep -L
'\*\*Findings\*\*: [1-9]' _audit/latest/findings/[0-9][0-9]-*.md`) so files with
three-digit prefixes are included; alternatively, switch to a regex-based grep
(e.g., `grep -P -L '\*\*Findings\*\*: [1-9]'
_audit/latest/findings/\d{2,}-.*\.md`) to explicitly match two-or-more digit
IDs.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: ef2ec22f-ae03-4710-8a65-97041214d1da
📒 Files selected for processing (1)
.claude/skills/codebase-audit/SKILL.md
📜 Review details
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
- GitHub Check: Analyze (javascript-typescript)
- GitHub Check: Analyze (python)
- GitHub Check: Analyze (go)
🧰 Additional context used
🪛 LanguageTool
.claude/skills/codebase-audit/SKILL.md
[style] ~351-~351: To elevate your writing, try using an alternative expression here.
Context: ...s its own file), so order-of-completion does not matter -- only that the pool stays saturated u...
(MATTERS_RELEVANT)
- Drop retired slot 14 from the src/ scope agent range: split '09-15' into '09-13, 15' with an explicit 'slot 14 retired' annotation so the row honours the 158-active-agent contract (coderabbit, line 29) - Widen zero-finding glob from [0-9][0-9]-*.md to [0-9][0-9]*-*.md so agent files with 3-digit IDs (100-159) are included; explain the two-or-more digit semantics inline (coderabbit, line 3858)
Aureliolo
pushed a commit
that referenced
this pull request
May 17, 2026
<!-- HIGHLIGHTS_START --> ## Highlights > _AI-generated summary (model: `openai/gpt-4.1-mini` via GitHub Models). Commit-based changelog below._ ### What you'll notice - Frontend WP-6 update with UX polish improves user interface and workflow. - Dashboard and training endpoint improvements enhance observability and dispatch behavior. - Web storybook now supports change detection for more responsive UI interactions. - Git hooks now isolated per worktree for cleaner repository management. - Providers automatically detect native streaming support in Litellm models. ### What's new - Added a new pipeline to convert Pydantic DTOs to TypeScript for better front-end compatibility. ### Under the hood - Refactored settings to three precedence categories, removing YAML tier for simpler configuration. - Completed RootConfig mirror coverage for enhanced configuration consistency. - Adopted API conventions with better query performance and forbidden extra fields for stricter validation. - Improved persistence, layer discipline, and restart safety in core work packages. - CI updated with split test jobs and tightened coverage gates for better test quality. - Switched to direct Trivy binary for security scans, removing previous Trivy action dependency. - Enhanced memory management with per-call processing options and better observability during speech-to-text encoding. - Various dependency updates for Python, infrastructure, and lock files maintain security and stability. - Removed TypeScript DTO type-tightening overlays to simplify type management. - Codebase audit tightened skill sets to prevent false positivity in class detection by 2026. <!-- HIGHLIGHTS_END --> :robot: I have created a release *beep* *boop* --- ## [0.8.5](v0.8.4...v0.8.5) (2026-05-17) ### Features * **codegen:** pydantic-to-typescript DTO pipeline + parity gate (closes [#1889](#1889)) ([#1909](#1909)) ([0265ef5](0265ef5)) * **storybook:** enable changeDetection + trim web/CLAUDE.md ([#1939](#1939)) ([3b1f4c0](3b1f4c0)) * **web,setup:** WP-6 frontend + UX polish ([#1941](#1941)) ([d9ca76d](d9ca76d)) ### Bug Fixes * correct invalid git for-each-ref syntax in post-merge-cleanup skill ([#1946](#1946)) ([69a1649](69a1649)) * dashboard polish, training endpoint dispatch, and observability cleanup ([#1911](#1911)) ([b61e9e8](b61e9e8)) * per-worktree git-hook isolation + hookify gate migration + MSW drift fix ([#1949](#1949)) ([e3f8495](e3f8495)) * **providers:** read supports_native_streaming from litellm model info ([#1942](#1942)) ([60364ca](60364ca)) * security and audit coverage (closes [#1883](#1883)) ([#1904](#1904)) ([d8ebf55](d8ebf55)) ### Performance * **ci:** mypy --num-workers=4 + enable ruff TID255 ([#1944](#1944)) ([484c1d3](484c1d3)) ### Refactoring * **ci:** drop aquasecurity/trivy-action, use direct trivy binary ([#1940](#1940)) ([df1f946](df1f946)) * **memory:** per-call processing_kwargs + observability for ST encode ([#1943](#1943)) ([3aa9d20](3aa9d20)) * Phase 7 follow-up — complete RootConfig mirror coverage (closes [#1907](#1907)) ([#1914](#1914)) ([605500b](605500b)) * **settings:** collapse precedence to three categories; drop YAML tier (closes [#1890](#1890)) ([#1910](#1910)) ([efd54c9](efd54c9)) * WP-3 API conventions + query performance + project-wide extra=forbid ([#1953](#1953)) ([504d579](504d579)), closes [#1918](#1918) * WP-4 settings + cross-cutting (clock seam, contextvars, dispatch, plugin surfaces) ([#1954](#1954)) ([7207d92](7207d92)) * **wp1:** persistence + layer discipline + restart safety ([#1945](#1945)) ([57586fb](57586fb)) ### Documentation * **wp5:** public-facing truth refresh ([#1924](#1924)) ([afb5cc5](afb5cc5)) ### CI/CD * split test job by marker with airtight aggregate coverage gate ([#1948](#1948)) ([0b818d5](0b818d5)), closes [#1938](#1938) [#1937](#1937) ### Maintenance * **codebase-audit:** tighten skill to prevent 2026-05-15 FP classes ([#1923](#1923)) ([9317ed1](9317ed1)) * Lock file maintenance ([#1913](#1913)) ([c08a355](c08a355)) * Lock file maintenance ([#1950](#1950)) ([8940ab1](8940ab1)) * remove TS DTO type-tightening overlays ([#1915](#1915)) ([d296214](d296214)), closes [#1906](#1906) * Update Infrastructure dependencies ([#1928](#1928)) ([d19fae5](d19fae5)) * Update Python dependencies ([#1929](#1929)) ([75cc2c8](75cc2c8)) * **wp7:** hygiene, stubs, test/CI/tooling, doc gaps, boundary patterns doc ([#1926](#1926)) ([c29eb32](c29eb32)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: synthorg-repo-bot[bot] <279117679+synthorg-repo-bot[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Hardens the
codebase-auditskill against the false-positive classes observed in the 2026-05-15 audit run. Pure documentation change to.claude/skills/codebase-audit/SKILL.md(+250 / -21).What changed
except A, B:findings as CRITICAL despite an existing rule; all were FALSE_POSITIVE per PEP 758.unused-python-exports) retired: regex cannot see Litestar DI, factory returns,__all__re-exports, isinstance /typing.Protocolstructural usage, or test fixture return types. 2026-05-15 run produced 84 findings, 100% FP. Prompt preserved in<!-- ... -->for reference. Replacement (vulture+ pre-push gate) is planned but not yet wired -- the retirement stands on its own merit because regex-based detection is unsalvageable.missing-settings-bridge) rewritten with operator-tunable-knob criteria, an exclusion list (shutdown drain, protocol-boundary values, JWT/audit retention, internal backoff bases), a 15-finding cap, and a verification requirement. 2026-05-15 FP rate ~70%.cli-reference-drift) requires multi-source grep evidence (docs/reference/,cli/CLAUDE.md,cli/README.md,docs/guides/) before flagging a flag as undocumented. 2026-05-15 misflagged--encrypt-secretsafter only greppingdocs/reference/.investigation.pyno-touch rule added withgit restore(not delete) safeguard.Pre-PR review
3 agents run:
tool-parity-checker.opencode/commands/codebase-audit.mdincludes@.claude/skills/codebase-audit/SKILL.mdverbatim; no parity drift.docs-consistencycomment-quality-rotFindings addressed
docs-consistency):SKILL.md:679claimed agent 14 was "Replaced byvulturein CI", whileSKILL.md:3715and reality both said the replacement is planned. Verifiedvultureis not inpyproject.toml,.pre-commit-config.yaml, or.github/workflows/. Aligned line 679 to the planned framing in a follow-up commit.Findings surfaced but outside PR scope
docs-consistency): the repo's working-treeinvestigation.pycurrently contains audit-script content (it was overwritten during the 2026-05-15 run -- the very incident this PR captures as a lesson). The new SKILL.md rule says togit restore investigation.pyto recover, but the file has never been tracked in git on any branch, so there is no committed version to restore from. Recommend a follow-up: either (a) commit a baseline / templateinvestigation.pyso thegit restorerecovery path actually works, or (b) update the SKILL.md text to reflect that the user maintains their own backup.Test plan
Review coverage note
3 agents run; 1 MAJOR finding fixed in-PR; 1 INFO surfaced for follow-up; 0 deferred. External-reviewer triage via
/aurelio-review-prafter CI green.