feat: add repository context and judge model features

[GangGreenTemperTatum]

big fan, love the research and thank you for the awesome inspiration @spaceraccoon 🙏 i wanted to try tackle this problem after some self-experience with it and in particularly quoting:

However, there was still a lot of false positives, including “bugs-but-not-really-exploitable-vulnerabilities”. This time, I tuned the context further:

Problem

reduces false positives in vulnerability detection. currently claude only sees the diff (max 15KB) without understanding what the code does, leading to potential misclassifications (i've had a few)

Solution

adds two opt-in features to improve detection accuracy (below)

Features:

• repository context: fetches up to 3 modified files (pre-patch) from parent commit to give claude understanding of what the code does, not just the diff
• judge model: second claude call that reviews primary detections and can reject them. only runs when primary analysis flags a vulnerability, saving api calls

Improvements:

• better detection accuracy through additional code context
• reduced false positives via adversarial review pattern
• graceful degradation on errors (missing files, judge failures)

Before & After Screenshots

test workflow:

Tests

• 24 unit tests pass (6 new tests added for new functions)
• typescript type checking passes
• new e2e test workflow (.github/workflows/test-action.yml) runs on PRs
• tests both features enabled and disabled for baseline comparison
• workflow analyzes last 3 commits from expressjs/express as integration test

to run tests locally:

npm test              # unit tests
npm run typecheck     # type checking
npm run build         # build dist/

Deploy Notes

new action inputs:

• enable-repo-context : (optional, default: false) fetch modified files for context (max 3 files, 3KB each)
• enable-judge : (optional, default: false) review detections with second model to reduce false positives
• judge-model : (optional, default: same as primary model) claude model to use for judge

[Copilot]

Pull request overview

Adds optional “repo context” and a “judge” pass to improve vulnerability-patch detection accuracy for this GitHub Action by giving the LLM more pre-patch code and enabling a second review step to reduce false positives.

Changes:

• Add new action inputs for repository context fetching and a judge model pass.
• Fetch up to 3 modified files’ contents (pre-patch) and inject into the analysis prompt.
• Add judgeAnalysis to validate positive detections and skip issue creation if the judge disagrees.

Reviewed changes

Copilot reviewed 9 out of 15 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/types.ts	Extends ActionInputs with repo-context/judge flags and judge model.
src/index.ts	Wires new inputs, fetches repo context, and conditionally runs judge before creating issues.
src/github.ts	Implements parsing modified paths from diff and fetching file content for context.
src/analyzer.ts	Adds repoContext to the analysis prompt and introduces judgeAnalysis.
src/__tests__/github.test.ts	Adds unit tests for getModifiedFilesContent.
src/__tests__/analyzer.test.ts	Adds unit tests for judgeAnalysis.
action.yml	Defines new action inputs and defaults.
README.md	Documents the new inputs and usage example.
.github/workflows/test-action.yml	Adds an e2e workflow that runs the action with features on/off.
.gitignore	Ignores CLAUDE.md.
dist/index.js	Built output reflecting new features.
dist/types.d.ts	Built type output for new inputs.
dist/github.d.ts	Built type output for getModifiedFilesContent.
dist/analyzer.d.ts	Built type output for judgeAnalysis and updated analyzeCommit signature.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 15 changed files in this pull request and generated 12 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 16 changed files in this pull request and generated 4 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 14 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 14 changed files in this pull request and generated 6 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 14 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[spaceraccoon]

Thanks for your contributions @GangGreenTemperTatum ! This looks amazing - I agree that full-file context will be helpful looking at how the OSS-Fuzz team also feeds this as part of their context. I'll target to finish my review over the Chinese New Year break and bring this in as soon as possible.

[GangGreenTemperTatum]

Thanks for your contributions @GangGreenTemperTatum ! This looks amazing - I agree that full-file context will be ..

Thanks so much @spaceraccoon ! Appreciate the kind words, of course no problem or rush at all! Again, thank you for the awesome work and inspiration, I'm heavily following your work and using this project, it's awesome.

Happy new year!! :) Enjoy the celebrations. Let me know if there's anything I can help with and have a great day!

[spaceraccoon]

Thanks for your patience @GangGreenTemperTatum ! I've added some comments, main points:

• Stricter parsing of judge output to a fixed type we define earlier, similar to VulnerabilityAnalysis (use a JudgeAnalysis type?)
• Handle edge case for spaces in git diff file paths
• Use type: boolean for the new Action inputs rather than string, so we can save on the need to cast them to boolean manually.

I'm happy to merge these in a separate working branch and update these myself as well if you want!

[GangGreenTemperTatum]

hey @spaceraccoon , thanks for being awesome as always!

made some changes in chore: pr feedback:

• stricter judge parsing: added a JudgeAnalysis type in types.ts and simplified judgeAnalysis() to parse directly to it using the same JSON.parse("{" + content.text) pattern as analyzeCommit(). on parse failure it returns agrees: true so it still fails gracefully (skips filtering).
• boolean inputs: changed both enable-repo-context and enable-judge to type: boolean with default: false in action.yml, and switched to core.getBooleanInput() in index.ts to avoid manual string casting.
• quoted paths in git diff: updated the extractModifiedPaths regex to handle both quoted ("a/path with spaces") and unquoted paths, with a test covering the quoted case.
• readme: updated defaults to false and changed example values from strings to bare booleans.

i hope this looks good, let me know if there's anything i can help with and i appreciate your time - it's been great working with you on this!

[spaceraccoon]

Thanks @GangGreenTemperTatum! Will merge and push as a new minor version release.

[spaceraccoon]

Closing and re-opening to retry CodeQL ref: https://github.com/orgs/community/discussions/159026#discussioncomment-13122154

[spaceraccoon]

Thanks a ton @GangGreenTemperTatum! I've got it running at https://github.com/spaceraccoon/vulnerability-spoiler-alert and will monitor for issues. I noticed a weird bug at https://github.com/spaceraccoon/vulnerability-spoiler-alert/actions/runs/22147833025/job/64030138126 (404-ed fetching one file, although the rest were fine), but will get to it later.

[GangGreenTemperTatum]

Thanks a ton @GangGreenTemperTatum! I've got it running at https://github.com/spaceraccoon/vulnerability-spoiler-alert and will monitor for issues. I noticed a weird bug at https://github.com/spaceraccoon/vulnerability-spoiler-alert/actions/runs/22147833025/job/64030138126 (404-ed fetching one file, although the rest were fine), but will get to it later.

absolutely @spaceraccoon , thank you!! yes, i am also watching the RSS on that so will let you know if theres anything funky I will let you know! interesting, i did some digging and can see the commit exists in nodejs/node.:

• Hash: d7a755153a1f3dd071d2ae8af8f577edbd717ad8
• Message: doc: remove incorrect mention of module in typescript.md
• Date: 2026-02-18
• PR: docs: Remove incorrect mention of module nodejs/node#61839

The 404is because the file tools/dep_updaters/update-test426-fixtures.sh doesn't exist at that commit — not because the commit itself is missing. That commit only touches docs (typescript.md), so it wouldn't have that shell script at that pat which i think makes sense and im noodling on it atm

[GangGreenTemperTatum]

hey @spaceraccoon , sorry for the additional ping - whilst writing the above i had an outstanding notification for spaceraccoon/vulnerability-spoiler-alert#32 - did you push up to the live site prior to that? (im thinking this will help us with the above thesis)

[spaceraccoon]

hey @spaceraccoon , sorry for the additional ping - whilst writing the above i had an outstanding notification for spaceraccoon/vulnerability-spoiler-alert#32 - did you push up to the live site prior to that? (im thinking this will help us with the above thesis)

Yep! I’ve already enabled the new context + judge, as well as the fix for the 404 problem before this issue finding