Repository navigation
test(judge): close accepted-type and ownership gaps #778
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 3 commits
Commits
Show all changes
38 commits
Select commit
Hold shift + click to select a range
ccedd00
test(judge): close accepted-type and ownership gaps
0721e55
feat: add cumulative threshold judge mode
79a8e83
fix: harden llm judge category bounds
8d89295
fix: normalize judge method validation
5c0dcd3
docs: distinguish proposed binding registry
11291fd
fix: harden judge result boundaries
seonghobae 50f3a3d
test(docs): pin integrated maturity states
seonghobae c54706c
docs: align maturity with protected main
seonghobae 9ebc322
docs: record inconclusive Strix evidence
seonghobae 6c42d4a
fix: reject unhashable criterion identifiers
seonghobae d938704
docs: record current Strix evidence boundary
seonghobae d1d262b
docs: bind Strix evidence and usage contract
seonghobae 54baec2
docs: use canonical maturity vocabulary
seonghobae 8741eb1
test(judge): reject criterion description subclasses
seonghobae 5697479
fix(judge): reject criterion description subclasses
seonghobae f04903e
test(judge): reject trace container subclasses
seonghobae 3eecfaf
docs: record central workflow evidence boundary
seonghobae e41afa4
Merge remote-tracking branch 'origin/codex/fast-judge-accepted-type-r…
seonghobae 630946c
fix(judge): isolate trace metadata hooks
seonghobae 4f8c052
docs: record workflow contract reachability finding
seonghobae 67d3cee
docs: record live gateway judge method sensitivity
seonghobae 22f9b02
docs(adr): record local judge queue saturation
seonghobae 2c4113a
docs(adr): record network transport boundary
seonghobae 35f6aab
fix(jmle): recognize objective convergence and add recovery evidence …
seonghobae e7ce079
docs(adr): record trusted coverage transport blocker
seonghobae 4e72fef
Merge remote-tracking branch 'origin/main' into codex/fast-judge-acce…
seonghobae e612550
test(jmle): exercise objective convergence guard
seonghobae 0fb9e46
Merge branch 'main' into codex/fast-judge-accepted-type-regression
opencode-agent[bot] 392e68f
docs(adr): record trusted Strix outage evidence
seonghobae e6e9221
Merge branch 'main' into codex/fast-judge-accepted-type-regression
opencode-agent[bot] 98849d4
docs(adr): preserve trusted Strix outage evidence
seonghobae 503f6b5
docs(research): preserve permissive OA judge-bias papers
seonghobae 1918cb1
merge main into judge and Rust-ownership correction line after #800
seonghobae eeaef39
merge main into judge correction line after #802
seonghobae 1a30f0e
Merge branch 'main' into codex/fast-judge-accepted-type-regression
opencode-agent[bot] 24e4f11
Merge branch 'main' into codex/fast-judge-accepted-type-regression
opencode-agent[bot] e2ea229
merge: sync with main
seonghobae b7b519d
Merge branch 'main' into codex/fast-judge-accepted-type-regression
seonghobae File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,159 @@ | ||
| # ADR-0014: Bounded LLM-judge category inputs and security-scan evidence | ||
|
|
||
| Status: **Proposed** | ||
| Date: 2026-08-12 | ||
| Supersedes: none | ||
| Superseded by: none | ||
|
|
||
| ## Context | ||
|
|
||
| The LLM-as-a-Judge adapter accepts category counts and category values at a | ||
| public Python boundary before projecting results into dichotomous or | ||
| polytomous IRT items. The current-head Strix run for PR #778 at | ||
| `0721e55ce5d889b7917aa7da2891367adf3430dc` reported a supposed integer | ||
| overflow in this path. Python's built-in integers do not wrap, so that claim | ||
| was not independently valid. The review did, however, expose a real adjacent | ||
| boundary issue: `isinstance(value, int)` admits an `int` subclass whose | ||
| comparison methods can return forged results, allowing the category-count | ||
| bound to be bypassed before a resource-amplifying operation. | ||
|
|
||
| The same trust-boundary concern applies to category values accepted by the | ||
| deterministic IRT projection. A model response parsed by `json.loads` contains | ||
| built-in scalar types, but callers can construct `LLMJudgeResult` directly. | ||
| Security evidence must therefore distinguish a real code defect from a model | ||
| or provider failure and must never weaken the fail-closed gate. | ||
|
|
||
| ## Decision drivers | ||
|
|
||
| - Reject untrusted or adversarial runtime scalar objects before comparisons, | ||
| arithmetic, or allocations. | ||
| - Preserve the existing dichotomous/polytomous response-matrix contract. | ||
| - Keep LLM security findings advisory until independently reproduced by code, | ||
| tests, or a structured scanner artifact. | ||
| - Keep provider rate limits separate from source-security conclusions. | ||
|
|
||
| ## Ownership and dependency direction | ||
|
|
||
| `fast-mlsirm` owns the judge result contract and its IRT projection. LLM | ||
| transport remains injected through `contextual-orchestrator`; neither a model | ||
| response nor a Strix narrative becomes numerical or merge authority. | ||
|
|
||
| ## Decision | ||
|
|
||
| 1. `category_count` accepts only an exact built-in Python `int` in | ||
| `2..MAX_JUDGE_CATEGORIES` (currently 64). Booleans, floats, subclasses, | ||
| negative values, and oversized integers fail before any derived list or | ||
| threshold array is materialized. | ||
| 2. Direct category values accept only exact built-in `int` or `float` values; | ||
| booleans, subclasses, non-finite values, fractional values, and out-of-range | ||
| values fail closed. | ||
| 3. Any cumulative-threshold result remains bounded by the validated category | ||
| count, and every IRT row still requires multiple criterion items and passes | ||
| the shared dichotomous/polytomous response-matrix validator. | ||
| 4. A Strix/provider failure is recorded with its exact head SHA, run URL, | ||
| failed step, backend error, and structured-report availability. A model-only | ||
| claim is not called a confirmed vulnerability without independent evidence. | ||
| 5. Required checks remain fail-closed. No scan failure is bypassed, and no | ||
| self-approval or protected-branch merge is manufactured. | ||
|
coderabbitai[bot] marked this conversation as resolved.
Resolved
|
||
|
|
||
| ## Invariants / acceptance evidence | ||
|
|
||
| 1. `tests/test_llm_judge.py::test_category_count_and_category_values_reject_runtime_subclasses` | ||
| rejects booleans, floats, values above 64, astronomically large integers, | ||
| and comparison-forging `int` subclasses. | ||
| 2. `judge()` and `LLMJudgeResult.to_irt_row()` share the same bounded category | ||
| validation before cumulative threshold allocation or IRT projection. | ||
| 3. PR #778's Strix log records an initial NVIDIA NIM 429, a fallback-model | ||
| report, no structured vulnerability artifact, and the exact finding. The | ||
| finding is treated as an adjacent hardening signal, not as proof of integer | ||
| wraparound. | ||
| 4. The full Python and Rust test suites, targeted Ruff checks, and exact-head | ||
| required checks must pass before this hardening is considered integrated. | ||
|
|
||
| ## Non-goals and claims not made | ||
|
|
||
| - This decision does not claim that Python integers overflow or that a single | ||
| model-generated security report proves a CVSS score. | ||
| - Exact scalar typing does not establish that LLM judgments are unbiased. | ||
| - This decision does not authorize keyword matching, direct provider calls, or | ||
| a single scalar response as an IRT dataset. | ||
|
|
||
| ## Consequences and trade-offs | ||
|
|
||
| ### Benefits | ||
|
|
||
| - Resource-amplifying category bounds cannot be bypassed through comparison | ||
| hooks or coercion protocols. | ||
| - Security and provider-availability evidence remains auditable and separate. | ||
| - The public judge path stays deterministic and compatible with the existing | ||
| multi-item IRT contract. | ||
|
|
||
| ### Costs / risks | ||
|
|
||
| - Legitimate custom numeric subclasses are rejected and must be normalized by | ||
| the caller before crossing the boundary. | ||
| - Strix runs remain dependent on external model quotas and can require a | ||
| rerun after a provider outage. | ||
|
|
||
| ## Alternatives considered | ||
|
|
||
| ### Coerce every value with `int()` or `float()` | ||
|
|
||
| Rejected because conversion hooks can execute caller code and can silently | ||
| change values at a security and measurement boundary. | ||
|
|
||
| ### Trust the Strix narrative as the vulnerability oracle | ||
|
|
||
| Rejected because the observed run lacked a structured report, began with a | ||
| provider 429, and made a false claim about Python integer wraparound. | ||
|
|
||
| ### Ignore the finding because the primary claim was false | ||
|
|
||
| Rejected because the same code admitted adversarial `int` subclasses that could | ||
| forge comparisons. The boundary is now hardened and regression-tested. | ||
|
|
||
| ## Failure, degraded, and recovery behavior | ||
|
|
||
| Malformed category inputs raise the package-owned `ValueError` or | ||
| `JudgeFormatError` before model output is accepted or an IRT row is returned. | ||
| Malformed model JSON, missing structured scan evidence, provider rate limits, | ||
| and failed required checks remain fail-closed. After a code change, rerun the | ||
| security workflow against the exact pushed head and record the new result; | ||
| never reinterpret a stale run as evidence for a new head. | ||
|
|
||
| ## Security and privacy implications | ||
|
|
||
| The change reduces coercion and resource-exhaustion risk without adding | ||
| credentials or retaining source text. Strix logs and provider errors may be | ||
| retained as CI evidence, but secrets must remain masked. The exact head SHA | ||
| and run URL are required for reproducibility; they do not grant merge | ||
| authority. | ||
|
|
||
| ## Compatibility, migration, and rollback | ||
|
|
||
| The accepted JSON schema and valid category range are unchanged. Callers that | ||
| used custom numeric subclasses must pass built-in scalars. Rollback is allowed | ||
| only with a superseding ADR and equivalent boundary tests; reverting to | ||
| coercive validation is not an acceptable compatibility path. | ||
|
|
||
| ## Verification and release evidence | ||
|
|
||
| - Run the targeted judge tests and Ruff on changed Python files. | ||
| - Run the full Python suite, Rust workspace tests, and binding tests. | ||
| - Re-run Strix and all required checks on the exact pushed PR head. | ||
| - Review current inline/general PR feedback and obtain an independent approval | ||
| before a protected merge. | ||
| - Preserve contextual-orchestrator routing for every LLM-as-a-Judge call. | ||
|
|
||
| ## Follow-ups | ||
|
|
||
| - Re-run PR #778's Strix workflow after this hardening and compare the exact | ||
| head SHA, provider status, structured artifact, and finding intersection. | ||
| - If a later scan reports a reproducible issue, create a superseding ADR or | ||
| append a new evidence record with the reproducer and regression test. | ||
|
|
||
| ## Reversal / supersession conditions | ||
|
|
||
| Supersede this ADR only if the public judge contract adopts a different | ||
| bounded scalar representation or if structured security evidence demonstrates | ||
| that the current exact-type boundary is insufficient. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.