Skip to content
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
ccedd00
test(judge): close accepted-type and ownership gaps
Aug 11, 2026
0721e55
feat: add cumulative threshold judge mode
Aug 11, 2026
79a8e83
fix: harden llm judge category bounds
Aug 11, 2026
8d89295
fix: normalize judge method validation
Aug 11, 2026
5c0dcd3
docs: distinguish proposed binding registry
Aug 11, 2026
11291fd
fix: harden judge result boundaries
seonghobae Aug 11, 2026
50f3a3d
test(docs): pin integrated maturity states
seonghobae Aug 12, 2026
c54706c
docs: align maturity with protected main
seonghobae Aug 12, 2026
9ebc322
docs: record inconclusive Strix evidence
seonghobae Aug 12, 2026
6c42d4a
fix: reject unhashable criterion identifiers
seonghobae Aug 12, 2026
d938704
docs: record current Strix evidence boundary
seonghobae Aug 12, 2026
d1d262b
docs: bind Strix evidence and usage contract
seonghobae Aug 12, 2026
54baec2
docs: use canonical maturity vocabulary
seonghobae Aug 12, 2026
8741eb1
test(judge): reject criterion description subclasses
seonghobae Aug 12, 2026
5697479
fix(judge): reject criterion description subclasses
seonghobae Aug 12, 2026
f04903e
test(judge): reject trace container subclasses
seonghobae Aug 12, 2026
3eecfaf
docs: record central workflow evidence boundary
seonghobae Aug 12, 2026
e41afa4
Merge remote-tracking branch 'origin/codex/fast-judge-accepted-type-r…
seonghobae Aug 12, 2026
630946c
fix(judge): isolate trace metadata hooks
seonghobae Aug 12, 2026
4f8c052
docs: record workflow contract reachability finding
seonghobae Aug 12, 2026
67d3cee
docs: record live gateway judge method sensitivity
seonghobae Aug 12, 2026
22f9b02
docs(adr): record local judge queue saturation
seonghobae Aug 12, 2026
2c4113a
docs(adr): record network transport boundary
seonghobae Aug 12, 2026
35f6aab
fix(jmle): recognize objective convergence and add recovery evidence …
seonghobae Aug 12, 2026
e7ce079
docs(adr): record trusted coverage transport blocker
seonghobae Aug 12, 2026
4e72fef
Merge remote-tracking branch 'origin/main' into codex/fast-judge-acce…
seonghobae Aug 12, 2026
e612550
test(jmle): exercise objective convergence guard
seonghobae Aug 12, 2026
0fb9e46
Merge branch 'main' into codex/fast-judge-accepted-type-regression
opencode-agent[bot] Aug 12, 2026
392e68f
docs(adr): record trusted Strix outage evidence
seonghobae Aug 12, 2026
e6e9221
Merge branch 'main' into codex/fast-judge-accepted-type-regression
opencode-agent[bot] Aug 12, 2026
98849d4
docs(adr): preserve trusted Strix outage evidence
seonghobae Aug 12, 2026
503f6b5
docs(research): preserve permissive OA judge-bias papers
seonghobae Aug 12, 2026
1918cb1
merge main into judge and Rust-ownership correction line after #800
seonghobae Aug 12, 2026
eeaef39
merge main into judge correction line after #802
seonghobae Aug 12, 2026
1a30f0e
Merge branch 'main' into codex/fast-judge-accepted-type-regression
opencode-agent[bot] Aug 12, 2026
24e4f11
Merge branch 'main' into codex/fast-judge-accepted-type-regression
opencode-agent[bot] Aug 12, 2026
e2ea229
merge: sync with main
seonghobae Aug 13, 2026
b7b519d
Merge branch 'main' into codex/fast-judge-accepted-type-regression
seonghobae Aug 13, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 10 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,14 +93,17 @@ print(fixed_item_calibration.best)
strict structured parsing. A judge result becomes an IRT row only through
LLMJudgeResult.to_irt_row() with at least two criteria, followed by
validate_irt_response_matrix() for a multi-item dichotomous or explicitly
categorized polytomous matrix. Equal-width score projection is experimental;
category-count and prompt-perturbation calibration are required. See
[ADR 0005](https://github.com/ContextualWisdomLab/contextual-orchestrator/blob/89bd5bf73319dd21f2be1f094eb2639bb8ead8f3/docs/planning/adrs/0005-irt-response-matrix-contract.md) and
[ADR 0006](https://github.com/ContextualWisdomLab/contextual-orchestrator/blob/89bd5bf73319dd21f2be1f094eb2639bb8ead8f3/docs/planning/adrs/0006-polytomous-llm-judge-bias-calibration.md) and
[ADR 0008](https://github.com/ContextualWisdomLab/contextual-orchestrator/blob/2b65d5c0f3d6bd64a9e05818f1f9286e98c334c1/docs/planning/adrs/0008-fast-judge-review-hardening.md).
categorized polytomous matrix. Equal-width direct K-way projection is
experimental; opt-in `category_method="cumulative_threshold"` evaluates each
ordered boundary with a strict Boolean vector and derives the category in the
adapter. Category-count and prompt-perturbation calibration remain required
for both methods. See
[ADR 0005](https://github.com/ContextualWisdomLab/contextual-orchestrator/blob/1b7dbd2a46533f41072def1fb94283147134cab5/docs/planning/adrs/0005-irt-response-matrix-contract.md),
[ADR 0006](https://github.com/ContextualWisdomLab/contextual-orchestrator/blob/1b7dbd2a46533f41072def1fb94283147134cab5/docs/planning/adrs/0006-polytomous-llm-judge-bias-calibration.md), and
[ADR 0008](https://github.com/ContextualWisdomLab/contextual-orchestrator/blob/1b7dbd2a46533f41072def1fb94283147134cab5/docs/planning/adrs/0008-fast-judge-review-hardening.md).
Cross-repository exact-head review, structured Strix evidence, and merge
policy are recorded in [contextual-orchestrator ADR 0004](https://github.com/ContextualWisdomLab/contextual-orchestrator/blob/befa094784e37947841948fb42016de7e6b965ab/docs/planning/adrs/0004-pr-review-merge-loop.md) and
[ADR 0009 dependency cooldown](https://github.com/ContextualWisdomLab/contextual-orchestrator/blob/befa094784e37947841948fb42016de7e6b965ab/docs/planning/adrs/0009-supply-chain-dependency-cooldown.md).
policy are recorded in [contextual-orchestrator ADR 0004](https://github.com/ContextualWisdomLab/contextual-orchestrator/blob/1b7dbd2a46533f41072def1fb94283147134cab5/docs/planning/adrs/0004-pr-review-merge-loop.md) and
[ADR 0009 dependency cooldown](https://github.com/ContextualWisdomLab/contextual-orchestrator/blob/1b7dbd2a46533f41072def1fb94283147134cab5/docs/planning/adrs/0009-supply-chain-dependency-cooldown.md).
- Standalone HTML reports for saved fit or dimensionality diagnostics.
- Automated benchmark evidence reports from release-acceptance timing.
- Release evidence index reports that tie dist artifact hashes, acceptance,
Expand Down
159 changes: 159 additions & 0 deletions docs/adr/0014-bounded-llm-judge-category-inputs.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,159 @@
# ADR-0014: Bounded LLM-judge category inputs and security-scan evidence

Status: **Proposed**
Date: 2026-08-12
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
Supersedes: none
Superseded by: none

## Context

The LLM-as-a-Judge adapter accepts category counts and category values at a
public Python boundary before projecting results into dichotomous or
polytomous IRT items. The current-head Strix run for PR #778 at
`0721e55ce5d889b7917aa7da2891367adf3430dc` reported a supposed integer
overflow in this path. Python's built-in integers do not wrap, so that claim
was not independently valid. The review did, however, expose a real adjacent
boundary issue: `isinstance(value, int)` admits an `int` subclass whose
comparison methods can return forged results, allowing the category-count
bound to be bypassed before a resource-amplifying operation.

The same trust-boundary concern applies to category values accepted by the
deterministic IRT projection. A model response parsed by `json.loads` contains
built-in scalar types, but callers can construct `LLMJudgeResult` directly.
Security evidence must therefore distinguish a real code defect from a model
or provider failure and must never weaken the fail-closed gate.

## Decision drivers

- Reject untrusted or adversarial runtime scalar objects before comparisons,
arithmetic, or allocations.
- Preserve the existing dichotomous/polytomous response-matrix contract.
- Keep LLM security findings advisory until independently reproduced by code,
tests, or a structured scanner artifact.
- Keep provider rate limits separate from source-security conclusions.

## Ownership and dependency direction

`fast-mlsirm` owns the judge result contract and its IRT projection. LLM
transport remains injected through `contextual-orchestrator`; neither a model
response nor a Strix narrative becomes numerical or merge authority.

## Decision

1. `category_count` accepts only an exact built-in Python `int` in
`2..MAX_JUDGE_CATEGORIES` (currently 64). Booleans, floats, subclasses,
negative values, and oversized integers fail before any derived list or
threshold array is materialized.
2. Direct category values accept only exact built-in `int` or `float` values;
booleans, subclasses, non-finite values, fractional values, and out-of-range
values fail closed.
3. Any cumulative-threshold result remains bounded by the validated category
count, and every IRT row still requires multiple criterion items and passes
the shared dichotomous/polytomous response-matrix validator.
4. A Strix/provider failure is recorded with its exact head SHA, run URL,
failed step, backend error, and structured-report availability. A model-only
claim is not called a confirmed vulnerability without independent evidence.
5. Required checks remain fail-closed. No scan failure is bypassed, and no
self-approval or protected-branch merge is manufactured.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved

## Invariants / acceptance evidence

1. `tests/test_llm_judge.py::test_category_count_and_category_values_reject_runtime_subclasses`
rejects booleans, floats, values above 64, astronomically large integers,
and comparison-forging `int` subclasses.
2. `judge()` and `LLMJudgeResult.to_irt_row()` share the same bounded category
validation before cumulative threshold allocation or IRT projection.
3. PR #778's Strix log records an initial NVIDIA NIM 429, a fallback-model
report, no structured vulnerability artifact, and the exact finding. The
finding is treated as an adjacent hardening signal, not as proof of integer
wraparound.
4. The full Python and Rust test suites, targeted Ruff checks, and exact-head
required checks must pass before this hardening is considered integrated.

## Non-goals and claims not made

- This decision does not claim that Python integers overflow or that a single
model-generated security report proves a CVSS score.
- Exact scalar typing does not establish that LLM judgments are unbiased.
- This decision does not authorize keyword matching, direct provider calls, or
a single scalar response as an IRT dataset.

## Consequences and trade-offs

### Benefits

- Resource-amplifying category bounds cannot be bypassed through comparison
hooks or coercion protocols.
- Security and provider-availability evidence remains auditable and separate.
- The public judge path stays deterministic and compatible with the existing
multi-item IRT contract.

### Costs / risks

- Legitimate custom numeric subclasses are rejected and must be normalized by
the caller before crossing the boundary.
- Strix runs remain dependent on external model quotas and can require a
rerun after a provider outage.

## Alternatives considered

### Coerce every value with `int()` or `float()`

Rejected because conversion hooks can execute caller code and can silently
change values at a security and measurement boundary.

### Trust the Strix narrative as the vulnerability oracle

Rejected because the observed run lacked a structured report, began with a
provider 429, and made a false claim about Python integer wraparound.

### Ignore the finding because the primary claim was false

Rejected because the same code admitted adversarial `int` subclasses that could
forge comparisons. The boundary is now hardened and regression-tested.

## Failure, degraded, and recovery behavior

Malformed category inputs raise the package-owned `ValueError` or
`JudgeFormatError` before model output is accepted or an IRT row is returned.
Malformed model JSON, missing structured scan evidence, provider rate limits,
and failed required checks remain fail-closed. After a code change, rerun the
security workflow against the exact pushed head and record the new result;
never reinterpret a stale run as evidence for a new head.

## Security and privacy implications

The change reduces coercion and resource-exhaustion risk without adding
credentials or retaining source text. Strix logs and provider errors may be
retained as CI evidence, but secrets must remain masked. The exact head SHA
and run URL are required for reproducibility; they do not grant merge
authority.

## Compatibility, migration, and rollback

The accepted JSON schema and valid category range are unchanged. Callers that
used custom numeric subclasses must pass built-in scalars. Rollback is allowed
only with a superseding ADR and equivalent boundary tests; reverting to
coercive validation is not an acceptable compatibility path.

## Verification and release evidence

- Run the targeted judge tests and Ruff on changed Python files.
- Run the full Python suite, Rust workspace tests, and binding tests.
- Re-run Strix and all required checks on the exact pushed PR head.
- Review current inline/general PR feedback and obtain an independent approval
before a protected merge.
- Preserve contextual-orchestrator routing for every LLM-as-a-Judge call.

## Follow-ups

- Re-run PR #778's Strix workflow after this hardening and compare the exact
head SHA, provider status, structured artifact, and finding intersection.
- If a later scan reports a reproducible issue, create a superseding ADR or
append a new evidence record with the reproducer and regression test.

## Reversal / supersession conditions

Supersede this ADR only if the public judge contract adopts a different
bounded scalar representation or if structured security evidence demonstrates
that the current exact-type boundary is insufficient.
1 change: 1 addition & 0 deletions docs/adr/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ A conversation, issue, PR body, design note, or paper summary is not an Accepted
| [0011](0011-canonical-pyo3-public-export-registry.md) | Proposed | Future Rust-backed features converge on one reviewed PyO3/public-export registry instead of competing extension initializers/import rewrites. |
| [0012](0012-purpose-limited-sensitive-data.md) | Accepted | Preserve valid measurement linkage through purpose-limited sensitive-data handling rather than blanket masking or raw-data proliferation. |
| [0013](0013-continuous-execution-and-documentation-governance.md) | Proposed | Keep autonomous work work-conserving and enforce one canonical cross-cutting documentation writer with explicit maturity states. |
| [0014](0014-bounded-llm-judge-category-inputs.md) | Proposed | Bound LLM-judge category inputs to exact built-in scalars and keep model/provider security evidence fail-closed and independently verifiable. |

## ADR completeness rule

Expand Down
1 change: 1 addition & 0 deletions docs/doctoring/fitstats_sx2_person_rust_required.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,4 +15,5 @@ Ordinary public S-X² and person-fit results are produced only by the compiled R
## Verification

- `tests/test_fitstats_rust_ownership_failclosed.py`
- `tests/test_cov_c_fitstats.py` incomplete-core regression
- `tests/test_fitstats.py` incomplete-core regression
9 changes: 9 additions & 0 deletions docs/documentation_coverage.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,15 @@ This matrix answers whether GitHub can reconstruct the current `fast-mlsirm` pro
- **REJECTED** — reviewed and intentionally excluded.
- **OUT_OF_SCOPE** — outside the reusable measurement-core boundary.

For compatibility with review reports, the human-readable status shorthand is
also normative: **IMPLEMENTED**, **ACTIVE PR**, **PLANNED**, and
**DOWNSTREAM**. The phrase **IMPLEMENTED / PLANNED extensions** means that a
protected-main primitive exists while the explicitly named broader extension
remains planned or partial; **ACTIVE PR** never means protected-main truth.
The **Canonical PyO3/public-export registry** is the source of truth for native
entrypoints; **PLANNED/partial** describes a capability that is not yet a
protected-main completion claim.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved

## Canonical documentation coverage

| Documentation capability | Canonical target | State | Current fitness / maintenance rule |
Expand Down
Loading
Loading