fix(heldout): declare judge-effect sample size - #1101
Merged
Merged
Conversation
RED: nested JSON fields still embed 95 in the name even though coverage is a run declaration. The next commit must rename those fields.
Rename nested *_ci95 JSON keys to *_interval. Declared coverage stays in bootstrap_confidence_level. IRT interval_95_coverage_rate diagnostics are unchanged. Production route/conduct defaults stay locked.
ADR 0045 is Proposed. Gap baseline, changelog, and doctoring keep coverage in the declared field rather than the JSON name.
CUSUM delay KPIs must fail closed without replications, horizon, change-point, and coverage, and must keep no-alarm replications as horizon-censored missed detections.
…lays Remove the abort-on-no-alarm and the baked-in 500/250/100 CUSUM screen. Wilson coverage comes from the declared interval and is stored as false_alarm_rate_upper_bound.
ADR 0046 is Proposed. Production route/conduct defaults stay locked.
Paired ranking timings must fail closed without a positive per-context repetition count, and the declared count must be the actual timing loop.
Remove the hidden 200-repetition timing default. The harness run still writes 200 as this run's choice and records latency_repetitions_per_context.
ADR 0047 is Proposed. Production route/conduct defaults stay locked.
Accuracy and decision-latency helpers must fail closed without a positive context_count, and the declared count must be the actual quality loop.
Remove the hidden 24-context TRAIN_CONTEXTS default from evidence construction, quality evaluation, and paired timings. The harness run still writes 24 as this run's choice.
ADR 0048 is Proposed. Production route/conduct defaults stay locked.
Logging-design evidence must fail closed without a positive trial_count, and the declared count must be the actual assignment loop.
Remove the hidden 24,000-trial logging default. The harness run still writes 24,000 as this run's choice and records trials.
ADR 0049 is Proposed. Production route/conduct defaults stay locked.
DIF evidence must fail closed without an even positive sample_size, and the declared count must be the two-group population.
Remove the hidden 4,000-row DIF default. Odd counts fail closed so the two cohorts stay equal. The harness run still writes 4,000 as this run's choice.
ADR 0050 is Proposed. Production route/conduct defaults stay locked.
Reliability evidence must fail closed without a positive sample_size, and the declared count must be the person population per case.
Remove the hidden 1,200-row reliability default. The harness run still writes 1,200 as this run's choice and records sample_size_per_case.
ADR 0051 is Proposed. Production route/conduct defaults stay locked.
Judge-severity evidence must fail closed without a positive sample_size, and the declared count must be the fully crossed person population.
Remove the hidden 1,000-row many-facet default. The harness run still writes 1,000 as this run's choice and records sample_size.
ADR 0052 is Proposed. Production route/conduct defaults stay locked.
|
Warning Review limit reachedNext included review available in 52 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (15)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
seonghobae
added
bug
ChatGPT Codex Connector
Signed-off-by: Seongho Bae <me@seonghobae.me>
Signed-off-by: Seongho Bae <me@seonghobae.me>
Signed-off-by: Seongho Bae <me@seonghobae.me>
Signed-off-by: Seongho Bae <me@seonghobae.me>
Signed-off-by: Seongho Bae <me@seonghobae.me>
Signed-off-by: Seongho Bae <me@seonghobae.me>
Signed-off-by: Seongho Bae <me@seonghobae.me>
Signed-off-by: Seongho Bae <me@seonghobae.me>
Contributor
Author
|
Normal predecessor integration at 8a04a5c retains original judge sample declaration cdbc6a0 and includes predecessor 3e85d32. No-detection censoring, alarm/detection denominators, null ineligible calibration candidate and retained calibration results now propagate through this stack. Merged-tree validation: 33 boundary/ADR identifier tests passed in 5.80 seconds; Ruff and diff whitespace checks passed; CodeGraph indexing completed. No force push or discarded delta. This is local regression evidence, not real-data accuracy, protected merge or release evidence. Required checks and reviews must use the new head. |
Preserve declared judge/drift sample-size contracts while absorbing main's changelog structure and newer gap-baseline receipts. Co-authored-by: Cursor <cursoragent@cursor.com>
After restacking onto main, keep main's seeded and mixed-denominator coverage while calling declared replications/horizon instead of the removed SEQUENTIAL_DRIFT_REPLICATIONS constant. Co-authored-by: Cursor <cursoragent@cursor.com>
seonghobae
changed the base branch from
codex/declared-reliability-sample-size-20260908
to
main
September 17, 2026 23:38
seonghobae
marked this pull request as ready for review
September 17, 2026 23:38
Keep declared judge/DIF/reliability contracts while requiring declared item-covariate sample_size from #1104 on the shared harness entry. Co-authored-by: Cursor <cursoragent@cursor.com>
This was referenced Sep 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack
Draft successor of #1100 (
9ff12b5d). Parents #1099, #1098, #1097, #1096, #1095, #1093, #1092, #1091, #1090, #1074, and psychometric KPI owner #1067 stay open. This PR does not close #1100.Exact head:
cdbc6a03605bff2d3a66b6404b324c3e376b30f2.Problem
Held-out judge-severity evidence hid
JUDGE_SAMPLE_SIZE = 1_000, so the fully crossed many-facet design used a repository-selected person population.Change
sample_sizeon_validate_judge_effects. Missing, boolean, or non-positive values fail closed.sample_size.Verification
Local smallest tests:
tests/test_psychometric_routing.py,tests/test_psychometric_benchmark_boundaries.py,tests/test_planning_adr_identifiers.py)This is declaration-contract evidence, not buyer-held-out accuracy, p95 latency, or protected merge.
Remaining no-heuristics work
Other harness sample sizes (exploration rate, item-covariate) remain later slices. No production policy or release is authorized. Parent #1067 still needs independent review.