(MOT-4305) feat(harness): discriminative judge-backed scenarios and scored hard-gate failures - #672
Merged
Merged
Conversation
A hard-gate failure is a quality result, not an execution error: the run keeps its objective partial credit (zero when criteria are judge-delegated), enters the aggregate median as a poor score, and shares the two-of-three tolerance used by score-only failures. Only technical failures stop further repetitions and fail the aggregate outright, in the runner and in the dashboard suite verdict.
…d rubrics Add design_tradeoff (contested scaling decision that punishes non-committal answers) and security_triage (subtle real vulnerabilities plus safe decoys that punish invented findings) to break the ceiling effect of the existing subjective scenarios. Anchor every judge-backed criterion description with explicit full/half/zero score bands to reduce judge variance.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
skill-check — worker0 verified, 51 skipped (no docs/).
Four for four. Nicely done. |
📝 WalkthroughWalkthroughChangesHarness E2E quality evaluation
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant ScenarioSuite
participant TextEvaluator
participant ScenarioReport
participant BenchmarkCollector
ScenarioSuite->>TextEvaluator: evaluate scenario response
TextEvaluator-->>ScenarioSuite: return criterion scores
ScenarioSuite->>ScenarioReport: record run results
ScenarioReport-->>BenchmarkCollector: provide suite status and metrics
Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 2📝 Generate docstrings 💡
🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
ytallo
changed the title
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two related changes to the harness E2E quality suite:
Hard-gate failures become scored quality results
A hard-gate failure no longer stops the scenario loop or fails the aggregate outright. The run keeps its objective partial credit (zero when every criterion is judge-delegated), enters the aggregate median as a poor score, and shares the two-of-three tolerance already used by score-only failures. Only technical failures (subject/judge/resource/infrastructure) still block further repetitions and fail the aggregate — in the runner and in the dashboard suite verdict (
collect_harness_e2e_benchmarks.py).Motivation: in execution 30736657213-1, one hard-gate-failed run (score 45) stopped the scenario at 2 of 3 requested runs and forced the failure, even though there was no execution error. A poor score above an error, whenever no execution error occurred.
Two new judge-backed scenarios + anchored rubrics
The existing subjective scenarios (
direct_answer,security_review) pin at median 100 — trivial tasks, no discrimination. This adds:design_tradeoff— a contested database-scaling decision with facts pulling in opposite directions and verifiable runway arithmetic in the hidden judge reference. The rubric punishes non-committal "it depends" answers and rewards honest costs of the chosen option plus concrete reversal conditions.security_triage— four Python snippets: two subtly exploitable (absolute-path bypass of anos.path.jointraversal check; non-constant-time HMAC comparison) and two safe decoys that pattern-match to SQLi/XSS.false_positive_control(35 pts) zeroes when a safe snippet is reported as vulnerable — measuring calibration, which the current suite cannot.Every judge-backed criterion description (new and existing scenarios) now carries explicit full/half/zero score anchors, reducing judge variance.
No CI/workflow changes needed: the scenario matrix comes from
harness-e2e listdynamically.Test plan
cargo test -p harness-e2e— 60/60, registry test validates all 7 specscargo run -p harness-e2e -- listincludesdesign_tradeoffandsecurity_triagecargo clippy -p harness-e2ecleanSummary by CodeRabbit
New Features
Bug Fixes
Fixes MOT-4305