(MOT-4277) fix(harness): align CI gating and transcript labels - #647
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
📝 WalkthroughWalkthroughThe change unwraps ChangesTranscript display normalization
E2E CI score-floor gating
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant HarnessWorkflow
participant E2eRunner
participant E2eReport
HarnessWorkflow->>E2eRunner: Set HARNESS_E2E_CI_SCORE_FLOOR
E2eRunner->>E2eReport: Evaluate fails_ci_gate(score_floor)
E2eReport-->>E2eRunner: Return CI gate result
E2eRunner-->>HarnessWorkflow: Pass, warn, or fail the workflow
Possibly related PRs
Suggested labels: Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
skill-check — worker0 verified, 50 skipped (no docs/).
Four for four. Nicely done. |
ytallo
changed the title
ytallo
marked this pull request as ready for review
July 30, 2026 19:53
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
harness/tests/e2e/src/report.rs (1)
633-656: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winAdd exact-floor and missing-score coverage.
The gate contract is boundary-sensitive: a score equal to the floor must pass, while a missing score must fail. Add tests for
median_score == 50and a non-technical quality failure whose score isNone.This follows the stated policy that scores at or above the floor are allowed while missing scores remain blocking.
Also applies to: 659-663
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@harness/tests/e2e/src/report.rs` around lines 633 - 656, Add coverage in advisory_ci_gate_uses_the_score_floor_for_quality_and_hard_gate_failures for a non-technical quality run with median_score equal to 50 that passes fails_ci_gate, and for a non-technical quality failure with no score that fails it. Keep the existing below-floor and hard-gate assertions unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/_harness-e2e.yml:
- Around line 17-25: Align the CI policy documentation across all three sites:
in .github/workflows/_harness-e2e.yml lines 17-25, describe that quality or
hard-gate failures at or above the configured ci_score_floor are advisory while
technical failures and missing scores remain blocking; in lines 339-345, include
missing scores in the blocking summary; and in harness/tests/e2e/README.md lines
109-113, refer to the configured floor instead of hard-coding 50.
---
Nitpick comments:
In `@harness/tests/e2e/src/report.rs`:
- Around line 633-656: Add coverage in
advisory_ci_gate_uses_the_score_floor_for_quality_and_hard_gate_failures for a
non-technical quality run with median_score equal to 50 that passes
fails_ci_gate, and for a non-technical quality failure with no score that fails
it. Keep the existing below-floor and hard-gate assertions unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: c5a354ce-4bc7-47b5-ae14-0c9618fce996
📒 Files selected for processing (8)
.github/benchmark-site/execution-transcript.js.github/benchmark-site/execution-transcript.test.cjs.github/workflows/_harness-e2e.yml.github/workflows/harness-e2e-daily.yml.github/workflows/harness-e2e-main.ymlharness/tests/e2e/README.mdharness/tests/e2e/src/main.rsharness/tests/e2e/src/report.rs
Comment on lines
+17
to
+25
| description: Only fail degraded results below the CI score floor | ||
| required: false | ||
| type: boolean | ||
| default: false | ||
| ci_score_floor: | ||
| description: Minimum median score allowed by the advisory CI policy | ||
| required: false | ||
| type: number | ||
| default: 50 |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Align all CI policy descriptions with the actual gate.
The implementation uses the configured floor (50 by default), while technical failures and missing scores remain blocking.
.github/workflows/_harness-e2e.yml#L17-L25: state that quality/hard-gate failures at or above the floor are advisory, but technical failures and missing scores still block..github/workflows/_harness-e2e.yml#L339-L345: include missing scores in the blocking summary.harness/tests/e2e/README.md#L109-L113: describe the configured floor rather than hard-coding 50.
📍 Affects 2 files
.github/workflows/_harness-e2e.yml#L17-L25(this comment).github/workflows/_harness-e2e.yml#L339-L345harness/tests/e2e/README.md#L109-L113
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/_harness-e2e.yml around lines 17 - 25, Align the CI policy
documentation across all three sites: in .github/workflows/_harness-e2e.yml
lines 17-25, describe that quality or hard-gate failures at or above the
configured ci_score_floor are advisory while technical failures and missing
scores remain blocking; in lines 339-345, include missing scores in the blocking
summary; and in harness/tests/e2e/README.md lines 109-113, refer to the
configured floor instead of hard-coding 50.
This was referenced Aug 3, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Root cause
Hard-gate failures were classified as unconditionally blocking even when the run had a score above the intended CI floor. A
reactive_automationexecution scoring 80% therefore exited with code 1 and marked the workflow as failed.Transcript entries also exposed the generic
agent_triggerwrapper even though the wrapped function id was available inarguments.function.Impact
Executions at or above 50% can remain visibly degraded without failing the pipeline. Scores below 50%, missing scores, provider or judge errors, resource limits, and infrastructure failures remain blocking.
Function calls in the transcript are labeled with actionable names such as
database::queryandstate::set, and their request panels omit the redundant{ function, payload }wrapper.Validation
cargo test --locked --manifest-path harness/Cargo.toml -p harness-e2e(60 passed)cargo clippy --locked --manifest-path harness/Cargo.toml -p harness-e2e --all-targets -- -D warningscargo fmt --manifest-path harness/Cargo.toml --all -- --checknode --test .github/benchmark-site/*.test.cjsgit diff --checkRefs MOT-4277
Summary by CodeRabbit
New Features
Documentation