chore: make Harness E2E score advisory

[ytallo]

Summary

• keep Harness E2E scores as reported quality data without using them as a CI exit gate
• keep hard-gate, technical, and empty-report failures blocking
• remove the obsolete score-advisory and score-floor configuration from the runner and workflows
• document the updated CI behavior

Why

Model scores are probabilistic quality signals. A low score should remain visible for evaluation and trend analysis without independently failing CI.

Validation

• cargo fmt --manifest-path harness/Cargo.toml --all -- --check
• cargo test --locked --manifest-path harness/Cargo.toml -p harness-e2e — 62 passed
• cargo clippy --locked --manifest-path harness/Cargo.toml -p harness-e2e --all-targets -- -D warnings
• YAML parsing for the affected workflows

Summary by CodeRabbit

• Bug Fixes

  • Updated end-to-end test gating so quality scores are advisory and do not independently fail CI.
  • Hard-gate failures, technical failures, and empty test scenarios remain blocking.
  • Removed configurable score thresholds from E2E execution and reporting.

• Documentation

  • Clarified score interpretation, CI gate behavior, and dashboard thresholds in the E2E documentation.

• Tests

  • Expanded coverage for advisory quality failures and blocking failure conditions.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
workers	Ready	Preview	Aug 3, 2026 9:53pm
workers-tech-spec	Ready	Preview	Aug 3, 2026 9:53pm

[coderabbitai]

Warning

Review limit reached

@ytallo, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 57 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: f6ebd437-5ef1-4f38-a913-e1c18df5cf65

📥 Commits

Reviewing files that changed from the base of the PR and between 2b37a89 and 3bc3c26.

📒 Files selected for processing (2)

• .github/workflows/_harness-e2e.yml
• harness/tests/e2e/README.md

📝 Walkthrough

Walkthrough

The E2E system removes configurable quality-advisory and score-floor settings. Quality-score failures no longer block CI. Empty, hard-gate, and technical failures remain blocking across the runner, workflows, tests, and documentation.

Changes

E2E quality gating

Layer / File(s)	Summary
CI blocking classification harness/tests/e2e/src/report.rs	E2eReport now blocks CI only for empty reports, hard-gate failures, or technical failures. Tests cover each blocking and non-blocking case.
Runner gate integration harness/tests/e2e/src/main.rs	The runner removes quality settings and uses has_ci_blocking_failure() to classify failures.
Workflow and documentation alignment .github/workflows/_harness-e2e.yml, .github/workflows/harness-e2e-*.yml, harness/tests/e2e/README.md	Workflows no longer pass quality settings. Summaries and documentation describe quality scores as advisory. Fixed thresholds remain evaluation criteria.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

• iii-hq/workers#637: Modifies the shared E2E workflow and scoring behavior.
• iii-hq/workers#647: Changes the same score-floor and advisory-gating implementation.
• iii-hq/workers#672: Updates E2E hard-gate and quality-score CI semantics.

Poem

I’m a rabbit watching scores glow,
Advisory numbers may come and go.
Hard gates stand firm, technical faults too,
Empty reports still block the queue.
CI hops onward with rules made clear.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: Harness E2E scores become advisory while blocking failures remain enforced.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch chore/remove-score-ci-gate

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

skill-check — worker

0 verified, 52 skipped (no docs/).

Layer	Result
structure	✓
vale	✓
ai	✓
render	✓

Four for four. Nicely done.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/_harness-e2e.yml:
- Around line 434-438: Update the CI gate summary in
.github/workflows/_harness-e2e.yml at lines 434-438 to state that empty reports,
hard-gate failures, and technical failures block CI. Update
harness/tests/e2e/README.md at lines 116-118 to document empty reports as
CI-blocking, and at lines 211-213 include empty reports in each live-model
workflow gate description.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 3983a993-8ba9-4be1-8eed-e9c71c4feebb

📥 Commits

Reviewing files that changed from the base of the PR and between 1795404 and 2b37a89.

📒 Files selected for processing (7)

• .github/workflows/_harness-e2e.yml
• .github/workflows/harness-e2e-daily.yml
• .github/workflows/harness-e2e-deployed.yml
• .github/workflows/harness-e2e-main.yml
• harness/tests/e2e/README.md
• harness/tests/e2e/src/main.rs
• harness/tests/e2e/src/report.rs

💤 Files with no reviewable changes (3)

• .github/workflows/harness-e2e-main.yml
• .github/workflows/harness-e2e-deployed.yml
• .github/workflows/harness-e2e-daily.yml