test(essay): add governed facets synthetic recovery evidence - #797
Conversation
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughThe pull request adds a deterministic synthetic essay many-facet recovery test. It generates governed scored records, calibrates injected respondent, task, and rater effects, checks recovery criteria, and documents the study scope and limitations. ChangesEssay facets synthetic recovery
Estimated code review effort: 3 (Moderate) | ~25 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
39ea756 to
dbe0c2f
Compare
![opencode-agent[bot]](https://avatars.githubusercontent.com/in/1549082?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
OpenCode cannot approve yet because required coverage evidence did not pass.
Review outcome
1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence
-
Problem: The required coverage-evidence job result was
failure, so OpenCode cannot establish approval sufficiency for this head. -
Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.
-
Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports
successwith required evidence or explicit no-source not-applicable evidence. -
Regression test: Keep the approval branch checking
needs.coverage-evidence.result == successbefore posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present. -
Result: REQUEST_CHANGES
-
Reason: coverage-evidence result was
failure, so required test/docstring evidence was not proven for current headdbe0c2f1eedb0e1a6d6fc6aeeade963de2e66c37. -
Head SHA:
dbe0c2f1eedb0e1a6d6fc6aeeade963de2e66c37 -
Workflow run: 31586438057
-
Workflow attempt: 1
Coverage evidence
Coverage evidence job did not run or did not publish coverage evidence.
Changed-File Evidence Map
flowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Changed file: CHANGELOG.md"]
S1 --> I1["repository behavior"]
I1 --> R1["Review risk: Changed file: CHANGELOG.md"]
R1 --> V1["required checks"]
Evidence --> S2["Docs (2 files)"]
S2 --> I2["operator or user guidance"]
I2 --> R2["Review risk: Docs (2 files)"]
R2 --> V2["docs review"]
Evidence --> S3["Test: test_scoring_essay_facets_synthetic_recovery.py"]
S3 --> I3["regression suite"]
I3 --> R3["Review risk: Test: test_scoring_essay_facets_synthetic_recovery.py"]
R3 --> V3["targeted test run"]
OpenCode Review Overview
Pull request overviewOpenCode cannot approve yet because required coverage evidence did not pass. Review outcome1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence
Coverage evidenceCoverage evidence job did not run or did not publish coverage evidence. Changed-File Evidence Mapflowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Docs (2 files)"]
S1 --> I1["operator or user guidance"]
I1 --> R1["Review risk: Docs (2 files)"]
R1 --> V1["docs review"]
Evidence --> S2["Test: test_scoring_essay_facets_synthetic_recovery.py"]
S2 --> I2["regression suite"]
I2 --> R2["Review risk: Test: test_scoring_essay_facets_synthetic_recovery.py"]
R2 --> V2["targeted test run"]
|
|
@cwl-noema-review Independent exact-head review requested for Review the bounded essay-facets synthetic recovery evidence only. Verify that truth/estimate identification alignment, respondent/task/rater connectedness, severity and task-difficulty ordering, convergence/failure handling, deterministic seeds, conservative interpretation and Rust-backed production arithmetic are accurately tested without treating correlation or one synthetic scenario as validity, fairness or deployment readiness. Every exact-head check is complete and successful and there are no review threads. Read-only review; do not modify or merge. |
|
Perform a read-only exact-head review of |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
tests/test_scoring_essay_facets_synthetic_recovery.py (1)
58-70: 🚀 Performance & Scalability | 🔵 Trivial | 💤 Low valueRemove the temporary squared-residual array.
The metric inputs are one-dimensional aligned vectors. Use the residual dot product for RMSE.
Proposed fix
- float(np.sqrt(np.mean(np.square(residual)))), + float(np.sqrt(np.dot(residual, residual) / residual.size)),🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_scoring_essay_facets_synthetic_recovery.py` around lines 58 - 70, Update the test helper _error_metrics to compute RMSE from the residual vector’s dot product divided by its length, rather than creating a temporary squared-residual array via np.square. Preserve the existing bias and MAE calculations and returned metric order.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@tests/test_scoring_essay_facets_synthetic_recovery.py`:
- Around line 58-70: Update the test helper _error_metrics to compute RMSE from
the residual vector’s dot product divided by its length, rather than creating a
temporary squared-residual array via np.square. Preserve the existing bias and
MAE calculations and returned metric order.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: db3b48d9-2444-42e5-9e9c-1b4f4c764e24
📒 Files selected for processing (3)
docs/changelog.d/782-essay-facets-synthetic-recovery.mddocs/doctoring/essay-facets-synthetic-recovery.mdtests/test_scoring_essay_facets_synthetic_recovery.py
There was a problem hiding this comment.
Pull request overview
OpenCode cannot approve yet because required coverage evidence did not pass.
Review outcome
1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence
-
Problem: The required coverage-evidence job result was
failure, so OpenCode cannot establish approval sufficiency for this head. -
Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.
-
Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports
successwith required evidence or explicit no-source not-applicable evidence. -
Regression test: Keep the approval branch checking
needs.coverage-evidence.result == successbefore posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present. -
Result: REQUEST_CHANGES
-
Reason: coverage-evidence result was
failure, so required test/docstring evidence was not proven for current heade9e224836c0966f12bc8ec982aa3efdb5d3aaaa9. -
Head SHA:
e9e224836c0966f12bc8ec982aa3efdb5d3aaaa9 -
Workflow run: 31626639854
-
Workflow attempt: 1
Coverage evidence
Coverage evidence job did not run or did not publish coverage evidence.
Changed-File Evidence Map
flowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Docs (2 files)"]
S1 --> I1["operator or user guidance"]
I1 --> R1["Review risk: Docs (2 files)"]
R1 --> V1["docs review"]
Evidence --> S2["Test: test_scoring_essay_facets_synthetic_recovery.py"]
S2 --> I2["regression suite"]
I2 --> R2["Review risk: Test: test_scoring_essay_facets_synthetic_recovery.py"]
R2 --> V2["targeted test run"]
1 participant
Supersedes draft #782. Local synthetic recovery test green. Merge when product gates pass.
Summary by CodeRabbit
Documentation
Tests