Repository navigation
contract(OMN-10846,OMN-10847,OMN-10848): add OCC contracts and DoD evidence for canary→routing feedback loop - #942
Merged
Conversation
…idence for canary→routing feedback loop tickets All three tickets delivered via PR #606 on OmniNode-ai/omnimarket (merge commit 1af2f16a1c7d10a1f9392288f6cd63a8a1473957). PRs #604 and #605 were closed as superseded after cherry-pick into #606. - OMN-10846: build_available_models_from_scores() bridge (capability_scores → routing policy) - OMN-10847: ModelMaterializeResult + routing_outcomes.quality_score population - OMN-10848: Proof-of-life integration test (canary → scores → routing, pure in-memory)
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (16)
✅ Files skipped from review due to trivial changes (12)
🚧 Files skipped from review as they are similar to previous changes (4)
📝 WalkthroughWalkthroughAdds three ticket contracts (OMN-10846/10847/10848) plus JSON evidence reports and YAML DOD/drift receipts that verify PR ChangesCanary Feedback Loop Tickets & Evidence
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@contracts/OMN-10846.yaml`:
- Around line 42-48: The DoD claim and the command check are mismatched: the
policy asserts "HandlerRoutingPolicy" is not modified but the check only
verifies existence of build_available_models_from_scores; update the command
under the check (or replace it) to explicitly verify no handler files or the
symbol HandlerRoutingPolicy were changed (e.g., run a git diff/name-only against
the PR and grep for files or symbols containing "HandlerRoutingPolicy" or
handler directory names) so the check fails if any handler-related files or the
HandlerRoutingPolicy symbol are modified; keep the existing
build_available_models_from_scores check only if needed and ensure the
id/description reflect the actual check.
In `@contracts/OMN-10848.yaml`:
- Around line 49-54: The check with id "dod-quality-score-nonzero" only greps
for the token "quality_score" and does not validate the numeric value; update
the check_value (under check_type "command") to run a command that extracts the
quality_score occurrences from the PR diff (the current check_value string) and
asserts the numeric value is greater than 0.0 (for example by parsing the diff
with awk/jq/grep -P and numeric comparison), so the contract actually verifies
quality_score > 0.0 rather than merely the presence of the token.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 49985dd4-dbd7-4d99-a973-6bcfd9742d34
📒 Files selected for processing (6)
contracts/OMN-10846.yamlcontracts/OMN-10847.yamlcontracts/OMN-10848.yamlevidence/OMN-10846/dod_report.jsonevidence/OMN-10847/dod_report.jsonevidence/OMN-10848/dod_report.json
jonahgabriel
enabled auto-merge (squash)
May 12, 2026 15:45
…vidence receipts for verify gate - Add dod-occ-pr evidence item to OMN-10846, OMN-10847, OMN-10848 contracts - Create drift/dod_receipts YAML for all dod_evidence items across all three tickets - Fix dod-no-handler-modification check_value to use grep '^+.*def' prefix - Fix dod-quality-score-nonzero check_value to assert quality_score > 0.0 - Address CodeRabbit threads on evidence description mismatches
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@drift/dod_receipts/OMN-10848/dod-quality-score-nonzero/command.yaml`:
- Around line 6-7: The probe's regex allows any "assert.*quality_score" to pass
without requiring a >0.0 check; update the check_value regex (and the other
identical occurrences around Lines 13-19) so the pattern requires an explicit
greater-than comparison. Replace the alternation with a pattern that enforces ">
0.0" in both raw and assert forms, e.g. use a single grep -E pattern like
(quality_score.*>\s*0\.0|assert.*quality_score.*>\s*0\.0) so PASS only occurs
when quality_score > 0.0 is actually asserted.
- Line 10: The receipt's commit_sha field in command.yaml is incorrect
(currently set to cfe9a908); update the commit_sha value to the actual head SHA
of the external PR (5a3dd07bf18cea9554bb71f9bcaa684bacdacdc7). Locate the
commit_sha key in the dod-quality-score-nonzero/command.yaml and replace the old
SHA with the new one so the provenance points to the correct
OmniNode-ai/omnimarket PR head.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: e996ed06-356c-49c9-860d-0530f521b829
📒 Files selected for processing (16)
contracts/OMN-10846.yamlcontracts/OMN-10847.yamlcontracts/OMN-10848.yamldrift/dod_receipts/OMN-10846/dod-bridge-function-exists/command.yamldrift/dod_receipts/OMN-10846/dod-no-handler-modification/command.yamldrift/dod_receipts/OMN-10846/dod-occ-pr/command.yamldrift/dod_receipts/OMN-10846/dod-pr-merged/command.yamldrift/dod_receipts/OMN-10847/dod-materialize-result-type/command.yamldrift/dod_receipts/OMN-10847/dod-occ-pr/command.yamldrift/dod_receipts/OMN-10847/dod-pr-merged/command.yamldrift/dod_receipts/OMN-10847/dod-quality-score-populated/command.yamldrift/dod_receipts/OMN-10848/dod-champion-selection/command.yamldrift/dod_receipts/OMN-10848/dod-full-chain-test/command.yamldrift/dod_receipts/OMN-10848/dod-occ-pr/command.yamldrift/dod_receipts/OMN-10848/dod-pr-merged/command.yamldrift/dod_receipts/OMN-10848/dod-quality-score-nonzero/command.yaml
✅ Files skipped from review due to trivial changes (11)
- drift/dod_receipts/OMN-10848/dod-pr-merged/command.yaml
- drift/dod_receipts/OMN-10847/dod-pr-merged/command.yaml
- drift/dod_receipts/OMN-10848/dod-full-chain-test/command.yaml
- drift/dod_receipts/OMN-10846/dod-pr-merged/command.yaml
- drift/dod_receipts/OMN-10847/dod-quality-score-populated/command.yaml
- drift/dod_receipts/OMN-10847/dod-materialize-result-type/command.yaml
- drift/dod_receipts/OMN-10846/dod-no-handler-modification/command.yaml
- drift/dod_receipts/OMN-10846/dod-bridge-function-exists/command.yaml
- drift/dod_receipts/OMN-10848/dod-champion-selection/command.yaml
- contracts/OMN-10848.yaml
- contracts/OMN-10846.yaml
🚧 Files skipped from review as they are similar to previous changes (1)
- contracts/OMN-10847.yaml
…with receipt-check pattern Switch dod_evidence check_values from inline `gh pr diff 606 -R omnimarket` commands to `grep -q '^status: PASS$' "$CONTRACT_REPO_DIR/drift/dod_receipts/...` to comply with the legacy-gh-pr lint rule (OMN-9350) that forbids hardcoded PR numbers in gh pr commands.
…upstream PR receipts Set commit_sha to 5a3dd07b (omnimarket PR #606 head SHA) in all receipts that document omnimarket evidence. Previously had the OCC branch SHA which misrepresents the provenance of the upstream evidence.
Remove the loose |assert.*quality_score fallback from the probe regex. The actual diff output confirms quality_score.*>.*0.0 matches directly, so the tighter form is both accurate and avoids false-positive via None-check.
…ceipt-check approach The description previously said "diff shows only X added" but the check reads a receipt file — mismatched claim/verification. Updated to say "verified via receipt" to match the actual check pattern.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
build_available_models_from_scores()capability_scores bridgeModelMaterializeResult+routing_outcomes.quality_scorepopulationAll three tickets delivered via PR #606 on OmniNode-ai/omnimarket (merge commit
1af2f16a1c7d10a1f9392288f6cd63a8a1473957). PRs #604 and #605 were closed as superseded after cherry-pick into #606.DoD Evidence
All checks run against PR #606 diff and verified PASS:
OMN-10846 — bridge function
build_available_models_from_scores()confirmed in diff, caller-side only (no handler modification)OMN-10847 —
ModelMaterializeResulttype andquality_scorecolumn population confirmed in diffOMN-10848 —
test_integration_feedback_loop.pyconfirmed in diff; qwen3-coder-30b selected as champion (0.94 vs 0.70 recall);quality_score > 0.0assertion presentTest plan
uv run validate-yaml contracts/OMN-10846.yaml contracts/OMN-10847.yaml contracts/OMN-10848.yaml→ all 3 PASSSummary by CodeRabbit