Repository navigation
Surface invalid label alongside quality gate evidence mismatch (OMN-18989) - #3560
Conversation
… gate mismatch (OMN-18989)
✅ Architectural Review — PASSEDErrors: 0 What this checks
No architectural violations found. Static architectural lint — no model inference (OMN-14176). |
✅ Hostile Reviewer — PASSEDBlocking findings (critical/error): 0 Gate semantics
Powered by omniintelligence.review_pairing.cli_review — node-based adversarial review via HandlerLlmCliSubprocess (OMN-8524) |
Repo Evidence Gate job "repo-evidence / dod-verify" failed at head fcc0b92 because the PR bound no evidence of its own. contracts/OMN-18989.yaml on dev (from PR #3447) only bound AC1. This commit adds acceptance criterion AC2 (the bus quality gate keeps DELIVERABLE_EVIDENCE_MISMATCH and appends the extractor contract_failure_reasons), an evidence_requirements entry, and dod_evidence item dod-omn18989-ac2-bus-gate-names-rejected-label running uv run pytest tests/unit/delegation/test_declared_label_set_membership_omn18989.py::test_bus_gate_names_rejected_labels_before_terminal_composition -q. Lab readback on h202: 4 passed at head; at merge base ed3b300 the test is absent (exit 4, no tests ran); head's test against base code: 2 failed, 2 passed.
The CI job "Tests (Split 2/20)" in run 37803216137 at head 509f157 failed on tests/ci/test_repo_evidence_caller_workflow.py::test_every_repo_contract_binds_every_criterion because the AC2 check_value in OMN-18989.yaml named the test as file.py::test_name, but the repo gate only accepts a selector token ending in .py. This change updates the AC2 dod_evidence check_value, the AC2 evidence_requirements command, and the AC2 falsifier in contracts/OMN-18989.yaml to name tests/unit/delegation/test_declared_label_set_membership_omn18989.py and select the test with -k test_bus_gate_names_rejected_labels_before_terminal_composition, matching the form used by other repo contracts. No code change.
File: src/omnimarket/nodes/node_delegation_quality_gate_reducer/handlers/handler_quality_gate.py, function _evaluate_response_contract. extract_deliverable is now called once, right after resolve_deliverable_contract; the DELIVERABLE_EVIDENCE_MISMATCH return reuses extraction.contract_failure_reasons; the later second call is removed. Reason: the hostile reviewer quorum (qwen3-review, gpt-oss-review) blocked the PR on a redundant extraction call. Behaviour is unchanged. Focused tests: test_declared_label_set_membership_omn18989.py 18 passed; test_quality_gate_response_contract_omn15193.py 15 passed; test_reasoning_preamble_veto_omn18379.py 32 passed.
|
Lab: head=7d50f0b7df20c8c45cb9bcc5f1658bf3798ae1c9 host=h200 lane=omn17427-3f28 exit=0 |
Cause
When production extraction withholds invalid JSON but grades raw provider text,
HandlerQualityGateIntentreturned only the evidence mismatch (DELIVERABLE_EVIDENCE_MISMATCH: cleaned content does not match the declared extraction evidence) and withheld the invalid label (INSTRUMENTEMENT_DEFECT).Change
node_delegation_quality_gate_reducer,HandlerQualityGateIntentandhandler_quality_gate._evaluate_response_contractpreserve the existing mismatch failure and append the existing extractorcontract_failure_reasons, so the invalid label now accompanies the mismatch. No new node, script or wire field.Tests
New focused cases
invalid,valid,invalid_then_validandvalid_mismatched_evidencerun through typedModelQualityGateIntent; existing terminal tests still prove refusal count and provider enum. RED first, same focused file: 2 failed, 15 passed, with the missing schema/token diagnostic forinvalidandinvalid_then_valid; the valid positive control passed. GREEN: 18 passed in 3.90s. Ruff check and format check passed for both changed files;git diff --checkpassed.Limitation
The focused handler replay uses a captured token, not a new real deployed delegate failing/passing an enum run. No runtime deployment; no new wire fields. Commit, push, PR creation, repository hooks, ledger and landing handoff belong to the launching host under remote instructions.
Lab: host=h202 lane=lab-fill-omn18989-10081430 command=env -u PYTHONPATH uv run pytest tests/unit/delegation/test_declared_label_set_membership_omn18989.py -q observed=18 passed
File-scoped pre-commit: ruff and ruff-format passed. Full repository hooks and CI will run on the launching host.
Evidence-Ticket: OMN-18989