Skip to content

Surface invalid label alongside quality gate evidence mismatch (OMN-18989) - #3560

Merged
jonahgabriel merged 7 commits into
devfrom
codex/lab-fill-omn18989-10081430
Oct 8, 2026
Merged

jonahgabriel merged 7 commits into
devfrom
codex/lab-fill-omn18989-10081430

Conversation

@jonahgabriel

@jonahgabriel jonahgabriel commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Cause

When production extraction withholds invalid JSON but grades raw provider text, HandlerQualityGateIntent returned only the evidence mismatch (DELIVERABLE_EVIDENCE_MISMATCH: cleaned content does not match the declared extraction evidence) and withheld the invalid label (INSTRUMENTEMENT_DEFECT).

Change

node_delegation_quality_gate_reducer, HandlerQualityGateIntent and handler_quality_gate._evaluate_response_contract preserve the existing mismatch failure and append the existing extractor contract_failure_reasons, so the invalid label now accompanies the mismatch. No new node, script or wire field.

Tests

New focused cases invalid, valid, invalid_then_valid and valid_mismatched_evidence run through typed ModelQualityGateIntent; existing terminal tests still prove refusal count and provider enum. RED first, same focused file: 2 failed, 15 passed, with the missing schema/token diagnostic for invalid and invalid_then_valid; the valid positive control passed. GREEN: 18 passed in 3.90s. Ruff check and format check passed for both changed files; git diff --check passed.

Limitation

The focused handler replay uses a captured token, not a new real deployed delegate failing/passing an enum run. No runtime deployment; no new wire fields. Commit, push, PR creation, repository hooks, ledger and landing handoff belong to the launching host under remote instructions.

Lab: host=h202 lane=lab-fill-omn18989-10081430 command=env -u PYTHONPATH uv run pytest tests/unit/delegation/test_declared_label_set_membership_omn18989.py -q observed=18 passed

File-scoped pre-commit: ruff and ruff-format passed. Full repository hooks and CI will run on the launching host.

Evidence-Ticket: OMN-18989

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

✅ Architectural Review — PASSED

Errors: 0
Warnings: 0

What this checks

Rule Description
ARCH-TOPIC-001 No hardcoded Kafka topic strings in handler code
ARCH-DI-001 No event_bus=None bypass
ARCH-DI-002 No direct Handler instantiation outside workflow_runner/adapters
ARCH-DI-003 No reinvented DI containers
ARCH-TOPIC-002 contract.yaml topics follow `onex.{cmd

No architectural violations found.

Static architectural lint — no model inference (OMN-14176).

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

✅ Hostile Reviewer — PASSED

Blocking findings (critical/error): 0
Total findings: 7
Models succeeded: qwen3-review,gpt-oss-review


Gate semantics

Verdict Meaning Blocks merge?
passed >=2 models succeeded, no critical/error findings No
blocked CRITICAL or ERROR findings found Yes
degraded Fewer than 2 models succeeded (infra unavailable/timeout) Yes (OMN-15110)

Powered by omniintelligence.review_pairing.cli_review — node-based adversarial review via HandlerLlmCliSubprocess (OMN-8524)

@jonahgabriel
jonahgabriel enabled auto-merge (squash) October 8, 2026 15:06
jonahgabriel and others added 6 commits October 8, 2026 11:15
Repo Evidence Gate job "repo-evidence / dod-verify" failed at head fcc0b92 because the PR bound no evidence of its own. contracts/OMN-18989.yaml on dev (from PR #3447) only bound AC1. This commit adds acceptance criterion AC2 (the bus quality gate keeps DELIVERABLE_EVIDENCE_MISMATCH and appends the extractor contract_failure_reasons), an evidence_requirements entry, and dod_evidence item dod-omn18989-ac2-bus-gate-names-rejected-label running uv run pytest tests/unit/delegation/test_declared_label_set_membership_omn18989.py::test_bus_gate_names_rejected_labels_before_terminal_composition -q. Lab readback on h202: 4 passed at head; at merge base ed3b300 the test is absent (exit 4, no tests ran); head's test against base code: 2 failed, 2 passed.
The CI job "Tests (Split 2/20)" in run 37803216137 at head 509f157 failed on tests/ci/test_repo_evidence_caller_workflow.py::test_every_repo_contract_binds_every_criterion because the AC2 check_value in OMN-18989.yaml named the test as file.py::test_name, but the repo gate only accepts a selector token ending in .py. This change updates the AC2 dod_evidence check_value, the AC2 evidence_requirements command, and the AC2 falsifier in contracts/OMN-18989.yaml to name tests/unit/delegation/test_declared_label_set_membership_omn18989.py and select the test with -k test_bus_gate_names_rejected_labels_before_terminal_composition, matching the form used by other repo contracts. No code change.
File: src/omnimarket/nodes/node_delegation_quality_gate_reducer/handlers/handler_quality_gate.py, function _evaluate_response_contract.
extract_deliverable is now called once, right after resolve_deliverable_contract; the DELIVERABLE_EVIDENCE_MISMATCH return reuses extraction.contract_failure_reasons; the later second call is removed.
Reason: the hostile reviewer quorum (qwen3-review, gpt-oss-review) blocked the PR on a redundant extraction call.
Behaviour is unchanged. Focused tests: test_declared_label_set_membership_omn18989.py 18 passed; test_quality_gate_response_contract_omn15193.py 15 passed; test_reasoning_preamble_veto_omn18379.py 32 passed.
@jonahgabriel

Copy link
Copy Markdown
Contributor Author

Lab: head=7d50f0b7df20c8c45cb9bcc5f1658bf3798ae1c9 host=h200 lane=omn17427-3f28 exit=0
Command: env -u PYTHONPATH uv run pytest tests/unit/delegation/test_declared_label_set_membership_omn18989.py tests/unit/delegation/test_quality_gate_response_contract_omn15193.py tests/unit/delegation/test_reasoning_preamble_veto_omn18379.py tests/ci/test_repo_evidence_caller_workflow.py -q -p no:cacheprovider
Observed: 70 passed in 6.28s.
Head 7d50f0b adds two commits: a merge of origin/dev that keeps dev's version 0.4.306 in pyproject.toml and uv.lock (the branch's 0.4.305 bump conflicted with dev), and a change so _evaluate_response_contract calls extract_deliverable once and the DELIVERABLE_EVIDENCE_MISMATCH return reuses extraction.contract_failure_reasons.
The second commit answers the hostile reviewer's blocking finding (redundant extraction call, agreed by qwen3-review and gpt-oss-review on the previous head).
The Tests (Split 2/20) red on the previous head was tests/ci/test_repo_evidence_caller_workflow.py failing on contracts/OMN-20728.yaml (AC2, AC3 unbound) from the base; dev bound them in 9a235b6, and the merge carries that.

@jonahgabriel
jonahgabriel merged commit a2a4d76 into dev Oct 8, 2026
142 checks passed
@jonahgabriel
jonahgabriel deleted the codex/lab-fill-omn18989-10081430 branch October 8, 2026 23:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant