Repository navigation
feat(OMN-20552): document delegation answers record a facts-grounding rubric verdict - #3399
Merged
Merged
Conversation
… rubric verdict The class rubric contract declared no document class, so every document attempt recorded UNDETERMINED no_rubric_for_class. onex delegate run de6357ab-d4e3-42de-b6a9-3ba186160860 (Qwen3.8-27B, quality 1.0) is the case: it named only identifiers the facts carry and gave two causes they never state. document now records claims_traceable, the summarization class's criterion, with a non-MET false-pass status, false-pass and false-refusal lines and a check-register entry. It refuses a path, backticked token or long quote the facts lack, which the gate floor does not check. Measured before shipping on the local accepted corpus (blind labels by Codex): it refused 265 of 1242 accepted document answers, 7 of a sampled 25 real. A lexical check of stated causes was built and measured and is not shipped: 14 of 73 refusals real at its loosest setting, at best 6 of 16. The recorded run's causes reuse the facts' words, so it is caught by the offline acceptance judge, whose reply is committed and replayed through the judge node's render and check. Tests: tests/unit/delegation/test_document_facts_grounding_omn20552.py. Onex-Lane: document-rubric-9143 Onex-Session: adbfb517db5b4a60a465904a4f6aee40
|
| Rule | Description |
|---|---|
| ARCH-TOPIC-001 | No hardcoded Kafka topic strings in handler code |
| ARCH-DI-001 | No event_bus=None bypass |
| ARCH-DI-002 | No direct Handler instantiation outside workflow_runner/adapters |
| ARCH-DI-003 | No reinvented DI containers |
| ARCH-TOPIC-002 | contract.yaml topics follow `onex.{cmd |
ARCH-DI-002: Direct Handler instantiation (warning)\n\n```\nsrc/omnimarket/nodes/node_delegate_skill_orchestrator/ports/port_local_delegation_dispatch.py:976: self._effect_handler = effect_handler or HandlerLlmDelegationCall()
src/omnimarket/nodes/node_delegate_skill_orchestrator/ports/port_local_delegation_dispatch.py:977: self._projection_handler = projection_handler or HandlerProjectionDelegation(
src/omnimarket/nodes/node_delegate_skill_orchestrator/ports/port_local_delegation_dispatch.py:3367: HandlerProjectionLlmCost().project(call_event, self._evidence_db)
src/omnimarket/nodes/node_local_dashboard_serve_effect/handlers/handler_local_dashboard_serve.py:227: handler=HandlerProjectionRead(topic_map=topics, row_source=source),\n```\n\nHandler classes should be wired only in workflow_runner.py or adapter modules. Direct instantiation elsewhere bypasses the contract system.\n\n
Static architectural lint — no model inference (OMN-14176).
✅ Hostile Reviewer — PASSEDBlocking findings (critical/error): 0 Gate semantics
Powered by omniintelligence.review_pairing.cli_review — node-based adversarial review via HandlerLlmCliSubprocess (OMN-8524) |
jonahgabriel
enabled auto-merge (squash)
October 5, 2026 00:10
…3 fails at the merge base The AC3 control found dod-document-rubric-ac3 passing at the merge base because the replay only exercised the pre-existing judge node and the fixture. It now also asserts the deterministic document rubric records claims_traceable PASS on the replayed document answers, a criterion the merge base does not declare for the class. Onex-Lane: document-rubric-9143 Onex-Session: adbfb517db5b4a60a465904a4f6aee40
jonahgabriel
added a commit
that referenced
this pull request
Oct 5, 2026
…der's POST, not from before handle() (#3404) * test(OMN-18852): time the silence-after-chunk deadline from the provider's POST, not from before handle() The test started its clock before HandlerInferenceIntent.handle(). That window also covers the served-model probe and the request build, which run before the deadline starts by design. In a cold process, the probe's first httpx.Client pays a one-time lazy httpcore import, measured at 0.25 s on .201. When #3399 moved the CI shard boundary so this test ran first in Split 11/20, dev fced10c failed twice (run 37249701095: 1.9452 s and 1.9269 s against 1.5). The handler's own timeout line read elapsed_seconds=1.000 both times. A later run on the same sha placed the test after its keep-alive sibling, and it passed in about 1.02 s. The clock now starts when the provider receives the POST. The 1.5 s bound is unchanged. A run with the executor bound removed still fails, at 1.61 s (0.6 s chunk plus a 1.0 s per-phase read timeout). A 0.6 s pre-deadline delay fails the old form and passes the new one. Onex-Lane: omnimarket-dev-deadline-2a21 Onex-Session: 522dbedf23184e26a900ceaab6c183cf * test(OMN-18852): add the repo-evidence contract for the deadline test fix omnimarket carries its own DoD contracts for the repo-evidence gate. This PR is test-only and cited OMN-18852 with no contracts/OMN-18852.yaml, so repo-evidence / dod-verify refused it. The contract binds AC2 (the inference deadline bound) to the deadline test file; the verifier reads it verified at the head, and the merge-base control takes the test-only path.
Patel230
added a commit
that referenced
this pull request
Oct 5, 2026
The PR read DIRTY against dev. The only conflict was the import block of port_local_delegation_dispatch.py: this branch imports ModelUsageCallEvent from omnimarket.models, and dev (#3399/#3404 line) imports dispatch_progress from the same node. They are independent, so both are kept in isort order. sqlite_database.py and the writer/store parity test merged without conflict. Dev's delegation_events reconcile (#3319) and this branch's usage-by-model-day reconcile run one after the other in _connect, each in its own BEGIN IMMEDIATE transaction. Onex-Lane: omn20006-usage-page Onex-Session: d99c149d-35af-4f1e-b791-28a28187bfc4
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Ticket OMN-20552:
delegation_class_rubrics.v1.yamllacked a document class, so every document attempt recorded rubricUNDETERMINEDwithno_rubric_for_class. Delegate runde6357ab-d4e3-42de-b6a9-3ba186160860(Qwen3.8-27B) scored quality 1.0 despite stating two causes absent from its facts.Change
Adds a document class with the
claims_traceablecriterion (the summarization class's own), a non-MET false-pass status, false-pass/false-refusal lines, and a check-register entry.claims_traceablerefuses paths, backticked tokens or long quotes the facts lack; the gate floor already refuses invented ids and numbers. The recorded run is a fixture; the offline acceptance judge rejects it as fabrication, and its reply is replayed in tests.What this does not do
test_rubric_not_met_recorded_only_shipped_config).Measurement (measured, not estimated)
claims_traceableover 1242 accepted local document answers: 780 PASS, 265 FAIL, 197 UNDETERMINED. A blind Codex label of 25 sampled FAILs found 7 real fabrications. The misses are checker defects: escaped quotes, a bare number written as a ref (#4288vs4288), derived ranges and timestamps. This is the first number on the document false-refusal line, and it is not near the floor.delegation-acceptance-judge.v1, render seedomn20552, Codex run7279913f-515e-4cf8-845b-ea77255b65eb) on the fixture batch: de6357ab REJECT fabrication; the hand-grounded rewrite ACCEPT. The test re-renders the batch, checks the prompt sha256 against the recorded one, and replays the reply through the node's check operation.Tests
tests/unit/delegation/test_document_facts_grounding_omn20552.py(12): contract, recorded-run verdict, refusal of an untraceable path, attempt-path recording, gate decides only when MET, judge replay, in-process port end to end.test_rubric_recorded_not_deciding_omn20165.py: the class-without-rubric case moves from document to planning.test_rubric_checks.py: the class set gains document.uv run python -m omnimarket.ranges check-register: OK, 72 checks.Lab proof
h202 (omnipc2), via the lab-run ssh path because h202 was not advertising on the bus (
placement: refused, only h200 seen). The run was at shaacf08e74990db41c3a18cd5e60c29b77bf1b3258with 439 passed in 23.57s across the new file, the rubric-check, gate-eval and acceptance-judge node suites, and the OMN-20165/20166 rubric suites, plus the register check. Not runtime-affecting for the lab's served behaviour: the shipped status records only, so no dev-lane refresh was needed or done.Delegation
03ff6a26-8d30-4320-b5e5-f149764e1c03(document, Qwen3.8-27B, deployed lane) drafted Why/Change and was accepted. Content check: every number and id is in the facts. One sentence was moved by hand.Coordinates with omnimarket#3382 (facts-first prompt, OMN-19432): the recording path grounds on the prompt as stated, facts included.
Evidence-Ticket: OMN-20552
Evidence-Source: OCC#12852