feat(OMN-20006): usage-by-model-day keeps measured cost apart from estimates

[Patel230]

Update 2026-10-05 08:20Z (head 55bdc6066)

• Head is now 55bdc6066 (was 24ff8612a). The two commits after 24ff8612a are merges of dev by headline: a15a50f6f (v0.4.301 open, chore(OMN-18010): post-release dev version bump to 0.4.301 #3409) and 55bdc6066 (91b8bb1, fix(OMN-20558): burn down the action-pin, transport-mock and runtime_profiles ratchets to plain checks #3408). The verification below covers 24ff8612a and was not re-run at 55bdc6066.
• Version at this head: 0.4.301 (read from pyproject.toml at 55bdc6066, 2026-10-05 08:16-08:17Z), not the 0.4.300 stated below.
• Merge order, superseding the list under "Not in this change": omnibase_infra#4559 merged first; then an omnibase_infra release that vendors node_projection_usage_by_model_day 0002 (the latest release read, v0.38.65 of 2026-10-04T04:30:05Z, does not); then this PR, which also bumps its omnibase-infra pin to that release; and omnidash#358 after omnidash#350. The Sibling Release Compatibility Gate requires the release before this PR: its failure at this head reads that 0002_usage_by_model_day_measured_cost.sql "is not in the pinned omnibase_infra release v0.38.65 ... and no omnibase_infra release vendors it yet" (job 111658862982).
• State at this head, same read: omnibase_infra#4559 is OPEN (head f32f25d82), omnidash#350 is OPEN. Merge state BLOCKED; failing contexts: Sibling Release Compatibility Gate, CI Summary, Coverage Sweep Gate, Repo Evidence Dependency (4), occ-preflight eligibility, repo-evidence / dod-verify, OCC Attestation Observe (report-only); only the Sibling gate's cause was read.

Summary

This is the data half of OMN-20006. usage-by-model-day.v1 now keeps measured cost apart from cost that was estimated or not recorded, the read is scoped to the reading tenant, and the local onex delegate path writes the usage rows the Usage page reads. Before this, an unmeasured run's cost was stored as a number, so no page could count it or leave it out (AC3, AC4).

What changed

• Each call carries its usage_source. ModelUsageCallEvent defaults it to unknown, so an event that never says how its cost was obtained is not taken for a measurement. The legacy alias API still reads as measured. The fold carries the source into ModelUsageCallDelta, and the store writes it on the call row.
• Each day-and-model row serves two new figures.
  • measured_cost_usd: the sum over measured calls, and NULL when there were none, never 0.
  • unmeasured_call_count: the count of the other calls.
  • cost_usd and call_count keep their meaning (every call).
• Both write paths agree. The SQLite store computes the rollup in handler_usage_by_model_day_store.py. The Postgres recount in handler_usage_by_model_day_writer.py uses SUM(cost_usd) FILTER (WHERE usage_source = 'measured') and COUNT(*) FILTER (WHERE usage_source <> 'measured').
• Migration 0002_usage_by_model_day_measured_cost.sql is additive only. Old rows read unknown, NULL and 0 until their key is recounted.
• Exposure contract.
  • It serves the two new columns and declares tenant_column: tenant_id.
  • The consumers: none opt-out is deleted, because the omnidash Usage page is now the declared reader, in omnidash's component registry, which is where omnibase_infra's exposure-reader-coverage gate resolves readers.
• Local store fix. The SQLite usage table lacked projection_cursor, so a served read of this exposure from any local store answered 503 projection_column_missing. Measured at omnimarket dev 8ee3d6103 with the read node, before this change. Now:
  • the store adds the missing columns on connect, inside one BEGIN IMMEDIATE;
  • it stamps the cursor on each insert and recount from a one-row sequence table, as the Postgres BIGSERIAL does from its sequence.
• Local delegate path. port_local_delegation_dispatch.py folds the run's calls into the usage tables, dated by the terminal's emitted_at. Each attempt that reached a provider is its own call, on its own model, with its own cost and usage source; tokens go on the deciding attempt's call only, as onex metering counts them (see the 2026-10-06 08:59Z update). A single-call run is keyed by its correlation id; several calls take <correlation id>:<n>. Each write is in its own guard, so a failed usage write is logged and never breaks the response or the call-metrics write.
• contracts/OMN-20006.yaml binds AC1 (data half), AC3 and AC4 to uv run pytest checks.
• Version 0.4.300, since 0.4.299 is published and reachable from this head (dev itself reads 0.4.299). 0.4.300 is the number the other open omnimarket PRs use.
• Not done here: no bus producer for llm-call-completed on deployed lanes, and cost_usd and call_count semantics are unchanged.

How it was verified

Update 2026-10-06 08:59Z: each attempt's cost is booked to its own model (head 21ef659)

The independent review (2026-10-06 08:13Z) found that the local path booked an escalated run's whole cost to the final model as one measured call. Two signed commits on f0e1f37 fix it:

• 8307a16: the dispatch loop records what each call consumed at the same two places it adds to the run's cumulative cost, so the two cannot drift. Each attempt that reached a provider becomes its own usage call on its own model, with its own cost_usd and its own usage_source (from the effect result, no longer hardcoded measured). A call that reached no provider carries no tokens and no cost and is not booked, so the per-model sum still equals the delegation row's cost_usd (AC2).
• 21ef659: tokens follow the one definition AC1 binds to. onex metering --json reads delegation_events, which records only the deciding attempt's tokens, so only the deciding attempt's call carries tokens and earlier attempts' calls carry 0. The token columns are NOT NULL on Postgres (migration 0000) and SQLite, and the row has no token usage source, so this 0 cannot be marked as "not counted" without a schema change. It is not a measured zero.
• Tests (4 added to tests/unit/nodes/node_delegate_skill_orchestrator/test_local_dispatch_usage_by_model_day.py): an escalated run (local, then cheap_cloud) gets per-model cost 0.001 and 0.010, the sum equals the delegation row's cost, and the window's usage tokens equal metering's (33, 44); a failed run across three models books each model's own cost and the tokens equal metering's (55, 66); an attempt whose cost was not measured reads measured_cost_usd NULL with 1 unmeasured call; a call that reached no provider is not a usage call.
  • RED at f0e1f37: 3 failed, 4 passed (the no-provider test passes there too and guards against regression). RED at 8307a16 for the token rule: 2 failed, 5 passed. GREEN at 21ef659: 7 passed.
• Mutations at 21ef659: book everything to the final model again, 3 failed; book every attempt's tokens, 3 failed. At 8307a16: always mark the call measured, 1 failed; count calls that reached no provider, 1 failed (not re-run at 21ef659).
• Related suites (tests/unit/nodes/node_delegate_skill_orchestrator, the usage-by-model-day tests, and every test that calls _project_evidence directly; -m "not kafka and not integration"): f0e1f37 553 passed, 3 skipped; 21ef659 557 passed, 3 skipped, 0 failed.
• Not run: the full -m "not kafka and not integration" suite. The run sat idle at 0% CPU with no output for 16 minutes and was stopped.
• Static checks: ruff check, ruff format --check, mypy --strict on the changed src, and pre-commit run --files (54 passed, 0 failed) each exit 0.
• ARCH-DI-002 warning left as is: the check flags any HandlerX( in a changed file; no factory or registry exists to use instead, and the same warning already fires at lines that predate this PR.

Update 2026-10-06 04:48Z: dev brought in, usage reconcile moved into #3370's store-step mechanism (head f0e1f37)

The counts below supersede those for 55bdc6066.

• Merge: signed merge of dev 0614c3a58 (no conflicts) as 63850c3f8. The pins equal dev's (omnibase-infra floor >=0.38.66).
• The change this PR promised for "if fix(OMN-19968): a read through the SQLite adapter no longer takes a write lock #3370 merges first": omnimarket#3370 is on dev, so commit f0e1f37 moves the usage-by-model-day reconcile into its recorded one-time store steps.
  • New step omn20006_usage_by_model_day_measured_cost, the SQLite counterpart of migration 0002.
  • _connect calls one _apply_store_steps, which runs each pending step in its own BEGIN IMMEDIATE transaction and commits it together with its record row.
  • Read-only stores: a read-only store takes no write and is read as it is, with one warning.
  • Busy stores: a busy store raises.
  • Recorded stores: a store that has recorded the step does nothing.
  • Removed: the connect-time schema probe, the dead commit guard and its stale comment.
• Why it matters: without the move, a read-only store written before this PR raised attempt to write a readonly database on every read. That is the case fix(OMN-19968): a read through the SQLite adapter no longer takes a write lock #3370 made readable.
• Tests:
  • New tests/unit/projection/test_sqlite_usage_by_model_day_store_step_omn20006.py (6 tests). RED on 63850c3f8: 5 failed, 1 passed (the passing one is a locked-store guard). GREEN: 6/6, with fix(OMN-19968): a read through the SQLite adapter no longer takes a write lock #3370's tests 35 passed.
  • fix(OMN-19968): a read through the SQLite adapter no longer takes a write lock #3370's _step_rows helper is scoped to its own step, because the store now records two steps. No assertion was loosened.
• Mutations, all caught: reverting the move (5 fail); treating any OperationalError as read-only (the locked-store test fails); a deferred BEGIN (the racing-first-opens test fails 5 of 5 runs).
• Suites (-m "not kafka and not integration"): 1289 passed, 5 skipped, 0 failed.
• Static checks: ruff check and format, mypy --strict on the changed src and pre-commit run --files each exit 0.
• Still red only on the Sibling Release Compatibility Gate (OMN-20381), which waits on omnibase_infra 0.38.67.

At head 24ff8612a, dev e51774cd7 (signed merges of dev at 76a5ce6e2 and 0201ce57a, the 0.4.300 bump 2262ee8f0, and the fix 24ff8612a):

• A defect the dev merge exposed, now fixed. Dev's llm_call_metrics relabel (fix(OMN-19968): the llm-cost writer speaks the shared usage-source vocabulary on SQLite and Postgres #3183) runs an UPDATE on every connect, and the sqlite3 module leaves a transaction open after it until _connect commits. This PR's usage reconcile issued BEGIN IMMEDIATE inside it, so a store written by the current release failed to open ("cannot start a transaction within a transaction"), and dev's concurrent-first-open test (fix(OMN-19976): the local store carries every column the delegation exposures declare #3319) failed 4 of 5 runs. The reconcile now commits that transaction first. test_a_store_written_by_the_previous_release_opens_and_keeps_its_relabel was red before the fix and is green after; the concurrent test passes 20 of 20; removing the commit fails both (concurrent 7 of 10 red); a rollback in its place fails the relabel assertion.
• Red, then green (this PR's test files except the parity file, which needs Postgres): dev with the tests overlaid 21 failed, 18 passed; branch 39 passed.
• Mutations: 4 of 4 caught at this head: the rollup counts every call as measured (5 red), no measured call reads 0 (3 red), the local path writes no usage (2 red), no reconcile on connect (3 red).
• Related suites (112 files: every test file naming projection, usage, llm_cost, sqlite, metering or cross-node, plus tests/projection, tests/unit/projection and tests/unit/nodes/node_delegate_skill_orchestrator; -m "not kafka and not integration", the 7 files that start a local Postgres or container left out): dev 2226 passed, 5 skipped; branch 2245 passed, 5 skipped.
• scripts/check_release_identity.py --base origin/dev: exit 1 at the merge commit, exit 0 after the bump. ruff, ruff format --check and strict mypy are clean on the changed files, and the full pre-commit run passed on each commit.
• Not re-run at this head: the real-Postgres parity and usage tests on the lakshman lane. The results below are from the earlier head.

Earlier, at head 2c2ed032c:

Head 2c2ed032c (signed), rebased on dev a1ae08f1f; the focused tests were re-run there (41 passed). The contract checks were re-run on base e71ca5378: 5, 15 and 3 failed there, and 5, 33 and 3 passed at the head. The suite counts below were measured on dev 7cb86dbe8, before dev's release commit, which changed no source.

• Red, then green. At dev 8ee3d6103 with the new tests overlaid: 19 failed and 18 passed. On the branch, the focused files pass (38).

• The contract's three checks fail at the merge base and pass at the head. Base is source 7cb86dbe8 with the branch tests overlaid, the way the repo-evidence gate re-controls added evidence:

  Check	Base	Head
  AC1	5 failed	5 passed
  AC3	15 failed, 18 passed	33 passed
  AC4	3 failed	3 passed

• Mutations: 9 of 9 caught. Each undoes one behaviour, and a test goes red:

  • the fold drops usage_source;
  • the rollup counts every call as measured;
  • no measured call reads 0;
  • the event defaults to measured;
  • no reconcile on connect;
  • no tenant_column;
  • the Postgres recount sums every call;
  • the local path writes no usage;
  • measured_cost_usd is not served.

• Related suites (every test file naming usage-by-model-day, the SQLite store, the local dispatch port, the topic map or bus_backed, plus tests/unit/projection, tests/unit/nodes/node_delegate_skill_orchestrator and tests/projection):

  • branch: 2100 passed, 53 skipped;
  • dev 7cb86dbe8: 2082 passed, 52 skipped;
  • the difference is the 18 new tests and the one new real-Postgres test, which skips without a database.

• Real Postgres, on the lakshman lane on .201 (port 45436), in throwaway schemas, 2026-10-04 about 14:28Z:

  • the two usage real-Postgres tests pass;
  • the parity test passes for usage_by_model_day (with every usage source) and metering_summary.
  • Two parity cases fail, delegation_events and llm_call_metrics, with invalid input value for enum usage_source_type: "MISSING". They fail the same way at dev 8ee3d6103 against the same database, so they are not caused by this change.

• Repo gates:

  • ruff check, ruff format --check and mypy src/omnimarket/ --strict are clean (4111 files);
  • tests/ci/test_repo_evidence_caller_workflow.py: 2 passed;
  • scripts/check_release_identity.py --base origin/dev: exit 0;
  • the registry-projection reconcile gate: OK;
  • the full pre-commit run on commit passed.

• The reader gate, run locally. omnibase_infra's exposure-reader-coverage gate over this branch's contracts plus omnimarket#3368's metering-summary contract:

  • with omnidash's branch registry: OK: 22 served bus_backed exposure(s), 0 without a reader;
  • with omnidash dev's registry: FAIL, naming exactly usage-by-model-day.v1 and metering-summary.v1.

• NULL parity. An independent review noted that the parity helper drops None values, so a NULL measured cost was never compared. The parity test now gives one day and model no measured call and compares (measured_cost_usd, unmeasured_call_count) per key explicitly. It passes on the lakshman lane at 16:12Z. With the "no measured call reads 0" mutant it fails: (Decimal('0'), 3) == (None, 3).

• Not run: the lab readback (AC1's window sum against onex metering --json, and AC2's per-model sum against the Overview spend on a fresh HOME). It needs this head installed on .201 and is the next step.

Failure paths

• No answer, or a failed write: the local usage write runs in its own try, and a raise is logged. Tested by test_a_failing_usage_write_never_breaks_the_delegation: the delegation and call-metrics rows still land and no usage row is written.
• Incomplete row: a key with no measured call serves NULL, not 0 (test_unmeasured_only_aggregate_has_typed_null_cost). A call that does not say how its cost was obtained is unmeasured.
• Size: n/a. No output crosses a transport cap.

Open defects

none. Risks from the independent review, not fixed here and named so the merger sees them:

• Known limit (follow-up): onex metering counts only the deciding attempt's tokens, so tokens spent by failed or escalated-from attempts are not counted anywhere. The fix belongs in the metering summary (one definition), so the CLI and the page move together. It is a follow-up and is not done here.
• The llm_call_metrics write from omnimarket#3069 still books an escalated run's whole cost to the final model. Out of this PR.
• A bus llm-call-completed event carries no usage_source, so it folds as unknown (Not measured) while llm_cost defaults it to measured. No deployed emitter exists yet.
• Day-and-model rows stored before 0002 read unmeasured_call_count 0 until recounted. 0 such rows on the lab and on dev today.
• A store opened read-only before any writer connects still lacks the new columns. The same was already true at dev, through the missing projection_cursor.

Local gates

• No gate was bypassed — no --no-verify, no SKIP=, no --no-gpg-sign, no core.hooksPath override. Every pre-commit and pre-push hook ran.
• Any hook that failed was fixed at the input, not worked around. Two hooks failed and each input was fixed:
  • release-identity: the version was bumped, to 0.4.294 after dev's v0.4.293 release, and again to 0.4.300 after dev's v0.4.299 release;
  • registry-projection reconcile: it read a cursor trigger's MAX(...) FROM usage_by_model_day as a registry read, so the cursor now comes from its own sequence table, as on Postgres.

Not in this change

• The omnidash Usage page, which reads these columns and is the declared reader, and the omnibase_infra vendoring of migration 0002 are separate PRs on this ticket.

• Merge order:

  1. omnibase_infra vendoring;
  2. omnidash;
  3. this PR, after an omnibase_infra release that contains 0002.

  omnidash goes before this one so the registry reader is in place before the opt-out deletion reaches omnibase_infra's pinned omnimarket.

• Evidence for AC1, AC3 and AC4 is in onex_change_control#12823: test_passes checks that fail at the merge base and pass at this head.

Overlap-Reviewed: #3371 — omnimarket#3371 (OMN-19976) moves _ensure_columns under the write lock; this PR adds the usage tables' own reconcile in _connect and does not touch _ensure_columns. Both also set the version to 0.4.300 in pyproject.toml and uv.lock, the same value. Both are needed; whichever merges second rebases.
Overlap-Reviewed: #3370 — omnimarket#3370 (OMN-19968) makes a read stop taking a write lock and runs one-time data steps, including the llm_call_metrics relabel, only on a write. This PR's usage reconcile commits the transaction that relabel leaves open in today's _connect; once #3370 moves the relabel, that commit finds no open transaction and does nothing. #3370 merged first, so this PR moved the usage reconcile into #3370's store-step mechanism (commit f0e1f37, see the 2026-10-06 update). Both set the version to 0.4.300.
Overlap-Reviewed: #3403 — omnimarket#3403 is the post-release bot bump of pyproject.toml and uv.lock to 0.4.300. This PR sets the same value because its src change needs it before #3403 lands; whichever merges second has no version change left.
Overlap-Reviewed: #3402 — omnimarket#3402 is the earlier post-release bot bump, whose head also reads 0.4.300 in pyproject.toml and uv.lock. Same value as this PR, so whichever merges second has no version change left.

Overlap-Reviewed: #3406 — omnimarket#3406 (OMN-20009): both PRs extend tests/test_omn19968_writer_store_parity.py to apply a follow-up migration (this PR usage-by-model-day 0002, #3406 metering-summary 0002) and refactor the same lines differently, so git merge-tree conflicts there (2026-10-05 06:59Z). Whichever merges second keeps both, with each table applying its own follow-up migrations. sqlite_database.py merges cleanly.

Overlap-Reviewed: #3407 — omnimarket#3407 (OMN-19977) also edits _project_evidence in port_local_delegation_dispatch.py: it records the terminal time inside the evidence try and refreshes metering after the except; this PR adds the usage fold inside the try. git merge-tree is clean (06:59Z).

Evidence-Ticket: OMN-20006
Evidence-Source: OCC#12823

[github-actions]

⚠️ Architectural Review — WARNINGS

Errors: 0
Warnings: 1

What this checks

Rule	Description
ARCH-TOPIC-001	No hardcoded Kafka topic strings in handler code
ARCH-DI-001	No event_bus=None bypass
ARCH-DI-002	No direct Handler instantiation outside workflow_runner/adapters
ARCH-DI-003	No reinvented DI containers
ARCH-TOPIC-002	contract.yaml topics follow `onex.{cmd

---

ARCH-DI-002: Direct Handler instantiation (warning)\n\n```\nsrc/omnimarket/nodes/node_delegate_skill_orchestrator/ports/port_local_delegation_dispatch.py:995: self._effect_handler = effect_handler or HandlerLlmDelegationCall()

src/omnimarket/nodes/node_delegate_skill_orchestrator/ports/port_local_delegation_dispatch.py:996: self._projection_handler = projection_handler or HandlerProjectionDelegation(
src/omnimarket/nodes/node_delegate_skill_orchestrator/ports/port_local_delegation_dispatch.py:3421: HandlerProjectionLlmCost().project(call_event, self._evidence_db)
src/omnimarket/nodes/node_delegate_skill_orchestrator/ports/port_local_delegation_dispatch.py:3513: HandlerProjectionUsageByModelDay().handle(usage_event),
src/omnimarket/nodes/node_projection_usage_by_model_day/handlers/handler_usage_by_model_day_writer.py:95: self._derive = HandlerProjectionUsageByModelDay()\n```\n\nHandler classes should be wired only in workflow_runner.py or adapter modules. Direct instantiation elsewhere bypasses the contract system.\n\n

Static architectural lint — no model inference (OMN-14176).

[github-actions]

✅ Hostile Reviewer — PASSED

Blocking findings (critical/error): 0
Total findings: 16
Models succeeded: qwen3-review,gpt-oss-review

---

Gate semantics

Verdict	Meaning	Blocks merge?
passed	>=2 models succeeded, no critical/error findings	No
blocked	CRITICAL or ERROR findings found	Yes
degraded	Fewer than 2 models succeeded (infra unavailable/timeout)	Yes (OMN-15110)

Powered by omniintelligence.review_pairing.cli_review — node-based adversarial review via HandlerLlmCliSubprocess (OMN-8524)

[onexbot-occ-writer]

OCC autobind did not rebind this PR.

This PR's evidence-source stamp line names OCC#12823, which is merged. A second companion, OCC#12830, was minted for this PR, but a merged companion is settled evidence — the merged-companion gate may already have passed against it — so the stamp was left pointing at OCC#12823 and nothing was written.

To clear this: OCC#12830 is redundant and can be closed. If the binding really should move, change the stamp by hand.

Reported by occ_companion_emitter (OMN-18089).

[Patel230]

Close-out for OMN-20006: merged 2026-10-07 08:29Z as 56b1b4b949, now the head of dev.

• Landed: usage-by-model-day.v1 keeps measured cost apart from estimated and unrecorded cost, through the SQLite store step and the projection migration vendored by omnibase_infra#4559 (released in 0.38.67). The pin moved to 0.38.67, and the clean-install probe passes --json to onex delegate.
• Last fixes before merge (f1501de66): both CodeQL review threads (alerts 1545 and 1546) were fixed in code, not dismissed. The store step now catches only sqlite3.OperationalError for the read-only case and keeps a bare rollback-and-raise for everything else; a new test fails with database is locked if that rollback is removed.
• Checks at the merged head: CI Summary and Release Staleness green (after v0.4.302); the one red context, occ-autobind / outcome, is non-required and a no-op ("No commits between": its evidence companion OCC#12823 had already merged).
• Left on the ticket: the AC3/AC4 lab readback with an unmeasured-cost run, the evidence contract's test paths (now on dev), and the AC boxes.

Thank you!

[Patel230]

AC3 and AC4 lab readback for OMN-20006, on the merged code: a fresh install on the .201 lab of omnimarket dev d761f582f8 (carries this PR, 56b1b4b949) and omnidash dev 70c1a23032 (carries omnidash#358), 2026-10-07 09:28–09:48Z.

How the unmeasured run was made. On the local delegate path every answered call is recorded as measured, so a real local run cannot be unmeasured; unmeasured usage comes from the hosted path. After two real local delegations, one call with usage_source=unknown and an unmeasured cost of 0.04 was folded through the installed code's own usage-by-model-day writer, under the install's tenant. The 0.04 is the control: had the page counted it, measured cost would read $0.04.

	before (2 measured runs)	after (+1 unmeasured call)
Usage page row	2026-10-07 · Qwen3.8-27B · 524 · 92 · $0.00 · none · 2	2026-10-07 · Qwen3.8-27B · 524 · 92 · $0.00 · 1 unmeasured · 3
Overview Spend	$0.0000	$0.0000
Overview Measured runs	2 of 2	2 of 2
served usage-by-model-day.v1	calls 2, unmeasured 0, measured cost 0.0	calls 3, unmeasured 1, measured cost 0.0, cost 0.04

• AC3: the unmeasured run is counted beside the totals ("1 unmeasured") and excluded from them: measured cost stays $0.00, not $0.04, and the Overview spend does not move.
• AC4: the unmeasured run shows as the typed "1 unmeasured", never $0 or a blank; the $0.00 is the two local runs' real measured zero.
• Both page reads HTTP 200, no alerts, no page errors (headless Chrome, 1440 wide).

With the 2026-10-06 readback (AC1, AC2), all four criteria now have a lab readback on merged code.

1 sources read: the two served exposures before and after, the CLI metering JSON, both page captures · 2 live at 09:48Z · 3 weakest claim checked: that the page reads the new row and not a stale one: the Unmeasured column moved none → 1 while measured cost stayed $0.00.

Thank you!