Skip to content

OMN-20794: lab alarm raises reach the ledger; failure-rate alerter covers three more repositories - #4779

Merged
jonahgabriel merged 4 commits into
devfrom
jonah/omn-20794-lab-alarm-ledger-row
Oct 9, 2026
Merged

jonahgabriel merged 4 commits into
devfrom
jonah/omn-20794-lab-alarm-ledger-row

Conversation

@jonahgabriel

@jonahgabriel jonahgabriel commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

This change addresses ticket OMN-20794 within epic OMN-20769. On 2026-10-09, the lab alarm raised container_restarts for omninode-runtime-effects at 10:25:05Z and delivered it to Slack only. The ledger got no row, and the orchestrator learned of the crash at 10:40Z from a stale projection.

scripts/lab_alarm.py now appends one FRICTION row (lane=lab-alarm) per newly raised alarm through the ledger's own append, onex-ledger, naming the condition, subject and evidence. The append does not wait on Slack consent. No new ssh or HTTP read was added. A failed append is kept in the state file, retried each tick, and makes the run exit as an alarm. A subject that clears and re-raises inside 360 minutes appends no second row. Additionally, omnibase_internal, omniclaude-internal and omnicursor are added to fleet_repos and scheduled_actions_repos in config/runner_routing_policy.yaml, to the App token repositories list and to the --scheduled-repo arguments of pr-ci-zombie-detector.yml, whose evaluating job runs on ubuntu-latest. The h201 reporter selects fleet_repos minus that share, so it does not evaluate them.

Ticket: OMN-20794 (epic OMN-20769).

Lab: host=h200 lane=mon-alarm-ledger-5d21 command=onex_lab_run.py --repo omnibase_infra --ref jonah/omn-20794-lab-alarm-ledger-row --host h200 -- uv run pytest tests/ci/test_lab_alarm_omn18867.py tests/ci/test_fleet_repos_include_internal_omn20794.py -k "lab_alarm_appends_ledger_row or crash_loop_reaches_ledger_replay or fleet_repos_include_internal" -q -rP observed: 10 passed, exit 0. The replay of the 2026-10-09 crash window (hourly ticks 09:25:05Z, 10:25:05Z, 11:25:05Z through run_once with a recording appender, i.e. a dry run that writes no ledger) appended exactly one row, stamped before 10:30Z: 2026-10-09T10:25:05Z | FRICTION | lane=lab-alarm | existing=OMN-20769 | cost=~0 lane-minutes at raise (accrues until the alarm clears) | condition=container_restarts | subject=omninode-runtime-effects | raised_at=2026-10-09T10:25:05+00:00 | detail=container omninode-runtime-effects has restarted 7 times, past its declared bound of 5; status restarting | evidence=restart counts read: omninode-runtime=0/5 running, omninode-runtime-effects=7/5 restarting | lab alarm raised container_restarts on omninode-runtime-effects; the detail and evidence cells carry what it read. The wider set (this ticket's tests plus the neighbouring alerter, routing-audit and zombie-detector suites, 9 files) ran on h200 at the same sha: 319 passed, 1 skipped (an existing brew-python skip), exit 0. head=385633230e640e676f9a292cfad45d11af7ee2eb

Acceptance criteria

AC Falsifier Evidence
AC1: a raised alarm appends exactly one ledger row; a re-raise inside the dedupe window appends none uv run pytest tests -q -k lab_alarm_appends_ledger_row test_lab_alarm_appends_ledger_row_once_per_raise raises, holds, recovers and re-raises inside the 360-minute window (one row), then re-raises after it (second row). Three sibling tests cover: no Slack consent still appends, one row per alarm, a failed append is loud and retried once. Ran RED (ImportError on the missing appender) before the implementation, GREEN on h200 at the head above.
AC2: replaying 10:16Z to 10:40Z raises the crash-loop alarm into the ledger before 10:30Z uv run pytest tests -q -k crash_loop_reaches_ledger_replay test_crash_loop_reaches_ledger_replay drives three real run_once ticks; the one row is stamped 2026-10-09T10:25:05Z, quoted in the Lab line.
AC3: the three repositories are in the GitHub-hosted share and the evaluator selects their scheduled runs there, not on h201 uv run pytest tests/ci -q -k fleet_repos_include_internal Six tests in tests/ci/test_fleet_repos_include_internal_omn20794.py: policy fleet_repos and scheduled_actions_repos, the workflow's --scheduled-repo arguments and App token scope, the host selection (--repos-from-policy) touching none of the three, and the workflow's own argv evaluating each as scheduled-only.

Evidence and limits stated plainly

  • The row was also appended with the real onex-ledger to a scratch ledger (not the canonical one) and accepted, and friction_guard.refusal_for_payload returned no refusal for it. The test file spells the guard's three obligations locally, because importing the other repository would skip where its clone is absent.
  • The restart counts in the replay (0, 7, 19) are reconstructions. The established facts are the tick time (10:25:05Z), the container and its manifest bound of 5.
  • The knowledge-base-internal plan (PR feat: DelegationIntentBridge completes delegation chain end-to-end [OMN-7604] #1180) was not readable from this host, so sections 3, 5 and T2 were not re-read; the brief's excerpt was the design input.
  • "The App token reads them" was not probed live (no GitHub reads were allowed). The basis is the policy's existing note that the onexbot installation selection is all, plus the mint step now requesting the three repositories. An unreadable repository renders as a loud source-unreadable finding and a red exit, so the first scheduled run settles it. That omnicursor and omniclaude-internal exist under the owner was not verified here either.
  • ledger_append_command is uv run --project ${OMNI_HOME}/../omnibase_internal onex-ledger. The installed launch agent's PATH carries ~/.local/bin and /opt/homebrew/bin; that uv resolves there on the operator Mac was not checked from this host. A failure would show as the loud ledger ... FAILED line and a non-zero exit, not as a silent miss.
  • No deploy, restart or credential change was made.

Evidence-Ticket: OMN-20794
Evidence-Source: OCC#13450

scripts/lab_alarm.py appends one FRICTION row (lane=lab-alarm, existing=OMN-20769, condition, subject, raised_at, detail, evidence) through the ledger's own append (onex-ledger, fixed argv, no shell). No new ssh or HTTP read. The append does not wait on Slack consent. A failed append is kept in the state file, retried each tick, and makes the run exit as an alarm. A subject that clears and re-raises inside ledger_dedupe_window_minutes (360) appends no second row.

config/lab_alarm.json gains ledger_append_command, ledger_row_ticket, ledger_dedupe_window_minutes.

config/runner_routing_policy.yaml: omnibase_internal, omniclaude-internal, omnicursor added to fleet_repos and scheduled_actions_repos.

.github/workflows/pr-ci-zombie-detector.yml: the three repositories added to the App token repositories list and to the --scheduled-repo arguments of the job that runs on ubuntu-latest.

Tests added: lab_alarm_appends_ledger_row, crash_loop_reaches_ledger_replay (replay of the 2026-10-09 10:25:05Z tick), fleet_repos_include_internal.
@jonahgabriel
jonahgabriel enabled auto-merge October 9, 2026 16:49
jonahgabriel pushed a commit to OmniNode-ai/onex_change_control that referenced this pull request Oct 9, 2026
…13450)

* evidence(OMN-20794): author OCC companion for OmniNode-ai/omnibase_infra#4779

OCC companion by node_pr_lifecycle_fix_effect (OMN-13317 F1 / OMN-13990 / OMN-14285). Product PR head 385633230e640e676f9a292cfad45d11af7ee2eb.

* evidence(OMN-20794): self-bind OCC#13450 + rebind contract_sha256

---------

Co-authored-by: omnimarket-bot <bot@omninode.ai>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hostile Reviewer — adversarial findings (OMN-17492)

Models succeeded: qwen3-review, local-studio-planner
Models failed: none
New finding threads: 2
Deduped (already posted on this PR): 0
Below quorum (one model only, reported not threaded): 11
Nit-level findings suppressed: 1

The model is the FINDER, never the gate: merge is gated only by the
deterministic Hostile Review Thread Gate, which blocks while
hostile-reviewer threads are unresolved. Resolve each thread after
addressing (or rejecting, with a reply) its finding.

Below quorum: 11 finding(s) raised by one model only (OMN-18479)

These are reported and NOT dropped, but they get no thread and do not block: a single model's finding no other model reproduced is not evidence enough to stop a merge. Read them; act on them if they are right.

  • [CRITICAL] scripts/lab_alarm.py (local-studio-planner) — Unvalidated Ledger Row Injection via Alarm Detail/Evidence | The render_ledger_row function sanitizes pipe characters and newlines in the detail and evidence fields using _ledger_cell. However
  • [MAJOR] scripts/lab_alarm.py, record_alarms_in_ledger (qwen3-review) — Ledger dedupe window pruned before pending rows are processed, causing duplicate appends | In record_alarms_in_ledger, the state.ledger_last_row dict is pruned of entries older than window befor
  • [MAJOR] config/lab_alarm.json, ledger_append_command (qwen3-review) — Ledger append command uses shell-expanded env var in argv, enabling command injection | The ledger_append_command in lab_alarm.json is `["uv", "run", "--project", "${OMNI_HOME}/../omnibase_interna
  • [MAJOR] scripts/lab_alarm.py, render_ledger_row (qwen3-review) — Ledger row cell sanitization does not prevent injection into the ledger's own grammar | _ledger_cell replaces | with / and collapses whitespace, but the ledger row is a pipe-delimited format. Th
  • [MAJOR] scripts/lab_alarm.py (local-studio-planner) — Insecure Command Construction in Ledger Appender | The make_ledger_appender function constructs the command line by concatenating the prefix, ledger path, and arguments. While it avoids shell interp
  • [MAJOR] scripts/lab_alarm.py (local-studio-planner) — Race Condition in Ledger Dedupe Window Logic | The record_alarms_in_ledger function uses a dedupe window based on state.ledger_last_row. It filters out alarms that were raised within the last `win
  • [MINOR] scripts/lab_alarm.py, run_once (qwen3-review) — Ledger append is synchronous and blocks the alarm run | The ledger append is performed synchronously in record_alarms_in_ledger, which is called before the Slack posting. If the ledger tool is slow
  • [MINOR] tests/ci/test_lab_alarm_omn18867.py (qwen3-review) — No test for ledger append failure with a non-zero exit code from the ledger tool | The test test_lab_alarm_ledger_appender_runs_the_ledger_tool_without_a_shell tests a refusing runner that returns
  • [MINOR] config/lab_alarm.json, ledger_append_command (qwen3-review) — Ledger append command is hardcoded to uv run, which may not be available in all environments | The ledger_append_command is `["uv", "run", "--project", "${OMNI_HOME}/../omnibase_internal", "onex-l
  • [MINOR] scripts/lab_alarm.py (local-studio-planner) — Inefficient State Loading and Saving for Large Alarms | The ModelAlarmState class loads and saves the entire state, including ledger_pending and ledger_last_row, to a JSON file on every tick. If
  • [MINOR] scripts/lab_alarm.py (local-studio-planner) — Tight Coupling Between Alarm and Ledger Logic | The lab_alarm.py script now has direct knowledge of the ledger's row format, lane, and cost structure. This creates a tight coupling between the alarm

Findings demoted from threads (anchor rejected)

  • [MINOR] hostile-reviewer (qwen3-review)

    Test for ledger row cell sanitization is incomplete | test_lab_alarm_ledger_row_cells_cannot_break_the_row tests that pipes and newlines are sanitized, but does not test other characters that could break the ledger's parser, such as backslashes, quotes, or Unicode characters that might be misinterpreted. The … truncation character is not tested for edge cases (e.g., what if the evidence is exactly 600 chars, or 601 chars?). | Evidence: def test_lab_alarm_ledger_row_cells_cannot_break_the_row(tmp_path: P

    Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

  • [MINOR] hostile-reviewer (local-studio-planner)

    Insufficient Coverage for Ledger Appender Edge Cases | The test test_lab_alarm_ledger_appender_runs_the_ledger_tool_without_a_shell covers basic success, refusal, and timeout scenarios. However, it does not test edge cases such as empty row content, rows with special characters that might cause issues in the command line (even if sanitized), or interactions with the ledger tool's specific error messages. The mock runner is simplistic and may not accurately reflect the behavior of the actual `onex-ledger

    Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

✅ Hostile Reviewer — REVIEWED

Critical findings: 1
Major findings: 5
Total findings: 15
Models succeeded: qwen3-review,local-studio-planner
Models unavailable: none

Action required: findings were posted as review threads. Address or reject each one, then resolve its thread — the Hostile Review Thread Gate fails while hostile-reviewer threads are unresolved (OMN-17492).


Semantics (OMN-17492 — the model finds, thread resolution gates)

Surface Meaning Blocks merge?
Review threads Per-finding, posted by the reviewer No (informational)
Hostile Review Thread Gate Deterministic: unresolved hostile-reviewer threads exist Fails until resolved (not yet a required context)
degraded verdict Fewer than 2 models succeeded (infra) Fails this job with a named reason

Powered by omniintelligence.review_pairing.cli_review — multi-model adversarial review: qwen3-review (Qwen3.8-27B), local-studio-planner (Qwen3.6-35B-A3B) (OMN-8468/OMN-8524/OMN-17492)

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hostile Reviewer — adversarial findings (OMN-17492)

Models succeeded: qwen3-review, local-studio-planner
Models failed: none
New finding threads: 4
Deduped (already posted on this PR): 0
Below quorum (one model only, reported not threaded): 7
Nit-level findings suppressed: 0

The model is the FINDER, never the gate: merge is gated only by the
deterministic Hostile Review Thread Gate, which blocks while
hostile-reviewer threads are unresolved. Resolve each thread after
addressing (or rejecting, with a reply) its finding.

Below quorum: 7 finding(s) raised by one model only (OMN-18479)

These are reported and NOT dropped, but they get no thread and do not block: a single model's finding no other model reproduced is not evidence enough to stop a merge. Read them; act on them if they are right.

  • [CRITICAL] scripts/lab_alarm.py (local-studio-planner) — Unvalidated Environment Variable Injection in Ledger Command | The ledger_append_command configuration is populated by expanding environment variables via expand_env. The resulting list of strings
  • [MAJOR] config/lab_alarm.json (qwen3-review) — Environment variable expansion in ledger command allows path injection | The ledger_append_command in config/lab_alarm.json contains ${OMNI_HOME}/../omnibase_internal which is expanded via `expa
  • [MAJOR] scripts/lab_alarm.py (local-studio-planner) — Sensitive Data Exposure in Ledger Row Evidence | The render_ledger_row function includes the detail and evidence fields from the alarm in the ledger row. These fields often contain raw output fr
  • [MAJOR] tests/ci/test_lab_alarm_omn18867.py (local-studio-planner) — Incomplete Test Coverage for Ledger Append Error Handling | The test test_lab_alarm_failed_ledger_append_is_loud_and_retried mocks the appender to raise LedgerAppendError. However, it does not ver
  • [MINOR] tests/ci/test_lab_alarm_omn18867.py (qwen3-review) — Missing test for ledger append failure with a valid appender | The test test_lab_alarm_failed_ledger_append_is_loud_and_retried uses a broken appender that always raises LedgerAppendError. However
  • [MINOR] scripts/lab_alarm.py (qwen3-review) — No validation that the ledger path exists before appending | The make_ledger_appender function does not check whether the ledger path exists before attempting to append. If the path does not exist,
  • [MINOR] scripts/lab_alarm.py (local-studio-planner) — Tight Coupling Between Alarm Logic and Ledger Format | The render_ledger_row function hardcodes the ledger row format (pipe-separated key-value pairs) and specific field names (lane, existing, `

Findings demoted from threads (anchor rejected)

  • [MAJOR] hostile-reviewer (qwen3-review)

    Ledger dedupe window logic may suppress legitimate re-alarms | The record_alarms_in_ledger function uses state.ledger_last_row to suppress re-alarms within the dedupe window. However, the window is based on the time the row was last written, not when the condition was last observed. If a condition flaps (clears and re-raises) within the window, the re-raise is suppressed. This is intended behavior, but the window is 360 minutes (6 hours). If a condition clears and re-raises after 5 hours, it will be sup

    Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

  • [MAJOR] hostile-reviewer (local-studio-planner)

    Incorrect Deduplication Logic for Retried Alarms | The record_alarms_in_ledger function processes state.ledger_pending (failed writes from previous ticks) before checking new alarms against state.ledger_last_row. If an alarm was raised, failed to write, and then cleared and re-raised within the dedupe window, the retry logic will still attempt to write the original alarm's row because it is in ledger_pending. This violates the intent of the dedupe window, which should suppress re-raises. The code

    Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

  • [MINOR] hostile-reviewer (qwen3-review)

    Ledger append is synchronous and may block the alarm run | The make_ledger_appender function runs the ledger append synchronously with a timeout of 120 seconds. If the ledger tool is slow or hangs, the alarm run will be blocked for up to 120 seconds. This could delay the delivery of the alarm to Slack and other downstream systems. The ledger append is a critical path for the alarm, but it should not block the entire run. Consider running the ledger append in a background thread or using a non-blocking mec

    Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

  • [MINOR] hostile-reviewer (local-studio-planner)

    Inefficient State Loading and Saving for Large Pending Queues | The ModelAlarmState.load and save methods serialize the entire ledger_pending dictionary to/from JSON on every tick. If a system experiences prolonged outages causing many failed ledger appends, this dictionary can grow significantly, leading to increased I/O latency and memory usage during state persistence. | Evidence: scripts/lab_alarm.py:2204-2236
    ledger_pending={
    str(k): {str(f): str(x) for f, x in v.items

    Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hostile Reviewer — adversarial findings (OMN-17492)

Models succeeded: qwen3-review, local-studio-planner
Models failed: none
New finding threads: 2
Deduped (already posted on this PR): 0
Below quorum (one model only, reported not threaded): 10
Nit-level findings suppressed: 1

The model is the FINDER, never the gate: merge is gated only by the
deterministic Hostile Review Thread Gate, which blocks while
hostile-reviewer threads are unresolved. Resolve each thread after
addressing (or rejecting, with a reply) its finding.

Below quorum: 10 finding(s) raised by one model only (OMN-18479)

These are reported and NOT dropped, but they get no thread and do not block: a single model's finding no other model reproduced is not evidence enough to stop a merge. Read them; act on them if they are right.

  • [CRITICAL] scripts/lab_alarm.py (qwen3-review) — Dedupe window check is dead code due to state mutation order | In record_alarms_in_ledger, the dedupe check if alarm.key in state.ledger_last_row is evaluated before the pending work is processed.
  • [CRITICAL] scripts/lab_alarm.py (local-studio-planner) — Unescaped user input in ledger row cells enables injection | The render_ledger_row function uses _ledger_cell to sanitize subject, detail, and evidence fields. The sanitization logic replaces pipe
  • [MAJOR] config/lab_alarm.json, scripts/lab_alarm.py (qwen3-review) — Environment variable expansion in ledger command allows path injection | The ledger_append_command in config/lab_alarm.json contains ${OMNI_HOME}/../omnibase_internal. The expand_env function
  • [MAJOR] scripts/lab_alarm.py (qwen3-review) — Ledger append timeout is hardcoded and may cause alarm delays | The make_ledger_appender function uses a default timeout_seconds of 120.0 seconds. The ledger append is called synchronously in `run
  • [MAJOR] tests/ci/test_lab_alarm_omn18867.py (qwen3-review) — Missing test for ledger append failure followed by condition clear and re-raise | The test test_lab_alarm_failed_ledger_append_is_loud_and_retried covers the case where the append fails and is retri
  • [MAJOR] scripts/lab_alarm.py (local-studio-planner) — Hardcoded timeout in ledger append command | The make_ledger_appender function hardcodes the --timeout argument to 60s when constructing the argv for the ledger tool. This value is not configura
  • [MINOR] scripts/lab_alarm.py (qwen3-review) — Ledger append is not idempotent | The ledger append is not idempotent. If the same alarm is raised and the append succeeds, but the state file is not saved (e.g., due to a crash between the append and
  • [MINOR] scripts/lab_alarm.py (qwen3-review) — No validation of ledger path existence | The make_ledger_appender function does not check if the ledger_path exists or is writable. If the path is invalid, the ledger tool will fail, and the error
  • [MINOR] scripts/lab_alarm.py (local-studio-planner) — Unbounded growth of ledger_pending state | The ledger_pending dictionary in ModelAlarmState stores alarms that failed to be written to the ledger. These entries are retried on every tick until the
  • [MINOR] tests/ci/test_lab_alarm_omn18867.py (local-studio-planner) — Test coverage for ledger append failure retry logic is incomplete | The test test_lab_alarm_ledger_append_failed_ledger_append_is_loud_and_retried verifies that a failed append is retried. However,

Findings demoted from threads (anchor rejected)

  • [MAJOR] hostile-reviewer (qwen3-review)

    Ledger row rendering does not escape the summary cell | The render_ledger_row function joins cells with |. The last cell is a summary string: f"lab alarm raised {alarm.condition.value} on {_ledger_cell(alarm.subject)}; the detail and evidence cells carry what it read". This cell is not passed through _ledger_cell, so it is not flattened or limited. If alarm.condition.value or alarm.subject contains a pipe character, it would break the row structure. However, _ledger_cell is used for `subject

    Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

  • [MAJOR] hostile-reviewer (local-studio-planner)

    Ledger dedupe window logic may suppress valid new alarms | The record_alarms_in_ledger function uses state.ledger_last_row to track the timestamp of the last written row for each alarm key. It filters out alarms that re-raise within window of a previously written row. However, if an alarm is cleared and then re-raises after the window has passed, it is treated as a new alarm. The issue arises if the alarm key is not unique enough or if the dedupe logic does not account for the fact that an alarm might

    Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hostile Reviewer — adversarial findings (OMN-17492)

Models succeeded: qwen3-review, local-studio-planner
Models failed: none
New finding threads: 0
Deduped (already posted on this PR): 0
Below quorum (one model only, reported not threaded): 14
Nit-level findings suppressed: 1

The model is the FINDER, never the gate: merge is gated only by the
deterministic Hostile Review Thread Gate, which blocks while
hostile-reviewer threads are unresolved. Resolve each thread after
addressing (or rejecting, with a reply) its finding.

Below quorum: 14 finding(s) raised by one model only (OMN-18479)

These are reported and NOT dropped, but they get no thread and do not block: a single model's finding no other model reproduced is not evidence enough to stop a merge. Read them; act on them if they are right.

  • [CRITICAL] scripts/lab_alarm.py (local-studio-planner) — Untrusted data passed to subprocess without shell escaping | The make_ledger_appender function constructs a command line by concatenating the user-provided row string directly into the argv list
  • [MAJOR] config/lab_alarm.json, scripts/lab_alarm.py (ModelAlarmConfig.load) (qwen3-review) — Shell variable expansion in ledger_append_command allows command injection | The ledger_append_command in config/lab_alarm.json contains ${OMNI_HOME}/../omnibase_internal. The `ModelAlarmConfig.
  • [MAJOR] scripts/lab_alarm.py (record_alarms_in_ledger) (qwen3-review) — ledger_last_row pruning uses wall-clock time, not the alarm's raised_at | In record_alarms_in_ledger, the state.ledger_last_row dict is pruned by comparing `now_dt - datetime.fromisoformat(stamped
  • [MAJOR] scripts/lab_alarm.py (ModelAlarmState, record_alarms_in_ledger) (qwen3-review) — ledger_pending entries are not pruned, causing unbounded state growth | The state.ledger_pending dict stores failed ledger appends for retry. If a ledger append keeps failing (e.g., the ledger tool
  • [MAJOR] scripts/lab_alarm.py (local-studio-planner) — Dedupe window logic allows duplicate rows for different conditions on same subject | The record_alarms_in_ledger function uses alarm.key as the deduplication key. If alarm.key is derived from th
  • [MAJOR] tests/ci/test_lab_alarm_omn18867.py (local-studio-planner) — Test relies on internal implementation details of ledger row parsing | The test test_lab_alarm_appends_ledger_row_once_per_raise and others parse the ledger row string using _ledger_cells, which s
  • [MINOR] scripts/lab_alarm.py (record_alarms_in_ledger) (qwen3-review) — ledger_last_row pruning iterates over all keys on every tick | The pruning of state.ledger_last_row in record_alarms_in_ledger iterates over all keys in the dict on every tick. If the number of di
  • [MINOR] tests/ci/test_lab_alarm_omn18867.py (qwen3-review) — No test for ledger_pending persistence across state file reloads | The ledger_pending dict is saved to and loaded from the state file. There is no test that verifies a pending entry survives a state
  • [MINOR] scripts/lab_alarm.py (ModelAlarmRun, exit_code) (qwen3-review) — exit_code checks for '; FAILED:' in run.ledger, but the summary format is fragile | The exit_code function checks "; FAILED:" in run.ledger to detect ledger append failures. The summary string is
  • [MINOR] tests/ci/test_lab_alarm_omn18867.py (test_crash_loop_reaches_ledger_replay) (qwen3-review) — Crash-loop replay test uses hardcoded tick times that may not align with the hourly timer | The test_crash_loop_reaches_ledger_replay test uses tick times 09:25:05, 10:25:05, and 11:25:05. The
  • [MINOR] scripts/lab_alarm.py (run_once) (qwen3-review) — Ledger append is a side effect of run_once, not a separate concern | The ledger append is embedded in run_once, which is a pure evaluation function. This couples the alarm evaluation to the ledger w
  • [MINOR] scripts/lab_alarm.py (local-studio-planner) — State file grows unbounded with ledger_pending entries | The ledger_pending dictionary in ModelAlarmState stores failed append attempts. These entries are only removed if the append succeeds. If t
  • [MINOR] scripts/lab_alarm.py (local-studio-planner) — Hardcoded timeout values in ledger appender | The make_ledger_appender function hardcodes the timeout for the ledger tool to 60s and the overall runner timeout to 120.0 seconds. These values are
  • [MINOR] scripts/lab_alarm.py (local-studio-planner) — Potential race condition in state file updates | The ModelAlarmState.load and save methods read and write a JSON file. If the alarm script is run concurrently (e.g., by multiple cron jobs or if th

@jonahgabriel
jonahgabriel added this pull request to the merge queue Oct 9, 2026
Merged via the queue into dev with commit 9f2b814 Oct 9, 2026
170 checks passed
@jonahgabriel
jonahgabriel deleted the jonah/omn-20794-lab-alarm-ledger-row branch October 9, 2026 20:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant