Repository navigation
fix(OMN-14531): data_flow_sweep fails closed on lag-probe failure + drops hardcoded LAN IP - #1746
Conversation
…rops hardcoded LAN IP Two structural-blindness defects in node_data_flow_sweep, the detector backing onex:data_flow_sweep (OMN-14531 detector-shelf audit): 1. Fail-open on rpk failure: probe_consumer_lag() treated any non-zero `rpk group describe` return code as lag=0 (healthy), so a broker/group probe failure was silently indistinguishable from "no lag" and could let an otherwise-broken flow classify FLOWING. It now returns (lag, lag_unknown) and fails CLOSED — collect_flow_metadata propagates consumer_lag_unknown, and the handler adds EnumFlowStatus.LAG_UNKNOWN so a failed probe is never classified healthy. 2. Hardcoded LAN IP (Rule #6): lane_target.py defaulted the .201 runtime host to the raw private IP 192.168.86.201 — unroutable off-network, the same bug class node_database_sweep already fixed under OMN-14526. Its own comment also claimed ONEX_DATA_FLOW_RUNTIME_HOST overrode the default, but the code never read that env var. Both are fixed: the default is now the Tailscale MagicDNS host, and ONEX_DATA_FLOW_RUNTIME_HOST is actually read (mirroring node_database_sweep's env-read so the two sweeps agree on which host they probe). RED->GREEN: added tests asserting rpk-failure fails closed (never FLOWING) and that the env var override actually takes effect — both failed against the pre-fix code, pass after. Also satisfies the pre-existing state-coverage-gate baseline debt for this node (summary output / result topic), promoted to a hard FAIL by --strict because this PR directly touches the node.
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 38 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (7)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
✅ Architectural Review — PASSEDErrors: 0 What this checks
No architectural violations found. Static architectural lint — no model inference (OMN-14176). |
✅ Hostile Reviewer — PASSEDBlocking findings (critical/error): 0 Gate semantics (pilot phase)
Powered by omniintelligence.review_pairing.cli_review — node-based adversarial review via HandlerLlmCliSubprocess (OMN-8524) |
Summary
Fixes two structural-blindness defects in
node_data_flow_sweep, the node backingonex:data_flow_sweep, found during the OMN-14531 detector-shelf audit (16/16 sweeps found structurally blind).Fail-open on rpk failure.
probe_consumer_lag()treated any non-zerorpk group describereturn code aslag=0— indistinguishable from a genuinely quiescent consumer group, so a broker/group probe failure could let an otherwise-broken flow classifyFLOWING(healthy). It now returns(lag, lag_unknown)and fails CLOSED:collect_flow_metadatapropagatesconsumer_lag_unknown, and the handler addsEnumFlowStatus.LAG_UNKNOWNso a failed probe is never classified healthy.Hardcoded LAN IP (Rule feat: PostgresDataSource adapter for data verification [OMN-7624] #6).
lane_target.pydefaulted the.201runtime host to the raw private IP192.168.86.201— unroutable off-network, the same bug classnode_database_sweepalready hit and fixed under OMN-14526. The module's own comment also claimedONEX_DATA_FLOW_RUNTIME_HOSToverrode the default, but the code never actually read that env var — the override was inert. Both are fixed: the default is now the Tailscale MagicDNS host (omninode-pc.tail75df5e.ts.net), andONEX_DATA_FLOW_RUNTIME_HOSTis now actually read (mirroringnode_database_sweep's_resolve_pg_runtime_host, which already reads this exact shared var, so the two sweeps agree on which host they probe).DoD evidence (OMN-14531)
RED→GREEN, captured against the pre-fix code before implementing:
test_probe_consumer_lag_fails_closed_on_rpk_failure/test_collect_flow_metadata_lag_probe_failure_fails_closed/test_handler_lag_unknown_flow_is_not_healthy— all failed against unpatched code (AttributeError/ValidationErroronconsumer_lag_unknown, then aFLOWINGfalse-green once the field existed), pass after the fix.test_dev_lane_resolves_to_remote_host/test_remote_lane_probe_uses_ssh_not_local_docker— updated to assert the Tailscale host; failed against the raw-IP default, pass after.test_runtime_host_env_var_overrides_default— new test provingONEX_DATA_FLOW_RUNTIME_HOSTactually takes effect; failed against unpatched code (env var was inert), passes after.Local verification:
uv run pytest tests/test_golden_chain_data_flow_sweep_collect.py tests/test_golden_chain_data_flow_sweep.py tests/test_data_flow_sweep_dispatch_resolution.py -v -m "not kafka"— 52/52 passed.uv run pytest tests/ -q -m "not kafka and not e2e and not live_model"(full suite) — 13259 passed, 19 skipped, 5 pre-existing failures unrelated to this change (test_no_cross_node_reach_in.py::test_no_new_cross_node_model_reach_ins— known OMN-14534-class cross-node reach-in debt; 3×test_pr_graphql_live.py— live GitHub GraphQL network tests). None touchnode_data_flow_sweep.uv run ruff format/ruff check --fix/mypy --strict src/omnimarket/nodes/node_data_flow_sweep/— clean.pre-commit run --all-fileson the changed set — all hooks passed, includingstate-coverage-gate(fixed a pre-existing baselined-debt promotion to hard-FAIL for this node by addingtest_contract_declares_summary_output/test_contract_publishes_result_topic, mirroring the existingtest_contract_declares_collect_inputconvention).Test plan
gh pr checksgreenOCC companion: owed to Codex per standing policy (agents do not author their own OCC companion/receipt).
Closes OMN-14531 (data_flow_sweep sub-fix; parent detector-shelf ticket remains open for the other 9 sweeps).
OCC Evidence
Evidence-Source: OCC#4108
Evidence-Ticket: OMN-14531
Evidence-Commit: 8c709084d03c5f2875fff8eedb2124052a03f3cb