Skip to content

fix(tools): drop exec-level OOMPolicy from systemd scope argv - #104132

Closed
UltraInstinct0x wants to merge 1 commit into
NousResearch:mainfrom
UltraInstinct0x:fix/scope-argv-no-oompolicy
Closed

UltraInstinct0x wants to merge 1 commit into
NousResearch:mainfrom
UltraInstinct0x:fix/scope-argv-no-oompolicy

Conversation

@UltraInstinct0x

Copy link
Copy Markdown

What does this PR do?

Scope units only accept cgroup resource properties. _systemd_scope_argv added --property OOMPolicy=kill, which systemd 249 rejects on scopes with Unknown assignment (rc=1). That failed the availability probe in _systemd_run_user_scope_available, so restart_safe_gateway_child_argv failed closed and blocked every gateway child: kanban workers (spawn_failed x N, tasks going blocked) and cron workers (hundreds of failure streaks, silent automations). Removing the exec-level property restores probe rc=0 while keeping MemoryAccounting and MemoryMax isolation.

Related Issue

No issue filed. Happy to link one if maintainers prefer an issue first.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)

Changes Made

  • tools/process_registry.py: drop OOMPolicy from _systemd_scope_argv with a note explaining why scopes reject it
  • tests/tools/test_process_registry.py: update spawn contract (OOMPolicy must be absent) and add scope-validity invariant test

How to Test

  1. On systemd 249 (Ubuntu 22.04): systemd-run --user --scope --quiet --unit=probe --collect --property=MemoryAccounting=yes --property=MemoryMax=1G -- /bin/true returns rc=0. Adding --property OOMPolicy=kill returns rc=1 with Unknown assignment.
  2. scripts/run_tests.sh tests/tools/test_process_registry.py tests/cron/test_restart_safe_worker.py: 122 passed, 0 failed.
  3. Live: after the patch plus gateway restart, a kanban worker assigned to a profile spawned successfully (previously spawn_failed with scope unavailable).

Scope units only accept cgroup resource properties. systemd 249
rejects OOMPolicy on scopes with Unknown assignment, which failed
the availability probe and blocked every gateway child (kanban
workers plus cron workers) with scope unavailable.

Keep MemoryAccounting and MemoryMax. Update the spawn test contract
and add a scope-validity invariant.
@alt-glitch alt-glitch added type/bug Something isn't working P1 High — major feature broken, no workaround comp/tools Tool registry, model_tools, toolsets comp/cron Cron scheduler and job management comp/gateway Gateway runner, session dispatch, delivery backend/local Local shell execution sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades duplicate This issue or pull request already exists labels Sep 6, 2026
@alt-glitch

Copy link
Copy Markdown

This was generated by AI during triage.

Duplicate of #102357 — same fix (drop OOMPolicy=kill from the systemd --scope argv in tools/process_registry.py) for the same probe-fails-closed bug tracked in #102486. Other open competitors: #102655, #103762, #103768 (version-gated variant), #103930. Leaving open for a maintainer to pick a canonical PR.

@kshitijk4poor

Copy link
Copy Markdown

Thanks @UltraInstinct0x — the kanban-worker angle (spawn_failed × N, tasks going blocked) is a useful addition to the cron-side reports; the same fail-closed raise feeds both.

This landed on main about an hour after you opened, via #104152 (611ee856c5, authored by @gkd2323c, whose #102357 was the earliest of six PRs on this bug, opened 2026-09-03). Identical production change — OOMPolicy=kill removed from the shared _systemd_scope_argv — plus the same test flip you made (spawn argv asserts no OOMPolicy=) and a matching pin on the captured probe argv. Your test_scope_argv_carries_no_exec_only_properties overlaps those. Closing as superseded; #102486 and #103693 closed with the merge. If kanban still shows spawn_failed on a current main gateway, please open a fresh issue with the scope probe stderr — that would be a different rejection than this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend/local Local shell execution comp/cron Cron scheduler and job management comp/gateway Gateway runner, session dispatch, delivery comp/tools Tool registry, model_tools, toolsets duplicate This issue or pull request already exists P1 High — major feature broken, no workaround sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants