Skip to content

fix(process_registry): drop OOMPolicy=kill from systemd scope units - #103762

Closed
aravindtri wants to merge 1 commit into
NousResearch:mainfrom
aravindtri:fix/systemd-oom-policy-scope-units
Closed

aravindtri wants to merge 1 commit into
NousResearch:mainfrom
aravindtri:fix/systemd-oom-policy-scope-units

Conversation

@aravindtri

Copy link
Copy Markdown

Problem

On Ubuntu 20.04 (systemd 245), cron jobs silently never run. The availability probe for restart-safe dispatch fails with:

Unknown assignment: OOMPolicy=kill
systemd-run --user --scope is unavailable

systemd <250 rejects OOMPolicy=kill on --scope units.

Fix

Remove OOMPolicy=kill from _systemd_scope_argv. MemoryMax + MemoryAccounting still bound the worker cgroup, and the kernel OOM killer still applies — no memory-limit regression.

Verification

  • systemd 245 (Ubuntu 20.04): scope spawn + probe now succeed; cron jobs dispatch.
  • Coverage: single code path shared by probe and real spawns.

systemd <250 (Ubuntu 20.04) rejects OOMPolicy=kill on scope units with
'Unknown assignment: OOMPolicy=kill', causing the availability probe to fail
and making restart-safe cron dispatch a no-op. MemoryMax + MemoryAccounting
still bound the worker cgroup; the kernel OOM killer applies.

Fixes cron jobs not running on Ubuntu 20.04 / systemd 245.

@crazyief crazyief left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use GitHub Commit suggestion (not copy-paste / amend). That records Co-authored-by: crazyief.

Suggestion: add unit tests for the OOMPolicy removal

The PR removes OOMPolicy=kill from _systemd_scope_argv but has zero test coverage for this function. I verified these tests pass 4/4 on the lab against the PR worktree.

diff --git a/tests/tools/test_systemd_scope_argv.py b/tests/tools/test_systemd_scope_argv.py
new file mode 100644
--- /dev/null
+++ b/tests/tools/test_systemd_scope_argv.py
@@ -0,0 +1,38 @@
+"""Tests for _systemd_scope_argv OOMPolicy removal (#103762)."""
+
+from __future__ import annotations
+
+import pytest
+
+
+class TestSystemdScopeArgv:
+    """_systemd_scope_argv must produce valid systemd-run argv without
+    OOMPolicy=kill (systemd <250 rejects it on scope units)."""
+
+    def test_oom_policy_not_present(self):
+        from tools.process_registry import _systemd_scope_argv
+        argv = _systemd_scope_argv("systemd-run", "test-unit", "echo", "hello")
+        oom_flags = [a for a in argv if "OOMPolicy" in a]
+        assert len(oom_flags) == 0
+
+    def test_memory_properties_still_present(self):
+        from tools.process_registry import _systemd_scope_argv
+        argv = _systemd_scope_argv("systemd-run", "test-unit", "echo", "hello")
+        assert any("MemoryMax" in a for a in argv)
+        assert any("MemoryAccounting" in a for a in argv)
+
+    def test_includes_collect_and_unit(self):
+        from tools.process_registry import _systemd_scope_argv
+        argv = _systemd_scope_argv("systemd-run", "test-unit", "echo", "hello")
+        assert "--collect" in argv
+        assert "--unit" in argv
+        assert "test-unit" in argv
+
+    def test_passes_through_command(self):
+        from tools.process_registry import _systemd_scope_argv
+        argv = _systemd_scope_argv("systemd-run", "test-unit", "echo", "hello")
+        assert "echo" in argv
+        assert "hello" in argv
+```

@alt-glitch alt-glitch added type/bug Something isn't working P1 High — major feature broken, no workaround comp/cron Cron scheduler and job management comp/tools Tool registry, model_tools, toolsets backend/local Local shell execution sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades duplicate This issue or pull request already exists labels Sep 5, 2026
@alt-glitch

Copy link
Copy Markdown

This was generated by AI during triage.

Duplicate of #102357, which makes the same change (drop OOMPolicy=kill from the systemd-run --scope argv in tools/process_registry.py) for #102486. #102508 and #102655 are also open for this cluster; a maintainer should pick one.

@kshitijk4poor

Copy link
Copy Markdown

Thanks @aravindtri — the Ubuntu 20.04 / systemd 245 evidence here is useful: it shows transient --scope units reject OOMPolicy=kill well below the v253 boundary some of the sibling PRs assumed, which is what settled the cluster on "drop the property" rather than "version-gate it".

Closing as a duplicate of #102357 (@gkd2323c, opened 2026-09-03), which makes the same change — remove OOMPolicy=kill from both the availability probe and _build_systemd_scope_argv in tools/process_registry.py — and is the earliest submission for #102486. #102357 is the one being reviewed and salvaged for the cluster; your systemd-245 data point is being cited in its landing notes. If #102357 does not land for some reason this will be reopened rather than lost.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend/local Local shell execution comp/cron Cron scheduler and job management comp/tools Tool registry, model_tools, toolsets duplicate This issue or pull request already exists P1 High — major feature broken, no workaround sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants