Skip to content

feat: add offload_job tool — systemd-run isolation for heavy workloads - #15873

Closed
realsigridjin wants to merge 3 commits into
NousResearch:mainfrom
realsigridjin:feat/offload-job-tool
Closed

feat: add offload_job tool — systemd-run isolation for heavy workloads#15873
realsigridjin wants to merge 3 commits into
NousResearch:mainfrom
realsigridjin:feat/offload-job-tool

Conversation

@realsigridjin

Copy link
Copy Markdown

Summary

New tool that runs memory-heavy/long-running shell commands (benchmarks, model inference, indexing) under systemd-run --user units instead of as children of hermes-gateway.service. Prevents OOM-killing the gateway.

Changes

File Change
tools/offload_job_tool.py New — Job runner with start/status/tail/list/cancel actions. Linux uses systemd-run --user for isolation; subprocess fallback. Metadata persisted in ~/.hermes/offload_jobs/.
tests/tools/test_offload_job_tool.py New — Tests for start→status→tail flow, list ordering, cancel, systemd path.
tools/terminal_tool.py Added _GATEWAY_OFFLOAD_PATTERNS and _gateway_local_offload_guidance() — refuses heavy commands (benchmarks, GPU workloads) when running inside hermes-gateway.service, redirecting to offload_job tool instead.
toolsets.py Registered offload_job in _HERMES_CORE_TOOLS and terminal toolset.

Why

  • Gateway-local terminal calls run as descendants of hermes-gateway.service
  • A model/benchmark process can OOM-kill the gateway itself
  • offload_job starts those commands under a separate user systemd unit, preserving gateway availability

Testing

  • pytest tests/tools/test_offload_job_tool.py — 5 tests covering fallback, list, cancel, systemd path

Scope

Minimal, focused change. No modifications to agent loop, no breaking changes to existing tools.

New tool that runs memory-heavy/long-running shell commands (benchmarks,
model inference, indexing) under systemd-run --user units instead of as
children of hermes-gateway.service. Prevents OOM-killing the gateway.

- tools/offload_job_tool.py: start/status/tail/list/cancel actions
- tools/terminal_tool.py: gateway-local offload guardrail patterns
- toolsets.py: register offload_job in core + terminal toolsets
- tests/tools/test_offload_job_tool.py: fallback, list, cancel, systemd tests
@realsigridjin
realsigridjin force-pushed the feat/offload-job-tool branch from 2eb1b72 to af9a9f8 Compare April 26, 2026 03:30
@alt-glitch alt-glitch added type/feature New feature or request P3 Low — cosmetic, nice to have comp/tools Tool registry, model_tools, toolsets tool/terminal Terminal execution and process management labels Apr 26, 2026
Subagents created via delegate_task previously did not inherit the
parent's fallback_model / _fallback_chain configuration. This meant
that when the primary provider (e.g. Claude) hit rate limits, child
agents would fail instead of falling back to the configured backup
provider (e.g. Kimi).

Pass the parent's fallback configuration through the AIAgent constructor
so _try_activate_fallback() works for subagents too.

Co-Authored-By: Claude
- Cross-platform home dir via dirs crate (no hardcoded paths)
- Auto workspace discovery from ~/.openclaw/workspace*
- Google Drive upload via rclone integration
- SQLite-safe backups (sqlite3 .backup)
- Compressed tar.zst archives
- Subcommands: backup, restore, list, upload, auto, setup
- Config file support at ~/.config/hbackup/config.toml
@teknium1

Copy link
Copy Markdown
Contributor

Thanks for the detailed PR — closing with reasoning.

Scope. Three unrelated changes in one PR: the offload_job tool, a delegate_task fallback_model inheritance fix, and a ~1000-line Rust hbackup CLI under tools/hbackup/. Each would need to stand on its own.

Stale against main — silently reverts #16177. The branch is 84 commits behind. The terminal_tool.py changes delete _resolve_container_task_id(), which we added today in #16177 (feat(terminal): collapse subagent task_ids to shared container). Cherry-picking this PR would revert the subagent-shared-container fix.

Core design: duplicates existing infra. terminal(background=True) + process already run long-lived commands and survive across turns. The underlying concern — heavy commands inheriting hermes-gateway.service's cgroup and OOM-killing the gateway — is a one-line MemoryMax= in the systemd unit file, not a 581-LOC tool with its own metadata store, job IDs, status/tail/list/cancel actions, and a refusal guard in terminal_tool. The right fix lives in the unit file and/or Linux cgroup config, not the tool surface.

Refusal heuristic is too aggressive. _GATEWAY_OFFLOAD_PATTERNS matches any command containing "benchmark" as a substring or two of {cuda, cudnn, cublas, ld_library_path, hf_home} — would refuse legitimate gateway-side runs like pytest -k benchmark or python -c 'import torch; print(torch.cuda.is_available())'.

hbackup. A Rust CLI with its own Cargo.toml, README, Google Drive uploader, and cron guide, but no hermes subcommand, no Python wiring, no tests. If you want to pursue this, it should be a separate project (or a focused PR that integrates it into hermes commands with tests) — it doesn't belong in tools/ as standalone.

Salvageable piece. The delegate_tool.py change — children inheriting the parent's fallback_model chain — is a reasonable 2-line fix on its own. If you'd like to re-submit that as a standalone PR against current main, happy to review.

Closing the rest.

@teknium1 teknium1 closed this Apr 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/tools Tool registry, model_tools, toolsets P3 Low — cosmetic, nice to have tool/terminal Terminal execution and process management type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants