Skip to content

fix(docker): preserve custom PATH entries in subprocess environment (#70905) - #70923

Open
webtecnica wants to merge 3 commits into
NousResearch:mainfrom
webtecnica:fix/docker-path-propagation
Open

fix(docker): preserve custom PATH entries in subprocess environment (#70905)#70923
webtecnica wants to merge 3 commits into
NousResearch:mainfrom
webtecnica:fix/docker-path-propagation

Conversation

@webtecnica

Copy link
Copy Markdown
Contributor

Fix: Preserve custom PATH entries in agent's subprocess execution context

Closes #70905

Problem

Custom PATH entries set via Dockerfile ENV, .env files, or config.yaml (terminal.env / docker_env) are silently dropped, causing command not found (exit 127) when the agent runs installed binaries.

Root cause

Two issues combined:

  1. _make_run_env (local.py): The filtering loop that strips Hermes provider credentials from the subprocess env had no fallback chain for PATH. If a future expansion of the blocklist accidentally filtered PATH, or the run_env PATH was empty, the custom entries would be lost.

  2. DockerEnvironment._build_init_env_args (docker.py): The -e env-var flags passed to docker exec during init_session only included docker_env overrides and forwarded/passthrough vars. PATH from the container's default environment (Dockerfile ENV) was NOT explicitly forwarded. When the login shell's profile scripts (/etc/profile, ~/.bashrc) truncated or reset PATH, the init_session snapshot would capture a narrower PATH than the container's default, silently losing custom entries on every subsequent execute() call.

Fix

  1. _make_run_env: Add a PATH propagation guard that restores PATH from os.environ if the filtering loop drops it, and falls back to os.environ when the run_env PATH is empty.

  2. _build_init_env_args: Explicitly include PATH from the host os.environ when it's not already overridden by docker_env, so the init_session snapshot always captures the full PATH.

  3. New test suite (tests/tools/test_path_propagation.py): 12 tests covering custom PATH preservation, _SANE_PATH fallback, terminal.env override, multiple custom entries, empty PATH recovery, and full execute() integration.

Testing

  • pytest tests/tools/test_path_propagation.py — 12/12 passed
  • pytest tests/tools/test_local_env_blocklist.py — all existing tests pass
  • pytest tests/tools/test_env_passthrough.py — all existing tests pass
  • pytest tests/tools/test_hermes_subprocess_env.py — all existing tests pass

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/cron Cron scheduler and job management comp/tools Tool registry, model_tools, toolsets tool/terminal Terminal execution and process management backend/docker Docker container execution backend/local Local shell execution labels Jul 24, 2026
@isak-ialogics

Copy link
Copy Markdown
Contributor

The current branch includes two unrelated commits before the PATH fix: 9a8a3d3 (the cron 401/403 change from #70913) and 8af2053 (the execution-ledger change from #70917). That is why this nominal PATH PR also modifies cron/executions.py, cron/scheduler.py, and cron tests, and the attribution/required-check gate is failing. Suggested next action: rebase onto current main (or recreate the branch) with only 6338165 and the PATH-related files, then rerun CI.

…D exhaustion (NousResearch#69567)

The cron execution ledger was leaking SQLite connections because `sqlite3.Connection` as a context manager only commits/rolls back -- it does NOT close the connection. Without explicit `conn.close()`, every ledger call (create, mark_running, finish, recover, list, latest) left an open connection and its WAL/SHM file descriptors.

Created a `_transaction()` context manager that wraps lock acquisition, connection open, schema initialization, transaction commit/rollback, and deterministic connection close in a `finally` block. Schema init runs inside the `try` too so PRAGMA/DDL failures after a successful connect still close the connection.

Key design:
- `_connect()` now only opens the connection (no schema)
- `_initialize_schema(conn)` handles PRAGMAs, DDL, and WAL setup
- `_transaction()` combines lock + connect + init + commit/rollback + close
- `apply_wal_with_fallback` is preserved (unlike PR NousResearch#69594 which replaced it with raw PRAGMA, losing NFS/SMB fallback)

All 7 call sites migrated from `with _lock, _connect() as conn:` to `with _transaction() as conn:`.

Added regression test `test_every_ledger_call_closes_sqlite_connection` that repeatedly calls all ledger functions and asserts the /proc/self/fd count for executions.db doesn't grow.
…ousResearch#70905)

Ensure that custom PATH entries set via Dockerfile ENV, .env files, or
config.yaml terminal.env / docker_env are preserved in the agent's
subprocess execution context.

Root cause
----------
Two issues combined to silently drop custom PATH entries:

1. _make_run_env (local.py): the filtering loop that strips Hermes
   provider credentials could, in edge cases with a future-expanded
   blocklist, leave the run_env dict without a PATH key.  Even when
   PATH was present, there was no fallback chain if it was empty.

2. DockerEnvironment._build_init_env_args (docker.py): the list of
   -e env-var flags passed to docker exec during init_session only
   included docker_env overrides and forwarded/passthrough vars.
   PATH from the container's default environment (Dockerfile ENV)
   was NOT explicitly forwarded.  If the login shell's profile
   scripts (/etc/profile, ~/.bashrc) truncated or reset PATH, the
   init_session snapshot would capture a narrower PATH than the
   container's default, silently losing custom entries on every
   subsequent execute() call.

Fix
---
1. _make_run_env: Add a PATH propagation guard that restores PATH
   from os.environ if the filtering loop drops it, and falls back to
   os.environ when the run_env PATH is empty.

2. _build_init_env_args: Explicitly include PATH from the host
   os.environ when it's not already overridden by docker_env, so
   the init_session snapshot always captures the full PATH.

3. Add comprehensive test suite (tests/tools/test_path_propagation.py):
   12 tests covering custom PATH preservation, _SANE_PATH fallback,
   terminal.env override, multiple custom entries, empty PATH recovery,
   and full execute() integration.

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for investigating the PATH-loss report. The Docker snapshot path is the right area to examine, but the current implementation does not preserve the container-defined value.

Problems

  • In this diff, tools/environments/docker.py:1518 reads os.environ["PATH"] from the Docker host. Main injects those arguments into docker exec at tools/environments/docker.py:1511-1523, while the container image defines its own runtime PATH at Dockerfile:415. The host PATH cannot carry a container-only Dockerfile entry such as /potato.
  • tests/tools/test_path_propagation.py:15-22 exercises only LocalEnvironment; it has no Docker backend or derived-image reproduction. This does not validate the reported snapshot failure.
  • As noted in the existing PR discussion, the reviewed diff still contains unrelated cron changes in cron/executions.py and cron/scheduler.py.

Suggested changes

  • Preserve the target container's pre-login PATH, then apply an explicit docker_env PATH override with defined precedence rather than forwarding the host PATH.
  • Add a Docker integration regression with a derived image whose ENV PATH includes /potato and verify its binary through the session snapshot path.

This is an automated hermes-sweeper review.

# Prefer the docker_env override when present; otherwise fall back to
# the host os.environ, which inside a Docker container carries the
# Dockerfile ENV PATH.
if "PATH" not in exec_env:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

os.environ here belongs to the Docker host, not the target container. The reported /potato entry is introduced by the target image's Dockerfile, so forwarding this value cannot preserve that container-only PATH and may replace it during the login-shell snapshot. Capture the container's pre-login PATH instead, then apply any explicit docker_env override.

@teknium1 teknium1 added sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 30, 2026
@GottZ

GottZ commented Aug 3, 2026

Copy link
Copy Markdown

This was generated by AI during triage.

Graph note (no action implied — a maintainer has already reviewed this thread).

Our triage graph places this PR in a complex with 1 related issue (#70905). They were checked against each other at the diff level and no consolidation is indicated — they address distinct causes.

Full neighbourhood: https://hermes-triage.gottz.de/?node=70923

This note exists so the relationship stays discoverable from the thread itself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend/docker Docker container execution backend/local Local shell execution comp/cron Cron scheduler and job management comp/tools Tool registry, model_tools, toolsets P2 Medium — degraded but workaround exists sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows tool/terminal Terminal execution and process management type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Unable to set path variable in a docker container

5 participants