Skip to content

fix(cron): give POSIX cron scripts the generation environment - #123548

Closed
vladneshrus-cpu wants to merge 1 commit into
NousResearch:mainfrom
vladneshrus-cpu:fix/cron-posix-pythonpath
Closed

vladneshrus-cpu wants to merge 1 commit into
NousResearch:mainfrom
vladneshrus-cpu:fix/cron-posix-pythonpath

Conversation

@vladneshrus-cpu

Copy link
Copy Markdown

fix(cron): POSIX cron scripts lose the generation environment

Problem

On POSIX, a cron job whose script is Python (.py) and imports a third-party library fails
with ModuleNotFoundError, even though the same script works when run by hand from the
generation environment.

Root cause: cron/scheduler_script.py. _windows_cron_python_invocation() returns

if sys.platform != "win32":
    return python_exe, {}

— an empty env overlay. The Windows branch instead builds
PYTHONPATH = <repo> + <generation site-packages> (plus VIRTUAL_ENV for uv venvs).

That overlay is what makes the child able to import anything beyond Hermes' own runtime:
the launcher (<install>/.hermes/bin/hermes) starts the managed "store" Python with
PYTHONPATH popped (os.environ.pop('PYTHONPATH', None)), and that interpreter's own
site-packages carries only Hermes itself. So the gateway's own process is fine (its
sys.path is built by the launcher), but any child it spawns for a cron script inherits
neither the paths nor the overlay on POSIX.

Practical impact: every cron .py script with third-party imports (Google API client,
YAML, requests, …) had to re-add the paths itself at the top of the file — a maintenance
trap that breaks on every environment migration.

Fix

  1. New _posix_dependency_env_overlay(): returns
    {"PYTHONPATH": "<repo><sep><generation site-packages>[<sep><inherited PYTHONPATH>]"}
    using the same pm.environments resolvers the Windows branch already trusts.
    It is gated on the generation environment actually existing, so source checkouts and
    portable runs (no managed environment) keep today's behaviour instead of getting a
    dangling PYTHONPATH. Any probe failure returns {} — never fails a job.
  2. The POSIX branch of _windows_cron_python_invocation() now returns that overlay instead
    of {}.
  3. _script_argv() keeps the .pth bootstrap Windows-only (env_overlay and sys.platform == "win32"): on POSIX the interpreter is already the managed Python, so the
    plain invocation plus PYTHONPATH is enough. Windows behaviour is byte-for-byte unchanged.

_run_job_script() already merged the overlay after the env sanitizer
(env.update(env_overlay)), so no change was needed there.

Note on .pth files

PYTHONPATH entries are not processed as .pth. The only .pth in the generation
environment that matters here is the editable install of Hermes itself, whose target (the
repo root) is added explicitly, so the overlay is sufficient on POSIX.

Verification

End-to-end, in a checkout whose _run_job_script was exercised by the same managed
interpreter the gateway uses (HERMES_HOME = throwaway home, PYTHONPATH popped, repo +
generation site-packages only on sys.path), with a probe job script doing
import googleapiclient, yaml:

  • before: Script exited with code 1 / ModuleNotFoundError: No module named 'googleapiclient'
  • after: PROBE_OK py=3.14.7 googleapiclient=<generation site-packages>/googleapiclient/__init__.py yaml=<…>/yaml/__init__.py

Unit/regression: tests/cron/test_cron_script.py — new TestPosixScriptDependencyOverlay
covers the overlay contents, inherited PYTHONPATH, the no-managed-environment case, the
plain POSIX invocation (no -c bootstrap) and an end-to-end child-process import. All five
fail without the fix (same ModuleNotFoundError in the child) and pass with it.

Scope / rollback

  • Windows: untouched (its branch is unchanged and still takes precedence).
  • POSIX installs without a managed generation environment: unchanged ({}).
  • Rollback: revert this commit; behaviour returns to the status quo.

Re-verified on current main

Patch applied to d0288be5b3 (clean git apply -3, no fuzz):

  • tests/cron/test_cron_script.py -k posix → 5 passed.
  • Regression sweep on the same tree: tests/cron/{test_estop,test_execution_ledger,test_scheduled_occurrence,test_atomic_paused_creation,test_worktree_maintenance_isolation}.py → 10 failed / 41 passed, byte-for-byte the same failure list as the unpatched tree — pre-existing, unrelated to this change.

Fixes #123547

The launcher starts the managed ("store") Python with PYTHONPATH popped, and that
interpreter's own site-packages carries only Hermes' own runtime. A cron .py script that
imports a third-party library therefore died with ModuleNotFoundError unless the script
re-added the paths itself: the Windows branch always installed a repo + generation
site-packages overlay, while the POSIX branch returned an empty dict.

Add _posix_dependency_env_overlay() (repo root + generation site-packages, plus any inherited
PYTHONPATH), gated on the managed environment actually existing so source checkouts and
portable runs keep today's behaviour. The interpreter stays sys.executable: it is already the
managed Python, so the Windows .pth bootstrap is not needed and stays Windows-only.

Covered by tests in tests/cron/test_cron_script.py, including an end-to-end case where a job
script imports a package that exists only in the generation environment.
@kiramakes

Copy link
Copy Markdown

🤖 Hermes Agent automated review: PR reviewed. Diff analyzed (164 lines changed). Check CI status and manual review recommended.

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/cron Cron scheduler and job management area/install-update Installer, updater, packaging, wheels, doctor needs-decision Awaiting maintainer decision before any implementation sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades labels Sep 26, 2026
@gryknight9

Copy link
Copy Markdown

Ran into the same behavior on a Linux install with a recent update (v46-era config). An hourly cron job using the bundled google-workspace skill (google_api.py gmail search ...) failed with ModuleNotFoundError: No module named 'googleapiclient' — while the exact same command worked when run by hand from a shell.

What I found while chasing it:

  • The gateway process and my interactive shells both resolve python to interpreters that do have the package, so the failure only reproduced in the cron worker's child environment.
  • hermes doctor reported a newly "staged runtime venv" under ~/.hermes/installs/<hash>/environments/<hash>/venv. That venv's site-packages carries only the Hermes runtime — import googleapiclient fails there, and it's what the worker's python resolves to.
  • Matches your root-cause write-up exactly: the POSIX branch returns an empty env overlay, so the child never gets the generation site-packages PYTHONPATH.

Workaround I used in the meantime, in case it helps others who can't wait for the merge:

uv pip install --python ~/.hermes/installs/<hash>/environments/<hash>/venv/bin/python \
  google-api-python-client google-auth-httplib2

After that the job's search ran clean and delivered normally. (Worth noting for the fix: the doctor check for the staged venv passes as "Runtime venv staged" without listing which third-party packages are present, which is what made this hard to spot — a hint in the doctor output, or a re-probe of a known cron job, would catch regressions like this earlier.)

👍 to the PR.

@kshitijk4poor

Copy link
Copy Markdown

Thanks @vladneshrus-cpu. This is superseded by #124973, which is armed for rebase-merge. It fixes the same POSIX managed-store cron-script ModuleNotFoundError (#123044 / #123440). It builds on #123051: the script runs on the selected venv's own interpreter and loads the live checkout through an in-process bootstrap. There's no PYTHONPATH overlay, because child processes inherit PYTHONPATH (the breakage in #123440). You're credited in the PR body. Closing; I'll reopen if #124973 doesn't land.

@gryknight9, thanks for the detailed report. After #124973 the script runs on that same staged venv's interpreter, so anything installed there imports fine, googleapiclient included. In your case the google extra was missing from the staged venv itself, which is why your uv pip install --python .../venv/bin/python workaround worked. The supported way to add it is hermes pm install --extra google, which syncs that install's venv. Your point that hermes doctor only reports "Runtime venv staged" without checking which extras are present is a fair one, but it's a separate issue from this fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/install-update Installer, updater, packaging, wheels, doctor comp/cron Cron scheduler and job management needs-decision Awaiting maintainer decision before any implementation P2 Medium — degraded but workaround exists sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

POSIX cron .py scripts lose the generation environment (ModuleNotFoundError)

5 participants