Skip to content

fix(install,update): restore installs and unblock the update runtime repair - #76562

Merged
OutThisLife merged 2 commits into
mainfrom
bb/npm-engine-outage
Aug 2, 2026
Merged

fix(install,update): restore installs and unblock the update runtime repair#76562
OutThisLife merged 2 commits into
mainfrom
bb/npm-engine-outage

Conversation

@OutThisLife

Copy link
Copy Markdown
Collaborator

Nobody can install or update Hermes right now. Fresh installs and hermes update both die at the first npm ci:

npm error code EBADENGINE
npm error notsup Required: {"node":">=26.0.0","npm":">=12.0.0"}
npm error notsup Actual:   {"node":"v24.15.0","npm":"11.12.1"}

Root cause

.npmrc sets engine-strict=true, so engines is a hard gate on every install — the installer's workspace step, the update dependency refresh, and CI alike.

The floor was raised to npm >=12.0.0, but no Node release bundles npm 12:

Node bundled npm
26.5.1 11.17.0
24.18.1 11.16.0
22.23.2 10.9.8
20.20.2 10.8.2

The requirement is unsatisfiable by any stock toolchain. The installer provisions a Node from nodejs.org and is then immediately unable to install anything with it — including the Node it just vendored.

The fix

engines.npm<11.10.0 || >=11.17.0. This still excludes the band the strictness was actually for. npm 11.10–11.16 honor min-release-age but ignore min-release-age-exclude, both set in .npmrc, so they apply the 14-day age gate to packages we deliberately exempted. Verified rather than assumed:

npm 11.12.1 → npm error notarget No matching version found for vite@8.2.0
              with a date before 7/18/2026
npm 11.17.0 → added 15 packages

engines.node>=20.0.0, toolchain floor back to Node 22. Nothing in the tree needs 26 — Vite 8.2.0 declares ^20.19.0 || >=22.12.0, Electron 40 declares >=12.20.55. Requiring 26 force-migrated every working install for no dependency reason. apps/desktop drops to Vite's own floor for the same reason, and the desktop bundle builds clean on Node 22.

install.sh gained a second gate. A system Node was accepted on version alone, so a machine with Node 24 + its bundled npm 11.16.0 — the bad band — passed the check and then failed npm ci. npm_supports_npmrc() now rejects that band and installs the managed Node instead.

CI, Docker and nix are hermetic and keep pinning Node 26 / npm 12. They provision their own toolchain and both satisfy the relaxed range.

Verification

Real downloads, temp HERMES_HOME, no mocks:

Path Result
Fresh install (managed node v22.23.2 / npm 10.9.8) npm ci — 209 packages
Existing managed tree (v22.22.3 / npm 10.9.8) npm ci — 209 packages
Node 26.5.1 + bundled npm 11.17.0 npm ci — 208 packages
System npm 11.12.1 (the reported case) ✓ EBADENGINE → recovery provisions managed tree → retry green
apps/desktop npm run build on Node 22 ✓ dist built, assert passes

tests/test_engines_satisfiable.py encodes the invariants that would have caught this: the npm floor must be met by an npm some shipping Node actually bundles, the node floor by the runtime install.sh provisions, the desktop floor by its own build toolchain, and the lockfile mirror must match the manifest. Restoring the broken values fails 5 of them, each naming the reason:

AssertionError: engines.npm is '>=12.0.0', which no shipping Node bundles
(checked {20: '10.8.2', 22: '10.9.8', 24: '11.16.0', 26: '11.17.0'}).
With engine-strict=true every fresh install fails at the first `npm ci`.

These are relationships, not snapshots — no version anyone expects to change is pinned.

Note on the revert plan

I dry-ran reverting #76459, #76306, #76295, #75037 against current main: three of the four conflict individually (only #76295, the one-line electron-headers sha, is clean). #75037 also carries the vulnerability patches, so reverting it reintroduces those. This PR is the narrow forward fix for the install break; the nix/security work can be unwound separately if still wanted.

Also worth separating from this outage — the cannot import name 'venv_python_path' from 'hermes_constants' line in the update logs comes from 83314ca38, not from any of the four. It is a caught warning on the managed-Python repair path, not the install failure.

…can meet

Fresh installs and `hermes update` both fail at the first `npm ci`:

    npm error code EBADENGINE
    npm error notsup Required: {"node":">=26.0.0","npm":">=12.0.0"}
    npm error notsup Actual:   {"node":"v24.15.0","npm":"11.12.1"}

`.npmrc` sets engine-strict=true, so `engines` is a hard gate on every
install. The floor was raised to npm >=12 — but no Node release bundles
npm 12: Node 26 ships 11.17.0, 24 ships 11.16.0, 22 ships 10.9.8. The
requirement is unsatisfiable by any stock toolchain, so the installer
provisions a Node and is immediately unable to install with it.

engines.npm becomes `<11.10.0 || >=11.17.0`. That still excludes the band
the strictness was actually for: npm 11.10-11.16 honor `min-release-age`
but ignore `min-release-age-exclude`, both set in .npmrc, so they apply the
14-day gate to packages we exempted. Verified rather than assumed — npm
11.12.1 fails `ETARGET ... vite@8.2.0 with a date before 7/18/2026` while
11.17.0 installs it.

engines.node returns to >=20.0.0 and the toolchain floor to Node 22.
Nothing in the tree needs 26: Vite 8.2.0 declares `^20.19.0 || >=22.12.0`
and Electron 40 declares `>=12.20.55`. Requiring 26 force-migrated every
working install for no dependency reason. apps/desktop drops to Vite's own
floor for the same reason; the desktop bundle builds clean on Node 22.

install.sh gained a second gate: a system Node was accepted on version
alone, so a machine with Node 24 + its bundled npm 11.16.0 (the bad band)
passed the check and then failed `npm ci`. npm_supports_npmrc() now rejects
that band and installs the managed Node instead.

CI, Docker and nix are hermetic and keep pinning Node 26 / npm 12 — they
provision their own toolchain, and both satisfy the relaxed range.

tests/test_engines_satisfiable.py encodes the invariants that would have
caught this: the npm floor must be met by an npm some shipping Node
bundles, the node floor by the runtime install.sh provisions, the desktop
floor by its own build toolchain, and the lockfile mirror must match.
Restoring the broken values fails 5 of them with the reason stated.

Verified end-to-end (real downloads, temp HERMES_HOME):
- fresh install: managed node v22.23.2 / npm 10.9.8 -> npm ci, 209 packages
- existing managed tree (v22.22.3 / npm 10.9.8) -> npm ci, 209 packages
- node 26.5.1 + bundled npm 11.17.0 -> npm ci, 208 packages
- system npm 11.12.1 (the reported case) -> EBADENGINE, recovery provisions
  a managed tree and retries green
- apps/desktop `npm run build` on Node 22 -> dist built, assert passes
@OutThisLife
OutThisLife requested a review from a team August 2, 2026 03:57
ethernet8023
ethernet8023 previously approved these changes Aug 2, 2026
@ethernet8023
ethernet8023 enabled auto-merge (rebase) August 2, 2026 04:04
`hermes update` aborted its managed-Python runtime repair with an error
that reads like a contradiction:

    ⚠ Managed Python runtime repair skipped: cannot import name
      'venv_python_path' from 'hermes_constants'
      (/home/teknium/.hermes/hermes-agent/hermes_constants.py)

The named file does contain the symbol. The module in memory does not.
main.py imports hermes_constants from the OLD checkout, `git pull` then
replaces that file on disk, and the freshly-pulled managed_uv runs its
lazy `from hermes_constants import venv_python_path` against the module
object Python already cached in sys.modules — the pre-upgrade one. The
ImportError reports the path of the new file, so it looks like the symbol
is missing from a file that plainly has it.

Same update-boundary class already documented on `_UvResult` for the
ensure_uv() arity skew. It fires on the first update from any release
older than 83314ca, which introduced the symbol.

Both lazy call sites now reload the module from disk on ImportError:

- hermes_cli/managed_uv.py::_venv_python — the reported path
- hermes_cli/gateway.py::get_python_path — same flaw, same fix; a gateway
  restarted mid-update hits it identically

Reload rather than a local fallback on purpose. Recomputing the layout
inline would hand-roll `Scripts`/`bin` a second time — exactly what #76105
deduped into venv_bin_dir()/venv_python_path(), and what
test_no_open_coded_venv_layout_remains_in_hermes_cli bans. Reloading fixes
the actual problem (a stale module) and keeps one owner for the layout.

hermes_cli/update_cmd.py imports the symbol at module scope, which is a
different failure mode (the module fails to import at all) and is already
covered by the installer's retry-once for the update-boundary crash.

Tests reproduce the stale-module state by deleting the attribute from the
imported hermes_constants: recovery resolves through the reloaded shared
helper (asserted via a sentinel, so an open-coded copy cannot pass), and
the normal path never reloads. Against origin/main's managed_uv they fail
with the exact reported ImportError.
@OutThisLife OutThisLife changed the title fix(install): restore installs broken by an unsatisfiable npm engine floor fix(install,update): restore installs and unblock the update runtime repair Aug 2, 2026
@OutThisLife

Copy link
Copy Markdown
Collaborator Author

Added a second commit — c51c92a2 — for the venv_python_path error in @Teknium's update log. I originally called it unrelated noise; it is a real update-boundary bug and it does block the managed-Python runtime repair, so it belongs here.

⚠ Managed Python runtime repair skipped: cannot import name
  venv_python_path from hermes_constants
  (/home/teknium/.hermes/hermes-agent/hermes_constants.py)

The named file does contain the symbol; the module in memory does not. main.py imports hermes_constants from the old checkout, git pull replaces the file, and the freshly-pulled managed_uv runs its lazy import against the module object already cached in sys.modules. The ImportError names the new file on disk, which is why it reads as a contradiction. Same class as the ensure_uv() arity skew already documented on _UvResult, and it fires on the first update from any release older than 83314ca38.

Reproduced standalone before touching anything:

1. updater imports old module; has symbol: False
2. git pull rewrote file; disk has symbol: True
3. lazy import in new managed_uv: ImportError ->
   cannot import name 'venv_python_path' from 'hermes_constants' (...)

Fixed at both lazy call sites — managed_uv._venv_python (the reported one) and gateway.get_python_path, which has the identical flaw and is hit by a gateway restarted mid-update.

Worth noting how: my first attempt recomputed <venv>/bin/python inline, and test_no_open_coded_venv_layout_remains_in_hermes_cli correctly failed it — that is exactly the duplication #76105 deduped away. So the recovery reloads the module from disk instead, which fixes the actual problem (a stale module) and keeps one owner for the layout. The regression test asserts recovery resolves through the reloaded shared helper via a sentinel, so an open-coded copy cannot pass it either.

Against origin/main's managed_uv the new tests fail with the exact reported ImportError; with the fix, 6 passed. test_update_zip_two_phase.py (the guard suite) stays green.

Note this is separate from the four PRs in the revert list — it comes from 83314ca38, so unwinding those would not have fixed it.

@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

૮ >ﻌ< ა ci review

ran on c51c92a

ℹ️ Info

Desktop E2E visual evidence · View test artifacts · View job

3 visual diffs.

inline evidence is publishing...

@OutThisLife
OutThisLife disabled auto-merge August 2, 2026 04:15
@OutThisLife
OutThisLife merged commit 8ced76d into main Aug 2, 2026
42 of 43 checks passed
@OutThisLife
OutThisLife deleted the bb/npm-engine-outage branch August 2, 2026 04:15
pull Bot pushed a commit to bryanwills/hermes-agent that referenced this pull request Aug 2, 2026
`hermes desktop` still failed with EBADENGINE demanding Node >=26 after
NousResearch#76562, on a machine whose `apps/desktop/package.json` already said
`^20.19.0 || >=22.12.0`. NousResearch#76562 fixed the manifest but not its mirror in
`package-lock.json`, and `npm ci` reads engines from the lockfile:

    package.json  apps/desktop -> {'node': '^20.19.0 || >=22.12.0'}
    package-lock  apps/desktop -> {'node': '>=26.0.0'}      <- what gated

Chasing that exposed a second, pre-existing problem: the floor NousResearch#76562
declared was too generous. Running the real `npm ci` against the whole
workspace on Node 22.21.1 fails on a transitive dependency —

    npm error notsup Not compatible with your version of node/npm:
      react-router@8.3.0
    npm error notsup Required: {"node":">=22.22.0"}

react-router 8.3.0 (a direct dependency of both `apps/desktop` and `web`)
declares `>=22.22.0`, which is tighter than Vite's `^20.19 || >=22.12` and
excludes all of Node 20. So `>=20.0.0` promised support the tree cannot
deliver: an install on Node 20 or early 22 passed the installer's gate and
then died inside `npm ci` on someone else's package.

All four engine declarations now state the floor the dependency tree
actually has, `>=22.22.0`: root `package.json`, `apps/desktop/package.json`,
and both of their `package-lock.json` mirrors. The installer gates move with
them (`node_satisfies_build` in install.sh, `Test-NodeVersionOk` in
install.ps1) so a too-old system Node is replaced with the managed one
*before* npm runs, and the failure a user does see names hermes-agent rather
than a transitive package. NODE_VERSION stays 22 — latest-v22.x is 22.23.2,
comfortably above the floor.

The invariant test gains the case that would have caught the mirror drift on
its own: the desktop assertion now pins the tightest floor a dependency
actually declares, and the managed-runtime check compares majors, since
install.sh fetches latest-v{major}.x rather than {major}.0.0.

Verified with real `npm ci --dry-run` over the full workspace:
- node 22.23.2 (what install.sh provisions) -> 1258 packages
- node 26.5.1                               -> 1189 packages
- node 22.21.1 (below the floor)            -> EBADENGINE naming
  hermes-agent, i.e. our own manifest, not react-router
teknium1 added a commit that referenced this pull request Aug 2, 2026
…ines churn (#76627)

The Playwright suite fails identically on every PR regardless of diff
(verified on a Python-only PR and a docs-only PR): the mock-backend
Electron window never gets a title, so boot/chat/setup/interim specs all
fail; only the dead-backend boot-failure path still passes. Breakage
window matches the Aug 1 night engines/npm churn (#76499/#76562/#76575).

Gated with 'false &&' in the job condition — delete that to re-enable.
Root-fix + re-enable tracked in #76627 (Ari).
randlee pushed a commit to randlee/hermes-agent that referenced this pull request Aug 11, 2026
…-outage

fix(install,update): restore installs and unblock the update runtime repair
randlee pushed a commit to randlee/hermes-agent that referenced this pull request Aug 11, 2026
`hermes desktop` still failed with EBADENGINE demanding Node >=26 after
NousResearch#76562, on a machine whose `apps/desktop/package.json` already said
`^20.19.0 || >=22.12.0`. NousResearch#76562 fixed the manifest but not its mirror in
`package-lock.json`, and `npm ci` reads engines from the lockfile:

    package.json  apps/desktop -> {'node': '^20.19.0 || >=22.12.0'}
    package-lock  apps/desktop -> {'node': '>=26.0.0'}      <- what gated

Chasing that exposed a second, pre-existing problem: the floor NousResearch#76562
declared was too generous. Running the real `npm ci` against the whole
workspace on Node 22.21.1 fails on a transitive dependency —

    npm error notsup Not compatible with your version of node/npm:
      react-router@8.3.0
    npm error notsup Required: {"node":">=22.22.0"}

react-router 8.3.0 (a direct dependency of both `apps/desktop` and `web`)
declares `>=22.22.0`, which is tighter than Vite's `^20.19 || >=22.12` and
excludes all of Node 20. So `>=20.0.0` promised support the tree cannot
deliver: an install on Node 20 or early 22 passed the installer's gate and
then died inside `npm ci` on someone else's package.

All four engine declarations now state the floor the dependency tree
actually has, `>=22.22.0`: root `package.json`, `apps/desktop/package.json`,
and both of their `package-lock.json` mirrors. The installer gates move with
them (`node_satisfies_build` in install.sh, `Test-NodeVersionOk` in
install.ps1) so a too-old system Node is replaced with the managed one
*before* npm runs, and the failure a user does see names hermes-agent rather
than a transitive package. NODE_VERSION stays 22 — latest-v22.x is 22.23.2,
comfortably above the floor.

The invariant test gains the case that would have caught the mirror drift on
its own: the desktop assertion now pins the tightest floor a dependency
actually declares, and the managed-runtime check compares majors, since
install.sh fetches latest-v{major}.x rather than {major}.0.0.

Verified with real `npm ci --dry-run` over the full workspace:
- node 22.23.2 (what install.sh provisions) -> 1258 packages
- node 26.5.1                               -> 1189 packages
- node 22.21.1 (below the floor)            -> EBADENGINE naming
  hermes-agent, i.e. our own manifest, not react-router
randlee pushed a commit to randlee/hermes-agent that referenced this pull request Aug 11, 2026
…ines churn (NousResearch#76627)

The Playwright suite fails identically on every PR regardless of diff
(verified on a Python-only PR and a docs-only PR): the mock-backend
Electron window never gets a title, so boot/chat/setup/interim specs all
fail; only the dead-backend boot-failure path still passes. Breakage
window matches the Aug 1 night engines/npm churn (NousResearch#76499/NousResearch#76562/NousResearch#76575).

Gated with 'false &&' in the job condition — delete that to re-enable.
Root-fix + re-enable tracked in NousResearch#76627 (Ari).
33hodl pushed a commit to 33hodl/hermes-agent that referenced this pull request Aug 12, 2026
…-outage

fix(install,update): restore installs and unblock the update runtime repair
33hodl pushed a commit to 33hodl/hermes-agent that referenced this pull request Aug 12, 2026
`hermes desktop` still failed with EBADENGINE demanding Node >=26 after
NousResearch#76562, on a machine whose `apps/desktop/package.json` already said
`^20.19.0 || >=22.12.0`. NousResearch#76562 fixed the manifest but not its mirror in
`package-lock.json`, and `npm ci` reads engines from the lockfile:

    package.json  apps/desktop -> {'node': '^20.19.0 || >=22.12.0'}
    package-lock  apps/desktop -> {'node': '>=26.0.0'}      <- what gated

Chasing that exposed a second, pre-existing problem: the floor NousResearch#76562
declared was too generous. Running the real `npm ci` against the whole
workspace on Node 22.21.1 fails on a transitive dependency —

    npm error notsup Not compatible with your version of node/npm:
      react-router@8.3.0
    npm error notsup Required: {"node":">=22.22.0"}

react-router 8.3.0 (a direct dependency of both `apps/desktop` and `web`)
declares `>=22.22.0`, which is tighter than Vite's `^20.19 || >=22.12` and
excludes all of Node 20. So `>=20.0.0` promised support the tree cannot
deliver: an install on Node 20 or early 22 passed the installer's gate and
then died inside `npm ci` on someone else's package.

All four engine declarations now state the floor the dependency tree
actually has, `>=22.22.0`: root `package.json`, `apps/desktop/package.json`,
and both of their `package-lock.json` mirrors. The installer gates move with
them (`node_satisfies_build` in install.sh, `Test-NodeVersionOk` in
install.ps1) so a too-old system Node is replaced with the managed one
*before* npm runs, and the failure a user does see names hermes-agent rather
than a transitive package. NODE_VERSION stays 22 — latest-v22.x is 22.23.2,
comfortably above the floor.

The invariant test gains the case that would have caught the mirror drift on
its own: the desktop assertion now pins the tightest floor a dependency
actually declares, and the managed-runtime check compares majors, since
install.sh fetches latest-v{major}.x rather than {major}.0.0.

Verified with real `npm ci --dry-run` over the full workspace:
- node 22.23.2 (what install.sh provisions) -> 1258 packages
- node 26.5.1                               -> 1189 packages
- node 22.21.1 (below the floor)            -> EBADENGINE naming
  hermes-agent, i.e. our own manifest, not react-router
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants