Skip to content

feat(state): persist last_prompt_tokens / last_completion_tokens per call - #1

Merged
OmarB97 merged 1 commit into
mainfrom
feat/state-db-last-prompt-tokens
May 25, 2026
Merged

feat(state): persist last_prompt_tokens / last_completion_tokens per call#1
OmarB97 merged 1 commit into
mainfrom
feat/state-db-last-prompt-tokens

Conversation

@OmarB97

@OmarB97 OmarB97 commented May 25, 2026

Copy link
Copy Markdown
Owner

Summary

Persist a per-call snapshot of the most-recent API call's prompt and
completion sizes on the sessions table. Cumulative input_tokens /
output_tokens mix every call together, so downstream consumers
(dashboards, monitoring tools) currently can't show "current
context-window size" without reconstructing it from per-call deltas
plus a cache model.

Motivation

MeshBoard's dashboard context chip was freezing on a static
system_prompt_size / 4 estimate for entire 30-minute dispatches
because state.db exposed no authoritative "current context size"
field. The dashboard shipped a workaround that uses system_prompt + session_input + session_output, but that double-counts when prompt
caching is active and misses tool-result re-introduction. The
authoritative fix is for Hermes to write the actual prompt_tokens
value the API returned for the most recent call.

Changes

hermes_state.py:

  • SCHEMA_SQL: two new columns on sessions:
    • last_prompt_tokens INTEGER DEFAULT 0
    • last_completion_tokens INTEGER DEFAULT 0
      The declarative _reconcile_columns() (line 514) will auto-ALTER TABLE ADD COLUMN on existing databases, no version bump needed.
  • update_token_counts() accepts new optional kwargs
    last_prompt_tokens / last_completion_tokens. They are SET on
    every update (never accumulated) in both absolute and increment
    modes — they snapshot the latest call only, not lifetime totals.
    Pass None (default) to leave the existing snapshot untouched
    (e.g. cost-only updates).

run_agent.py:

  • The CLI commit path at line ~13237 now passes
    last_prompt_tokens=prompt_tokens and
    last_completion_tokens=completion_tokens (the raw values from the
    canonical usage record) on every per-call token persistence.

Tests

4 new tests in tests/test_hermes_state.py:

  • test_last_prompt_tokens_is_set_not_incremented — multiple updates
    set, not accumulate
  • test_last_prompt_tokens_omitted_keeps_previous_valueNone
    default leaves the snapshot untouched (the cost-only-update case)
  • test_last_prompt_tokens_in_absolute_mode — the gateway path also
    sets, not accumulates
  • test_existing_db_reconciles_last_prompt_tokens_columns
    declarative reconciliation adds the columns on pre-existing DBs

Test run

$ python3 -m pytest tests/test_hermes_state.py tests/test_hermes_state_wal_fallback.py
============================= 231 passed in 13.64s =============================

(226 pre-existing + 5 new — the test that uses pytest fixture db
exercises the absolute-mode path and the reconciliation alongside
the new explicit tests.)

Compatibility

  • Existing databases: declarative reconciliation adds the columns
    with DEFAULT 0 on the next Hermes startup. No migration script
    needed.
  • Existing code: update_token_counts() signature is backward-
    compatible. The new kwargs default to None.
  • Forward-compat with the dashboard: MeshBoard's
    session_context_usage._extract_context_usage direct_keys check
    already reads last_prompt_tokens from state.db rows
    (line ~298). Once this PR ships, the chip starts showing
    authoritative current-context size automatically — no dashboard
    change required.

…call

Cumulative input_tokens / output_tokens are the only token-related
state.db columns today, which means downstream consumers (dashboards,
monitoring) can't show "current context-window size" without
reconstructing it from per-call deltas + a cache model.

Add a per-call snapshot of the most-recent API call's prompt /
completion sizes:

- New SCHEMA_SQL columns `last_prompt_tokens` and
  `last_completion_tokens` on the sessions table. Both INTEGER
  DEFAULT 0; the declarative reconciliation in _reconcile_columns
  picks them up on existing databases via ALTER TABLE ADD COLUMN,
  no version bump needed.
- update_token_counts() accepts optional last_prompt_tokens /
  last_completion_tokens kwargs. They are SET on every update
  (never accumulated) in both absolute and increment modes — they
  represent the latest call only, not lifetime totals. Pass None
  (default) to leave the existing snapshot untouched for
  cost-only updates.
- run_agent.py wires `prompt_tokens` and `completion_tokens` from
  the canonical usage record through on every API call commit.

Tests:
- last_prompt_tokens is SET not accumulated across calls
- passing None keeps the prior value (no zero-out on cost-only updates)
- absolute mode also sets, not accumulates
- the declarative reconciliation adds the columns on pre-existing DBs

Motivated by: the MeshBoard dashboard's "context" chip was freezing on
the system-prompt-size estimate for an entire 30-min dispatch because
state.db exposed no authoritative "current context size" field. With
this column the chip can render the real last-prompt size directly
(via the existing direct_keys path in session_context_usage.py).
@github-actions

Copy link
Copy Markdown

🔎 Lint report: feat/state-db-last-prompt-tokens vs origin/main

ruff

Total: 0 on HEAD, 0 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 0 pre-existing issues carried over.

ty (type checker)

Total: 8386 on HEAD, 8386 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 4424 pre-existing issues carried over.

Diagnostics are surfaced as warnings — this check never fails the build.

@OmarB97
OmarB97 merged commit 511c914 into main May 25, 2026
12 of 14 checks passed
OmarB97 pushed a commit that referenced this pull request May 28, 2026
Pre-filter fallback_providers when the agent starts up — entries whose
provider has no credentials (no API key in env, no custom_providers
definition, no base_url with env var set) are dropped silently from the
chain and a startup diagnostic lists them with a hint naming the env
var that would activate them.

Before: runtime fallback loop discovered each failure one-by-one via
401/402/429 errors → wall of red logs, multiple seconds of latency,
confusing UX (the #1 source of "why does my fallback keep 401ing").

After: chain is clean from the first failed API call; startup prints
e.g. "Fallback chain: skipped 4 unconfigured: deepseek, minimax, zai,
kimi-moonshot" with per-entry hints like "set DEEPSEEK_API_KEY in
~/.hermes/.env" or "`hermes auth add nous`".
OmarB97 pushed a commit that referenced this pull request Jun 4, 2026
…ch#37677)

Anthropic enforces two independent ceilings per image:
1. 5 MB encoded byte size
2. 8000 px longest side

Hermes only guarded #1. A tall screenshot (e.g. 1200x12000 at 0.06 MB)
passes every byte check but fails the pixel check, returning a
non-retryable HTTP 400 that permanently bricks the conversation thread.

Fixes:
- error_classifier: add 'image dimensions exceed' pattern to
  _IMAGE_TOO_LARGE_PATTERNS so the 400 is classified as image_too_large
  and triggers the shrink/retry path instead of falling through to
  non-retryable error.
- conversation_compression: check pixel dimensions (via Pillow) even
  when byte size is under the 4 MB target. If max(dims) > 8000, force
  shrink.
- vision_tools._resize_image_for_vision: add optional max_dimension param.
  When set, images exceeding the pixel cap are downscaled even if they're
  under the byte budget. The resize loop now checks both byte AND pixel
  limits before accepting a candidate.

Closes NousResearch#37677
OmarB97 pushed a commit that referenced this pull request Jun 8, 2026
…bes + test-leak fix (NousResearch#40909)

* fix(gateway,windows): reliability — supervisor task, JOB breakaway, status --deep

Three coordinated fixes for the Windows gateway reliability story:

1. CREATE_BREAKAWAY_FROM_JOB on every detached spawn

   The 'hermes update' triggered from the Electron Desktop GUI ran inside
   Electron's job object. Without breakaway, the post-update gateway
   watcher spawned by update — already DETACHED_PROCESS — was still
   reaped when Electron's job tore down, so the gateway never came back
   after a GUI-initiated update. Adds CREATE_BREAKAWAY_FROM_JOB (0x01000000)
   to:
     - hermes_cli/_subprocess_compat.py::windows_detach_flags() — used by
       every helper that calls windows_detach_popen_kwargs(), including
       launch_detached_profile_gateway_restart()
     - The watcher subprocess's own respawn snippet in
       hermes_cli/gateway.py (inlined flags so the watcher's child
       respawn also breaks away)

   _spawn_detached() in gateway_windows.py already had the flag; this
   change brings the rest of the codebase to parity.

2. Per-minute supervisor Scheduled Task — Windows equivalent of
   systemd Restart=always

   Introduces hermes_cli/gateway_supervisor.py and registers it as a
   second Scheduled Task ('Hermes_Gateway_Supervisor', SC MINUTE /MO 1,
   LIMITED rights) alongside the existing ONLOGON task. Every minute,
   the supervisor uses the same gateway.status.get_running_pid() probe
   as 'hermes gateway status' and, if no gateway is alive, calls
   gateway_windows._spawn_detached() (which now includes BREAKAWAY) to
   bring one back.

   Covers every crash mode, not just 'machine rebooted': taskkill,
   OOM, GUI update SIGTERM, parent job teardown. Cheap — one pythonw
   startup per minute when down, one PID-existence check per minute
   when up.

   Wired into both the schtasks-success and Startup-folder-fallback
   install paths via _install_supervisor_best_effort(), and removed in
   uninstall(). Best-effort: a failing supervisor install logs a
   warning but doesn't roll back the primary install.

3. 'hermes gateway status --deep' shows per-probe PASS/FAIL

   Replaces the existing terse '--deep' output (which only printed
   paths) with an actual diagnostic table:
     [1] PID file present
     [2] Lock file held by a live process
     [3] get_running_pid() result
     [4] _pid_exists(pid) — OS-level liveness
     [5] gateway_state.json (state + age)
     [6] Last lifecycle event from gateway-exit-diag.log

   When the high-level summary disagrees with reality, the user can
   see exactly which signal is lying.

Test-leak fix
-------------

tests/hermes_cli/test_gateway_wsl.py::TestGatewayCommandWSLMessages
monkey-patched is_linux/is_wsl/supports_systemd_services to simulate
WSL but did NOT stub is_windows(). On a Windows host, the dispatcher
in _gateway_command_inner takes the is_windows() branch BEFORE the
WSL guidance branch, so the test invoked gateway_windows.install()
for real. install() writes to %APPDATA%\...\Startup\Hermes_Gateway.cmd
— the REAL user Startup folder, never sandboxed by tmp_path — pointing
at the test's pytest-of-<user>/pytest-<N>/.../gateway-service/ wrapper.
When pytest tore down the tmp_path, every subsequent Windows login
flashed a cmd.exe window that failed to find the missing target.

Stubs is_windows=False on all four affected tests:
  test_install_wsl_no_systemd
  test_start_wsl_no_systemd
  test_status_wsl_running_manual
  test_status_wsl_not_running

Defense-in-depth: _build_startup_launcher() now prefixes the launcher
with 'if not exist <target> exit /b 0', so any future stale Startup
entry silently no-ops instead of flashing a console window.

Status enhancements
-------------------

- status() now reports supervisor task presence alongside the existing
  schtasks/Startup info, and nudges the user to reinstall if the
  supervisor isn't registered.
- Deep mode dumps both the supervisor task name + script path.

* fix(gateway,windows): drop the per-minute supervisor task — keep breakaway + deep probes

Earlier in this branch we added a per-minute schtasks-based supervisor to
respawn the gateway after crashes / GUI-update SIGTERMs. The implementation
flashed a brief console window on every firing, which stole window focus.
We tried several variants:

  - cmd.exe wrapper invoking pythonw  -> flashes (cmd.exe is console-subsystem)
  - schtasks /TR pointing at pythonw  -> flashes (uv venv launcher pythonw is
    actually subsystem=Console, not GUI; it respawns the real pythonw)
  - schtasks /TR pointing at base uv  -> still flashes (Task Scheduler-side
    conhost preallocation; documented Windows quirk)
  - XML registration with <Hidden>true>  -> still flashes (<Hidden> only hides
    the task in the Task Scheduler UI, not the spawned window)

Researched what leading projects do:

  - Ollama: GUI-subsystem tray exe + Startup-folder shortcut. No supervisor.
  - Tailscale: real Windows Service via SCM. Session 0, no console possible.
  - Syncthing: --no-console flag inside the binary + Startup folder.
  - openclaw: VBS Run(..., 0, False) wrapper. Suppresses the *window* but
    Super User Q971162 confirms focus-steal still occurs in some cases.

None of these use a per-minute polling scheduled task. The 'auto-restart on
crash' responsibility belongs INSIDE the daemon (Tailscale's in-process
recovery / Ollama's monitor+worker pair) OR is delegated to the Windows
Service Control Manager — not Task Scheduler.

So this commit drops the supervisor entirely. The CREATE_BREAKAWAY_FROM_JOB
fix in _subprocess_compat.py (from commit c1e5fa4) survives — that is the
*real* fix for problem #2 (GUI-update kills gateway): the post-update
watcher in launch_detached_profile_gateway_restart() now breaks out of
Electron's job object, so the gateway respawn watcher survives the GUI
quit and successfully respawns the gateway.

Surviving from c1e5fa4:
  * CREATE_BREAKAWAY_FROM_JOB in hermes_cli/_subprocess_compat.py (fixes #2)
  * Inlined breakaway flag in the watcher respawn snippet in gateway.py
  * hermes gateway status --deep PASS/FAIL probes (fixes #1 — visibility)
  * 'if not exist <target> exit /b 0' guard in _build_startup_launcher
    (fixes #3 — silent no-op for stale Startup entries)
  * tests/hermes_cli/test_gateway_wsl.py is_windows=False stubs (root cause
    of #3 — pytest WSL tests no longer leak Startup entries on Win hosts)

Removed in this commit:
  * hermes_cli/gateway_supervisor.py (entire file)
  * Supervisor section in hermes_cli/gateway_windows.py (~180 lines):
      get_supervisor_task_name, get_supervisor_script_path,
      _build_supervisor_cmd_script, _write_supervisor_script,
      _install_supervisor_task, is_supervisor_task_registered,
      _install_supervisor_best_effort
  * _install_supervisor_best_effort() calls in install() (3 spots)
  * supervisor cleanup block in uninstall()
  * supervisor display lines in status() / status(deep=True)

Future direction (out of scope for this PR): the right place for Windows
'Restart=always' semantics is a real Windows Service installed via
pywin32's win32serviceutil.ServiceFramework — session-0 isolation, SCM
auto-restart, no console window possible. That's a meaningful next-PR
project, not a band-aid.

Tests: 51 pass / 2 pre-existing failures in
tests/hermes_cli/test_gateway_{windows,wsl}.py (the 2 failures are
TestSupportsSystemdServicesWSL cases that fail on origin/main too —
unrelated to this PR).
OmarB97 pushed a commit that referenced this pull request Jun 19, 2026
Phase 1 of the pluggable cron-scheduler refactor (Axis B — the trigger).
No call-site changes; this phase only makes the abstraction exist + tested
in isolation.

Task 1.1: cron/scheduler_provider.py — the EXPERIMENTAL CronScheduler ABC.
  Required surface is name + start; is_available()/stop() carry safe defaults.
  is_available has a no-network invariant. Docstring marks it experimental
  until the Chronos provider (Phase 4) validates the shape.
Task 1.2: InProcessCronScheduler wraps the historical 60s ticker loop, calling
  cron.scheduler.tick(sync=False) exactly as the raw ticker does. Uses
  stop_event.wait(interval) for responsive stop (both raw tickers already do).

Tests: ABC-is-abstract, default-is_available, the InProcess loop drives tick
and stops, stop() no-op, and test_abc_growth_stays_additive (the forward-compat
guard: required abstractmethods must stay exactly {name, start}, so the three
Phase-4 hooks land as NON-abstract additions).

tick() internals in cron/scheduler.py are byte-unchanged (only new file added).
Phase 0 characterization tests still green. Full tests/cron/: 445 passed.
OmarB97 pushed a commit that referenced this pull request Jun 30, 2026
* fix(windows): harden gateway scheduled task

* fix(windows): launch gateway scheduled task via console-less wscript

The Scheduled Task ran the gateway through cmd.exe, which allocates a
console. During logon Windows broadcasts CTRL_CLOSE_EVENT to console
process groups, reaping cmd.exe and the half-initialized gateway with
STATUS_CONTROL_C_EXIT (0xC000013A) - which Task Scheduler treats as a
user cancel, so RestartOnFailure never fires and the gateway vanishes on
every reboot (issue NousResearch#45599 root cause #1).

Add a console-less .vbs launcher (wscript.exe -> pythonw.exe, both
GUI-subsystem) mirroring the gateway.cmd env + argv, and point the task
action at it. The .cmd stays for the Startup-folder fallback and /Run.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Jeff <jeffrobodie@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
OmarB97 pushed a commit that referenced this pull request Jul 5, 2026
…ture

get_copilot_api_token now returns (api_token, base_url); the auth-remove
suppression test still mocked it as a bare string, mis-unpacking into the
credential-pool seed path and failing with 'No credential #1'.
OmarB97 pushed a commit that referenced this pull request Jul 5, 2026
…_id signature churn

Two independent bugs evicted the cached gateway AIAgent on every turn,
preventing the prompt cache from ever warming:

1. Model normalization mismatch: the post-run fallback-eviction check
   compared _agent.model (stripped in AIAgent.__init__) against the raw
   _resolve_gateway_model() config string. For vendor-prefixed config on
   native providers (e.g. 'deepseek/deepseek-v4-pro' vs 'deepseek-v4-pro')
   this was always unequal, so the agent was evicted after every
   successful run. Normalize _cfg_model the same way (skip aggregators).

2. Discord triggering message_id leaked into the cached system prompt via
   build_session_context_prompt()'s Discord IDs block. message_id changes
   every turn, so the agent-cache signature (computed from the ephemeral
   prompt) changed every Discord turn -> rebuild every message. The id is
   now injected per-turn into the user message (where per-turn content
   belongs and does not touch the cache signature); the cached IDs block
   carries a static pointer to it, preserving reply/react/pin via the
   discord tools.

Adapted from NousResearch#28846. Bug #1 fix is the contributor's; bug #2 reworked to
be non-destructive (keeps the triggering-id capability instead of deleting
it). Redundant auto-reset eviction (already on main via NousResearch#9893/NousResearch#48031) and
the wrong-premise reset_context_note plumbing from the original PR were
dropped.

Co-authored-by: Hermes Agent <hermes@nousresearch.com>
OmarB97 pushed a commit that referenced this pull request Jul 5, 2026
… fail on '(empty)' sentinel

Two related bugs caused subagent delegation to silently return empty summaries
with 0 tokens when the user configured delegation.provider=bedrock alongside
delegation.base_url=https://bedrock-runtime.<region>.amazonaws.com.

Root cause #1 — misrouting in _resolve_delegation_credentials():
  The configured_base_url branch unconditionally forced provider='custom' and
  api_mode='chat_completions', only specializing for chatgpt.com, anthropic,
  and kimi hosts. Bedrock (and other native-SDK providers) fell through as
  'custom' + chat_completions, which then POSTed OpenAI-shaped JSON at
  Bedrock's native API. Bedrock rejected the payload and returned nothing,
  which looked like an empty LLM response to the child agent.

  Fix: when provider is one of {bedrock, vertex, google, google-genai}, skip
  the base_url short-circuit and fall through to resolve_runtime_provider(),
  which knows how to construct the proper SDK client. base_url can still be
  forwarded through that path for regional overrides.

Root cause #2 — '(empty)' sentinel accepted as success:
  After N retries of empty LLM responses, run_agent.py emits the literal
  string '(empty)' as final_response. _run_single_child then hit
  `elif summary:` — '(empty)' is truthy, so status became 'completed' and
  the parent surfaced a blank result with no error. Users saw api_calls=4,
  tokens=0, duration~0.4s, status=completed.

  Fix: treat final_response.strip() == '(empty)' as a failure so the parent
  surfaces it instead of silently accepting zero-content 'success'.

Both paths were reproduced in a live Hermes TUI session on us-west-2 Bedrock
(provider=bedrock, model=us.anthropic.claude-sonnet-4-6) and are covered by
new tests in tests/tools/test_delegate.py.
OmarB97 pushed a commit that referenced this pull request Jul 20, 2026
…onnect ladder can't freeze silently (NousResearch#66377)

The Telegram gateway could go silently deaf for hours: the reconnect ladder
stalled mid-way (e.g. "attempt 4/10, reconnecting in 40s" then nothing) while
the process stayed active(running), so Restart=always never fired.

Root class: every recovery path — the ladder's re-entry
(_schedule_polling_recovery), the pending-update probe (_probe_pending_updates),
and PTB's error callback — gates new recovery on _polling_error_task.done(). If
that single task wedges on any hung await, all recovery returns early forever
and nothing retries.

The heartbeat loop is a separate task, so make it an independent, cause-agnostic
watchdog: if the same recovery task stays in-flight past
_POLLING_ERROR_TASK_STUCK_TIMEOUT (300s — well beyond a healthy ladder attempt's
bounded stop+drain+start+backoff), force a retryable-fatal so the background
reconnector rebuilds the adapter instead of relying on the frozen ladder. This
guarantees progress regardless of *where* the stall is (issue direction #1),
tracked locally so no task-assignment site needs to change.

Also salvages @koduri-mahesh-bhushan-chowdary's NousResearch#66492 (drain-await timeout),
which closes the one concrete wedge vector documented in the incident
(_drain_polling_connections' unbounded shutdown()/initialize() on a wedged
CLOSE-WAIT pool). The watchdog covers the rest of the class.

Co-authored-by: Koduri Mahesh Bhushan Chowdary <mkoduri73@gmail.com>
OmarB97 pushed a commit that referenced this pull request Jul 20, 2026
…reaming

Two real render-cost wins found by inspection (no behavior change):

1. Sidebar re-rendered on every stream token. $sessionStates is republished on
   every message delta (tens/sec during a turn), and the derived ID computeds
   ($workingSessionIds, $attentionSessionIds, $backgroundRunningSessionIds)
   allocated a fresh array each time. nanostores notifies on !==, so the whole
   ChatSidebar + every mounted row re-rendered per token even when the working/
   attention/background set was unchanged. Return the previous array reference
   when the contents match → nanostores skips the notify unless the set actually
   changes. Turns streaming from O(visible rows)/token into O(0) for the sidebar.

2. Tool rows normalized the FULL uncapped detail every render. `looksRedundant`
   (lowercase + whitespace-collapse over the entire read_file/terminal payload)
   ran twice in the ToolEntry render body, so every completed tool re-normalized
   its whole output on every stream tick of the running message. Memoize on the
   view fields so it recomputes only when the tool's content changes.

Both are correctness-preserving (stable refs + memoization). The CI stream
scenario drives $messages directly, not the publishSessionState path, so it
won't reflect #1 — verified by inspection.
OmarB97 pushed a commit that referenced this pull request Jul 20, 2026
Blocking #1 — gateway-connecting-overlay.tsx reduced-motion regression:
the top `if (reduce) setPhase('gone')` fired unconditionally on mount
whenever reduce-motion was on, so every OS reduced-motion user lost the
CONNECTING overlay during cold boot entirely (jumped to 'gone' before the
gateway was even open). The intent was to skip the exit *choreography*,
not to skip showing the overlay. Removed the unconditional top block and
the redundant nested preview block; kept only the third branch
(`gatewayState === 'open' && shownRef.current` → `reduce ? 'gone' :
'text-out'`) which correctly gates the short-circuit on connect. Also
fixed `if(reduce)` missing-space, 6-space misindent, and the same 3-line
comment pasted three times.

Nit #1 — tsconfig excludes e2e, so specs were never typechecked in CI.
Added tsconfig.e2e.json (extends base, includes e2e/ + playwright.config.ts,
adds @playwright/test types) and wired it into the typecheck script. This
surfaced three latent type errors that are fixed in the same commit:
  - fix-electron-tracing.ts: `app._context` and `electron._playwright` are
    private APIs — added `as any` on the access before the existing cast.
  - playwright.config.ts: `reducedMotion: 'reduce'` directly under `use:`
    is not a valid UseOptions property in playwright 1.58; it's a
    BrowserContextOption accessed via `contextOptions: { reducedMotion:
    'reduce' }`. The old form was silently ignored at runtime, so
    reduced-motion emulation wasn't actually active — screenshots could
    catch overlays mid-fade (exactly what the comment warned about).

Nit #2 — fix-electron-tracing.ts reaches into Playwright internals
(_playwright, _allContexts, _context) with no public contract. Added a
header comment calling out the `@playwright/test` exact pin (=1.58.2) so a
future bump knows to re-verify the private symbols still exist.

Nit #3 — main.ts TEST_WORKER_INDEX block had stray 6-space indentation.

Verified: tsc -p . && tsconfig.electron && tsconfig.e2e → 0 errors;
vitest boot-failure-overlay (3/3) + boot-failure-reauth (21/21) pass;
npm run build clean; playwright e2e/boot-failure.spec.ts 2/2 pass.
OmarB97 pushed a commit that referenced this pull request Jul 29, 2026
…native extension)

unicode61 indexes a CJK run as ONE token, so 2-char Korean terms (일본,
구글, 우리, ...) can never match it and the trigram tokenizer needs >=3
chars per term — any query containing a 1-2 char CJK token falls through
to a LIKE full-table scan (measured 3-6.4s CPU per query on a 6.8GB
production state.db; the #1 base cost behind a 12.4s session_search
average on CJK workloads).

This ships a ~250-line loadable FTS5 tokenizer (no deps) that wraps
unicode61: maximal CJK runs inside its tokens are re-emitted as
overlapping character bigrams (Lucene CJKAnalyzer semantics), everything
else passes through unchanged. FTS5 phrase semantics turn consecutive
sub-tokens into exact substring matching down to 2-char terms at index
speed.

Build: native/fts5_cjk/build.sh -> ~/.hermes/lib/libfts5_cjk.so
(override: HERMES_FTS5_CJK_SO).

Salvaged from PR NousResearch#65544; the schema integration lands separately on the
v23 external-content layout.
OmarB97 pushed a commit that referenced this pull request Jul 29, 2026
…add same-pid self-reclaim guard

Hardening on top of the salvaged dead-PID lease reclamation from PR NousResearch#65775
(@the3asic):

- Probe via psutil.pid_exists (hard dependency; CONTRIBUTING.md critical
  rule #1) with the contributor's os.kill(pid, 0) POSIX probe retained
  only as a scaffold-phase fallback when psutil is missing.
- Same-process holders (pid == os.getpid()) are never probed and never
  self-reclaimed — another thread's live lease is owned by the lease
  refresher/release path.
- Any probe doubt (exceptions, permission errors) conservatively keeps
  the lease until normal TTL expiry; Windows stays TTL-only.
- Tests: psutil-first dead-pid reclaim (probe call pinned), os.kill
  fallback path, probe-doubt keeps lease, same-pid no self-reclaim,
  legacy holder + Windows paths assert NO probe via either API.
OmarB97 pushed a commit that referenced this pull request Jul 29, 2026
…ch#67140)

The background write guard decided ownership from `isinstance(usage_rec, dict)`,
so a local skill with NO usage record passed. That successful write called
bump_patch(), which created a `created_by: null` record — and the identical
write was refused from then on. "Allowed exactly once, then never" is a race
with our own bookkeeping, not a policy. Reproduced on main: patch #1 succeeds,
patch #2 with the same arguments is refused.

Option B from the issue. Option A (split `session_review` from
`scheduled_curator` and let the session fork patch user-owned skills it
consulted) would widen autonomous write permission onto skills the user owns
with no user present to consent — wrong direction for a no-user-present actor.

- skill_manager_tool: missing and explicit-null records now resolve
  IDENTICALLY, both fail closed. The refusal names the reason and points at
  `hermes curator adopt <name>`.
- background_review: both review prompts told the reviewer to patch any skill
  consulted in the session and claimed pinned skills could be improved, while
  enforcement refused both. Prompts now list pinned, external, and user-owned
  skills as protected, and tell the reviewer to RECOMMEND adoption instead of
  attempting a write that will be refused.
- skill_usage: document that `created_by` is a curator-management policy flag,
  not a provenance claim, and add `is_curator_managed()` so call sites read as
  the question they ask. Field name retained — it is on disk in every
  `.usage.json` and renaming would strand those records.
- curator CLI: `hermes curator list-unmanaged` itemizes unmanaged skills with
  the reason each is unmanaged (completes the NousResearch#67139 spec).

Foreground writes are untouched: a user-directed edit to a user-owned skill
still works, including on pinned skills.

Sibling tests: 9 failures in test_skill_manager_tool.py were fixtures that
created record-less skills to exercise OTHER guards (consolidation-delete,
read-before-write) and relied on ownership falling through. Fixed at the
fixture, since the real curator only ever operates on managed sediment. One
test asserted the old "manually authored" wording; rewritten to assert the
behavior contract instead of the string.

Validation: 274 targeted tests + all 7 background-review files (60 tests) pass.
E2E on a temp HERMES_HOME (30 checks) covers the flip, foreground writes,
adoption unblocking, pin semantics, prompt/enforcement parity, and the new verb.
Each new test sabotage-verified: revert the fix, confirm it goes red.

Fixes NousResearch#67140
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant