fix(windows): residual encoding gaps — MCP stdio, gateway update file-I/O, STT/TTS, desktop spawn, bootstrap ver-stub - #71014
Merged
Merged
Conversation
On Windows, pipe I/O can deliver non-UTF-8 bytes at chunk boundaries, causing `UnicodeDecodeError` when the MCP SDK's `TextReceiveStream` uses `errors="strict"`. Set `encoding_error_handler="replace"` on `StdioServerParameters` so undecodable bytes become U+FFFD instead of crashing.
Follow-up to the salvaged #38985: guard the 4 bare read_text/write_text sites its allowlist missed (google_chat thread-count store + oauth JSON) and add whatsapp/google_chat to the AST guard test's file list.
…desktop backend env Two gaps found auditing the decode-crash cluster: 1. suppress_platform_ver_console() only ran in hermes_cli.main processes; slash workers, tui_gateway/entry, run_agent, batch_runner, and cli.py import only hermes_bootstrap and were exposed to both the console flash and (on Python 3.11.0/3.11.1, which lack CPython's encoding='locale' fix) a UnicodeDecodeError inside platform.win32_ver() under PEP 540 — the crash #69413 reported. Move the stub into hermes_bootstrap so every entry point gets it; the _subprocess_compat copy stays for non-bootstrap callers. 2. The desktop Electron spawn built the backend env without PYTHONUTF8, so anything the Python child emitted before hermes_bootstrap ran (interpreter startup errors, pre-bootstrap tracebacks) decoded with the locale default. Re-port of PR #56499's env half (echoriver89) to backend-env.ts (original targeted the deleted backend-env.cjs); explicit user setting wins.
teknium1
force-pushed
the
hermes/hermes-dc3471e3
branch
from
July 24, 2026 21:07
c64f465 to
8ac0353
Compare
alt-glitch
added
type/bug
Contributor
૮ >ﻌ< ა ci reviewran on 8ac0353 all good! |
This was referenced Jul 24, 2026
17 tasks
This was referenced Aug 3, 2026
rlaope
added a commit
to rlaope/hermes-agent
that referenced
this pull request
Aug 3, 2026
Bare Path.read_text()/write_text() fall back to locale.getpreferredencoding() — cp1252/cp936 on Windows — so the UTF-8 agent_summary.json artifact writes mojibake or raises UnicodeDecodeError there. Pin encoding="utf-8" on the persist path and its test read, per the NousResearch#71014 read_text campaign.
1 task
This was referenced Aug 7, 2026
teknium1
pushed a commit
that referenced
this pull request
Aug 8, 2026
…indows
`tests/hermes_cli/test_plugins_cmd.py::TestNoAutoActivation::test_compressor_default_ignores_plugin`
fails on every Windows machine:
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8f in
position 47744: character maps to <undefined>
The test reads `run_agent.py` back as text to assert a removed comment is
gone, but called `open()` with no `encoding=`. Python then falls back to
the locale preferred encoding, which is cp1252 on a default Windows
install rather than UTF-8. `run_agent.py` contains nine bytes cp1252
leaves undefined, so the read raises before the assertion is reached. On
Linux and macOS the preferred encoding is UTF-8 and the same line is
fine, which is why CI never caught it.
That one line is the only active failure. The rest of this change closes
the same gap in the files it touches, which `scripts/check-windows-footguns.py`
flags and which the #71014 read_text campaign has been working through
elsewhere in the tree:
- `tests/hermes_cli/test_plugins_cmd.py`: nine bare `write_text`/`read_text`
calls writing YAML manifests, config and plugin sources
- `tests/tools/test_web_tools_truncate.py`: reads stored extracted web text,
which is arbitrary content from the internet
- `tests/stress/test_atypical_scenarios.py`: writes and reads worker task
ids and a barrier file
All three files are now clean under `check-windows-footguns.py`.
Reads go through `Path.read_text(encoding="utf-8")` rather than
`open(...).read()`, which also closes the handle instead of leaving it to
the garbage collector. On Windows a live handle blocks tmpdir cleanup, so
that part is not cosmetic either.
No new test. The repaired test is the regression coverage: it fails
before this change and passes after, on Windows.
sliamh11
added a commit
to sliamh11/hermes-agent
that referenced
this pull request
Aug 9, 2026
…ced reload (#2) * Inspired by Cursor: MCP config context variables (${userHome}, ${workspaceFolder}, ...) * fix(video): read analyze inputs through terminal backend * fix(video): route terminal-backend reads through the shared media resolver Follow-up on the salvaged commit: replace the hand-rolled file_ops python3 exec-read with tools.image_source.resolve_image_source(permitted=('video',)), so video_analyze gets the same pipeline as vision_analyze — media-cache host reads, bounded head -c sandbox exec (no python3 dependency in the sandbox image, no unbounded base64 stream), lazy env bring-up (#62825), the credential-read guard, and the 50MB ingest cap. * fix(vision): retry container exec-read for Docker cold-start, surface stderr (#76566) Under the Docker terminal backend, vision_analyze's first exec-read sometimes returned empty / non-zero against a freshly started container, producing 'could not read <path> inside the sandbox' on a file the agent could cat seconds later. Cold pipe setup on the first exec against a new container, not a permissions or mount problem. Retry once after a short delay (150 ms covers Docker exec warm-up without making a real failure feel sluggish). When every attempt still fails, fold the container's first stderr line into the raised error so the user can tell 'no such file' from 'permission denied' instead of staring at one opaque message. Tests cover the retry-then-succeed path, the diagnostic-on-exhausted path, and confirm the existing single-attempt raise is preserved. * feat: --resume latest keyword and --in DIR launch flag --resume latest resolves the most recent session through the same workspace-scoped MRU lookup as -c (TUI source first under --tui, with classic-CLI fallback). --in DIR chdirs before session resolution so the lookup keys off DIR's workspace, and pins the session there by skipping the recorded-cwd restore. Requested by @Jeff9James: hermes --tui --resume latest --in ./dir * feat(skills): add email-inbox-triage * chore(skills/email-inbox-triage): tighten to hardline standards - description 219 -> 58 chars - author credits Ben Barclay (benbarclay) first - modern section order; trimmed template safety boilerplate into step-local rules and a skill-specific verification checklist - tests at tests/skills/test_email_inbox_triage_skill.py (9 passing) - docs regen scoped: per-skill page + one catalog row + one sidebar line * fix(image_gen): confine generation source images to the terminal backend image_generate and video_generate forwarded model-supplied local paths to provider plugins, which read them off the HOST filesystem regardless of terminal backend — inconsistent with the confinement boundary vision/video analysis enforce (GHSA-gpxw-6wxv-w3qq), and broken for sandbox-only files. New dispatch-layer chokepoint (_confine_source_images): under a non-local backend, path-like image_url / reference_image_urls resolve through tools.image_source (media-cache host reads, bounded in-sandbox exec-read, lazy env bring-up, credential guard, 50MB cap) and reach every provider as data: URLs — which all backends already accept. URLs/data: pass through; local backend is a no-op. xai_video_edit/extend already require public HTTPS URLs, so no change needed there. * fix(security): redact terminal exception results and ACP stderr logs (#77484) Closes the last two emission gaps from #77484: - tools/terminal_tool.py: both exception paths (generic except and TERMINAL_DEGRADED_MODE=fail) returned raw str(e) + traceback.format_exc() to the model — only the logger copy was redacted. Exception text can embed the failing command line and any secrets inline in it; both fields now pass through redact_sensitive_text. - acp_adapter/entry.py: _setup_logging cleared root handlers and installed a plain logging.Formatter, bypassing redaction entirely on ACP stderr. Now uses RedactingFormatter like every other logging surface. The other three gaps from #77484 (process(list), *_KEY regex variants, control-char splits) were fixed in #80964/#80965. * fix(read_file): warn when PDF pages yield no text (scanned-image coverage gap) anydoc converts the PDF text layer only and emits no image placeholders or page markers, so a mostly-scanned PDF extracts 'successfully' into section headers with empty bodies — silent data loss the model cannot detect. Count per-page text via poppler pdftotext and prepend an EXTRACTION COVERAGE WARNING naming the empty pages and the recovery path (pdftoppm + vision_analyze, or the ocr-and-documents skill). Found on a 311-page HOA resale package where 198 scanned pages (CC&Rs, Bylaws, Articles, insurance certs) vanished without a trace. * fix(terminal-tool): redact terminal error result fields Force-redact every terminal exception and traceback field before JSON serialization, including environment creation, background startup, exhausted foreground retries, and the outer catch-all. Preserve the current command-aware, opt-out-respecting redact_terminal_output(output, command) behavior for successful output. * Port from superagent-ai/grok-cli: description-aware slash-menu fuzzy scoring * Port from superagent-ai/grok-cli: directory-chain AGENTS.md loading * feat: add new FAL video families and image models Video (plugins/video_gen/fal): Seedance 2.5, MiniMax H3, Seedance 2.0 Mini, FLUX 3, Grok Imagine 1.5, Gemini Omni Flash (i2v-only). New family capability flags: - duration_int: endpoints that take duration as a JSON integer - resolution_aliases: maps 720p/1080p-style values onto non-standard enums (H3's 768P/2K/4K) - image_drop_keys: strips keys the family's i2v endpoint rejects (aspect_ratio on Seedance 2.5 / H3 / Grok 1.5) Image (tools/image_generation_tool): Seedream 5.0 Pro (+edit) and Lite, Ideogram V4 instant + fast, Qwen Image 3 (+edit), MAI Image 2.5 Pro, Nano Banana 2 Lite (+edit), Recraft V4.1. Every new endpoint live-tested against fal.run through the real payload builders + submit path: 18/18 pass (t2v, i2v, t2i, and edit probes). Note: several new endpoints return HTTP 409 from the Nous Portal FAL proxy allowlist until it is updated portal-side; BYOK FAL_KEY works today and the existing 4xx guidance message covers it. * Inspired by Cursor: fail-closed hook semantics + exit-code-2 blocking * fix(learn): extend existing skills during relearning * fix(learn): process large sources incrementally * fix(tools): preserve document extraction boundaries * fix(read_extract): keep scanned-PDF coverage warning on the backend bytes path The salvaged bytes path (_extract_anydoc_bytes) bypassed the coverage check added in #81680. Materialize transferred PDF bytes in a host temp file for the pdftotext scan, and name the backend-visible path in the recovery command rather than the temp file. * docs: document read_file document extraction and the scanned-PDF coverage warning * fix: post-merge audit follow-ups for #81138/#81139/#81141/#81148 Four fix-forwards from the adversarial post-merge audit of the Aug 7 unreviewed merge batch: - estop (#81148): is_engaged() now fails SAFE (engaged) on stat errors; the gateway estop gate lets recognized slash commands and replies owned by in-flight work (update prompts, clarify, slash-confirm, tool approvals, running sessions) through instead of consuming them; new gateway /pause [reason|off] command gives messaging-only operators an in-band engage/resume path (busy_policy=dispatch so it works mid-run). - cron monitor mode (#81138): execution-mode invariants (monitor x no_agent, monitor_script x monitor_url, no_agent-requires-script) now have ONE owner (_validate_job_mode_invariants) called from BOTH create_job and update_job, so the create-time invariant can no longer be silently violated through the update door. - cron notepad (#81139): remove_job now clears the job's notepad rows (clear_notepad was dead code -> orphaned KV state forever); clear is best-effort and no-ops without creating notepad.db. - delegation batch gate (#81141): template-marker regex narrowed to multi-word placeholder shapes only (<feature name>, {file_path}) so generics (Vec<T>), HTML tags, JSON snippets, glob braces and f-string style no longer reject legitimate batches; duplicate-goal rejection removed (best-of-N fan-outs are legitimate). * fix(api-server): mark replayed tool calls completed in Responses output items The non-streaming /v1/responses path built function_call and function_call_output output items with no status field (and no item id), while the SSE streaming path correctly emits status in_progress -> completed. Spec-strict OpenAI clients reading the non-streaming output array could interpret the status-less function_call items as pending calls the CLIENT must execute — but these tools were already executed server-side by the Hermes agent and are replayed for structured tool UI only. Reported by a community user whose GPT-5.6 client concluded 'a server should not tell an OpenAI client to execute a tool the server already executed itself'. - _extract_output_items now stamps status: completed and spec-shaped item ids (fc_/fco_) on replayed items, matching the streaming path - test updated to pin status + id shape - docs example updated + explicit note that output tool calls are replayed, never pending * feat(skills): add github-issue-to-pr * chore(skills/github-issue-to-pr): de-router, fold in maintainer issue-to-PR discipline Rewrote from a sibling-skill routing table into a skill that carries its own procedure, and folded in generalized rules from maintainer practice: - full-thread reads (gh issue view --comments; newest comment = live state) - duplicate-PR sweep (issue number + keyword variants) before any code - design-intent check via git log -p -S alongside premise reproduction - fix the class: sweep sibling call sites into the same PR - sabotage run: prove the regression test fails without the fix - open the PR immediately (PR dispatches CI; CI latency is the long pole) - close the loop: comment the issue with the PR link Also: description 205 -> 59 chars, author credits Ben Barclay first, modern section order, boilerplate trimmed, tests (10) incl. a router-pattern guard, scoped docs regen. * feat(image_gen): add FAL Nano Banana 2 model * test: convert NB2 catalog snapshot test to invariants; live-verified t2i+edit Follow-up on the cherry-picked contribution from @michaelsam94 (#51794): replace display-string/exact-value snapshot assertions with invariant checks per the no-change-detector-tests policy. Live-tested fal-ai/nano-banana-2 and fal-ai/nano-banana-2/edit through the real payload builders: both pass. * fix(read_file): surface document extraction failures instead of the generic binary-file error When extraction of a binary document format (.pdf, .docx, .xlsx, Office, EPUB…) fails for a specific reason — the anydoc size cap, an encrypted or malformed file — read_file previously swallowed the ExtractionError at debug level and fell through to the generic 'Cannot read binary file' guard, so the agent never saw the actionable reason (e.g. 'Document too large to convert (N bytes, limit is 52,428,800)'). read_file now returns the specific extraction failure for binary document formats. Fallthrough behavior is preserved where a raw read is still useful: .ipynb (plain JSON) and converter-unavailable PDFs keep their historical raw-read path, and the 'Unsupported document type' shape (no extra information) keeps the generic guard. Follow-up to #80004, where the size-cap message was being generated but never reached the agent. * fix(docker): read attached binary files in backend (#76577) * fix(docker): close the cold-container and multi-backend gaps in attachment delivery Follow-ups on the salvaged commit: 1. get_cache_directory_mounts() now CREATES missing staging dirs instead of skipping them. Docker snapshots the mount list at container creation, so a dir born later (first attachment, first clipboard image) dangled for the life of a persistent container. Empty bind mount costs nothing. 2. to_agent_visible_cache_path() translates per-backend instead of docker-only: docker/modal -> /root/.hermes, ssh/daytona/vercel_sandbox -> ~/.hermes (shell-expanded remotely; bytes arrive via file sync), local/ singularity keep the host path (apptainer auto-binds the host home). Mirrors the proven _agent_cache_base_for_env heuristics. Updated the two mount-list tests pinning the old skip behavior; added per-backend translation coverage. * feat(read_extract): label each unreadable PDF gap with its preceding section text The coverage warning listed bare page ranges, which tells the agent WHERE the gaps are but not WHAT they contain — its only options were guessing or OCRing everything. Each gap is now labeled with the last text extracted before it (usually a section divider page), so the agent can decide which gaps it actually needs and render/OCR only those. Gap list capped at 20 entries with a summary line for pathological alternating documents. * test(tests): stabilize write_json concurrent serialization flake * fix(slack): insert resolved display names literally when humanizing mentions _humanize_user_mentions rewrites <@UID> to @DisplayName by passing the resolved name as re.sub's replacement, where re parses it as a template. A display name is arbitrary user-set text, so the escapes in it are the user's characters, not regex syntax: dev\ops -> re.error: bad escape \o a\1b -> re.error: invalid group reference 1 \g<0> -> expands to the whole match, silently putting the opaque <@UID> back — the token this method exists to remove The trigger-text call site sits in _handle_slack_message outside any try, and both Bolt event handlers await it bare, so the raise takes the whole inbound message down: every message mentioning that person is dropped. Pass the replacement as a function instead — re does no template parsing on the return value, so the name lands verbatim. Same shape the Matrix adapter already uses for its outbound mention rewrite. * feat(skills): add google-workspace-daily-brief * chore(skills/google-workspace): fold daily-brief into references/, not a sibling skill The brief is single-connector (every command comes from google-workspace), so it ships as references/daily-brief.md with a pointer + load trigger in SKILL.md — progressive disclosure instead of a new skill-index entry. Contributor's procedure preserved (half-open day windows, mail-to-meeting linking with fuzzy-match discipline, 7-section brief, bounded actions); credit noted in the reference header. Tests (8) guard the wiring and disciplines. Version 1.1.0 -> 1.2.0. * fix(curator): protect cron skills referenced by absolute path 4c2961c51 added referenced_skill_names() so the curator never archives a skill a cron job depends on — paused jobs and infrequent schedules would otherwise age their skills out and the next run fails to load them. 62972060c then taught the scheduler that jobs may store ABSOLUTE skill paths, normalizing them through normalize_skill_lookup_name before skill_view. The protection set kept returning the raw string, so it now holds a full path while the curator matches it against bare skill names. Those jobs silently lost their protection: the skill is archived, and the next fire logs a warning and runs the job without its instructions. Canonicalize each reference the same way the scheduler resolves it, with a deferred import and a verbatim fallback so a resolver failure can never drop a name (referenced_skill_names has exactly one caller, the curator's protection lookup, so nothing else sees the change). * fix(docs): retain prior builds' hashed assets across Pages deploys Pages serves exactly the newest artifact, so every push-triggered deploy deleted the previous build's content-hashed JS/CSS while edge caches (max-age=300, stale-while-revalidate=3600) kept serving HTML that referenced them. With deploys landing every ~15-30 min, docs pages spent most of the day pointing at 404'd bundles — search (pure client JS) was the loudest casualty. Fix: keep a rolling 14-day pool of hashed assets (en + zh-Hans) in the Actions cache and union-merge it into each deploy artifact, current build authoritative on collision (cp --update=none). Stale HTML and already-open tabs now keep resolving across any number of deploys. * fmt(js): `npm run fix` on merge (#81849) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * feat(skills): add meeting-action-items * chore(skills/meeting-action-items): tighten to hardline standards - description 178 -> 59 chars - author credits Ben Barclay (benbarclay) first - dropped phantom 'Linear' connector from prose (points at notion/ github-issues/user's tracker instead) - Hermes-tool framing (read_file for transcripts) - template boilerplate folded into step-local rules and skill-specific verification - tests at tests/skills/test_meeting_action_items_skill.py (10 passing, incl. phantom-connector guard and reconcile-before-create discipline) - docs regen scoped: per-skill page + one catalog row + one sidebar line * feat(agent): read_window_below tool — which OS window is underneath the desktop app Desktop-gated (desktop_ui toolset) metadata-only window awareness: the agent can ask which application window sits directly behind the Hermes window (app, title, bounds — never pixels). Rides the same blocking bridge as read_terminal: the gateway emits window.read.request and the renderer answers window.read.respond. * feat(desktop): answer window.read.request with the window below New electron/window-below.ts: pure z-order picker (walks past our own pid, first other-process window whose bounds overlap ours) over get-windows' front-to-back enumeration, with the Linux xprop stacking order reversed to match (EWMH _NET_CLIENT_LIST_STACKING is bottom-to-top). Main answers the hermes:window:readBelow IPC; on macOS other apps' titles pass through only when Screen Recording is already granted — never prompted for. * build(desktop): stage get-windows like node-pty get-windows@9.3.0 (MIT, zero runtime deps on macOS/Linux) is external to the esbuild bundle and staged into dist/node_modules per target platform: the universal Swift helper on macOS, the prebuilt N-API binding on Windows (fail-closed magic-byte validation), nothing on Linux (xprop at runtime). The staged lib/windows.js is rewritten to load the binding directly so @mapbox/node-pre-gyp's tree stays out of the package. * test: pin read_window_below into the toolset + post-hook contracts, appease eslint The desktop_ui and post-hook ownership contract tests enumerate their tool sets exactly — add read_window_below to both (plus the executor-path parametrize case). Lint: sorted type import, explicit GetWindowsModule type instead of an import() annotation, curly + blank-line style. * fix(build): win32 get-windows staging must skip the tarball's bundled darwin binding The published tarball ships lib/binding/napi-9-darwin-unknown-arm64 on every platform, so a real Windows host has both it and the downloaded win32 binding — the classify-everything gate threw on the darwin dir and killed every Windows pack. Stage only bindings naming the target platform (classify still rejects impostors), stop copyGlobByExt from recursing into lib/binding, and add a version tripwire so a get-windows bump fails the build until the lib/windows.js rewrite is re-verified. Also from review: the renderer answers window.read.respond with empty text when the IPC invoke rejects (older shell / main-side throw) instead of stalling the tool's 30s timeout; the tool schema discloses that sibling Hermes windows are skipped; docs gain read_window_below in both references. * fix(tts): split long speech by provider and platform limits Salvage of PR #17973 by @TKCen (Sebastian Hänisch), re-implemented on current main to preserve speed/instructions/provider params, prepare_spoken_text normalization, OPUS_VOICE_PLATFORMS, is_write_denied path security, microsecond timestamps, and the streaming-TTS gate. - Split long TTS text into provider-safe chunks instead of truncating - Pack generated audio against platform upload limits (Discord 10MB, Telegram 50MB, configurable via tts.delivery_profiles) - Combine chunks with ffmpeg (OGG/Opus re-encoded, MP3 stream-copied) - Multi-file delivery when combination fails or would exceed limits - Remove hard [:4000] truncation from all callers (cli.py, voice.py, gateway/run.py, gateway/platforms/base.py) - Gemini TTS raises ValueError instead of silently truncating when composed prompt exceeds the provider limit Simplify-code fixes: removed dead all_touched_paths set, added try/finally for scratch file cleanup on exception, clean error response on chunk failure instead of leaking stale file_path. * fix(desktop): an empty HUD thread shouldn't paint a blank panel A fresh thread has nothing to show, but the HUD showed a slab of frosted glass above the bar anyway. Vibrancy is the window's whole content view, so it frosts the full rectangle — fine while the band always filled the window, wrong the moment there is no transcript to fill it. It stays off until there is something to back. The sheet had a 12px floor for the same reason: the breathing room above the first row was added in CSS, so a zero-row transcript still measured 12. It is folded into the measured height now and only applies when there are rows. * fix(gateway): support Docker /workspace media paths in gateway delivery Translate MEDIA paths under configured Docker volume mounts (and the default persistent /workspace) to host paths before media delivery validation, using longest container-prefix match so host:/workspace and /output export mounts work. * fix(gateway): widen container->host media translation to home, cache, and in-process gateways Follow-ups on the salvaged commit (#37207 by @charzhou): - Persistent /root home mount translates too: an agent writing /root/out.png produced a real host file under <sandbox>/docker/default/home the gateway could not find. - /root/.hermes cache mounts translate to the HOST cache (longest-prefix beats the home mount), so MEDIA:<agent_visible_image> paths deliver. - /root/.hermes/* OUTSIDE a cache mount never translates through the home mount: those are the sandbox's credential copies (.env, auth.json) that sit outside the host-side denylist prefixes — fail closed. - Run the idempotent terminal-config->env bridge before mount parsing so in-process gateways (Desktop backend, hermes serve) see the active backend and docker_volumes (covers #42299's /output case there too). * feat: add DCP context engine Cherry-picked from PR #20774 by @jmmaloney4 (jmmaloney4@gmail.com). Original commits: 4420e0b0, 44247545, bac2955d. DCP-style model-guided context engine behind context.engine: dcp. Adds compress tool, outbound API-call transforms, automatic dedup/purge, and DCP-compatible config surface. Closes #20717 Co-Authored-By: Jack Maloney <jmmaloney4@gmail.com> * fix: rewire DCP context engine to current main architecture Fixes 13 issues found in PR #20774 review: 1. Wiring: engine selection moved from run_agent.py to agent/agent_init.py (where init_agent lives on current main). Transform hook moved from run_agent.py to agent/conversation_loop.py (where run_conversation lives). 2. Prompt caching: replace copy.deepcopy with copy-on-write (shallow list copy + clone only messages that are mutated). Use last_prompt_tokens from update_from_response instead of re-estimating tokens every call. System extension injection is idempotent (one-time cache break). 3. Signature mismatch: _message_signature renamed to _content_signature and now excludes tool_calls/tool_call_id from the hash. This prevents mismatches when _canonicalize_api_tool_calls re-serializes argument JSON with sort_keys=True on the API copy. 4. update_model: accepts api_mode parameter (required by agent_init.py). 5. Reconciled with select_context: transform_api_messages is a separate hook that runs AFTER select_context and sanitization, before prompt-cache marker placement. Both hooks coexist with clear ordering. 6. Dedup/purge: kept as DCP-specific strategies (different semantics from ContextCompressor._prune_old_tool_results — DCP deduplicates by tool+args signature, not by content hash). 7. Removed copy.deepcopy: replaced with shallow list copy + copy-on-write via _clone_if_needed. Only messages that are actually mutated get cloned. 8. Removed redundant _ensure_refs call: _match_api_messages_to_refs no longer calls _ensure_refs (the caller already called it). 9. _message_key still uses index (needed for positional ref assignment), but _content_signature is cached per id(msg) to avoid re-hashing. 10. _inject_nudge: only injects into user messages, never falls back to non-user messages (prevents role semantics violations). 11. Memory: _evict_inactive_blocks bounds blocks_by_id to _MAX_INACTIVE_BLOCKS (50) deactivated blocks. 12. Merged _range_tool_schema and _message_tool_schema into a single _compress_tool_schema. Merged _handle_range_compress and _handle_message_compress into _handle_compress. 13. Dropped DCP_CONTEXT_ENGINE_PR_SPEC.md (temporary file, not for tree). Config defaults kept minimal in hermes_cli/config_defaults.py (only the keys the engine actually reads, not the full DCP-compatible surface). Closes #20717 * Revert "feat: add DCP context engine" This reverts commit d7072ab914de0bd3c98cd5cf006fd193a5ee752d. * Revert "fix: rewire DCP context engine to current main architecture" This reverts commit 9841a6c65161b9253c342f04b20f3a8bdb63884c. * fix: clean up SkillEvaluator Tier 1 security findings in bundled skills Findings from scanning skills/ + optional-skills/ with NVIDIA SkillEvaluator's deterministic Tier 1 checks (PII/secrets, unicode smuggling, script lint): - pixel-art, pokemon-player: remove hardcoded /home/teknium/ personal paths (use ~ / portable phrasing); pokemon-player no longer claims machine-specific state as fact - kanban-video-orchestrator: replace <path> angle-bracket token in frontmatter credits (flagged as XML-in-frontmatter prompt injection) - comfyui, hermes-agent, unsloth, 1password, actual-setup: rephrase placeholder secrets so they no longer pattern-match real credentials (your-* placeholder convention, comment markers, {env:...} form) - docker-management, pytorch-lightning: drop user:pass@ from example connection strings (env/secret-manager guidance instead) - evm: break up Keccak round constant that Luhn-validates as a credit card number (digit-group underscores, value unchanged) All targeted skills now pass pii+unicode+lint 3/3 except unsloth, which retains scanner false positives only (Colab notebook IDs read as Bitcoin addresses; an email inside a quoted upstream system prompt). * feat: replace Anthropic office document skills with clean-room MIT implementations The bundled docx, xlsx, powerpoint, and pdf skills were adapted from Anthropic's document skills and carried their proprietary LICENSE.txt (no derivatives, no redistribution). Flagged as critical license findings by the SkillEvaluator Tier 1 scan of our skill tree. This replaces all four with clean-room rewrites: - Authored from scratch against library knowledge only (python-docx, openpyxl, python-pptx, pypdf/reportlab/pdfplumber — all MIT/BSD) by isolated subagents given functional specs, with an explicit prohibition on reading the prior skill content or anthropics/skills; session transcripts retained as provenance evidence. - MIT licensed (LICENSE file per skill), author: Nous Research. - Each skill: SKILL.md to house standards + argparse helper scripts with UTF-8-explicit I/O + its own e2e pytest suite (fixtures built on the fly, non-ASCII round-trips run under LC_ALL=C). - All four pass SkillEvaluator Tier 1 pii+unicode+lint 3/3. tests/skills/test_office_document_skills.py rewritten against the new contracts: MIT/no-Anthropic-text invariants, scripts documented in SKILL.md, argparse CLI shape, and a no-locale-default-open() check (which caught and fixed a real gap: pdfplumber text reads are fine, but the invariant scan now guards every future script). Docs pages regenerated for the four skills (scoped; unrelated generator drift excluded). Honest capability deltas vs the old versions are documented per SKILL.md (e.g. tracked-changes accept/reject and OOXML XSD validation are not reimplemented; form flattening limits stated). * feat: extend clean-room office skills toward full parity Same clean-room discipline as the initial rewrite (isolated subagents, functional specs only, predecessor content banned including via git history; transcripts retained). All additions test-proven. docx (13->29 tests): - docx_revisions.py: tracked changes list/accept/reject (all or by id), incl. tables and headers/footers, via direct oxml manipulation - docx_comments.py: list/add/delete comments (native python-docx >=1.2 API with XML fallback), anchored-text extraction - docx_validate.py: package health check (rels, images, styles, CRC) with JSON severity report — explicitly not XSD validation - docx_edit.py: run normalization; TOC + PAGE/NUMPAGES field insertion xlsx (5->12 tests): - xlsx_restructure.py: reference-aware insert/delete rows/cols — rewrites formulas on all sheets (absolute refs, ranges, cross-sheet, quoted names), shifts merges/autofilter/freeze/validation/CF ranges, tables, defined names; JSON report incl. honest not_shifted list - native Excel tables, named ranges, hyperlinks, cell notes, sheet protection (documented as strippable, not security) - xlsx_recalc.py: headless LibreOffice recalc with graceful degrade powerpoint (11->21 tests): - pptx_render.py: all slides -> PNGs (soffice + pdftoppm/pdftocairo), wired to vision_analyze review loop in SKILL.md - run-merge normalize before replace (identical-format splits lossless) - surgical chart ops (series/category/title) wrapping replace_data - slide duplication with rel remap (clean refusal on chart slides) - backgrounds, hyperlinks, slide numbers, footers, notes editing pdf (8->21 tests): - pdf_make_form.py: JSON spec -> AcroForm (text/checkbox/radio/dropdown) - pdf_form_layout.py: pre-build layout lint (bounds/overlap/pairing) + rendered box overlay for vision_analyze review - pdf_page_image.py + shared _raster.py: pypdfium2 -> pdftoppm chain, graceful degrade; connected to scanned-PDF triage flow - pdf_stamp.py: text/image stamps at coordinates (rotation/opacity) - pdf_meta.py: DocInfo metadata + attachments round-trip Gates re-verified independently: 83 skill tests green under LC_ALL=C, repo invariant suite 29/29, SkillEvaluator pii+unicode+lint 3/3 x4. * fix(desktop): don't frost the HUD window the sheet isn't covering The frost is native vibrancy, which is the window's content view rather than an element — it fills the whole rectangle and nothing in the page can clip it to the sheet. That was only ever right while the sheet covered the window; anywhere it falls short, the difference is frost over empty space. On a fresh thread the sheet is zero and the difference is the entire window, which is the grey slab that appears the moment you put the caret in the composer. Gating the caller's `engaged` was not enough, and is why the first attempt at this missed: the hook turns the frost on for a focused composer by itself, independent of what the caller passes. The veto belongs inside, next to that check. * fmt(js): `npm run fix` on merge (#81914) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * feat(skills): advisory SKILL.md convention linter on create Adds tools/skill_linter.py — a soft companion to the hard frontmatter validator. It encodes the CONTRIBUTING 'Skill authoring standards (HARDLINE)' conventions that today only a human reviewer catches: - shell-utility references in prose (`grep`/`sed`/`cat`...) that should name the native tool (search_files/patch/read_file) - missing version/author/license/metadata.hermes block - name != directory, invalid name format - description over the 60-char prompt budget, marketing words - dangling references/ links, forbidden scaffolding files - POSIX-only script primitives without a platforms: gate Findings are ADVISORY. skill_manage(create) attaches them as lint_warnings + lint_hint in the success result; nothing is blocked (the hard rejects already run in _validate_frontmatter). A CLI (python -m tools.skill_linter <dir>) exits 1 only on ERROR-severity findings so CI can gate on structural breakage without failing on nits. Calibrated against the bundled skills/ tree: 76 advisory findings, exit 0, no false positives after excluding repo-root scripts/ refs. Inspired by MiniMax Code's skill-creator lint step; adapted to our existing validator + skill_utils rather than a parallel system. * fix: suppress windows-footgun false positives in linter pattern list The _POSIX_PRIMITIVES tuple holds search-pattern STRINGS the linter greps for in skill scripts — 'os.setsid' / 'signal.SIGKILL' are data, not calls. Add inline # windows-footgun: ok suppressions. * fix(api-server): resolve reasoning for the request's model, not model.default e81d18dfb collapsed six per-surface copies of reasoning resolution onto resolve_reasoning_config() and, in its own words, "fixes the gateway resolving reasoning against config model.default instead of the session's effective model". It did not touch gateway/platforms/api_server.py, which kept that defect. _create_agent() called GatewayRunner._load_reasoning_config() with no model on its first line — before the model precedence chain (browser lock -> session /model -> session row -> route -> per-request -> defaults) has run. Per-model agent.reasoning_overrides therefore keyed off model.default on the one surface where every request names its own model: a request for a model with an override silently got the global effort instead. Resolve after the chain settles, so the override follows the model the request actually runs. An explicit per-request reasoning parameter still takes precedence over config. The existing test stub for _load_reasoning_config took no arguments (it mirrored the old call); it now matches the real signature, as the sibling stub in the same file already did. * feat(docs): replace local lunr search with Algolia DocSearch The local-search plugin shipped a ~16 MB client-side lunr index that every visitor downloaded and hydrated before their first result — slow on any connection, painful on poor ones, and another lazy-loaded chunk that died during deploy skew windows. DocSearch answers from Algolia's servers: no client index, instant results at any docs size. - themeConfig.algolia with public search-only credentials (admin key is not in the repo); contextualSearch keeps en/zh-Hans results separated via the crawler's docusaurus_tag facets - drop @easyops-cn/docusaurus-search-local from package.json + lockfile - index live and verified: 9,404 records, query 'telegram' returns 374 hits with correct URLs * feat(skills): add product-price-monitor * chore(skills/product-price-monitor): cron-recipe shape + price-watch blueprint Skill polish (hardline standards): - description 199 -> 58 chars; author credits Ben Barclay (benbarclay) first - moved research/ -> productivity/ (consumer task, not research) - restructured into Setup (foreground, once) / Tick (each scheduled run) phases with explicit cronjob(action='create') wiring and a state file at ~/.hermes/price-watches/ - dropped phantom 'flight-research' related_skills/prose refs - Hermes-tool framing (web_extract, browser_navigate) Blueprint half: - new 'price-watch' Automation Blueprint (item/condition/interval_h/ deliver slots) loading the skill via skills=(...), [SILENT] no-alert path, catalog now 15 blueprints; blueprints index regenerated Tests: 12 skill tests incl. setup/tick split, state discipline, blueprint registration + schedule resolution; existing blueprint catalog suite green (33 total across both files). * fix(skills): pin text-mode file I/O to UTF-8 in comfyui and pdf skill scripts The bundled comfyui and pdf skills read and write text files with the locale-default codec. Both declare platforms: [linux, macos, windows], so these paths run on hosts where that codec is not UTF-8 (cp1252 on US Windows, cp936 on Chinese Windows, ASCII under LC_ALL=C). Readers (the live bugs): - run_workflow.py load_schema() and the main() workflow read parse user-authored JSON. A non-ASCII label crashes json.load with UnicodeDecodeError under a non-UTF-8 locale, and a file saved from a Windows GUI editor carries a UTF-8 BOM that json.load rejects with JSONDecodeError. Both are read as utf-8-sig, which is BOM-tolerant and identical to utf-8 on BOM-less input. This differs from adecb0d1a, which used plain utf-8 for the pdf form JSON; those payloads are agent-authored and BOM-free by construction, these are not. - hardware_check.py reads /proc/version and /proc/meminfo. Both are Linux-gated so Windows never reaches them, but the C locale defaults to ASCII, so they pin plain utf-8. No BOM is possible on /proc. Writers (not currently broken): - extract_form_structure.py and extract_form_field_info.py write their JSON with json.dump, whose default ensure_ascii=True keeps the bytes pure ASCII. Pinned anyway because the codec is the writer's contract, not a property of what the caller happens to dump. wf_path.open() is a Path.open() site that check-windows-footguns.py deliberately does not flag (per the rule comment: "Path.open() is ALSO affected ... and can be audited separately"). It is fixed here because it is the same bug 156 lines from a site the checker does flag, and line 623 of the same file already uses read_text(encoding="utf-8"). Adds tests/skills/test_comfyui_skill.py with contract assertions plus two live regressions that run load_schema in a child interpreter under LC_ALL=C with PYTHONUTF8=0, and extends the office skill tests with writer contract assertions. All 8 new tests fail without this change. Note that pyproject.toml exempts skills/** from ruff PLW1514 (unspecified-encoding) because skill scripts are partly user-authored. This change does not touch that exemption; the sites are fixed by hand, the same way adecb0d1a did. * fix(skills): widen BOM-tolerant reads to all comfyui workflow-JSON call paths The salvaged fix covered run_workflow.py and hardware_check.py. The same locale-default read of user-authored workflow JSON exists in five sibling scripts (auto_fix_deps, check_deps, extract_schema, health_check, run_batch) — same bug class, same utf-8-sig fix. Invariant test extended to pin all nine read sites. The pdf half of the original PR is superseded: those scripts were replaced wholesale by the clean-room rewrite (#81890), which ships UTF-8-explicit I/O enforced by its own invariant test. * feat(compression): native OpenAI Responses server-side compaction for gpt-5.6 Opt-in via compression.codex_responses_native (default: false). When enabled, gpt-5.6-family models on the direct OpenAI API (api.openai.com) or a ChatGPT Codex subscription send context_management=[{type: compaction, compact_threshold: N}] on Responses requests. OpenAI compacts server-side and returns an encrypted compaction output item; Hermes captures it into the existing codex_reasoning_items sidecar and replays it on later turns in place of the pruned history — inheriting persistence, session replay, the cross-issuer guard, and the encrypted-replay kill switch with zero new state. Scope is deliberately hard-gated (agent/native_compaction.py, re-checked per request): gpt-5.6 family only — gpt-5.1/5.2 fail server-side on the field (HTTP 500 / stream stall, no structured rejection; live-verified) — and direct OpenAI/Codex routes only; xAI, GitHub/Copilot, OpenRouter, relays, and local servers never see the field. Hermes' local compression stays armed as the fallback owner: the native threshold is clamped ~8K tokens below the local trigger so the server compacts first, and a structured provider rejection of context_management disables native compaction for the session and retries without it (one-shot guard in TurnRetryState). Live-verified E2E on api.openai.com/gpt-5.6: server compaction fired at a 4K threshold, checkpoints captured and replayed, recall preserved across 3 turns; gpt-5.1 with the flag enabled stays clean (field never sent). Direction credit: PR #76950 by @laryhorb explored native Responses compaction; this is a minimal reimplementation on current main. * feat(skills): add weekly-review-planning * chore(skills/weekly-review-planning): hardline polish + wire task blueprints to their skills Skill polish: - description 208 -> 57 chars; author credits Ben Barclay (benbarclay) first - connector framing (google-workspace, obsidian, notion, email-inbox-triage) - modern section order; boilerplate folded into step-local rules Blueprint wiring (completes the batch's recipe integration): - weekly-review blueprint loads weekly-review-planning; prompt follows the skill's seven-section shape, drafts-only - morning-brief blueprint loads google-workspace; prompt points at references/daily-brief.md when connected - important-mail blueprint loads email-inbox-triage - blueprints index regenerated Tests: 13 skill tests + two catalog invariants (every blueprint skills= entry resolves to a real bundled skill; the four task blueprints are wired to their procedure skills). 32 green across both files. * feat(config): resolve ephemeral prompt from display.personality Keep agent.system_prompt user-owned; named personalities resolve as an ephemeral overlay via display.personality. Co-authored-by: kyssta-exe <kyssta-exe@users.noreply.github.com> Co-authored-by: EMT5320 <1908937833@qq.com> * fix(personality): stop writing personality into agent.system_prompt Persist display.personality only; apply rendered text as an in-session overlay across CLI, TUI config.set, and gateway /personality. Co-authored-by: kyssta-exe <kyssta-exe@users.noreply.github.com> Co-authored-by: EMT5320 <1908937833@qq.com> * test(personality): regression coverage for #81791 Assert config.set and /personality preserve manual agent.system_prompt, and that startup resolution prefers display.personality. Co-authored-by: kyssta-exe <kyssta-exe@users.noreply.github.com> Co-authored-by: EMT5320 <1908937833@qq.com> * feat(skills): add social-media-content-calendar * chore(skills/social-media-content-calendar): tighten to hardline standards, ship optional - description 210 -> 57 chars; author credits Ben Barclay (benbarclay) first - optional-skills/creative/ (marketing vertical, narrowest audience of the batch) - dropped phantom 'image-generation-workflow' ref; visuals via the image_generate tool - honest handoff language: platforms without connectors end at approved drafts marked handed-off, never claimed as published - tests (10) incl. phantom-ref and honest-handoff guards - docs regen scoped: per-skill page + one catalog row + one sidebar line * fix: ensure utf-8 encoding in jobs.json * fix(gateway): write cron delivery output files as UTF-8 Cron and agent output that contains emoji, CJK, or accented text is silently lost on Windows. When a job's output exceeds the platform limit (MAX_PLATFORM_OUTPUT = 4000), DeliveryRouter._deliver_to_platform saves the full text to disk and sends a truncated preview with a "full output saved to ..." pointer. That save used Path.write_text(content) with no encoding, so on Windows it encodes through the platform code page (cp1252) and raises UnicodeEncodeError on any non-ASCII character. The exception propagates out of _deliver_to_platform and deliver() records the target as failed, so the whole truncate-and-send path aborts: the user receives nothing — even though an ASCII payload of the same size would deliver fine — and the promised backup file is never written. The sibling local-file path (_deliver_local) had the identical defect. The Windows-footgun CI gate misses this because it only inspects open() / Path.open(), not Path.write_text(). Both writes now pass encoding="utf-8" explicitly so output is persisted consistently across platforms. Fixes silent loss of non-ASCII cron/agent output on Windows. The two on-disk writes in the delivery router (`_deliver_to_platform`'s full output save and `_deliver_local`'s file save) now write UTF-8 instead of the platform-default code page, so emoji/CJK/accented output is saved and delivered the same on Windows as on macOS/Linux. N/A - [x] 🐛 Bug fix (non-breaking change that fixes an issue) - `gateway/delivery.py`: pass `encoding="utf-8"` to the `write_text` call in `_save_full_output` (oversized-output backup) and the one in `_deliver_local` (local file delivery). - `tests/gateway/test_delivery.py`: add two regression tests that simulate a non-UTF-8 Windows code page and assert oversized non-ASCII output is still delivered and the backup/local files round-trip as UTF-8. 1. `scripts/run_tests.sh tests/gateway/test_delivery.py` — 25 passing. 2. Revert either `encoding="utf-8"` argument and re-run: the two new tests fail with `UnicodeEncodeError` from the cp1252 codec, proving they catch the regression. 3. `python scripts/check-windows-footguns.py gateway/delivery.py` and `ruff check gateway/delivery.py tests/gateway/test_delivery.py` both pass. - [x] I've read the Contributing Guide - [x] My commit messages follow Conventional Commits - [x] I searched for existing PRs to make sure this isn't a duplicate - [x] My PR contains only changes related to this fix - [x] I've run the gateway delivery tests and all tests pass - [x] I've added tests for my changes - [x] I've tested on my platform: macOS 15 (Darwin 25.5) - [x] I've updated relevant documentation (README, docs/, docstrings) — or N/A - [x] I've updated cli-config.yaml.example if I added/changed config keys — or N/A - [x] I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A - [x] I've considered cross-platform impact (Windows, macOS) — this fix is specifically about Windows code-page encoding - [x] I've updated tool descriptions/schemas if I changed tool behavior — or N/A * test(cron): regression coverage for Windows encoding cluster - CJK/emoji round-trip + human-readable jobs.json (PRs #52302/#29754) - emoji through no_agent script stdout capture (issue #42384) - truncated/invalid UTF-8 script stdout must not raise (#47393) * chore: contributor mapping for zcj1122-rgb * fix: restore corrupted warning emoji in batch_runner checkpoint handler * fix(batch): normalize checkpoint warning emoji spacing (salvage follow-up for #32982/#66680) * fix(telegram): honor UTF-16 entity offsets * fix(gateway): tolerate invalid UTF-8 update output (cherry picked from commit 1dee620462c43daacd88783f446c32c6354f5b02) (cherry picked from commit 295f32dad9b6ad9c3cc61bc0f0e4941ee0ba7617) * fix(update): handle UnicodeDecodeError in interactive update prompts Ports #68497 forward onto current main per teknium1's review. input() can raise UnicodeDecodeError when the terminal encoding cannot decode the byte sequence (e.g. a non-UTF-8 locale, or an embedded terminal). The prior port targeted hermes_cli/main.py, the pre-refactor location -- the update pipeline moved to hermes_cli/update_cmd.py in 927463efcc. Per review, fixed all three interactive update prompts that call input() directly, not just the one this originally targeted: 1. Config-migration prompt (update_cmd.py:~3989): extends the existing except EOFError to also catch UnicodeDecodeError, prints an actionable 'hermes config migrate' hint, and falls through to the skip branch (response=n). 2. Stash-restore prompt (_restore_stashed_changes, ~line 971): the raw input() call here had NO exception guard at all -- not even for EOFError. Added a try/except covering both EOFError and UnicodeDecodeError, falling back to the existing skip-restore path (changes remain safely in git stash, restorable manually). 3. Upstream-remote prompt (_sync_with_upstream_if_needed, ~line 1274): already caught (EOFError, KeyboardInterrupt) but not UnicodeDecodeError -- added it to the existing tuple. Also dropped the incorrect #12884 reference (a TUI sticky-scroll report, unrelated to this update-encoding issue, per the review). 4 new tests pass covering all three call sites (config-migration prompt via cmd_update end to end, stash-restore and upstream-remote prompts via direct unit tests against their own functions), plus an EOFError sanity test confirming the stash-restore fix doesn't regress that case either (it had no guard before). 6/6 in the full tests/hermes_cli/test_update_yes_flag.py file (no regression). * test(update): strengthen UnicodeDecodeError regression to assert_not_called() Follow-up per review of #74631. The prior assertion (call_count == 0 OR interactive != True) also passed if an unintended non-interactive migration occurred, which the safe fallback (response='n') is supposed to prevent entirely. Replaced with mock_migrate.assert_not_called(). 6/6 pass in the full tests/hermes_cli/test_update_yes_flag.py file. * fix(email): never let unknown or malformed charsets abort the IMAP fetch Unknown charset labels (QQ Mail's RFC 1428 'unknown-8bit' placeholder, misspelled names, garbage encoded-word charsets) raised LookupError from bytes.decode — errors='replace' only guards decode errors, not a missing codec — aborting the whole fetch batch. UIDs are marked seen before the fetch, so the crash permanently dropped every message in the batch. - _safe_decode(): alias table (unknown-8bit→utf-8, gb2312/gbk→gb18030, ks_c_5601-1987→cp949, ...) then utf-8, then latin-1 last resort. - _decode_header_value(): wraps decode_header() so a malformed RFC 2047 header degrades to the raw string instead of crashing. - _extract_text_body(): all three decode sites now use _safe_decode. Fixes #35901, fixes #55381, fixes #55383. * test(gateway): regression tests for UTF-16 chunk limits at the Telegram boundary (#55844) * chore: contributor email mapping for salvaged commits * fix(environments): surrogateescape-safe stdin piping, always close stdin (#79178) * fix(environments): surface stdin write failures as stdin_error (#79178) * fix(file_operations): reject unencodable surrogates early, hash with surrogateescape (#79178) * test(file_operations): pin early surrogate rejection over the backstop (#79178) * fix(process_registry): surrogateescape-safe PTY stdin writes (#79178) * fix(cli): scrub lone surrogates before oneshot stdout write Prevent UnicodeEncodeError when model text contains U+D800-range surrogates by sanitizing to U+FFFD before writing to UTF-8 stdout. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent,gateway): class-level lone-surrogate chokepoints (#80366 #55143 #55309 #50959 #19819) Own the surrogate-crash class at three chokepoints instead of leaf sites: - finalize_turn scrubs final_response once where model text leaves the conversation loop — covers oneshot stdout (#80366), NIM/any-provider responses (#19819), and every delivery consumer of the turn result. - _sanitize_gateway_final_response scrubs at the gateway chat-surface boundary — Telegram utf16_len (#55309) and Signal formatting (#55143) can no longer see a lone surrogate; raw-text surfaces keep passthrough. - run_conversation walks the fully-built api_kwargs with _sanitize_structure_surrogates so tool descriptions (session_search, #50959) and every other request-body leaf are JSON-encodable before any provider sees them. Regression tests pin all three chokepoints plus helper semantics. Cherry-picked alongside #79240 (TheophilusChinomona) and #80374 (rainbowgore) whose commits precede this one with authorship preserved. * fix(cli): read .env as utf-8-sig so a BOM doesn't drop the first key PowerShell 5.1 Set-Content -Encoding UTF8 and Windows Notepad write a UTF-8 BOM. load_dotenv(encoding="utf-8") kept U+FEFF on the first key name, so the canonical name was absent from os.environ and Hermes looked unconfigured with no error. utf-8-sig strips the BOM and is a no-op for BOM-less UTF-8; latin-1 fallback unchanged. * fix(cli): strip UTF-8 BOM on latin-1 .env fallback path utf-8-sig only covers the primary decode. BOM + invalid UTF-8 (e.g. PowerShell BOM + cp1252 body) forced latin-1, which kept EF BB BF as part of the first key name and dropped the canonical name. Strip the BOM before latin-1 decode and load via stream so override= is preserved. * fix(cli): apply BOM-safe .env decoding to hermes send's private loader send_cmd._load_hermes_env intentionally reimplements a minimal dotenv load (no secret-source pulls, no sanitize rewrite, get_hermes_home path resolution incl. Windows/profile override), so the shared-loader BOM fix is mirrored in place: utf-8-sig primary read, BOM strip before the latin-1 stream fallback. Claude-Session: https://claude.ai/code/session_01JPmJz5u1Bvtw4cCRvRWnYr * fix(auth): read auth stores as UTF-8 to prevent credential loss on Windows The auth store readers (_load_auth_store, _import_codex_cli_tokens, and the shared Nous store reader) called Path.read_text() with no encoding, so bytes were decoded with locale.getpreferredencoding() — cp1252 on Windows. The stores are *written* as UTF-8 (os.fdopen(..., encoding="utf-8")), so any non-ASCII byte (a CJK or emoji credential label, an accented display name in OAuth state) raised UnicodeDecodeError on read. Worst case: _load_auth_store's broad except then copied the file to .corrupt and returned an empty store, silently wiping every provider credential on the next launch. The sibling reader at line 2161 already used read_text(encoding="utf-8"), confirming the omission was unintentional. Use utf-8-sig (matching the .env handling in config.py) so a BOM from a Notepad-edited file is tolerated too. Adds regression tests covering the UTF-8 round-trip with a non-ASCII label, BOM tolerance, no-corrupt-on-valid-load, and that the readers pass an explicit encoding (guard against future regressions). Verified the tests fail when the fix is reverted. Closes no issue — found via cross-platform code audit (the bug is not in the issue tracker). * fix(auth): cover remaining auth.json readers across modules Follow-up to the auth.json UTF-8 read fix in this PR. A repo-wide scan for the same bug class found three more callers that read ~/.hermes/auth.json via Path.read_text() with no encoding — same Windows cp1252 hazard: - agent/auxiliary_client.py _read_nous_auth: a non-ASCII byte raised UnicodeDecodeError, the broad except swallowed it, and Nous silently stopped being available as the auxiliary (vision/summarization) provider. - tools/xai_http.py has_xai_credentials: same failure mode — xAI OAuth silently looked absent on Windows. - hermes_cli/main.py is_setup_complete: same; has a config.yaml fallback so the impact is milder, but the read is still wrong. All three now use read_text(encoding="utf-8-sig"), matching _save_auth_store's write encoding. A repo-wide grep confirms there are no remaining json.loads(...read_text()) reads of auth.json without an explicit encoding. Tests: rewrote the Windows-encoding regression tests to actually exercise the bug on POSIX too — a new windows_default_encoding fixture forces a no-encoding read_text() to decode as cp1252 (the Windows default), and _write_utf8 now emits real non-ASCII UTF-8 bytes (ensure_ascii=False) so the bytes actually trip cp1252. Verified each test fails when its fix is reverted (including the two new sibling-reader tests). * test(auth): cover the two remaining Windows-encoding readers Address review feedback on #58158: the regression suite covered four of the changed readers but not _read_shared_nous_state (auth.py) or _has_any_provider_configured (main.py), which also read UTF-8 stores the Windows cp1252 default can corrupt. Add a non-ASCII UTF-8 regression case for each, reusing the existing windows_default_encoding fixture and _write_utf8 helper: - _read_shared_nous_state: a nous_auth.json with an accented display_name and valid tokens must round-trip intact (not return None). Pins HERMES_SHARED_AUTH_DIR to tmp to satisfy the shared-store seat belt. - _has_any_provider_configured: an auth.json whose active provider carries a CJK label must still report a configured provider (the read must not raise into the swallowing except). get_auth_status is faked so the result is driven by the read, and provider env vars are cleared to reach the auth.json branch. Both tests are RED-verified — they fail when the respective read_text(encoding=...) is reverted. * fix(gateway): read auth.json as UTF-8 in _read_nous_provider_state tools/managed_tool_gateway._read_nous_provider_state read auth.json with a bare read_text(), which on Windows decodes as cp1252 and raises on any non-ASCII byte (e.g. an accented Nous provider label). The broad except swallowed it and returned None, so the gateway treated Nous as unconfigured — the same Windows UTF-8 hazard the other auth.json readers in this PR already fix. Add encoding="utf-8-sig" (consistent with the sibling readers) plus a non-ASCII regression test reusing the windows_default_encoding fixture. This covers the one auth.json reader in #66782 not already handled here (tools/managed_tool_gateway.py:40); the other two readers #66782 touches (agent/auxiliary_client.py, tools/xai_http.py) are already fixed in this PR. RED-verified. * fix(tools): make json_parse tolerate UTF-8 BOM (salvage #57870) json_parse used json.loads(strict=False), which relaxes control characters but rejects a leading UTF-8 BOM (U+FEFF). Windows CLI tools and some files prepend a BOM, causing JSONDecodeError on otherwise valid JSON output. Strip a leading BOM before calling json.loads when the input is a string with a U+FEFF prefix. Original PR by @woxinwuhen713-bit (#57870). Co-Authored-By: Claude <noreply@anthropic.com> * test(tools): cover UTF-8 BOM input in json_parse sandbox helper Review asked for a BOM-prefixed JSON case alongside the existing control-character coverage. The sandbox script now also feeds json_parse a \ufeff-prefixed document and asserts the parsed value round-trips (fails against the pre-fix helper, passes with the BOM strip). * fix(auth): read .env as utf-8-sig in the dotenv-vs-shell detector _remove_env_source() decides whether a credential var lives in ~/.hermes/.env or the shell by scanning the .env with env_path.read_text(errors="replace") — no encoding. read_text() with no encoding falls back to the system locale (cp1252/GBK on Windows) and never strips a BOM. The canonical .env readers in hermes_cli/config.py all use encoding="utf-8-sig" precisely because 'users may edit .env in Notepad which adds one' (a BOM), and doctor.py documents that .env is written as UTF-8 everywhere. This sibling reader diverged: on a Notepad-edited .env the BOM prefixes the first line, so line.strip().startswith(f"{env_var}=") is False for the first variable — the detector reports a .env-backed key as a phantom shell export and prints a misleading 'still set in your shell environment' hint on . Match the canonical reader (utf-8-sig + errors=replace). Adds a regression test with a BOM'd .env. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: finish the missing-encoding sweep — BOM-tolerant reads for user-edited stores Complements the cherry-picked contributor fixes and closes out the remaining sites of the 'missing explicit encoding' bug class, which is now permanently gated by ruff PLW1514 (enabled repo-wide in pyproject.toml and enforced by the blocking `ruff check .` step in .github/workflows/lint.yml): - tools/memory_tool.py: read MEMORY.md/USER.md via utf-8-sig so a Notepad BOM never glues U+FEFF onto the first entry (issue #10878, PR #10888 by @easyvibecoding — strict-decode contract of _read_raw_checked preserved rather than errors="replace", so undecodable files still refuse read-modify-write instead of being lossily rewritten). Regression tests included. - tools/skills_tool.py: SKILL.md and skill file reads pinned to utf-8-sig + errors="replace" — deterministic across platforms instead of the locale fallback proposed in PR #51701 (superseded: falling back to cp1252/GBK makes the same skill render differently per host); .env reader aligned with the canonical utf-8-sig dialect in hermes_cli/config.py. - agent/shell_hooks.py, hermes_cli/main.py, gateway/slash_commands.py: explicit utf-8 on the remaining fdopen/open text-mode sites flagged by the AlexFucuson9 sweep series (#56033 #56940 #65565 #66782 #66791). Co-authored-by: easyvibecoding <easyvibecoding@users.noreply.github.com> Co-authored-by: AlexFucuson9 <AlexFucuson9@users.noreply.github.com> Co-authored-by: flyingdoubleg <wangzhe00zju@gmail.com> Co-authored-by: LeonSGP43 <cine.dreamer.one@gmail.com> * test: pin module-level _AUTH_JSON_PATH to tmp store in salvaged windows-encoding test * feat(lint): close the fdopen + chained-call gaps in the encoding footgun gate ruff PLW1514 (already enforced repo-wide via the blocking lint step) covers open()/Path.open()/read_text()/write_text() but NOT os.fdopen — the exact hole the AlexFucuson9 sweep PRs (#56033 #56940 #65565) kept patching by hand. Add an fdopen rule to check-windows-footguns.py, which also runs as a blocking CI step, so a bare text-mode fdopen fails CI. Also fix a false-negative in the read_text/write_text rule: chained forms like `read_text()[:4000]` or `read_text().splitlines()` never end the line with `)` and slipped past the multi-line-call heuristic. Replace the endswith check with a paren-balance walk (keeps multi-line calls with encoding= on a continuation line unflagged — verified against the full tree). This makes the rule the effective standing replacement for the standalone checker proposed in PR #66669: R1-style coverage now lives in PLW1514 + this script, both blocking in .github/workflows/lint.yml. Sabotage-verified: reverting agent/shell_hooks.py's fdopen encoding or tools/skills_tool.py's read_text encoding now fails the gate. Co-authored-by: AlexFucuson9 <AlexFucuson9@users.noreply.github.com> Co-authored-by: Paulo Nascimento <pnascimento9596@gmail.com> * fix(tests): read and write test files as UTF-8 so the suite runs on Windows `tests/hermes_cli/test_plugins_cmd.py::TestNoAutoActivation::test_compressor_default_ignores_plugin` fails on every Windows machine: UnicodeDecodeError: 'charmap' codec can't decode byte 0x8f in position 47744: character maps to <undefined> The test reads `run_agent.py` back as text to assert a removed comment is gone, but called `open()` with no `encoding=`. Python then falls back to the locale preferred encoding, which is cp1252 on a default Windows install rather than UTF-8. `run_agent.py` contains nine bytes cp1252 leaves undefined, so the read raises before the assertion is reached. On Linux and macOS the preferred encoding is UTF-8 and the same line is fine, which is why CI never caught it. That one line is the only active failure. The rest of this change closes the same gap in the files it touches, which `scripts/check-windows-footguns.py` flags and which the #71014 read_text campaign has been working through elsewhere in the tree: - `tests/hermes_cli/test_plugins_cmd.py`: nine bare `write_text`/`read_text` calls writing YAML manifests, config and plugin sources - `tests/tools/test_web_tools_truncate.py`: reads stored extracted web text, which is arbitrary content from the internet - `tests/stress/test_atypical_scenarios.py`: writes and reads worker task ids and a barrier file All three files are now clean under `check-windows-footguns.py`. Reads go through `Path.read_text(encoding="utf-8")` rather than `open(...).read()`, which also closes the handle instead of leaving it to the garbage collector. On Windows a live handle blocks tmpdir cleanup, so that part is not cosmetic either. No new test. The repaired test is the regression coverage: it fails before this change and passes after, on Windows. * fix(tests): Windows-aware path-list split and UTF-8 progress output in parallel runner Two Windows bugs in scripts/run_tests_parallel.py: - --files/--paths/HERMES_TEST_PATHS were split on ':', which shreds absolute Windows paths at the drive letter ('C:\repo\tests' -> ['C', '\repo\tests']): the drive letter became a phantom discovery root and the rooted remainder only resolved by WindowsPath re-anchoring it onto repo_root's drive. New _split_pathspec() keeps drive-letter colons glued to their path and accepts ';' (os.pathsep) on Windows, while ':'-joined lists (CI generate job) keep working. - With piped stdout (CI, subprocess capture) Windows encodes the runner's output as the ANSI code page, so printing the per-file progress glyphs raised UnicodeEncodeError inside the executor done-callback and every pro…
ma1138569845
pushed a commit
to ma1138569845/dechnicAuditor-agent
that referenced
this pull request
Aug 10, 2026
…indows
`tests/hermes_cli/test_plugins_cmd.py::TestNoAutoActivation::test_compressor_default_ignores_plugin`
fails on every Windows machine:
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8f in
position 47744: character maps to <undefined>
The test reads `run_agent.py` back as text to assert a removed comment is
gone, but called `open()` with no `encoding=`. Python then falls back to
the locale preferred encoding, which is cp1252 on a default Windows
install rather than UTF-8. `run_agent.py` contains nine bytes cp1252
leaves undefined, so the read raises before the assertion is reached. On
Linux and macOS the preferred encoding is UTF-8 and the same line is
fine, which is why CI never caught it.
That one line is the only active failure. The rest of this change closes
the same gap in the files it touches, which `scripts/check-windows-footguns.py`
flags and which the NousResearch#71014 read_text campaign has been working through
elsewhere in the tree:
- `tests/hermes_cli/test_plugins_cmd.py`: nine bare `write_text`/`read_text`
calls writing YAML manifests, config and plugin sources
- `tests/tools/test_web_tools_truncate.py`: reads stored extracted web text,
which is arbitrary content from the internet
- `tests/stress/test_atypical_scenarios.py`: writes and reads worker task
ids and a barrier file
All three files are now clean under `check-windows-footguns.py`.
Reads go through `Path.read_text(encoding="utf-8")` rather than
`open(...).read()`, which also closes the handle instead of leaving it to
the garbage collector. On Windows a live handle blocks tmpdir cleanup, so
that part is not cosmetic either.
No new test. The repaired test is the regression coverage: it fails
before this change and passes after, on Windows.
randlee
pushed a commit
to randlee/hermes-agent
that referenced
this pull request
Aug 11, 2026
…indows
`tests/hermes_cli/test_plugins_cmd.py::TestNoAutoActivation::test_compressor_default_ignores_plugin`
fails on every Windows machine:
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8f in
position 47744: character maps to <undefined>
The test reads `run_agent.py` back as text to assert a removed comment is
gone, but called `open()` with no `encoding=`. Python then falls back to
the locale preferred encoding, which is cp1252 on a default Windows
install rather than UTF-8. `run_agent.py` contains nine bytes cp1252
leaves undefined, so the read raises before the assertion is reached. On
Linux and macOS the preferred encoding is UTF-8 and the same line is
fine, which is why CI never caught it.
That one line is the only active failure. The rest of this change closes
the same gap in the files it touches, which `scripts/check-windows-footguns.py`
flags and which the NousResearch#71014 read_text campaign has been working through
elsewhere in the tree:
- `tests/hermes_cli/test_plugins_cmd.py`: nine bare `write_text`/`read_text`
calls writing YAML manifests, config and plugin sources
- `tests/tools/test_web_tools_truncate.py`: reads stored extracted web text,
which is arbitrary content from the internet
- `tests/stress/test_atypical_scenarios.py`: writes and reads worker task
ids and a barrier file
All three files are now clean under `check-windows-footguns.py`.
Reads go through `Path.read_text(encoding="utf-8")` rather than
`open(...).read()`, which also closes the handle instead of leaving it to
the garbage collector. On Windows a live handle blocks tmpdir cleanup, so
that part is not cosmetic either.
No new test. The repaired test is the regression coverage: it fails
before this change and passes after, on Windows.
blut-agent
pushed a commit
to blut-agent/hermes-agent-fork
that referenced
this pull request
Aug 11, 2026
…indows
`tests/hermes_cli/test_plugins_cmd.py::TestNoAutoActivation::test_compressor_default_ignores_plugin`
fails on every Windows machine:
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8f in
position 47744: character maps to <undefined>
The test reads `run_agent.py` back as text to assert a removed comment is
gone, but called `open()` with no `encoding=`. Python then falls back to
the locale preferred encoding, which is cp1252 on a default Windows
install rather than UTF-8. `run_agent.py` contains nine bytes cp1252
leaves undefined, so the read raises before the assertion is reached. On
Linux and macOS the preferred encoding is UTF-8 and the same line is
fine, which is why CI never caught it.
That one line is the only active failure. The rest of this change closes
the same gap in the files it touches, which `scripts/check-windows-footguns.py`
flags and which the NousResearch#71014 read_text campaign has been working through
elsewhere in the tree:
- `tests/hermes_cli/test_plugins_cmd.py`: nine bare `write_text`/`read_text`
calls writing YAML manifests, config and plugin sources
- `tests/tools/test_web_tools_truncate.py`: reads stored extracted web text,
which is arbitrary content from the internet
- `tests/stress/test_atypical_scenarios.py`: writes and reads worker task
ids and a barrier file
All three files are now clean under `check-windows-footguns.py`.
Reads go through `Path.read_text(encoding="utf-8")` rather than
`open(...).read()`, which also closes the handle instead of leaving it to
the garbage collector. On Windows a live handle blocks tmpdir cleanup, so
that part is not cosmetic either.
No new test. The repaired test is the regression coverage: it fails
before this change and passes after, on Windows.
33hodl
pushed a commit
to 33hodl/hermes-agent
that referenced
this pull request
Aug 12, 2026
…indows
`tests/hermes_cli/test_plugins_cmd.py::TestNoAutoActivation::test_compressor_default_ignores_plugin`
fails on every Windows machine:
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8f in
position 47744: character maps to <undefined>
The test reads `run_agent.py` back as text to assert a removed comment is
gone, but called `open()` with no `encoding=`. Python then falls back to
the locale preferred encoding, which is cp1252 on a default Windows
install rather than UTF-8. `run_agent.py` contains nine bytes cp1252
leaves undefined, so the read raises before the assertion is reached. On
Linux and macOS the preferred encoding is UTF-8 and the same line is
fine, which is why CI never caught it.
That one line is the only active failure. The rest of this change closes
the same gap in the files it touches, which `scripts/check-windows-footguns.py`
flags and which the NousResearch#71014 read_text campaign has been working through
elsewhere in the tree:
- `tests/hermes_cli/test_plugins_cmd.py`: nine bare `write_text`/`read_text`
calls writing YAML manifests, config and plugin sources
- `tests/tools/test_web_tools_truncate.py`: reads stored extracted web text,
which is arbitrary content from the internet
- `tests/stress/test_atypical_scenarios.py`: writes and reads worker task
ids and a barrier file
All three files are now clean under `check-windows-footguns.py`.
Reads go through `Path.read_text(encoding="utf-8")` rather than
`open(...).read()`, which also closes the handle instead of leaving it to
the garbage collector. On Windows a live handle blocks tmpdir cleanup, so
that part is not cosmetic either.
No new test. The repaired test is the regression coverage: it fails
before this change and passes after, on Windows.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the five residual Windows encoding gaps left after #70850/#70875 closed the subprocess
text=Trueclass — each verified line-level against current main by a premise-audit pass before building.Salvages three contributor fixes with authorship preserved:
encoding_error_handler="replace"toStdioServerParameters— the SDK always exposed the knob, we never set it, so any MCP server emitting a locale-mismatched byte crashed the stdio reader on Windows ([Bug]: MCP client TextReceiveStream uses errors='strict' — fails on Windows with UnicodeDecodeError #46099).encoding="utf-8"on all 26 bareread_text/write_textsites in the gateway update path (gateway/run.pyupdate watcher/notification, slash_commands, status, delivery, dead_targets, qqbot) + the Discord/Telegram/Feishu/WhatsApp update-response writers, with an AST guard test ([Bug]: Windows gateway hermes update flow crashes with UnicodeEncodeError/UnicodeDecodeError (bare read_text/write_text under cp1252/GBK) #37423). These sit insideexcept OSErrorblocks;UnicodeDecodeErroris aValueErrorsubclass, so a GBK/cp1252 locale crashed the gateway command handler.popen_kwargsdict sites the fix(windows): close out the text-mode subprocess decode bug class — salvage #55339/#60741/#60751 + full sweep + linter #70875 AST sweep missed because kwargs are built indirectly —_run_command_stt(tools/transcription_tools.py) and_run_command_tts(tools/tts_tool.py).Plus two fixes of our own:
backend-env.cjs): the desktop Electron spawn now setsPYTHONUTF8=1in the backend child's env (backend-env.ts), covering output emitted beforehermes_bootstrapruns inside the child. Explicit user setting wins; vitest case added.suppress_platform_ver_console()moved intohermes_bootstrap: previously onlyhermes_cli.mainprocesses got theplatform._syscmd_verstub — slash workers,tui_gateway/entry,run_agent,batch_runnerwere exposed to the console flash and, on Python 3.11.0/3.11.1 (no CPythonencoding="locale"fix), aUnicodeDecodeErrorinsideplatform.win32_ver()under PEP 540 (Windows subprocess reader thread crashes on non-UTF-8 output #69413). Now every entry point gets it at import. This supersedes PR fix(windows): patch platform._syscmd_ver to survive non-UTF-8 output under PEP 540 #69522, whose wrapper approach TypeError'd on no-arg calls.Validation
0x90byte through_run_command_tts, bootstrap stubtests/gateway/+tests/tools/+tests/plugins/+ bootstrap (21,772 tests)apps/desktopvitestbackend-env.test.ts(incl. new PYTHONUTF8 cases)Closes #46104, #38985 (salvaged with credit). Fixes #46099, #37423, #69413.
Infographic