feat(tools): read vision max_tokens from auxiliary.vision config - #15430
Closed
phonezawphyo wants to merge 195 commits into
Closed
phonezawphyo wants to merge 195 commits into
phonezawphyo wants to merge 195 commits into
Conversation
The vision_analyze_tool hardcoded max_tokens to 2000, ignoring any auxiliary.vision.max_tokens value set in config.yaml. Users with local vision models (vLLM, llama.cpp) or high-capacity APIs could not leverage their full output capacity for OCR or detailed image analysis. Changes: - Read max_tokens from auxiliary.vision.max_tokens config, falling back to the previous hardcoded default of 2000. - Mirrors the existing pattern for timeout and temperature, which are already read from auxiliary.vision.*. Fixes: vision outputs were silently truncated regardless of config.
alt-glitch
added
type/feature
NousResearch#15438) /model gpt-5.5 on openai-codex showed 'Context: 1,050,000 tokens' because the display block used ModelInfo.context_window directly from models.dev. Codex OAuth actually enforces 272K for the same slug, and the agent's compressor already runs at 272K via get_model_context_length() — so the banner + real context budget said 272K while /model lied with 1M. Route the display context through a new resolve_display_context_length() helper that always prefers agent.model_metadata.get_model_context_length (which knows about Codex OAuth, Copilot, Nous caps) and only falls back to models.dev when that returns nothing. Fix applied to all 3 /model display sites: cli.py _handle_model_switch gateway/run.py picker on_model_selected callback gateway/run.py text-fallback confirmation Reported by @emilstridell (Telegram, April 2026).
…aths (NousResearch#15444) * fix(nix): use --rebuild in fix-lockfiles to bypass cached FOD store paths fix-lockfiles checked npm lockfile hashes by running `nix build .#<attr>.npmDeps`, but fetchNpmDeps is a fixed-output derivation — if the old store path exists locally, Nix returns it from cache without re-fetching. This caused the script to report "ok" even when hashes were stale, while CI (with no cache) failed with a hash mismatch. Adding --rebuild forces Nix to re-derive and verify the output hash against the declared one, catching staleness regardless of local cache state. Also updates the tui and web npm deps hashes that were stale. * fix(nix): regenerate ui-tui lockfile to add missing @emnapi entries npm ci was failing because @emnapi/core and @emnapi/runtime were missing from ui-tui/package-lock.json despite being required as peer deps by @napi-rs/wasm-runtime (via @rolldown/binding-wasm32-wasi). Running npm install --package-lock-only adds the missing entries. The npmDepsHash reverts to its previous value since fetchNpmDeps was already fetching these packages as transitive dependencies.
PgCryptoStore.__init__ defaults _device_id to "" and put_account writes
that blank value into crypto_account. The UPSERT's ON CONFLICT DO UPDATE
clause deliberately does not touch device_id, so once the row is written
blank it stays blank forever — breaking every downstream device-scoped
olm operation. Peers' to-device olm ciphertext can't match our identity
key, no megolm sessions ever land, and the user sees "hermes is in the
room but never responds to encrypted messages".
Fix: call put_device_id(client.device_id) immediately after
crypto_store.open() and before olm.load(). This sets the store's
in-memory _device_id so the first put_account INSERT writes the correct
value from the start.
Observable symptoms without the fix, on a fresh crypto.db:
- crypto_account.device_id = ""
- crypto_tracked_user: 0 rows
- crypto_device: 0 rows
- crypto_olm_session: 0 rows
- crypto_megolm_inbound_session: 0 rows
- "No one-time keys nor device keys got when trying to share keys"
warning on every startup
- "olm event doesn't contain ciphertext for this device" DecryptionError
on any inbound to-device event
- Encrypted room messages arrive but never decrypt
After the fix (wiped crypto.db + restart):
- device_id populated with actual runtime device (e.g. CZIKTRFLOV)
- all counts populate from sync as expected
- encrypted DMs flow normally
Who hits this: anyone with a fresh crypto.db — includes first-time matrix
E2EE setup, nio→mautrix migrations (since matrix.py removes the legacy
pickle on startup, creating a fresh SQLite store), and anyone who wipes
crypto.db to start over. Existing installs that somehow already have a
non-blank device_id would be unaffected, but no prior code path writes
it correctly, so that set is likely empty.
Two adjustments to make CI pass:
- In gateway/platforms/matrix.py: `DeviceID` is `NewType("DeviceID", str)`,
so passing `client.device_id` directly (already a str) works identically
at runtime. The explicit import was cosmetic and tripped CI environments
where `mautrix.types` doesn't re-export DeviceID at the expected path
("cannot import name 'DeviceID' from 'mautrix.types' (unknown location)").
- In tests/gateway/test_matrix.py: add `put_device_id` to the hand-written
`PgCryptoStore` fake so the three encryption-path tests
(test_connect_with_access_token_and_encryption,
test_connect_uses_configured_device_id_over_whoami,
test_connect_registers_encrypted_event_handler_when_encryption_on) can
exercise the new crypto-store binding without AttributeError.
On Windows WSL2, ConPTY implicitly enables mouse event injection when the alternate screen buffer (DEC 1049) is entered, causing raw escape sequences to appear in the transcript as ghost characters. Fix (two parts): 1. ConPTY fix: send DISABLE_MOUSE_TRACKING immediately after entering alt screen when mouse tracking is off (AlternateScreen.tsx) 2. Runtime toggle: add /mouse [on|off|toggle] slash command with config persistence (display.tui_mouse) so users can manage this at runtime The env var HERMES_TUI_DISABLE_MOUSE continues to work as the initial default, but can now be overridden via /mouse and persisted to config. Closes: upstream ConPTY mouse injection issue Credits: OutThisLife / PR NousResearch#13716 for the toggle concept
fix(tui): proactive mouse disable on ConPTY + /mouse toggle command
…parent TUI (NousResearch#15491) Subagents run inside a ThreadPoolExecutor. The CLI's interactive approval callback lives in tools/terminal_tool.py's threading.local(), which worker threads do not inherit. When a subagent hits a dangerous-command guard, prompt_dangerous_approval() falls back to input() from the worker thread, deadlocking against the parent's prompt_toolkit TUI that owns stdin. Fix: install a non-interactive callback into every subagent worker thread via ThreadPoolExecutor(initializer=set_approval_callback, initargs=(cb,)). The callback is config-gated by delegation.subagent_auto_approve: false (default) -> _subagent_auto_deny (safe; matches leaf tool blocklist) true -> _subagent_auto_approve (opt-in YOLO for cron/batch) Both emit a logger.warning audit line. Gateway sessions are unaffected because they resolve approvals via tools/approval.py's per-session queue, not through these TLS callbacks. Diagnosis credit: @MorAlekss (NousResearch#14685). - hermes_cli/config.py: DEFAULT_CONFIG.delegation.subagent_auto_approve: False - cli-config.yaml.example: documented, commented (default) - tools/delegate_tool.py: _subagent_auto_deny, _subagent_auto_approve, _get_subagent_approval_callback, wired into the child timeout executor - tests/tools/test_delegate.py: 7 tests covering defaults, truthy coercion, and TLS scoping in the worker thread
…hboard (NousResearch#15530) The web-dashboard.md and dashboard-plugins.md pages had overlapping, partial coverage of the theme and plugin systems. Themes were split across two pages; the plugin docs had a minimal manifest reference but no step-by-step guide, no slot catalog, and no theme+plugin demo. New: user-guide/features/extending-the-dashboard.md — single navigable reference for all three extension layers (themes, UI plugins, backend plugins). Includes: - Theme quick-start + full schema (palette, typography, layout, layout variants, assets, componentStyles, colorOverrides, customCSS) - Plugin quick-start + full schema (manifest, SDK, slots, tab.override, tab.hidden, backend routes, custom CSS) - 10-slot shell catalog with locations - Plugin discovery + load lifecycle - Combined theme+plugin walkthrough (Strike Freedom cockpit demo) - API reference + troubleshooting web-dashboard.md: trimmed to core tool docs (pages, REST API, CORS, development). Theme/plugin content now points to the new page with a built-in themes summary table. dashboard-plugins.md: deleted (merged into extending-the-dashboard.md). sidebars.ts: swap 'dashboard-plugins' → 'extending-the-dashboard' under the Management group. No user-facing behavior change; docs-only.
update_model() recalculated threshold_tokens but left tail_token_budget and max_summary_tokens at their __init__ values. When switching from a 200K model to 32K, the tail budget stayed at ~20K tokens (62% of 32K) instead of the intended ~10%. Adds budget recalculation in update_model() and 2 regression tests.
…_platform_tools YAML parses bare numeric toolset names (e.g. 12306:) as int, causing TypeError in sorted() since the read path normalizes to str but the save path did not. The no_mcp sentinel was preserved in existing entries even when the user re-enabled MCP servers, causing MCP to stay silently disabled.
…15608) Root installs on Linux now put the code at /usr/local/lib/hermes-agent and the hermes command at /usr/local/bin/hermes. HERMES_HOME (~/.hermes) stays state-only. Matches Claude Code / Codex CLI / OpenClaw, keeps Docker bind-mounted /root/ volumes lean, and puts the command on every shell's default PATH without touching shell RC files. - Non-root users and macOS root: unchanged - Existing root installs at $HERMES_HOME/hermes-agent: preserved in-place (detected via .git dir) — no auto-migration, no breakage - Explicit --dir / $HERMES_INSTALL_DIR: always wins, never overridden - Termux: unchanged (package manager manages /data/data/...) Requested by @souly9999 (Discord). Our own Dockerfile already uses this split (code at /opt/hermes, data at /opt/data volume); the user-install path now matches.
The tool schema promised 'On update, pass an empty array to clear' but the update branch ignored the context_from kwarg entirely — users could set the field at create time and never modify or clear it afterward. - tools/cronjob_tools.py: handle context_from in the update branch the same way script/enabled_toolsets/workdir are handled: normalize str/list to refs, validate each referenced job exists (same check the create branch does), store as list-or-None to match create_job()'s shape. Empty string or empty list clears the field. - tests/cron/test_cron_context_from.py: 6 new tests covering add/change/ clear (both shapes)/bad-ref/preserve-across-unrelated-update.
The reverse-mapping loop in _get_platform_tools only checked CONFIGURABLE_TOOLSETS, silently dropping platform-specific toolsets like discord and feishu_doc whose tools were in the composite but had no configurable key. Add a second pass over TOOLSETS that picks up unclaimed toolsets whose tools are present in the resolved composite.
Split the monolithic discord_server tool (14 actions) into two: - discord: core actions (fetch_messages, search_members, create_thread) that are useful for the agent's normal operation. Auto-enabled on the discord platform via the pipeline fix. - discord_admin: server management actions (list channels/roles, pins, role assignment) that require explicit opt-in via hermes tools. Added to CONFIGURABLE_TOOLSETS and _DEFAULT_OFF_TOOLSETS.
The feishu_doc and feishu_drive tools were registered in the tool registry but never added to the hermes-feishu composite toolset. The pipeline fix from the prior commit now recovers them automatically once they are in the composite.
Groundwork for injecting raw platform identifiers into the agent's system prompt. Currently only `thread_id` is exposed as a raw ID — callers in a Discord thread had to guess `channel_id == thread_id` (which happens to work because threads are channels in Discord's REST API) and had no way to reference the parent channel, guild, or the triggering message. Adds three optional fields: - `guild_id` — Discord guild / Slack workspace / Matrix server scope - `parent_chat_id` — parent channel when chat_id refers to a thread - `message_id` — ID of the triggering message (pin/reply/react) Extends `BasePlatformAdapter.build_source()` to accept + forward them and teaches `to_dict`/`from_dict` to serialize them. Behaviourally a no-op: nothing reads the fields yet and they default to None.
…onSource Discord knows all four identifiers for every inbound message — guild, channel (or thread), parent channel when in a thread, and the triggering message. Pass them into ``SessionSource`` via the new ``build_source()`` kwargs so downstream code (context-prompt builder, delivery, logging) can use them without re-resolving from discord.py objects. For auto-threaded messages, remember the original channel as the parent before swapping ``chat_id`` to the freshly created thread. Behavioural: still a no-op — nothing consumes these fields yet.
load_gateway_config() has a side effect: when config.yaml contains platform-gating keys (slack.require_mention, slack.strict_mention, slack.free_response_channels, slack.allow_bots, slack.reactions, plus analogous keys for discord/telegram/whatsapp/dingtalk/matrix), it calls os.environ[KEY] = ... to bridge them to env-var form. monkeypatch.delenv doesn't track direct os.environ mutations made inside the test body, so tests that call load_gateway_config() leak those env vars into later tests on the same xdist worker. The failure mode is flaky seed-dependent: test_top_level_message_requires_mention_ even_with_session (and siblings in TestThreadReplyHandling) pass when SLACK_REQUIRE_MENTION is unset but fail when a leaked value of 'false' is present. Add the gating env vars to _HERMES_BEHAVIORAL_VARS so the hermetic autouse fixture blanks them on every test setup, closing the leak regardless of which test sets them.
…picker lists (NousResearch#16197) Removes deepseek/deepseek-v4-pro and deepseek/deepseek-v4-flash from OPENROUTER_MODELS and _PROVIDER_MODELS['nous'], then regenerates website/static/api/model-catalog.json so the hosted picker JSON drops them too. Direct-API deepseek provider support is unchanged.
…nnels send_message(target='slack:<channel_id>') failed with "Could not resolve" because _parse_target_ref had no Slack branch — Slack's uppercase alphanumeric IDs fell through to channel-name resolution, which only matched by name. As a fallback, the agent would retry with bare target='slack' and post to the home channel instead. Three fixes: - _parse_target_ref recognizes Slack IDs (C/G/D/U/W prefix) as explicit targets so the name-resolver is bypassed entirely. - resolve_channel_name tries a case-sensitive raw-ID match before the existing name match, so any platform's IDs resolve cleanly. - _build_slack now actually calls users.conversations against each workspace's AsyncWebClient (paginated), instead of only returning session-history entries. This populates the directory with public and private channels the bot has joined, so action='list' shows them and they can also be addressed by name. Errors from one workspace don't block others. build_channel_directory becomes async (Slack web calls require it). The two async-context callers in gateway/run.py are awaited; the cron ticker thread call bridges via asyncio.run_coroutine_threadsafe. Slack bot needs channels:read and groups:read scopes for full enumeration; missing scopes degrade gracefully per-workspace. addressing NousResearch#15927
Slack's chat.postMessage API rejects user IDs (U...) and workspace IDs (W...) — they are not valid conversation IDs. Posting to them fails because the API requires a channel ID (C/G/D). To DM a user, the sender must first call conversations.open to obtain a D... ID. Tighten _SLACK_TARGET_RE from [CGDUW] to [CGD] so the send path rejects U/W values as explicit targets and instead falls through to channel- name resolution (where they'll fail with a clear 'could not resolve' error rather than silently getting stuck in a retry loop on the API). Flip the corresponding regression test to assert U/W values are not explicit. Matches the narrower regex briandevans proposed in NousResearch#15939. Co-authored-by: briandevans <brian@bde.io>
…arent
The Slack thread-context fetcher used to drop every message with a
bot_id, which silently erased the thread parent whenever a cron job (or
any other bot) had posted it. As a result, replies to a cron-posted
summary lost all context and the agent answered as if from a blank
thread.
Changes:
1. gateway/platforms/slack.py::_fetch_thread_context
- Keep the thread parent even when it was posted by a bot
(e.g. cron summaries, third-party integrations).
- Only skip *our own* prior bot replies to avoid circular context,
matching the per-workspace bot user id via _team_bot_user_ids so
multi-workspace deployments stay correct.
- Keep non-self bot children (useful third-party context).
2. gateway/platforms/slack.py::_handle_slack_message
- Populate MessageEvent.reply_to_text for thread replies (parity
with Telegram/Discord/Feishu/WeCom). gateway.run uses this field
to inject a [Replying to: "..."] prefix when the parent is not
already in the session history, which is exactly the scenario
triggered by cron-generated thread parents.
- New helper _fetch_thread_parent_text reuses the existing thread-
context cache (and its 60s TTL) to avoid duplicate
conversations.replies calls; falls back to a cheap limit=1 fetch
when the cache is cold.
Tests:
- Updated TestSlackThreadContext::test_skips_bot_messages to reflect
the new behaviour (self-bot child dropped, third-party bot kept).
- Added:
* test_fetch_thread_context_includes_bot_parent
* test_fetch_thread_context_excludes_self_bot_replies
* test_fetch_thread_context_multi_workspace
* test_fetch_thread_context_current_ts_excluded (regression guard)
* test_fetch_thread_parent_text_from_cache
* test_slack_reply_to_text_set_on_thread_reply
* test_slack_reply_to_text_none_for_top_level_message
Full Slack suite: 176 passed (was 169).
Slack Connect channels return file objects with file_access="check_file_info" and no url_private_download field (see https://docs.slack.dev/reference/objects/file-object/#slack_connect_files). These stub objects must be resolved via files.info before download can proceed. Without this the agent silently skips attachments posted in Slack Connect channels. Call files.info on every file whose file_access is check_file_info, replace the stub with the full file object, and let the existing download path continue. Warn and skip on files.info failures. Closes NousResearch#11095.
Temporary background review agents can initialize Hindsight-backed memory clients, but close() alone skips provider teardown. Shut the memory provider down before closing so aiohttp sessions do not leak at process exit. Made-with: Cursor
…search#16099 Background review fork now inherits session_id, credential_pool, and status_callback from the parent (added in NousResearch#16099 after this PR was written). Extend the bare-agent helper so the regression test keeps reaching the cleanup assertions instead of failing in the runtime resolver. Signed-off-by: Teknium <8425893+teknium1@users.noreply.github.com>
Translate Slack attachment failures into actionable user-facing notices
instead of generic download errors. When a scope/auth/permission issue
breaks attachment processing, the user sees:
[Slack attachment notice]
- Slack attachment access failed for photo.jpg. Missing scope:
files:read. Update the Slack app scopes/settings and reinstall
the app to the workspace.
Two helpers do the translation:
_describe_slack_api_error — handles SlackApiError responses
(missing_scope, invalid_auth, file_not_found, access_denied, etc.)
_describe_slack_download_failure — handles httpx.HTTPStatusError
(401/403/404) and Slack-returns-HTML-sign-in fallbacks
Wired into three existing call sites:
- the Slack Connect files.info path (PR NousResearch#11111) so scope errors
surface instead of being logged as generic "files.info failed"
- the image, audio, and document download paths so 401/403 and
HTML-body responses translate into actionable notices
Adjustment from original PR: dropped _probe_slack_file_access_issue,
the proactive pre-download files.info probe. It added one extra
Slack API call per attachment even on healthy ones, and overlapped
with the existing files.info call from PR NousResearch#11111. The post-failure
translation path covers the same user-facing diagnostic value
without the per-message tax.
Also documents files:read scope more prominently in the Slack setup
guide and troubleshooting table.
Contributed back from https://github.com/xinbenlv/zn-hermes-agent.
Closes NousResearch#7015.
Co-authored-by: xinbenlv <zzn+pa@zzn.im>
HindsightEmbedded.close() delegates to its sync client.close(). When Hermes created/used that client on the shared async loop, closing it from the main thread raises 'attached to a different loop' before aiohttp releases the session — so the ClientSession / TCPConnector leak past provider teardown. Close the embedded inner async client on the shared loop first via _run_sync(inner_client.aclose()), then let the wrapper's sync close() do its daemon/UI bookkeeping. Salvage of NousResearch#14605: test placement rebased — appended TestShutdown class after TestSharedEventLoopLifecycle (which landed on main after the PR was written). Original author attribution preserved.
…ousResearch#16208) After NousResearch#14798 made cron honor per-platform `hermes tools` config, the `_DEFAULT_OFF_TOOLSETS` filter silently stripped `homeassistant` from cron jobs for users who'd been relying on the previous blanket toolset. Norbert's HA cron reports regressed as a result. The HA toolset is already runtime-gated by its `check_fn` (requires HASS_TOKEN to register any tools). When HASS_TOKEN is set the user has explicitly opted in — `_DEFAULT_OFF_TOOLSETS` adds nothing in that case, so stop double-gating and restore HA for cron / cli / other platforms without an explicit saved toolset list. moa and rl stay off by default (original NousResearch#14798 goal preserved). Fixes HA cron regression reported by Norbert.
Slack's modern composer sends messages with a 'blocks' array that
contains rich_text elements. When a user forwards or quotes another
message, the quoted content shows up in the rich_text_quote children
of that array — and is NOT included in the plain 'text' field. The
agent saw only the lossy plain text and was blind to forwarded /
quoted content. Same story for link unfurl previews (Notion, docs,
GitHub, etc.) which Slack puts in the 'attachments' array.
Two fixes in the inbound handler:
1. _extract_text_from_slack_blocks walks rich_text / rich_text_quote /
rich_text_list / rich_text_preformatted trees and renders readable
text ('> quoted', '• bullet', code fences), dedupes against the
plain text field, and appends the extracted content so the agent
sees everything.
2. Link unfurl / attachment preview extraction reads title, url,
body, and footer from the 'attachments' array and appends a
'📎 [title](url)\n body\n _footer_' section per preview.
Skips is_msg_unfurl to avoid echoing our own Slack replies back.
Routing is careful not to trust augmented text: mention gating
(is_mentioned) and slash-command detection both run against the
original 'text' field, so forwarded content containing '<@bot>' or
'/deploy' in a quote can't trick the bot into responding in a
channel it shouldn't or classifying a normal message as a command.
Adjustment from original PR: dropped _serialize_slack_blocks_for_agent,
which inlined a redacted JSON dump of non-rich_text blocks (section,
accessory, actions, etc.) — the agent would see the raw Block Kit
structure for UI-heavy alerts. It added up to 6000 characters to the
prompt context on every qualifying message with no opt-out. The
rich_text extraction and attachment unfurls cover the common bug-fix
case (quoted/forwarded content + link previews) without the prefill
tax. If a user needs block inspection later, it can return as a
config opt-in.
Also updates the Slack platform notes in session.py to accurately
describe what the gateway inlines.
Collaborator
|
Thanks for the focused config-parity work. Automated hermes-sweeper review found that this change adds a user-facing
This is an automated hermes-sweeper review. Closed as not-planned per standing maintainer policy ( |
teknium1
added
the
sweeper:not-planned
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
19 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The
vision_analyze_toolhardcodedmax_tokensto 2000, ignoring anyauxiliary.vision.max_tokensvalue set in config.yaml. Users with local vision models (vLLM, llama.cpp) or high-capacity APIs could not leverage their full output capacity for OCR or detailed image analysis.Changes
max_tokensfromauxiliary.vision.max_tokensconfig, falling back to the previous hardcoded default of 2000.timeoutandtemperature, which are already read fromauxiliary.vision.*.Test Plan
max_tokensFixes
Vision outputs were silently truncated regardless of config.
Scope:
tools/vision_tools.pyonly.Type: Feature (config parity with existing timeout/temperature wiring).
Breaking: No — backward compatible via fallback default.