Integrate upstream tiered tool disclosure (#67034) into Hopebox v0.19 - #4
Merged
latentoperator merged 12 commits intoJul 27, 2026
Merged
Conversation
… disclosure Deferred MCP/plugin tools become invisible once the tool_search bridge activates — live benchmarking (48 runs, Claude Haiku 4.5) showed models substituting visible core tools (terminal/web_search/browser) for deferred capabilities or declaring them nonexistent instead of searching: 16/24 task success vs 24/24 with eager loading. Skills never had this failure mode because every skill keeps a ~21-token name+description listing line in the system prompt. This ports that exact pattern to the tool bridge: when tool_search activates, a grouped manifest of every deferred tool (name + first sentence of description, clipped to 60 chars, grouped per MCP server / toolset) is embedded in the tool_search bridge description. - tools/tool_search.py: build_catalog_listing() with deterministic ordering (byte-stable across assemblies -> prompt prefix stays cacheable); token-budget fallbacks full -> names-only -> legacy bare count; bridge_tool_schemas(listing=...) embeds it and instructs the model to skip tool_search when the exact name is visible (one fewer round-trip per use) - config: tools.tool_search.listing auto|on|off (default auto), listing_max_tokens (default 4000, clamped 200..20000); legacy bool shapes keep working - tests: 8 new tests (config parsing/clamps, short-desc clipping, deterministic rendering, budget fallbacks, bridge embedding, assembly on/off paths); full file green (47 passed) - docs: tool-search.md config table + rationale - scripts/tool_search_livetest2.py: benchmark harness v2 with real per-call token accounting (normalize_usage spy) and a third 'listing' mode for A/B/C comparison
…eplayed (Opus 4.8) Replays the actual tool schemas captured from Epic's UE 5.8 ModelContextProtocol + AllToolsets plugins (830 tools / 52 toolsets) as live registry tools with mocked editor responses, then benchmarks eager vs bare-bridge vs bridge+listing at two scales (62-tool editor subset, full 830) on Claude Opus 4.8 (1M ctx; eager at 830 does not fit any 200K model — first call requests ~266K tokens). Headline (full 830, mean per task, rescored): eager 8/8 at 810,578 input tokens ($4.05); bare bridge 16/16 at 160,844 ($0.80); listing 16/16 at 257,264 ($1.29). Frontier model erases the accuracy gap in every mode; cost is the differentiator. At 62 tools eager wins on cost — consistent with the auto-threshold design. Also parameterizes livetest harness model + listing_max_tokens via env/args (TS_UE_MODEL, TS_UE_SCALE, TS_UE_MODES, TS_UE_LISTING_MAX).
…4 + windows-footguns)
… error mocks, strict scoring Scenarios target real confusion clusters in Epic's UE 5.8 catalog (StaticMesh vs SkeletalMesh set_material, three tag systems, CurveTable vs DataTable rows, Niagara Component vs System variables, four capture variants, zero-keyword phrasing). Mocks return realistic editor errors on wrong-type calls; scoring is strict (clean solve = correct tool with zero distractor calls; first-call accuracy tracked separately). Key result: first-call selection is unreliable in EVERY mode — eager with all 199K of schemas in context managed 2/10 — but clean solves stay 75-95% because agents probe (get_components, get_material_slots) before committing. The probe loop works through the 3-tool bridge at 1/4 the cost of eager ($1.60-1.69 vs $6.49/task, Opus 4.8). On Haiku the listing beats bare bridge 18/20 vs 15/20 (core-tool substitution again). Zero distractor invocations across all 50 Opus runs.
…e the listing's structural advantage Bridge vs listing only (Opus 4.8, 830 real UE schemas, 3 reps/cell). Excluding one both-modes mock artifact: listing 24/24 vs bridge 20/24, searches/task 0.2 vs 4.0. Bridge failures: core-tool substitution at frontier tier (ran the host test suite via terminal instead of discovering RunTests, 2/3 reps), up to 8 searches to prove a negative, and search-vocabulary misses on paraphrase. Listing asserts absence in zero searches and answers a 5-way capability survey in 1 API call.
…scale the listing with catalog size
Tier 0: no MCP/plugin tools -> everything eager (pass-through).
Tier 1: deferred tools whose catalog listing fits min(threshold_pct%
of context, listing_max_tokens) -> bridge + skills-style
listing, degrading to names-only over budget.
Tier 2: listing over budget even names-only (Cloudflare's flat API
surface: 3,320 tools, names alone ~32K tokens) -> bare bridge,
discovery through tool_search only.
The old activation threshold (defer only when schemas > threshold_pct
of context) let mid-size catalogs ride eager and pay full schema cost;
with servers like Cloudflare (~597K tokens of schema, would not even
fit a 200K window) the binary gate is the wrong shape. Activation is
now driven purely by deferrable-tool presence; threshold_pct is
repurposed as the listing budget's context-relative leg.
- AssemblyResult gains tier + listing_form for observability
- listing_max_tokens default 4000 -> 20000 (cap 60000) so an 830-tool
catalog keeps a names-only listing while Cloudflare-scale drops to
bare bridge
- E2E verified against real captures: Linear 24 tools -> tier 1 full,
Epic UE 5.8 830 tools -> tier 1 names-only, Cloudflare 3,320 tools
-> tier 2 (both 200K and 1M context)
…verse-map at dispatch
Cloudflare's flat API MCP ships 61 property keys that violate Anthropic's
^[a-zA-Z0-9_.-]{1,64}$ pattern (query-filter params like 'issue_class~neq'
and 'meta.<field>[<operator>]'). One bad key anywhere in the tools array
400s the ENTIRE request — measured live: Anthropic, Bedrock, Google Vertex,
and Azure all rejected an eager 3,320-tool Cloudflare request at validation,
before token limits even applied.
- schema_sanitizer: rename non-conforming property keys deterministically
(bad chars -> '_', 64-char truncation, collision dedup with numeric
suffixes), nested schemas included; required[] remapped alongside
- unrename_tool_args(): reverse map applied in coerce_tool_args at dispatch,
so the MCP server receives the original wire names; recurses into object
values and array items
- deterministic on both sides: the rename map is recomputed from the
registry's original schema at dispatch time, no state carried
E2E on the real capture: 61 -> 0 violations across 3,320 tools; round-trip
verified on get_accounts_intel_attacksurfacereport_issues (5 '~neq' keys).
…ion; 5% default budget
Teknium review changes on the tiered policy:
1. threshold_pct default 10 -> 5 (listing budget = min(5% of context,
listing_max_tokens)); unknown-context fallback 20K -> 10K.
2. Tier 2 no longer leaves the model blind: when even names-only doesn't
fit, the bridge description now carries a one-line-per-server summary
('cloudflare (3320 tools)') plus an instruction to search FIRST rather
than substitute a generic tool or claim the capability is missing —
the measured tier-2 failure mode (core-tool substitution) at zero
meaningful token cost (~50 tokens/server).
3. Listing degradation is now PER SERVER, largest first: one oversized
server (Cloudflare) collapses to its summary line while small
co-attached servers (Linear) keep their full per-tool listings
('mixed' form). Previously global: attaching Cloudflare next to
Linear silently cost Linear its listing. Greedy fit is deterministic
(size then label) so the rendered block stays byte-stable per catalog
— prompt-prefix cache safe.
E2E on real captures (defaults, 200K ctx): linear alone -> tier 1 full;
unreal alone -> tier 2 groups (5% budget) / tier 1 names at 1M;
cloudflare alone -> tier 2 groups; linear+cloudflare -> tier 1 MIXED
(linear fully listed, cloudflare summarized). 48/48 tests.
The include/exclude filter matched exact names only — glob-style entries
('*_radar_*') silently matched nothing, so a Cloudflare flat-mode config
meant to trim 3,320 tools to ~1,900 actually registered 3,319. Unmatched
patterns produced no warning.
- matches_name_filter(): exact membership first (O(1) for literal lists),
then fnmatch.fnmatchcase for entries containing * ? [ — same pattern
semantics as approvals.deny. Entries without metacharacters stay
strictly literal ('docs' never matches 'docs_search').
- _should_register() uses it for both include and exclude (symmetric)
- hermes mcp tools picker (mcp_config.py) pre-selection uses the same
matcher so the UI agrees with runtime registration
E2E against the live Cloudflare capture with the real exclude list:
3,320 -> 1,905 surviving (1,415 excluded); radar/DLP gone,
purge_cache/dns_records kept. 220/220 mcp_tool tests (4 new).
…o tiered disclosure CI slices 2 and 7 caught three tests broken by always-defer: - /tools (CLI show_tools + TUI gateway tools.show) now passes skip_tool_search_assembly=True — it's a discovery/inspection surface, so users verifying an MCP installed must see deferred tools, not a collapsed bridge row. This also fixes test_slash_worker_mcp_discovery (profile MCP tool visible in /tools). - test_plugins.py::test_plugin_tools_in_definitions: 'visible' becomes 'reachable' — direct schema OR listed in the bridge description; scope-exclusion assertions unchanged (not direct AND not listed). - test_discord_tool.py dynamic-schema-rebuild test reads the pre-assembly list (the rebuilt schema is what tool_describe serves). Banner/status tool counts intentionally keep the post-assembly view — they reflect what the model actually sees.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Replays all 11 contributor-authored commits from NousResearch#67034 onto the current Hopebox v0.19 production base.
The integration adds:
/toolsHopebox reconciliation
The PR applied cleanly except for one stale-base import conflict in
tools/mcp_tool.py. The upstream PR parent already had an unrelatederrnoimport that v0.19 does not carry; this branch retains only the PR-requiredfnmatchimport.No Hopebox behavior was removed. Core Kanban tools remain non-deferrable. All 13 live profile configs parse under the new configuration and retain their existing always-defer posture with a 20K catalog-listing budget at a 200K context window.
Validation
/toolsand preloaded-skill tests passedgit diff --checkand merge-marker inspection passedDeployment status
Not deployed. The production branch and 26 running Hermes services remain on
e9f8f8f15127.The cutover is intentionally gated because
hopewell-hermes-backup.serviceis failing on Dante's malformedmessages_fts_trigramindex. A consistent database copy was repaired successfully with the standard FTS5 rebuild command, but the live database has not been modified.