docs(architecture): audit Tools and capabilities group against implementation - #4799
Conversation
…entation Audit the three Tools and capabilities documents from the apache#3522 documentation audit against the current implementation (22715e8). mcp-runtime-architecture-draft.zh-CN.md: the load-bearing contracts resolve as written (tool naming mcp__{server}__{tool} with the 64-char hash suffix, categoryHint network_send default, the 30s/60s/15s/600s timeout matrix, the stdio env allowlist, the four package boundaries). The PermissionEngine mention is the one the tracking issue already verified as correct. No drift; added the standard frontmatter. skill-catalog-policy.md: every named symbol resolves, and the budget constants match the code exactly (MAX_SKILLS_PROMPT_CHARS 18000, MIN/MAX_SKILLS_PROMPT_TOKENS 4_000/8_000, SKILLS_PROMPT_CONTEXT_RATIO 0.02). No drift; added the standard frontmatter. web-search-provider-capability.md: one drift fixed — the execution surfaces table still listed the retired Headless Harbor surface and its MAKA_WEB_SEARCH_ENABLED opt-in, which apache#2605 removed on 2026-08-11 when Headless became the minimal Eval kernel; the row now records the retirement instead of presenting the surface as current. The rest verifies (routeWebSearchTools, model-web-search.ts, providerExecuted replay, the DeepSeek V4 Flash anthropic-compatible rule). Added the standard frontmatter. Refs apache#3522 Generated-by: GLM-5.3-Flash (ZCode)
774aea4 to
c2d3697
Compare
hqhq1025
left a comment
There was a problem hiding this comment.
Reviewed exact head c2d36977b9bde51b0c615d8f06a9f4d9f365d423.
Actual change: adds current-status metadata to the MCP runtime and Skill catalog documents, and replaces the retired Headless Harbor row in the web-search document with an Eval-subject explanation. I checked the three documents against the exact-head implementation, focused tests/contracts, and current main.
Validation: hosted exact-head test, ASF header audit, focused Biome formatting, and git diff --check pass. The current-main synthetic merge tree is clean. The MCP protocol/timeouts/tool projection and Skill catalog/invocation contracts named in the PR resolve in code. I found one implementation mismatch in the changed Eval paragraph.
I did not independently rerun the historical live-provider searches, performance measurements, or revalidate every external provider-matrix statement against its public API.
Automated review notice: This comment was posted by an automated review agent operated by hqhq1025. It is not an independent human review and does not replace one.
hqhq1025
left a comment
There was a problem hiding this comment.
Re-reviewed exact head c7665057639a646dc7685a42ebc3eb2cface4152 after the merge from main.
The existing P2 remains on this head. docs/web-search-provider-capability.md:87-92 says current Eval subjects can enable provider-native search in their own configuration. The exact-head Eval contract says the opposite: packages/eval/README.md:184-197 removes named and native web tools from every benchmark subject; harbor-external-subject.ts:94-108,198-203,326-330 installs explicit deny policies; and provider-web-tool-surface.test.ts:24-47 verifies the metering proxy strips native web-search tools.
The merge commit did not change any of the three PR documents or the cited Eval implementation relative to the previously reviewed commit. The existing inline thread remains current and is not duplicated here. The MCP runtime and Skill catalog claims still match the inspected implementation.
Validation: hosted test passed on this exact head; repository format, ASF headers, and git diff --check passed; the provider web-tool surface regression passed 2/2; and current main produced a clean synthetic merge tree identical to the reviewed tree. The exact-head Eval aggregate build was unavailable under the reused dependency layout because @maka/runtime-host/client and /protocol were not resolved, so I relied on the focused source test and hosted CI for that path. I did not rerun historical live-provider or performance measurements, or revalidate every external provider claim against its API.
Automated review notice: This comment was posted by an automated review agent operated by hqhq1025. It is not an independent human review and does not replace one.
… doc Review (hqhq1025) flagged the Eval paragraph for documenting an enablement path that does not exist: current benchmark subjects cannot turn on provider-native search in their own configuration. packages/eval/README.md removes WebSearch, WebFetch, and FetchURL from every subject and the metering proxy structurally strips named and native web tools from external-harness requests. Reword the retired Headless Harbor opt-in rationale as the reason Eval subjects have web search removed outright for parity, keeping the enablement story historical. Generated-by: GLM-5.3-Flash (ZCode)
hqhq1025
left a comment
There was a problem hiding this comment.
Re-reviewed exact head 2f25ebf88b96875564e86f2ccf4a224264ae12b2.
The previous P2 is fixed. docs/web-search-provider-capability.md now states the current Eval contract accurately: benchmark subjects have no web-search enablement path, their provider-visible tool lists remove WebSearch, WebFetch, and FetchURL, and the external-harness metering proxy strips both named and provider-native web tools.
I found no remaining P0-P3 issues. The incremental commit changes only that paragraph; the MCP and Skill documents and the cited implementation are unchanged from the prior review. The focused Eval web-tool regression passed 2/2, hosted exact-head test passed, ASF headers and git diff --check passed, and a synthetic merge onto current main (12ebc45e6fe43b2eebd7dfe61f5c1fa92c0c873a) was clean with the PR patch preserved. I did not rerun live-provider searches, historical performance experiments, or the complete external-provider API matrix.
Automated review notice: This comment was posted by an automated review agent operated by hqhq1025. It is not an independent human review and does not replace one.
Astro-Han
left a comment
There was a problem hiding this comment.
Thanks for the update. Additional independent review of 2f25ebf88b96875564e86f2ccf4a224264ae12b2: the corrected Eval no-web contract is present. One minor omission remains in the MCP environment inventory.
AI disclosure: Codex agents performed this review; the coordinating agent checked the cited evidence. This is not an independent human review.
中文说明
Eval 禁用 Web 工具的修正已到位;MCP 环境继承清单仍有一处轻微遗漏。
`buildStdioEnvironment` (packages/mcp/src/index.ts) inherits LOGNAME alongside USER by default, but the security section's allowlist omitted it, understating what a stdio server receives. Documentation-only correction from the P3 review on apache#4799. Generated-by: GLM-5.3-Flash (ZCode)
Astro-Han
left a comment
There was a problem hiding this comment.
Thanks for the change. Approving exact head 57b4427091213daa5c4bd0388e38e1a4a7f5a636 following the completed review and Astro-Han’s explicit acceptance of this merge batch. Current checks pass and no review threads remain unresolved.
Align tools and capability documentation with current Eval web-tool exclusions and MCP environment inheritance, including LOGNAME. Preserve historical provider evidence separately from current execution surfaces. Documentation only.
AI assistance: Codex performed the review and final-state verification; Astro-Han authorized approval and merge.
中文
感谢改动。基于已完成的审查和 Astro-Han 对本批次的明确认可,批准当前精确 head;检查通过,讨论已结清。此前说明的验证边界与后续事项保持不变。本次由 Codex 执行审查和状态核对,Astro-Han 授权批准与合并。
Summary
Audits the Tools and capabilities group (3 documents) from #3522 against the current implementation (22715e8).
docs/architecture/mcp-runtime-architecture-draft.zh-CN.md— no drift. Load-bearing contracts resolve as written: tool namingmcp__{serverId}__{toolName}with the 64-char hash-suffix rule (mcp-tools.ts:224), thecategoryHint: network_senddefault (mcp-tools.ts:141), the 30s/60s/15s/600s timeout matrix (packages/mcp/src/index.ts:125-128), the stdio env allowlist withLC_*/XDG_*prefixes (index.ts:2422-2433), and the four package boundaries. ItsPermissionEnginemention is the one the tracking issue already verified as correct.docs/skill-catalog-policy.md— no drift. All named symbols resolve (selectSkillsForContext,SkillSelectionReportdecision values,BUNDLED_SKILL_CATALOG,prepareSkillInvocationMessage,SkillInvocationReceipt,too_many_requests,SkillSearch,blocked_path/read_failed,.maka/skills-state.jsonv2 withneedsReview), and the budget constants match the code exactly (skills-context.ts:50-53).docs/web-search-provider-capability.md— one drift fixed: the execution-surfaces table still listed the retired Headless Harbor surface and itsMAKA_WEB_SEARCH_ENABLED=trueopt-in. That env var was introduced by feat(search): add provider-native web search #2152 (2026-08-05) and removed by refactor: replace Headless with minimal Eval kernel #2605 (2026-08-11) when Headless became the minimal Eval kernel; on the current tree it exists only in this document. The row now records the retirement, and the opt-in rationale is reworded to apply to eval subjects. Everything else verifies (routeWebSearchTools,packages/core/src/model-web-search.ts,providerExecutedreplay, the DeepSeek V4 Flash anthropic-compatible rule);web_search_20250305lives in the conformance/backend test suites as described.All three documents gained the standard frontmatter block with
last_verified; none has a bilingual pair (translation_status: source-only).Verification
mcp-tools.ts,packages/mcp/src/index.tspackages/runtime/src/skills-context.tsgit log -S MAKA_WEB_SEARCH_ENABLED/ repo-wide greppackages/*,apps/*npm run format:check/npm run check:asf-headersAI use
Prepared with ZCode (GLM-5.3-Flash): read the three documents, traced each checkable claim to code, and drafted the corrections. The commit carries the
Generated-bytrailer.Checklist
Refs #3522in the commit messagelast_verifiedset to the audit date (2026-09-04)