Skip to content

docs(architecture): audit Tools and capabilities group against implementation - #4799

Merged
Astro-Han merged 5 commits into
apache:mainfrom
ggbdpq:docs/audit-tools-a4
Sep 7, 2026
Merged

Astro-Han merged 5 commits into
apache:mainfrom
ggbdpq:docs/audit-tools-a4

Conversation

@ggbdpq

@ggbdpq ggbdpq commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Summary

Audits the Tools and capabilities group (3 documents) from #3522 against the current implementation (22715e8).

  • docs/architecture/mcp-runtime-architecture-draft.zh-CN.md — no drift. Load-bearing contracts resolve as written: tool naming mcp__{serverId}__{toolName} with the 64-char hash-suffix rule (mcp-tools.ts:224), the categoryHint: network_send default (mcp-tools.ts:141), the 30s/60s/15s/600s timeout matrix (packages/mcp/src/index.ts:125-128), the stdio env allowlist with LC_*/XDG_* prefixes (index.ts:2422-2433), and the four package boundaries. Its PermissionEngine mention is the one the tracking issue already verified as correct.
  • docs/skill-catalog-policy.md — no drift. All named symbols resolve (selectSkillsForContext, SkillSelectionReport decision values, BUNDLED_SKILL_CATALOG, prepareSkillInvocationMessage, SkillInvocationReceipt, too_many_requests, SkillSearch, blocked_path/read_failed, .maka/skills-state.json v2 with needsReview), and the budget constants match the code exactly (skills-context.ts:50-53).
  • docs/web-search-provider-capability.mdone drift fixed: the execution-surfaces table still listed the retired Headless Harbor surface and its MAKA_WEB_SEARCH_ENABLED=true opt-in. That env var was introduced by feat(search): add provider-native web search #2152 (2026-08-05) and removed by refactor: replace Headless with minimal Eval kernel #2605 (2026-08-11) when Headless became the minimal Eval kernel; on the current tree it exists only in this document. The row now records the retirement, and the opt-in rationale is reworded to apply to eval subjects. Everything else verifies (routeWebSearchTools, packages/core/src/model-web-search.ts, providerExecuted replay, the DeepSeek V4 Flash anthropic-compatible rule); web_search_20250305 lives in the conformance/backend test suites as described.

All three documents gained the standard frontmatter block with last_verified; none has a bilingual pair (translation_status: source-only).

Verification

Claim Command / route Result
MCP contracts grep mcp-tools.ts, packages/mcp/src/index.ts all constants and rules match
Skill budget constants grep packages/runtime/src/skills-context.ts 18000 / 4_000 / 8_000 / 0.02 exact
Retired Harbor opt-in git log -S MAKA_WEB_SEARCH_ENABLED / repo-wide grep introduced #2152, removed #2605; only the doc mentions it today
Web-search symbols grep across packages/*, apps/* all resolve
Drift greps tracking-issue patterns on the three docs zero unexpected matches
Format / headers npm run format:check / npm run check:asf-headers exit 0 / changed files pass

AI use

Prepared with ZCode (GLM-5.3-Flash): read the three documents, traced each checkable claim to code, and drafted the corrections. The commit carries the Generated-by trailer.

Checklist

  • One subsystem group only (Tools and capabilities, 3 documents)
  • Refs #3522 in the commit message
  • last_verified set to the audit date (2026-09-04)
  • Docs only, no code changes

@github-actions github-actions Bot added the effort/S Under 100 readable lines label Sep 4, 2026
…entation

Audit the three Tools and capabilities documents from the apache#3522
documentation audit against the current implementation (22715e8).

mcp-runtime-architecture-draft.zh-CN.md: the load-bearing contracts
resolve as written (tool naming mcp__{server}__{tool} with the 64-char
hash suffix, categoryHint network_send default, the 30s/60s/15s/600s
timeout matrix, the stdio env allowlist, the four package boundaries).
The PermissionEngine mention is the one the tracking issue already
verified as correct. No drift; added the standard frontmatter.

skill-catalog-policy.md: every named symbol resolves, and the budget
constants match the code exactly (MAX_SKILLS_PROMPT_CHARS 18000,
MIN/MAX_SKILLS_PROMPT_TOKENS 4_000/8_000, SKILLS_PROMPT_CONTEXT_RATIO
0.02). No drift; added the standard frontmatter.

web-search-provider-capability.md: one drift fixed — the execution
surfaces table still listed the retired Headless Harbor surface and
its MAKA_WEB_SEARCH_ENABLED opt-in, which apache#2605 removed on 2026-08-11
when Headless became the minimal Eval kernel; the row now records the
retirement instead of presenting the surface as current. The rest
verifies (routeWebSearchTools, model-web-search.ts, providerExecuted
replay, the DeepSeek V4 Flash anthropic-compatible rule). Added the
standard frontmatter.

Refs apache#3522

Generated-by: GLM-5.3-Flash (ZCode)
@ggbdpq
ggbdpq force-pushed the docs/audit-tools-a4 branch from 774aea4 to c2d3697 Compare September 4, 2026 22:48
@ggbdpq ggbdpq changed the title docs: audit Tools and capabilities group against implementation docs(architecture): audit Tools and capabilities group against implementation Sep 4, 2026

@hqhq1025 hqhq1025 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed exact head c2d36977b9bde51b0c615d8f06a9f4d9f365d423.

Actual change: adds current-status metadata to the MCP runtime and Skill catalog documents, and replaces the retired Headless Harbor row in the web-search document with an Eval-subject explanation. I checked the three documents against the exact-head implementation, focused tests/contracts, and current main.

Validation: hosted exact-head test, ASF header audit, focused Biome formatting, and git diff --check pass. The current-main synthetic merge tree is clean. The MCP protocol/timeouts/tool projection and Skill catalog/invocation contracts named in the PR resolve in code. I found one implementation mismatch in the changed Eval paragraph.

I did not independently rerun the historical live-provider searches, performance measurements, or revalidate every external provider-matrix statement against its public API.

Automated review notice: This comment was posted by an automated review agent operated by hqhq1025. It is not an independent human review and does not replace one.

Comment thread docs/web-search-provider-capability.md Outdated

@hqhq1025 hqhq1025 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed exact head c7665057639a646dc7685a42ebc3eb2cface4152 after the merge from main.

The existing P2 remains on this head. docs/web-search-provider-capability.md:87-92 says current Eval subjects can enable provider-native search in their own configuration. The exact-head Eval contract says the opposite: packages/eval/README.md:184-197 removes named and native web tools from every benchmark subject; harbor-external-subject.ts:94-108,198-203,326-330 installs explicit deny policies; and provider-web-tool-surface.test.ts:24-47 verifies the metering proxy strips native web-search tools.

The merge commit did not change any of the three PR documents or the cited Eval implementation relative to the previously reviewed commit. The existing inline thread remains current and is not duplicated here. The MCP runtime and Skill catalog claims still match the inspected implementation.

Validation: hosted test passed on this exact head; repository format, ASF headers, and git diff --check passed; the provider web-tool surface regression passed 2/2; and current main produced a clean synthetic merge tree identical to the reviewed tree. The exact-head Eval aggregate build was unavailable under the reused dependency layout because @maka/runtime-host/client and /protocol were not resolved, so I relied on the focused source test and hosted CI for that path. I did not rerun historical live-provider or performance measurements, or revalidate every external provider claim against its API.

Automated review notice: This comment was posted by an automated review agent operated by hqhq1025. It is not an independent human review and does not replace one.

… doc

Review (hqhq1025) flagged the Eval paragraph for documenting an enablement
path that does not exist: current benchmark subjects cannot turn on
provider-native search in their own configuration. packages/eval/README.md
removes WebSearch, WebFetch, and FetchURL from every subject and the metering
proxy structurally strips named and native web tools from external-harness
requests. Reword the retired Headless Harbor opt-in rationale as the reason
Eval subjects have web search removed outright for parity, keeping the
enablement story historical.

Generated-by: GLM-5.3-Flash (ZCode)

@hqhq1025 hqhq1025 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed exact head 2f25ebf88b96875564e86f2ccf4a224264ae12b2.

The previous P2 is fixed. docs/web-search-provider-capability.md now states the current Eval contract accurately: benchmark subjects have no web-search enablement path, their provider-visible tool lists remove WebSearch, WebFetch, and FetchURL, and the external-harness metering proxy strips both named and provider-native web tools.

I found no remaining P0-P3 issues. The incremental commit changes only that paragraph; the MCP and Skill documents and the cited implementation are unchanged from the prior review. The focused Eval web-tool regression passed 2/2, hosted exact-head test passed, ASF headers and git diff --check passed, and a synthetic merge onto current main (12ebc45e6fe43b2eebd7dfe61f5c1fa92c0c873a) was clean with the PR patch preserved. I did not rerun live-provider searches, historical performance experiments, or the complete external-provider API matrix.

Automated review notice: This comment was posted by an automated review agent operated by hqhq1025. It is not an independent human review and does not replace one.

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update. Additional independent review of 2f25ebf88b96875564e86f2ccf4a224264ae12b2: the corrected Eval no-web contract is present. One minor omission remains in the MCP environment inventory.

AI disclosure: Codex agents performed this review; the coordinating agent checked the cited evidence. This is not an independent human review.

中文说明

Eval 禁用 Web 工具的修正已到位;MCP 环境继承清单仍有一处轻微遗漏。

Comment thread docs/architecture/mcp-runtime-architecture-draft.zh-CN.md
`buildStdioEnvironment` (packages/mcp/src/index.ts) inherits LOGNAME
alongside USER by default, but the security section's allowlist omitted
it, understating what a stdio server receives. Documentation-only
correction from the P3 review on apache#4799.

Generated-by: GLM-5.3-Flash (ZCode)

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the change. Approving exact head 57b4427091213daa5c4bd0388e38e1a4a7f5a636 following the completed review and Astro-Han’s explicit acceptance of this merge batch. Current checks pass and no review threads remain unresolved.

Align tools and capability documentation with current Eval web-tool exclusions and MCP environment inheritance, including LOGNAME. Preserve historical provider evidence separately from current execution surfaces. Documentation only.

AI assistance: Codex performed the review and final-state verification; Astro-Han authorized approval and merge.

中文

感谢改动。基于已完成的审查和 Astro-Han 对本批次的明确认可,批准当前精确 head;检查通过,讨论已结清。此前说明的验证边界与后续事项保持不变。本次由 Codex 执行审查和状态核对,Astro-Han 授权批准与合并。

@Astro-Han
Astro-Han merged commit c802759 into apache:main Sep 7, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

effort/S Under 100 readable lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants