perf(tools): compact delegate_task description by deduping against param schema - #77372
Merged
Conversation
…ram schema The top-level delegate_task description repeated content the model already receives through parameter descriptions: the concurrency limit (tasks param), the full nesting clause (role param), context-passing guidance (goal/context params), and background semantics (background param). Every API call paid for the duplication (~4,000 chars). The description now carries only what exists nowhere else in the schema: use/don't-use routing (execute_code, cronjob), the no-poll rule, the non-durability warning, the self-report verification contract with concrete verbs, the language-passing example, the leaf blocked-tool list, and model inheritance. 3,963 -> 1,704 chars (~570 tokens saved per API call), and the top-level text is now static (dynamic limits flow only through the two param descriptions, which are already rebuilt per get_definitions() call). A/B benchmark across 4 models (gpt-4o, gpt-4o-mini, claude-haiku-4.5, llama-3.3-70b) showed the naive compaction in PR #72813 regressed weaker models on exactly the passages it cut (side-effect verification 8/8->0/8 on gpt-4o-mini; language passing 3/3->0/3 on haiku-4.5). This version keeps those benchmark-sensitive hooks verbatim. Tests pin the contracts at keyword level (not prose-literal) plus a size ceiling, and verify dynamic limits still reach the model via the tasks/role param descriptions. Refs #72737, supersedes the delegate_task half of PR #72813.
…ption Round-2 A/B (gpt-4o-mini, 6 reps) showed two passages could not survive paraphrase: the DO-NOT-USE list needs the arrow-list shape with the 'no reasoning needed' qualifier (prose form regressed mechanical-work routing 6/6->1/6), and the self-report rule needs the concrete 'claiming uploaded successfully may be wrong' framing (without it, side-effect verification regressed 6/6->2/6). With both restored: 30/42 vs 30/42 on gpt-4o-mini and intent-parity on claude-haiku-4.5. Final size: 1,900 chars (from 3,963).
Contributor
૮ >ﻌ< ა ci reviewran on 0b723f1 all good! |
alt-glitch
added
type/perf
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The
delegate_tasktop-level description now carries only guidance that exists nowhere else in the schema — 3,963 → 1,900 chars (~570 tokens saved on every API call), with A/B-verified behavior parity.Root cause of the bloat: the top-level text repeated content the model already receives through parameter descriptions — the concurrency limit (
tasksparam), the full nesting clause (roleparam, verbatim duplicate), context-passing guidance (goal/contextparams), and background semantics (backgroundparam).Changes
tools/delegate_tool.py:_build_top_level_description()rewritten to be static and duplication-free; dynamic limits (max_concurrent_children,max_spawn_depth) now flow to the model exclusively through thetasks/roleparam descriptions, which were already rebuilt perget_definitions()call.tests/tools/test_delegate.py: keyword-level contract test (size ceiling + every top-level-only contract) and a test proving dynamic limits reach the model via param descriptions and no stale limits remain in the static text.website/docs/reference/tools-reference.md: description excerpt refreshed.What the compact text keeps (top-level-only contracts)
Use/don't-use routing (
execute_code,cronjob), the no-poll rule, the non-durability warning (/stop//new), the self-report verification contract with concrete verbs ("fetch the URL, stat the file"), the"respond in Chinese"language example, the leaf blocked-tool list, and model inheritance/pinning.Validation
A/B benchmark over OpenRouter (7 delegation-routing scenarios, tool-call grading, both arms carrying the real param descriptions):
The haiku delta is a single flaky scenario (language passing, which OLD also failed 1/4); every routing, no-poll, verification, and context-passing scenario is at parity. Two earlier compaction drafts DID regress (mechanical-work routing 6/6→1/6 when the DO-NOT-USE list was collapsed to prose; verification 6/6→2/6 when the "may be wrong" framing was cut) — those benchmark-sensitive passages are restored verbatim. The naive compaction in #72813 showed the same two regressions plus loss of
context-field usage.scripts/run_tests.sh tests/tools/test_delegate.py tests/agent/test_prompt_builder.py -q— 118 passed, 0 failed.model_tools.get_tool_definitions()against tempHERMES_HOMEconfirms all contract keywords present and dynamic limits flowing via params.Refs #72737. Supersedes the
delegate_taskhalf of #72813 (thanks @JeremyDev87 for the direction and the size-regression test idea; the skill-preamble half is reviewed separately).Infographic