Skip to content

perf(core): clear tool results to a low watermark to preserve prompt cache #8464

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
doudouOUC merged 9 commits into from
Aug 4, 2026
Merged

perf(core): clear tool results to a low watermark to preserve prompt cache #8464

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
a66751b
perf(core): clear tool results to a low watermark to preserve prompt …
doudouOUC Aug 3, 2026
86db1b7
fix(core): harden size-cleanup protection against zero-char and pendi…
doudouOUC Aug 3, 2026
8404ddc
qwen: address PR review feedback (#8464)
doudouOUC Aug 3, 2026
8dae440
qwen: address PR review feedback (#8464)
doudouOUC Aug 3, 2026
69f4712
qwen: address PR review feedback (#8464)
doudouOUC Aug 4, 2026
e596dc1
qwen: address PR review feedback (#8464)
doudouOUC Aug 4, 2026
3411e86
qwen: address PR review feedback (#8464)
doudouOUC Aug 4, 2026
0c5842f
qwen: address PR review feedback (#8464)
doudouOUC Aug 4, 2026
efb8e39
Merge branch 'main' into perf/microcompact-low-watermark
wenshao Aug 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 15 additions & 15 deletions docs/users/configuration/settings.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -302,21 +302,21 @@ The `extra_body` field allows you to add custom parameters to the request body s

#### context

| Setting | Type | Description | Default |
| ----------------------------------------------------------- | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------- |
| `context.fileName` | string or array of strings | The name of the context file(s). | `undefined` |
| `context.autoCompactThreshold` | number | Target fraction of the context window at which auto-compaction triggers. Must be greater than 0 and at most 1. Default is `0.85` (85%). Acts as a ceiling on the trigger: on large windows it is the effective trigger (~85%), while on smaller windows compaction may fire earlier to leave room to summarize. Replaces the old `model.chatCompression.contextPercentageThreshold`. | `undefined` (uses internal 0.85) |
| `context.importFormat` | string | The format to use when importing memory. | `undefined` |
| `context.includeDirectories` | array | Additional directories to include in the workspace context. Specifies an array of additional absolute or relative paths to include in the workspace context. Missing directories will be skipped with a warning by default. Paths can use `~` to refer to the user's home directory. This setting can be combined with the `--include-directories` command-line flag. | `[]` |
| `context.loadFromIncludeDirectories` | boolean | Controls the behavior of the `/memory refresh` command. If set to `true`, `QWEN.md` files should be loaded from all directories that are added. If set to `false`, `QWEN.md` should only be loaded from the current directory. | `false` |
| `context.fileFiltering.respectGitIgnore` | boolean | Respect .gitignore files when searching. | `true` |
| `context.fileFiltering.respectQwenIgnore` | boolean | Respect .qwenignore and configured custom ignore files when searching. | `true` |
| `context.fileFiltering.customIgnoreFiles` | array | Project-root-relative ignore files to use instead of the default compatibility files (`.agentignore`, `.aiignore`) when `respectQwenIgnore` is enabled. `.qwenignore` is always included. | `[".agentignore", ".aiignore"]` |
| `context.fileFiltering.enableRecursiveFileSearch` | boolean | Whether to enable searching recursively for filenames under the current tree when completing `@` prefixes in the prompt. | `true` |
| `context.fileFiltering.enableFuzzySearch` | boolean | When `true`, enables fuzzy search capabilities when searching for files. Set to `false` to improve performance on projects with a large number of files. | `true` |
| `context.clearContextOnIdle.toolResultsThresholdMinutes` | number | Minutes of inactivity before clearing old tool result content. Use `-1` to disable the idle trigger. | `60` |
| `context.clearContextOnIdle.toolResultsNumToKeep` | integer | Integer number of most-recent compactable tool results to preserve when clearing. Values below 1 are floored to 1. | `5` |
| `context.clearContextOnIdle.toolResultsTotalCharsThreshold` | number | Total compactable tool result output characters allowed in history before clearing oldest results. Use `-1` to disable the size trigger. This is a soft threshold: protected recent tool results may keep the total above it. | `500000` |
| Setting | Type | Description | Default |
| ----------------------------------------------------------- | -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- |
| `context.fileName` | string or array of strings | The name of the context file(s). | `undefined` |
| `context.autoCompactThreshold` | number | Target fraction of the context window at which auto-compaction triggers. Must be greater than 0 and at most 1. Default is `0.85` (85%). Acts as a ceiling on the trigger: on large windows it is the effective trigger (~85%), while on smaller windows compaction may fire earlier to leave room to summarize. Replaces the old `model.chatCompression.contextPercentageThreshold`. | `undefined` (uses internal 0.85) |
| `context.importFormat` | string | The format to use when importing memory. | `undefined` |
| `context.includeDirectories` | array | Additional directories to include in the workspace context. Specifies an array of additional absolute or relative paths to include in the workspace context. Missing directories will be skipped with a warning by default. Paths can use `~` to refer to the user's home directory. This setting can be combined with the `--include-directories` command-line flag. | `[]` |
| `context.loadFromIncludeDirectories` | boolean | Controls the behavior of the `/memory refresh` command. If set to `true`, `QWEN.md` files should be loaded from all directories that are added. If set to `false`, `QWEN.md` should only be loaded from the current directory. | `false` |
| `context.fileFiltering.respectGitIgnore` | boolean | Respect .gitignore files when searching. | `true` |
| `context.fileFiltering.respectQwenIgnore` | boolean | Respect .qwenignore and configured custom ignore files when searching. | `true` |
| `context.fileFiltering.customIgnoreFiles` | array | Project-root-relative ignore files to use instead of the default compatibility files (`.agentignore`, `.aiignore`) when `respectQwenIgnore` is enabled. `.qwenignore` is always included. | `[".agentignore", ".aiignore"]` |
| `context.fileFiltering.enableRecursiveFileSearch` | boolean | Whether to enable searching recursively for filenames under the current tree when completing `@` prefixes in the prompt. | `true` |
| `context.fileFiltering.enableFuzzySearch` | boolean | When `true`, enables fuzzy search capabilities when searching for files. Set to `false` to improve performance on projects with a large number of files. | `true` |
| `context.clearContextOnIdle.toolResultsThresholdMinutes` | number | Minutes of inactivity before clearing old tool result content. Use `-1` to disable the idle trigger. | `60` |
| `context.clearContextOnIdle.toolResultsNumToKeep` | integer | Integer number of most-recent compactable tool results to preserve when clearing. Values below 1 are floored to 1. | `5` |
| `context.clearContextOnIdle.toolResultsTotalCharsThreshold` | number | Total compactable tool result output characters allowed in history before clearing oldest results. When exceeded, oldest results are cleared down to half this threshold (best effort) so later turns keep reusing the provider prompt cache instead of rewriting history every turn. Use `-1` to disable the size trigger. This is a soft threshold: protected recent tool results may keep the total above it. | `500000` |

#### Troubleshooting File Search Performance

Expand Down
2 changes: 1 addition & 1 deletion packages/cli/src/config/settingsSchema.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1879,7 +1879,7 @@ const SETTINGS_SCHEMA = {
requiresRestart: false,
default: DEFAULT_TOOL_RESULTS_TOTAL_CHARS_THRESHOLD as number,
description:
'Total compactable tool result output characters allowed in history before clearing oldest results. Use -1 to disable. This is a soft threshold: protected recent tool results may keep the total above it.',
'Total compactable tool result output characters allowed in history before clearing oldest results. When exceeded, oldest results are cleared down to half this threshold (best effort) to preserve the provider prompt cache on later turns. Use -1 to disable. This is a soft threshold: protected recent tool results may keep the total above it.',
showInDialog: false,
},
},
Expand Down
67 changes: 63 additions & 4 deletions packages/core/src/core/client.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3509,7 +3509,7 @@ describe('Gemini Client (client.ts)', () => {
functionResponse: {
id: 'pending-shell',
name: 'run_shell_command',
response: { output: 'Y'.repeat(50_000) },
response: { output: 'Y'.repeat(140_000) },
},
},
],
Expand All @@ -3527,14 +3527,70 @@ describe('Gemini Client (client.ts)', () => {
compacted[1]!.parts![0]!.functionResponse!.response!['output'],
).toBe('[Old tool result content cleared]');
expect(clear).not.toHaveBeenCalled();
expect(markReadEvictedFromHistory).toHaveBeenCalledTimes(1);
// Three reads are blanked while clearing down to the 250K watermark.
expect(markReadEvictedFromHistory).toHaveBeenCalledTimes(3);
expect(mockClientDebugLogger.info).toHaveBeenCalledWith(
expect.stringContaining(
'[TOOL-RESULT MC] tool result chars 620000 > 500000',
),
);
expect(mockClientDebugLogger.info).toHaveBeenCalledWith(
expect.stringContaining(
'[TOOL-RESULT MC] tool result chars 530000 > 500000',
'history now 120000 (+140000 pending), target 250000 (soft-exceeded)',
),
);
Comment thread
doudouOUC marked this conversation as resolved.
});

it('omits the soft-exceeded marker when clearing lands exactly on the watermark', async () => {
// Pins the marker's absence at the boundary: virtual total after
// clearing == watermark must NOT be flagged (kills the `>=` and
// always-true mutants of the marker condition).
const { clear, markReadEvictedFromHistory } = mockFileReadCacheStub();
const { history } = await makeReadFileResponses(3, 150_000);
const setHistory = vi.fn();
client['chat'] = {
addHistory: vi.fn(),
getHistory: vi.fn().mockReturnValue(history),
setHistory,
} as unknown as GeminiChat;
vi.mocked(mockConfig.getClearContextOnIdle).mockReturnValue({
toolResultsThresholdMinutes: 60,
toolResultsNumToKeep: 1,
toolResultsTotalCharsThreshold: 500_000,
});
client['lastApiCompletionTimestamp'] = Date.now();
mockClientDebugLogger.info.mockClear();

const stream = client.sendMessageStream(
[
{
functionResponse: {
id: 'pending-shell-exact',
name: 'run_shell_command',
response: { output: 'Y'.repeat(100_000) },
},
},
],
new AbortController().signal,
'prompt-toolresult-watermark-boundary',
{ type: SendMessageType.ToolResult },
);
for await (const _ of stream) {
/* drain */
}

// 550K total → clear two 150K reads → 150K committed + 100K pending
// sits exactly on the 250K watermark.
expect(setHistory).toHaveBeenCalled();
expect(clear).not.toHaveBeenCalled();
expect(markReadEvictedFromHistory).toHaveBeenCalledTimes(2);
expect(mockClientDebugLogger.info).toHaveBeenCalledWith(
expect.stringContaining('history now 360000 (+50000 pending)'),
expect.stringContaining(
'history now 150000 (+100000 pending), target 250000',
),
);
expect(mockClientDebugLogger.info).not.toHaveBeenCalledWith(
expect.stringContaining('(soft-exceeded)'),
);
});

Expand Down Expand Up @@ -3576,6 +3632,9 @@ describe('Gemini Client (client.ts)', () => {
expect(mockClientDebugLogger.info).toHaveBeenCalledWith(
expect.stringContaining('cleared 0 tool result(s)'),
);
expect(mockClientDebugLogger.info).toHaveBeenCalledWith(
expect.stringContaining('target 250000 (soft-exceeded)'),
);
expect(mockClientDebugLogger.info).toHaveBeenCalledWith(
expect.stringContaining('history now 800000'),
);
Expand Down
Loading
Loading