Skip to content

fix(core): support per-provider stream idle timeout #9795

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
wenshao merged 1 commit into from
Aug 24, 2026
Merged

fix(core): support per-provider stream idle timeout #9795

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions docs/users/configuration/model-providers.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -317,6 +317,7 @@ Most local inference servers (vLLM, Ollama, LM Studio, etc.) provide an OpenAI-c
"baseUrl": "http://localhost:11434/v1",
"generationConfig": {
"timeout": 300000,
"streamIdleTimeoutMs": 600000,
"maxRetries": 1,
"contextWindowSize": 32768,
"samplingParams": {
Expand Down Expand Up @@ -358,6 +359,13 @@ Most local inference servers (vLLM, Ollama, LM Studio, etc.) provide an OpenAI-c
}
```

For queued or slow local OpenAI-compatible servers, `streamIdleTimeoutMs`
controls how long this model may stay silent between streamed chunks. It
overrides the global `QWEN_STREAM_IDLE_TIMEOUT_MS` value for the selected
provider entry; set it to `0` to disable the idle guard. The separate 15-minute
stream lifetime cap still applies unless `QWEN_STREAM_MAX_LIFETIME_MS` is raised
or disabled.

For local servers that don't require authentication, you can use any placeholder value for the API key:

```bash
Expand Down
5 changes: 3 additions & 2 deletions docs/users/configuration/settings.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -202,6 +202,7 @@ These settings are read from operator scopes only (User, System, and SystemDefau
"model": {
"generationConfig": {
"timeout": 60000,
"streamIdleTimeoutMs": 300000,
"contextWindowSize": 128000,
"modalities": {
"image": true
Expand Down Expand Up @@ -232,10 +233,10 @@ These settings are read from operator scopes only (User, System, and SystemDefau

Two guards bound a streaming response, each accepting `0` to disable. Neither is implemented by the Anthropic/Gemini generators, which leave the drip-fed shape below unbounded.

- `QWEN_STREAM_IDLE_TIMEOUT_MS` (default `240000`) bounds inactivity _between_ streamed chunks: a stream that goes silent for this long is aborted as a retryable `ETIMEDOUT`.
- `streamIdleTimeoutMs` (default `240000`) bounds inactivity _between_ streamed chunks: a stream that goes silent for this long is aborted as a retryable `ETIMEDOUT`. For provider-backed models, set it under the matching `modelProviders[providerId][].generationConfig`; for runtime models, use `model.generationConfig`. An explicit model value takes precedence over `QWEN_STREAM_IDLE_TIMEOUT_MS`, and `0` disables the idle guard.
- `QWEN_STREAM_MAX_LIFETIME_MS` (default `900000`) caps the _total_ upstream-wait time of one streaming response regardless of chunk flow — the bound a drip-fed stream that never completes cannot reset.

These are **environment variables (or, for embedders, `ContentGeneratorConfig.streamIdleTimeoutMs` / `streamMaxLifetimeMs`) only — there is no settings.json key**; writing `"streamMaxLifetimeMs"` into settings.json has no effect. Upgrade notes: a deployment that previously set `QWEN_STREAM_IDLE_TIMEOUT_MS=0` — or passed `streamIdleTimeoutMs: 0` in `ContentGeneratorConfig` — to opt out of stream aborts now also needs `QWEN_STREAM_MAX_LIFETIME_MS=0` (or `streamMaxLifetimeMs: 0`) to keep that; and the 15-minute lifetime cap bounds even a stream whose idle timeout you raised above it (e.g. `QWEN_STREAM_IDLE_TIMEOUT_MS=1800000`) — raise the cap likewise, or set it to `0`, if you rely on a longer window.
`streamMaxLifetimeMs` remains available only through `QWEN_STREAM_MAX_LIFETIME_MS` or, for embedders, `ContentGeneratorConfig.streamMaxLifetimeMs`; writing it into `settings.json` has no effect. The 15-minute lifetime cap still bounds a stream whose idle timeout you raise above it. Raise the lifetime environment variable likewise, or set it to `0`, if you rely on a longer window. Disabling `streamIdleTimeoutMs` alone does not disable this lifetime cap.

**max_tokens (output token limit):**

Expand Down
15 changes: 15 additions & 0 deletions packages/cli/src/config/settingsSchema.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -278,6 +278,21 @@ describe('SettingsSchema', () => {
expect(model.maxToolCallsPerTurn.type).toBe('integer');
});

it('should define streamIdleTimeoutMs as a bounded generation setting', () => {
const streamIdleTimeout =
getSettingsSchema().model.properties.generationConfig.properties
?.streamIdleTimeoutMs;

expect(streamIdleTimeout).toMatchObject({
type: 'integer',
default: undefined,
minimum: 0,
maximum: 2_147_483_647,
requiresRestart: false,
showInDialog: false,
});
});

it('should define stopHookBlockingCap schema override as a positive integer', () => {
expect(
getSettingsSchema().stopHookBlockingCap.jsonSchemaOverride,
Expand Down
13 changes: 13 additions & 0 deletions packages/cli/src/config/settingsSchema.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1707,6 +1707,19 @@ const SETTINGS_SCHEMA = {
parentKey: 'generationConfig',
showInDialog: false,
},
streamIdleTimeoutMs: {
type: 'integer',
label: 'Stream Idle Timeout',
Comment on lines +1710 to +1712

@qwen-code-ci-bot qwen-code-ci-bot Aug 23, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] R1-1: The settings knob this PR adds takes precedence over QWEN_STREAM_IDLE_TIMEOUT_MS (per resolveStreamGuardMs in packages/core/src/core/openaiContentGenerator/pipeline.ts and the docs added here), but the StreamInactivityTimeoutError message in that same file (lines 192-196) still tells users to set the env var. That advice is silently ineffective whenever the timeout came from settings/modelProviders: an operator who sets model.generationConfig.streamIdleTimeoutMs: 300000 (the exact configuration the new docs recommend) and then hits a long SSE silence gets "Set QWEN_STREAM_IDLE_TIMEOUT_MS to increase this window (or 0 to disable it)" on every retry attempt; exporting that env var (even to 0) changes nothing because the explicit config field wins, and nothing hints that a settings value is overriding it. In a daemon deployment this turns a one-step fix into an open-ended debugging session.

Evidence: at the reviewed commit, npx vitest run src/core/openaiContentGenerator/pipeline.test.ts -t "stream is silent past the idle timeout|explicit streamIdleTimeoutMs config take precedence" in packages/core → Tests 2 passed | 158 skipped (160) — one test asserts the env-only advice string fires from an explicit-config timeout, and the other proves config beats env (env stubbed 1000 vs config 5000: the guard does not trip at 1000 ms and trips at 5000 ms).

Fix location: packages/core/src/core/openaiContentGenerator/pipeline.ts (~line 192) — name the settings knobs in the message, or make it source-aware (the pipeline resolves config.streamIdleTimeoutMs in its constructor and knows whether it was explicit):

super(
  `No stream activity for ${idleMs}ms after ${chunksReceived} chunks ` +
    `(stream lifetime: ${streamLifetimeMs}ms). Set ` +
    `model.generationConfig.streamIdleTimeoutMs (or the modelProviders ` +
    `entry's generationConfig.streamIdleTimeoutMs) — or ` +
    `${QWEN_STREAM_IDLE_TIMEOUT_MS_ENV} when no settings value is set — ` +
    `to increase this window (or 0 to disable it).`,
);
中文说明

本 PR 新增的设置项优先级高于 QWEN_STREAM_IDLE_TIMEOUT_MS（见 packages/core/src/core/openaiContentGenerator/pipeline.ts 中的 resolveStreamGuardMs 以及本 PR 新增的文档），但同一文件中的 StreamInactivityTimeoutError 错误信息（第 192-196 行）仍然只提示用户设置环境变量。当超时值来自 settings/modelProviders 时，该提示会静默失效：运维人员按新文档推荐设置 model.generationConfig.streamIdleTimeoutMs: 300000 后，若遇到较长的 SSE 静默，每次重试都会看到 "Set QWEN_STREAM_IDLE_TIMEOUT_MS to increase this window (or 0 to disable it)"；此时导出该环境变量（即使设为 0）也不会有任何效果，因为显式配置优先于环境变量，而错误信息中没有任何线索表明有设置值在覆盖它。在守护进程部署场景下，这会把一步就能解决的问题变成无期限的排查。

证据：在被审提交上运行 npx vitest run src/core/openaiContentGenerator/pipeline.test.ts -t "stream is silent past the idle timeout|explicit streamIdleTimeoutMs config take precedence"（位于 packages/core）→ Tests 2 passed | 158 skipped (160) —— 其中一个测试断言仅提及环境变量的提示文案会在显式配置超时场景下出现，另一个测试证明配置优先于环境变量（env 设为 1000、配置为 5000：守卫不在 1000 ms 触发，而在 5000 ms 触发）。

修复位置：packages/core/src/core/openaiContentGenerator/pipeline.ts（约第 192 行）—— 在错误信息中同时提及设置项，或让提示感知来源（pipeline 在构造函数中解析 config.streamIdleTimeoutMs，知道该值是否为显式配置），参考上方代码块。

— qwen3.8-max via Qwen Code /review (v0.22.0)

category: 'Generation Configuration',
requiresRestart: false,
default: undefined as number | undefined,
description:
'Maximum inactivity between streamed chunks for OpenAI-compatible models, in milliseconds. Set to 0 to disable the idle guard. For provider-backed models, configure this field in the selected modelProviders entry.',
minimum: 0,
maximum: 2_147_483_647,
parentKey: 'generationConfig',
showInDialog: false,
},
maxRetries: {
type: 'number',
label: 'Max Retries',
Expand Down
1 change: 1 addition & 0 deletions packages/core/src/models/constants.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ type ContentGeneratorConfig =
export const MODEL_GENERATION_CONFIG_FIELDS = [
'samplingParams',
'timeout',
'streamIdleTimeoutMs',
'maxRetries',
'retryInitialDelayMs',
'retryMaxDelayMs',
Expand Down
5 changes: 5 additions & 0 deletions packages/core/src/models/content-generator-config.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ describe('buildAgentContentGeneratorConfig', () => {
samplingParams: { temperature: 0.7, top_p: 0.9 },
reasoning: { effort: 'high' as const },
timeout: 30000,
streamIdleTimeoutMs: 300000,
maxRetries: 3,
contextWindowSize: 128000,
extra_body: { custom: 'value' },
Expand All @@ -68,6 +69,7 @@ describe('buildAgentContentGeneratorConfig', () => {
expect(result.samplingParams).toEqual({ temperature: 0.7, top_p: 0.9 });
expect(result.reasoning).toEqual({ effort: 'high' });
expect(result.timeout).toBe(30000);
expect(result.streamIdleTimeoutMs).toBe(300000);
expect(result.maxRetries).toBe(3);
expect(result.contextWindowSize).toBe(128000);
expect(result.extra_body).toEqual({ custom: 'value' });
Expand Down Expand Up @@ -101,6 +103,7 @@ describe('buildAgentContentGeneratorConfig', () => {
expect(result.samplingParams).toBeUndefined();
expect(result.reasoning).toBeUndefined();
expect(result.timeout).toBeUndefined();
expect(result.streamIdleTimeoutMs).toBeUndefined();
expect(result.maxRetries).toBeUndefined();
expect(result.contextWindowSize).toBeUndefined();
expect(result.extra_body).toBeUndefined();
Expand Down Expand Up @@ -152,6 +155,7 @@ describe('buildAgentContentGeneratorConfig', () => {
envKey: 'REGISTRY_API_KEY',
generationConfig: {
samplingParams: { temperature: 0.5 },
streamIdleTimeoutMs: 600000,
contextWindowSize: 200000,
Comment on lines 157 to 159

@qwen-code-ci-bot qwen-code-ci-bot Aug 23, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] R1-2: The registry-overlay path in applyResolvedModelConfig (packages/core/src/models/content-generator-config.ts, guarded by registryValue !== undefined) never receives a falsy registry value in any test. The documented streamIdleTimeoutMs: 0 disable case is pinned only in modelConfigResolver.test.ts, which exercises a different function (resolveGenerationConfig, guarded by field in). The code is correct today, but the one-line mutation registryValue !== undefined → registryValue survives the entire relevant test population green (measured: 153/153 pass under the mutation), which would silently drop a provider's 0 so agent content generators fall back to the env var / 240 s default and abort streams the operator explicitly told to leave unbounded.

Evidence (mutation probe in an isolated scratch tree): baseline 18/18; with the mutation, 153/153 passed across content-generator-config.test.ts + modelConfigResolver.test.ts + modelRegistry.test.ts; adding the streamIdleTimeoutMs: 0 case below fails with expected undefined to be +0 under the mutation (the probe flips) and passes against the real code.

Suggested fix — pin the falsy branch with a sibling case in the with registry-resolved model describe block:

it('should apply a falsy registry-resolved streamIdleTimeoutMs', () => {
  const config = createMockConfig(parentConfig, {
    ...resolvedModel,
    generationConfig: {
      ...resolvedModel.generationConfig,
      streamIdleTimeoutMs: 0,
    },
  });

  const result = buildAgentContentGeneratorConfig(
    config,
    'registry-model-id',
    { authType: 'anthropic' },
  );

  expect(result.streamIdleTimeoutMs).toBe(0);
});
中文说明

applyResolvedModelConfig 中的 registry 覆盖路径（packages/core/src/models/content-generator-config.ts，守卫条件为 registryValue !== undefined）没有任何测试覆盖 falsy 值。文档中记载的 streamIdleTimeoutMs: 0 禁用场景只在 modelConfigResolver.test.ts 中被固定，而该文件测试的是另一个函数（resolveGenerationConfig，守卫条件为 field in）。代码目前是正确，但单行变异 registryValue !== undefined → registryValue 可以在全部相关测试保持绿色的情况下存活（实测：变异后 153/153 通过），这会静默丢弃 provider 配置的 0，导致 agent 内容生成器回退到环境变量 / 240 秒默认值，从而中止运维人员明确要求不设上限的流。

证据（在隔离 scratch tree 中进行的变异探针）：基线 18/18；应用变异后，content-generator-config.test.ts + modelConfigResolver.test.ts + modelRegistry.test.ts 共 153/153 通过；加入下方的 streamIdleTimeoutMs: 0 用例后，在变异代码上以 expected undefined to be +0 失败（探针翻转），在真实代码上通过。

建议修复 —— 在 with registry-resolved model describe 块中新增一个同级用例来固定 falsy 分支（见上方代码块）。

— qwen3.8-max via Qwen Code /review (v0.22.0)

reasoning: { effort: 'medium' as const },
},
Expand Down Expand Up @@ -182,6 +186,7 @@ describe('buildAgentContentGeneratorConfig', () => {
expect(result.apiKeyEnvKey).toBe('REGISTRY_API_KEY');
// Registry generation config applied
expect(result.samplingParams).toEqual({ temperature: 0.5 });
expect(result.streamIdleTimeoutMs).toBe(600000);
expect(result.contextWindowSize).toBe(200000);
expect(result.reasoning).toEqual({ effort: 'medium' });
// Fields not in registry stay cleared (cross-provider)
Expand Down
9 changes: 9 additions & 0 deletions packages/core/src/models/modelConfigResolver.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -328,6 +328,7 @@ describe('modelConfigResolver', () => {
apiKey: 'key',
generationConfig: {
timeout: 60000,
streamIdleTimeoutMs: 300000,
maxRetries: 5,
samplingParams: {
temperature: 0.7,
Expand All @@ -338,12 +339,14 @@ describe('modelConfigResolver', () => {
});

expect(result.config.timeout).toBe(60000);
expect(result.config.streamIdleTimeoutMs).toBe(300000);
expect(result.config.maxRetries).toBe(5);
expect(result.config.retryInitialDelayMs).toBeUndefined();
expect(result.config.retryMaxDelayMs).toBeUndefined();
expect(result.config.samplingParams?.temperature).toBe(0.7);

expect(result.sources['timeout'].kind).toBe('settings');
expect(result.sources['streamIdleTimeoutMs'].kind).toBe('settings');
expect(result.sources['samplingParams'].kind).toBe('settings');
});

Expand All @@ -354,6 +357,7 @@ describe('modelConfigResolver', () => {
settings: {
generationConfig: {
timeout: 30000,
streamIdleTimeoutMs: 300000,
retryInitialDelayMs: 60_000,
retryMaxDelayMs: 300_000,
},
Expand All @@ -368,16 +372,21 @@ describe('modelConfigResolver', () => {
baseUrl: 'https://api.example.com',
generationConfig: {
timeout: 60000,
streamIdleTimeoutMs: 0,
retryInitialDelayMs: 3_000,
retryMaxDelayMs: 30_000,
},
},
});

expect(result.config.timeout).toBe(60000);
expect(result.config.streamIdleTimeoutMs).toBe(0);
expect(result.config.retryInitialDelayMs).toBe(3_000);
expect(result.config.retryMaxDelayMs).toBe(30_000);
expect(result.sources['timeout'].kind).toBe('modelProviders');
expect(result.sources['streamIdleTimeoutMs'].kind).toBe(
'modelProviders',
);
expect(result.sources['retryInitialDelayMs'].kind).toBe(
'modelProviders',
);
Expand Down
2 changes: 2 additions & 0 deletions packages/core/src/models/modelRegistry.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,7 @@ describe('ModelRegistry', () => {
name: 'GPT-4 Turbo',
baseUrl: 'https://api.openai.com/v1',
generationConfig: {
streamIdleTimeoutMs: 600000,
samplingParams: {
temperature: 0.8,
max_tokens: 4096,
Expand All @@ -176,6 +177,7 @@ describe('ModelRegistry', () => {

expect(model?.generationConfig.samplingParams?.temperature).toBe(0.8);
expect(model?.generationConfig.samplingParams?.max_tokens).toBe(4096);
expect(model?.generationConfig.streamIdleTimeoutMs).toBe(600000);
// No defaults are applied - only the configured values are present
expect(model?.generationConfig.samplingParams?.top_p).toBeUndefined();
expect(model?.generationConfig.timeout).toBeUndefined();
Expand Down
1 change: 1 addition & 0 deletions packages/core/src/models/types.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ export type ModelGenerationConfig = Pick<
ContentGeneratorConfig,
| 'samplingParams'
| 'timeout'
| 'streamIdleTimeoutMs'
| 'maxRetries'
| 'retryInitialDelayMs'
| 'retryMaxDelayMs'
Expand Down
6 changes: 6 additions & 0 deletions packages/vscode-ide-companion/schemas/settings.schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -763,6 +763,12 @@
"description": "Request timeout in milliseconds.",
"type": "number"
},
"streamIdleTimeoutMs": {
"description": "Maximum inactivity between streamed chunks for OpenAI-compatible models, in milliseconds. Set to 0 to disable the idle guard. For provider-backed models, configure this field in the selected modelProviders entry.",
"type": "integer",
"minimum": 0,
"maximum": 2147483647
},
"maxRetries": {
"description": "Maximum number of retries for failed requests.",
"type": "number"
Expand Down
Loading