refactor(core): shrink the content generator interface

.github/workflows/.size-baseline

@@ -16,7 +16,7 @@
 3480 audio-capture-prebuilds.yml
 9023 auto-minimize-spam.yml
 4638 build-and-publish-image.yml
-29715 cd-cua-driver.yml
+42519 cd-cua-driver.yml
 2076 cd-mobile-mcp.yml
 69782 ci.yml
 1482 codeql.yml

docs/design/2026-07-22-lazy-google-genai-loading.md

@@ -22,14 +22,21 @@ Provider implementations continue to use the official SDK classes. In particular
 
 `createContentGenerator()` still validates configuration, preloads the runtime fetch implementation, and performs Qwen OAuth credential acquisition at its current point in the session lifecycle. It returns a private lazy `ContentGenerator` whose memoized loader constructs the selected provider and wraps it in `LoggingContentGenerator` on the first asynchronous content-generator operation.
 
-All four asynchronous operations share the same loader promise:
+All three asynchronous operations share the same loader promise:
 
 - `generateContent`
 - `generateContentStream`
-- `countTokens`
 - `embedContent`
 
-Concurrent first calls therefore import and construct the provider once. `useSummarizedThinking()` remains synchronous and is supplied from the selected provider's known behavior: true for Gemini/Vertex and false for OpenAI, Qwen OAuth, and Anthropic.
+Concurrent first calls therefore import and construct the provider once.
+
+> **Update (2026-08, PR #9676)**: `countTokens` and `useSummarizedThinking`
+> were removed from the `ContentGenerator` interface — no production caller
+> used either, and the removal narrows every provider and test double to the
+> operations something actually calls. This doc originally listed four shared
+> asynchronous operations (including `countTokens`) and a synchronous
+> `useSummarizedThinking()` supplied from each provider's known behavior; both
+> are gone from the interface, the lazy wrapper, and the four providers.
 
 Qwen OAuth credential acquisition remains eager within `createContentGenerator()`. An expired or missing cached credential therefore continues to reject ACP session creation rather than producing an apparently usable session that fails only on its first prompt.
 
@@ -49,7 +56,7 @@ The serve fast-path metafile guard adds `@google/genai` to the ACP forbidden-pac
 
 There are three direct production creation paths. `Config.refreshAuth()` owns the main-session generator. `BaseLlmClient` owns cached per-model generators for routed side requests. `createRuntimeContentGeneratorView()` owns dedicated generators used by the in-process agent backend, subagent manager, and forked agents. Each path stores and consumes only the `ContentGenerator` interface, so the private lazy wrapper preserves its ownership and routing boundary.
 
-The interface consumers call only `generateContent`, `generateContentStream`, `countTokens`, `embedContent`, and `useSummarizedThinking`. The main chat path, prompt hooks, memory/goal/side queries, vision routing, subagents, and session resume do not inspect the concrete provider or unwrap `LoggingContentGenerator`; a repository-wide search found no production `instanceof` or `getWrapped()` caller. MCP tool discovery is separate from generator ownership and keeps the SDK-provided `mcpToTool` adapter behind its own first-use import.
+The interface consumers call only `generateContent`, `generateContentStream`, and `embedContent`. The main chat path, prompt hooks, memory/goal/side queries, vision routing, subagents, and session resume do not inspect the concrete provider or unwrap `LoggingContentGenerator`; a repository-wide search found no production `instanceof` or `getWrapped()` caller. MCP tool discovery is separate from generator ownership and keeps the SDK-provided `mcpToTool` adapter behind its own first-use import.
 
 ## Alternatives rejected
 
@@ -71,7 +78,7 @@ The interface consumers call only `generateContent`, `generateContentStream`, `c
 
 ## Verification
 
-Unit tests cover helper parity, deferred construction, Qwen credential timing, single-flight behavior, provider-specific summarized-thinking values, deferred module failures, and MCP discovery behavior. The bundled metafile must show `@google/genai` absent from the ACP static closure while retaining it in dynamic provider/MCP chunks.
+Unit tests cover helper parity, deferred construction, Qwen credential timing, single-flight behavior, deferred module failures, and MCP discovery behavior. The bundled metafile must show `@google/genai` absent from the ACP static closure while retaining it in dynamic provider/MCP chunks.
 
 The 2C4G acceptance run follows #7264: 30 paired serial cold starts, `channel.initialize` P50/P95, process-to-first-session, preheated/warm behavior, concurrent first sessions, telemetry on/off, and peak RSS. Because this change moves work later, it additionally records session-response-to-first-token and process-to-first-token for an immediate first prompt. A startup win that is fully repaid as a first-token regression is reported rather than treated as a successful optimization.
 

docs/design/web-shell/web-shell-image-drag-and-drop.md

@@ -502,9 +502,14 @@ oversized placeholder。它们验证既有服务端契约；Web Shell helper 测
 候选顺序和 encoded-data 剩余预算。
 
 BMP 以 `image/bmp` 进入缩略图 data URL 和 daemon image block。Core 的
-`SUPPORTED_IMAGE_MIME_TYPES` 明确包含 `image/bmp`，`ImageTokenizer` 解析 BMP 尺寸，
-OpenAI converter 把启用 image modality 的 `inlineData` 原样构造成
-`data:image/bmp;base64,...`；Gemini 路径保留相同 `inlineData`。因此 V1 不在浏览器转码。
+`SUPPORTED_IMAGE_MIME_TYPES` 明确包含 `image/bmp`，OpenAI converter 把启用 image modality 的
+`inlineData` 原样构造成 `data:image/bmp;base64,...`；Gemini 路径保留相同 `inlineData`。
+因此 V1 不在浏览器转码。
+
+> **2026-08-24 同步注记（PR #9676）**：request-tokenizer 估计器簇（含 `ImageTokenizer`
+> 及其 BMP 尺寸解析）已作为孤儿代码删除。BMP 支持现在仅依赖 `SUPPORTED_IMAGE_MIME_TYPES`
+> 接受清单与 converter 透传；token 计数使用 `compactionInputSlimming.ts` 中的固定
+> `DEFAULT_IMAGE_TOKEN_ESTIMATE`。下文对 BMP 路径的 E2E/人工验收要求不变。
 浏览器若不能解码缩略图，不影响附件数据传输，但 E2E 必须覆盖 Chromium 解码，
 Firefox/Linux 必须完成人工验收。
 
@@ -634,8 +639,8 @@ Web Shell 输入层静默改变格式。
 
 BMP 的下游回归不只停在 mock HTTP 入参：在既有 ACP session prompt 转换测试中加入
 `image/bmp`，验证最终 Core canonical content 保持
-`inlineData.mimeType === 'image/bmp'`；OpenAI/Gemini converter/tokenizer 聚焦测试验证
-各自既有图片路径，Anthropic 聚焦测试明确断言 BMP 转为 unsupported-media 文本。daemon
+`inlineData.mimeType === 'image/bmp'`；OpenAI/Gemini converter 聚焦测试验证
+各自既有图片路径（tokenizer 估计器簇已随 PR #9676 删除，见上文同步注记），Anthropic 聚焦测试明确断言 BMP 转为 unsupported-media 文本。daemon
 已有结构化 `413` 测试，Core 已有 inline-media within/over limit 测试；本功能不复制
 production 限制，只确认 Web Shell 对这些既有失败语义的状态保留。
 

packages/cli/src/commands/review/lib/assets.ts

@@ -70,9 +70,8 @@ export const ASSET_HEADER_BYTES = 16;
  *
  * A sibling signature table lives in core: `sniffFileKind` in
  * `packages/core/src/utils/binary-content.ts` (best-effort kind detection for
- * fetched web content, deliberately looser) and the dimension extractors in
- * `packages/core/src/utils/request-tokenizer/imageTokenizer.ts`. Admitting or
- * correcting a format here means checking those sites too.
+ * fetched web content, deliberately looser). Admitting or correcting a format
+ * here means checking that site too.
  */
 export function sniffImageFormat(header: Uint8Array): ImageFormat | null {
   const at = (i: number): number => header[i] ?? -1;

packages/cli/src/ui/AppContainer.test.tsx

@@ -5080,9 +5080,9 @@ describe('AppContainer State Management', () => {
         lastCompletedUuid: 'a1',
       };
 
-      vi.spyOn(mockConfig, 'getContentGenerator').mockReturnValue({
-        useSummarizedThinking: vi.fn(() => false),
-      } as unknown as ReturnType<typeof mockConfig.getContentGenerator>);
+      vi.spyOn(mockConfig, 'getContentGenerator').mockReturnValue(
+        {} as unknown as ReturnType<typeof mockConfig.getContentGenerator>,
+      );
       vi.spyOn(mockConfig, 'initialize').mockResolvedValue(undefined);
       vi.spyOn(mockConfig, 'getResumedSessionData').mockReturnValue(
         resumeSessionData as ReturnType<

packages/cli/src/ui/components/StandaloneSessionPicker.test.tsx

@@ -701,7 +701,6 @@ describe('SessionPicker', () => {
       getIdeMode: () => false,
       isTrustedFolder: () => false,
       getToolRegistry: () => ({ getTool: () => undefined }),
-      getContentGenerator: () => ({ useSummarizedThinking: () => false }),
     } as unknown as Config;
     const PREVIEW_SETTINGS_STUB = {
       merged: { ui: {} },

packages/cli/src/ui/components/StandaloneSessionPicker.tsx

@@ -37,7 +37,6 @@ const PREVIEW_CONFIG_STUB = {
   getIdeMode: () => false,
   isTrustedFolder: () => false,
   getToolRegistry: () => ({ getTool: () => undefined }),
-  getContentGenerator: () => ({ useSummarizedThinking: () => false }),
 } as unknown as Config;
 
 const PREVIEW_SETTINGS_STUB = {

packages/core/src/agents/runtime/agent-headless.test.ts

@@ -72,9 +72,7 @@ vi.mock('../../core/contentGenerator.js', async (importOriginal) => {
     createContentGenerator: vi.fn().mockResolvedValue({
       generateContent: vi.fn(),
       generateContentStream: vi.fn(),
-      countTokens: vi.fn().mockResolvedValue({ totalTokens: 100 }),
       embedContent: vi.fn(),
-      useSummarizedThinking: vi.fn().mockReturnValue(false),
     }),
     createContentGeneratorConfig: vi.fn().mockReturnValue({
       model: DEFAULT_QWEN_MODEL,

packages/core/src/config/config.test.ts

@@ -5181,7 +5181,6 @@ describe('Server Config (config.ts)', () => {
       vi.mocked(createContentGenerator).mockResolvedValue({
         generateContent: vi.fn(),
         generateContentStream: vi.fn(),
-        countTokens: vi.fn(),
         embedContent: vi.fn(),
       } as unknown as ContentGenerator);
 
@@ -5232,7 +5231,6 @@ describe('Server Config (config.ts)', () => {
       vi.mocked(createContentGenerator).mockResolvedValue({
         generateContent: vi.fn(),
         generateContentStream: vi.fn(),
-        countTokens: vi.fn(),
         embedContent: vi.fn(),
       } as unknown as ContentGenerator);
 
@@ -5266,7 +5264,6 @@ describe('Server Config (config.ts)', () => {
       vi.mocked(createContentGenerator).mockResolvedValue({
         generateContent: vi.fn(),
         generateContentStream: vi.fn(),
-        countTokens: vi.fn(),
         embedContent: vi.fn(),
       } as unknown as ContentGenerator);
 
@@ -5823,7 +5820,6 @@ describe('Server Config (config.ts)', () => {
       vi.mocked(createContentGenerator).mockResolvedValue({
         generateContent: vi.fn(),
         generateContentStream: vi.fn(),
-        countTokens: vi.fn(),
         embedContent: vi.fn(),
       } as unknown as ContentGenerator);
 

packages/core/src/core/__tests__/openaiTimeoutHandling.test.ts

@@ -363,24 +363,5 @@ describe('OpenAIContentGenerator Timeout Handling', () => {
         generator.generateContent(request, 'test-prompt-id'),
       ).rejects.toThrow(/Request timeout after \d+s/);
     });
-
-    it('should fall back to character-based estimation if countTokens fails', async () => {
-      const timeoutError = new Error('Request timeout');
-      mockOpenAIClient.chat.completions.create.mockRejectedValue(timeoutError);
-
-      // Mock countTokens to throw error
-      const mockCountTokens = vi.spyOn(generator, 'countTokens');
-      mockCountTokens.mockRejectedValue(new Error('Count tokens failed'));
-
-      const request = {
-        contents: [{ role: 'user' as const, parts: [{ text: 'Hello world' }] }],
-        model: 'gpt-4',
-      };
-
-      // Should not throw due to token counting failure
-      await expect(
-        generator.generateContent(request, 'test-prompt-id'),
-      ).rejects.toThrow(/Request timeout after \d+s/);
-    });
   });
 });

packages/core/src/core/anthropicContentGenerator/anthropicContentGenerator.test.ts

@@ -6,30 +6,19 @@
 
 import { getEventListeners } from 'node:events';
 import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
-import type {
-  CountTokensParameters,
-  GenerateContentParameters,
-} from '@google/genai';
+import type { GenerateContentParameters } from '@google/genai';
 import { FinishReason, GenerateContentResponse } from '@google/genai';
 import type { ContentGeneratorConfig } from '../contentGenerator.js';
 import {
   DEFAULT_TIMEOUT,
   DISABLED_REQUEST_TIMEOUT_MS,
 } from '../openaiContentGenerator/constants.js';
 
-// Mock the request tokenizer module BEFORE importing the class that uses it.
-const mockTokenizer = {
-  calculateTokens: vi.fn(),
-  dispose: vi.fn(),
-};
 const mockReportAnthropicRequest = vi.hoisted(() => vi.fn());
 const mockReportAnthropicFollowingRequest = vi.hoisted(() => vi.fn());
 const mockReportAnthropicResponse = vi.hoisted(() => vi.fn());
 const mockReportAnthropicEvent = vi.hoisted(() => vi.fn());
 
-vi.mock('../../utils/request-tokenizer/index.js', () => ({
-  RequestTokenEstimator: vi.fn(() => mockTokenizer),
-}));
 vi.mock('../../telemetry/gen-ai-request.js', () => ({
   reportAnthropicRequest: mockReportAnthropicRequest,
   reportAnthropicFollowingRequest: mockReportAnthropicFollowingRequest,
@@ -107,16 +96,6 @@ describe('AnthropicContentGenerator', () => {
     savedMaxOutputTokensEnv = process.env[MAX_OUTPUT_TOKENS_ENV];
     delete process.env[MAX_OUTPUT_TOKENS_ENV];
 
-    mockTokenizer.calculateTokens.mockResolvedValue({
-      totalTokens: 50,
-      breakdown: {
-        textTokens: 50,
-        imageTokens: 0,
-        audioTokens: 0,
-        otherTokens: 0,
-      },
-      processingTime: 1,
-    });
     anthropicState = anthropicMockState;
 
     anthropicState.createImpl.mockReset();
@@ -3305,58 +3284,6 @@ describe('AnthropicContentGenerator', () => {
     });
   });
 
-  describe('countTokens', () => {
-    it('counts tokens using the request tokenizer', async () => {
-      const { AnthropicContentGenerator } = await importGenerator();
-      const generator = new AnthropicContentGenerator(
-        {
-          model: 'claude-test',
-          apiKey: 'test-key',
-          timeout: 10_000,
-          maxRetries: 2,
-          samplingParams: {},
-          schemaCompliance: 'auto',
-        },
-        mockConfig,
-      );
-
-      const request: CountTokensParameters = {
-        contents: [{ role: 'user', parts: [{ text: 'Hello world' }] }],
-        model: 'claude-test',
-      };
-
-      const result = await generator.countTokens(request);
-      expect(mockTokenizer.calculateTokens).toHaveBeenCalledWith(request);
-      expect(result.totalTokens).toBe(50);
-    });
-
-    it('falls back to character approximation when tokenizer throws', async () => {
-      const { AnthropicContentGenerator } = await importGenerator();
-      mockTokenizer.calculateTokens.mockRejectedValueOnce(new Error('boom'));
-      const generator = new AnthropicContentGenerator(
-        {
-          model: 'claude-test',
-          apiKey: 'test-key',
-          timeout: 10_000,
-          maxRetries: 2,
-          samplingParams: {},
-          schemaCompliance: 'auto',
-        },
-        mockConfig,
-      );
-
-      const request: CountTokensParameters = {
-        contents: [{ role: 'user', parts: [{ text: 'Hello' }] }],
-        model: 'claude-test',
-      };
-
-      const content = JSON.stringify(request.contents);
-      const expected = Math.ceil(content.length / 4);
-      const result = await generator.countTokens(request);
-      expect(result.totalTokens).toBe(expected);
-    });
-  });
-
   describe('generateContentStream', () => {
     it('emits tool preparation metadata before the complete function call', async () => {
       const { AnthropicContentGenerator } = await importGenerator();

packages/core/src/core/anthropicContentGenerator/anthropicContentGenerator.ts

@@ -6,8 +6,6 @@
 
 import Anthropic from '@anthropic-ai/sdk';
 import type {
-  CountTokensParameters,
-  CountTokensResponse,
   EmbedContentParameters,
   EmbedContentResponse,
   GenerateContentParameters,
@@ -29,7 +27,6 @@ type MessageCreateParamsNonStreaming =
   Anthropic.MessageCreateParamsNonStreaming;
 type MessageCreateParamsStreaming = Anthropic.MessageCreateParamsStreaming;
 type RawMessageStreamEvent = Anthropic.RawMessageStreamEvent;
-import { RequestTokenEstimator } from '../../utils/request-tokenizer/index.js';
 import { safeJsonParse } from '../../utils/safeJsonParse.js';
 import { AnthropicContentConverter } from './converter.js';
 import { buildAnthropicUsageMetadata } from './usage.js';
@@ -445,41 +442,11 @@ export class AnthropicContentGenerator implements ContentGenerator {
     return drainThenCleanup();
   }
 
-  async countTokens(
-    request: CountTokensParameters,
-  ): Promise<CountTokensResponse> {
-    try {
-      const estimator = new RequestTokenEstimator();
-      const result = await estimator.calculateTokens(request);
-
-      return {
-        totalTokens: result.totalTokens,
-      };
-    } catch (error) {
-      debugLogger.warn(
-        'Failed to calculate tokens with tokenizer, ' +
-          'falling back to simple method:',
-        error,
-      );
-
-      const content = JSON.stringify(request.contents);
-      const totalTokens = Math.ceil(content.length / 4);
-      return {
-        totalTokens,
-      };
-    }
-  }
-
   async embedContent(
     _request: EmbedContentParameters,
   ): Promise<EmbedContentResponse> {
     throw new Error('Anthropic does not support embeddings.');
   }
-
-  useSummarizedThinking(): boolean {
-    return false;
-  }
-
   private buildHeaders(useProxyIdentity: boolean): Record<string, string> {
     // Beta headers are computed per-request in buildPerRequestHeaders so they
     // stay in sync with what the request body actually carries — see #3788

packages/core/src/core/client.test.ts

@@ -424,26 +424,6 @@ vi.mock(
 );
 import { microcompactHistory } from '../services/microcompaction/microcompact.js';
 
-// Mock RequestTokenizer to use simple character-based estimation
-vi.mock('../utils/request-tokenizer/requestTokenizer.js', () => ({
-  RequestTokenizer: class {
-    async calculateTokens(request: { contents: unknown }) {
-      // Simple estimation: count characters in JSON and divide by 4
-      const totalChars = JSON.stringify(request.contents).length;
-      return {
-        totalTokens: Math.floor(totalChars / 4),
-        breakdown: {
-          textTokens: Math.floor(totalChars / 4),
-          imageTokens: 0,
-          audioTokens: 0,
-          otherTokens: 0,
-        },
-        processingTime: 0,
-      };
-    }
-  },
-}));
-
 /**
  * Array.fromAsync ponyfill, which will be available in es 2024.
  *
@@ -562,7 +542,6 @@ describe('Gemini Client (client.ts)', () => {
       generateContent: mockGenerateContentFn,
       generateContentStream: vi.fn(),
       batchEmbedContents: vi.fn(),
-      countTokens: vi.fn().mockResolvedValue({ totalTokens: 100 }),
     } as unknown as ContentGenerator;
 
     // Because the GeminiClient constructor kicks off an async process (startChat)