diff --git a/.github/workflows/.size-baseline b/.github/workflows/.size-baseline index 9bfcc18fb48..c5979b7cd32 100644 --- a/.github/workflows/.size-baseline +++ b/.github/workflows/.size-baseline @@ -16,7 +16,7 @@ 3480 audio-capture-prebuilds.yml 9023 auto-minimize-spam.yml 4638 build-and-publish-image.yml -29715 cd-cua-driver.yml +42519 cd-cua-driver.yml 2076 cd-mobile-mcp.yml 69782 ci.yml 1482 codeql.yml diff --git a/docs/design/2026-07-22-lazy-google-genai-loading.md b/docs/design/2026-07-22-lazy-google-genai-loading.md index 26f145269a0..80449d6dffa 100644 --- a/docs/design/2026-07-22-lazy-google-genai-loading.md +++ b/docs/design/2026-07-22-lazy-google-genai-loading.md @@ -22,14 +22,21 @@ Provider implementations continue to use the official SDK classes. In particular `createContentGenerator()` still validates configuration, preloads the runtime fetch implementation, and performs Qwen OAuth credential acquisition at its current point in the session lifecycle. It returns a private lazy `ContentGenerator` whose memoized loader constructs the selected provider and wraps it in `LoggingContentGenerator` on the first asynchronous content-generator operation. -All four asynchronous operations share the same loader promise: +All three asynchronous operations share the same loader promise: - `generateContent` - `generateContentStream` -- `countTokens` - `embedContent` -Concurrent first calls therefore import and construct the provider once. `useSummarizedThinking()` remains synchronous and is supplied from the selected provider's known behavior: true for Gemini/Vertex and false for OpenAI, Qwen OAuth, and Anthropic. +Concurrent first calls therefore import and construct the provider once. + +> **Update (2026-08, PR #9676)**: `countTokens` and `useSummarizedThinking` +> were removed from the `ContentGenerator` interface — no production caller +> used either, and the removal narrows every provider and test double to the +> operations something actually calls. This doc originally listed four shared +> asynchronous operations (including `countTokens`) and a synchronous +> `useSummarizedThinking()` supplied from each provider's known behavior; both +> are gone from the interface, the lazy wrapper, and the four providers. Qwen OAuth credential acquisition remains eager within `createContentGenerator()`. An expired or missing cached credential therefore continues to reject ACP session creation rather than producing an apparently usable session that fails only on its first prompt. @@ -49,7 +56,7 @@ The serve fast-path metafile guard adds `@google/genai` to the ACP forbidden-pac There are three direct production creation paths. `Config.refreshAuth()` owns the main-session generator. `BaseLlmClient` owns cached per-model generators for routed side requests. `createRuntimeContentGeneratorView()` owns dedicated generators used by the in-process agent backend, subagent manager, and forked agents. Each path stores and consumes only the `ContentGenerator` interface, so the private lazy wrapper preserves its ownership and routing boundary. -The interface consumers call only `generateContent`, `generateContentStream`, `countTokens`, `embedContent`, and `useSummarizedThinking`. The main chat path, prompt hooks, memory/goal/side queries, vision routing, subagents, and session resume do not inspect the concrete provider or unwrap `LoggingContentGenerator`; a repository-wide search found no production `instanceof` or `getWrapped()` caller. MCP tool discovery is separate from generator ownership and keeps the SDK-provided `mcpToTool` adapter behind its own first-use import. +The interface consumers call only `generateContent`, `generateContentStream`, and `embedContent`. The main chat path, prompt hooks, memory/goal/side queries, vision routing, subagents, and session resume do not inspect the concrete provider or unwrap `LoggingContentGenerator`; a repository-wide search found no production `instanceof` or `getWrapped()` caller. MCP tool discovery is separate from generator ownership and keeps the SDK-provided `mcpToTool` adapter behind its own first-use import. ## Alternatives rejected @@ -71,7 +78,7 @@ The interface consumers call only `generateContent`, `generateContentStream`, `c ## Verification -Unit tests cover helper parity, deferred construction, Qwen credential timing, single-flight behavior, provider-specific summarized-thinking values, deferred module failures, and MCP discovery behavior. The bundled metafile must show `@google/genai` absent from the ACP static closure while retaining it in dynamic provider/MCP chunks. +Unit tests cover helper parity, deferred construction, Qwen credential timing, single-flight behavior, deferred module failures, and MCP discovery behavior. The bundled metafile must show `@google/genai` absent from the ACP static closure while retaining it in dynamic provider/MCP chunks. The 2C4G acceptance run follows #7264: 30 paired serial cold starts, `channel.initialize` P50/P95, process-to-first-session, preheated/warm behavior, concurrent first sessions, telemetry on/off, and peak RSS. Because this change moves work later, it additionally records session-response-to-first-token and process-to-first-token for an immediate first prompt. A startup win that is fully repaid as a first-token regression is reported rather than treated as a successful optimization. diff --git a/docs/design/web-shell/web-shell-image-drag-and-drop.md b/docs/design/web-shell/web-shell-image-drag-and-drop.md index 9bdeb841e87..72229a6cffc 100644 --- a/docs/design/web-shell/web-shell-image-drag-and-drop.md +++ b/docs/design/web-shell/web-shell-image-drag-and-drop.md @@ -502,9 +502,14 @@ oversized placeholder。它们验证既有服务端契约；Web Shell helper 测 候选顺序和 encoded-data 剩余预算。 BMP 以 `image/bmp` 进入缩略图 data URL 和 daemon image block。Core 的 -`SUPPORTED_IMAGE_MIME_TYPES` 明确包含 `image/bmp`，`ImageTokenizer` 解析 BMP 尺寸， -OpenAI converter 把启用 image modality 的 `inlineData` 原样构造成 -`data:image/bmp;base64,...`；Gemini 路径保留相同 `inlineData`。因此 V1 不在浏览器转码。 +`SUPPORTED_IMAGE_MIME_TYPES` 明确包含 `image/bmp`，OpenAI converter 把启用 image modality 的 +`inlineData` 原样构造成 `data:image/bmp;base64,...`；Gemini 路径保留相同 `inlineData`。 +因此 V1 不在浏览器转码。 + +> **2026-08-24 同步注记（PR #9676）**：request-tokenizer 估计器簇（含 `ImageTokenizer` +> 及其 BMP 尺寸解析）已作为孤儿代码删除。BMP 支持现在仅依赖 `SUPPORTED_IMAGE_MIME_TYPES` +> 接受清单与 converter 透传；token 计数使用 `compactionInputSlimming.ts` 中的固定 +> `DEFAULT_IMAGE_TOKEN_ESTIMATE`。下文对 BMP 路径的 E2E/人工验收要求不变。 浏览器若不能解码缩略图，不影响附件数据传输，但 E2E 必须覆盖 Chromium 解码， Firefox/Linux 必须完成人工验收。 @@ -634,8 +639,8 @@ Web Shell 输入层静默改变格式。 BMP 的下游回归不只停在 mock HTTP 入参：在既有 ACP session prompt 转换测试中加入 `image/bmp`，验证最终 Core canonical content 保持 -`inlineData.mimeType === 'image/bmp'`；OpenAI/Gemini converter/tokenizer 聚焦测试验证 -各自既有图片路径，Anthropic 聚焦测试明确断言 BMP 转为 unsupported-media 文本。daemon +`inlineData.mimeType === 'image/bmp'`；OpenAI/Gemini converter 聚焦测试验证 +各自既有图片路径（tokenizer 估计器簇已随 PR #9676 删除，见上文同步注记），Anthropic 聚焦测试明确断言 BMP 转为 unsupported-media 文本。daemon 已有结构化 `413` 测试，Core 已有 inline-media within/over limit 测试；本功能不复制 production 限制，只确认 Web Shell 对这些既有失败语义的状态保留。 diff --git a/packages/cli/src/commands/review/lib/assets.ts b/packages/cli/src/commands/review/lib/assets.ts index 77431eac5e4..21e7d3724c3 100644 --- a/packages/cli/src/commands/review/lib/assets.ts +++ b/packages/cli/src/commands/review/lib/assets.ts @@ -70,9 +70,8 @@ export const ASSET_HEADER_BYTES = 16; * * A sibling signature table lives in core: `sniffFileKind` in * `packages/core/src/utils/binary-content.ts` (best-effort kind detection for - * fetched web content, deliberately looser) and the dimension extractors in - * `packages/core/src/utils/request-tokenizer/imageTokenizer.ts`. Admitting or - * correcting a format here means checking those sites too. + * fetched web content, deliberately looser). Admitting or correcting a format + * here means checking that site too. */ export function sniffImageFormat(header: Uint8Array): ImageFormat | null { const at = (i: number): number => header[i] ?? -1; diff --git a/packages/cli/src/ui/AppContainer.test.tsx b/packages/cli/src/ui/AppContainer.test.tsx index 1a1e46bf4b6..192b96d09a9 100644 --- a/packages/cli/src/ui/AppContainer.test.tsx +++ b/packages/cli/src/ui/AppContainer.test.tsx @@ -5080,9 +5080,9 @@ describe('AppContainer State Management', () => { lastCompletedUuid: 'a1', }; - vi.spyOn(mockConfig, 'getContentGenerator').mockReturnValue({ - useSummarizedThinking: vi.fn(() => false), - } as unknown as ReturnType); + vi.spyOn(mockConfig, 'getContentGenerator').mockReturnValue( + {} as unknown as ReturnType, + ); vi.spyOn(mockConfig, 'initialize').mockResolvedValue(undefined); vi.spyOn(mockConfig, 'getResumedSessionData').mockReturnValue( resumeSessionData as ReturnType< diff --git a/packages/cli/src/ui/components/StandaloneSessionPicker.test.tsx b/packages/cli/src/ui/components/StandaloneSessionPicker.test.tsx index 1671e261c76..70d09c4704f 100644 --- a/packages/cli/src/ui/components/StandaloneSessionPicker.test.tsx +++ b/packages/cli/src/ui/components/StandaloneSessionPicker.test.tsx @@ -701,7 +701,6 @@ describe('SessionPicker', () => { getIdeMode: () => false, isTrustedFolder: () => false, getToolRegistry: () => ({ getTool: () => undefined }), - getContentGenerator: () => ({ useSummarizedThinking: () => false }), } as unknown as Config; const PREVIEW_SETTINGS_STUB = { merged: { ui: {} }, diff --git a/packages/cli/src/ui/components/StandaloneSessionPicker.tsx b/packages/cli/src/ui/components/StandaloneSessionPicker.tsx index 1a245e0ab06..aecbbdb7b4b 100644 --- a/packages/cli/src/ui/components/StandaloneSessionPicker.tsx +++ b/packages/cli/src/ui/components/StandaloneSessionPicker.tsx @@ -37,7 +37,6 @@ const PREVIEW_CONFIG_STUB = { getIdeMode: () => false, isTrustedFolder: () => false, getToolRegistry: () => ({ getTool: () => undefined }), - getContentGenerator: () => ({ useSummarizedThinking: () => false }), } as unknown as Config; const PREVIEW_SETTINGS_STUB = { diff --git a/packages/core/src/agents/runtime/agent-headless.test.ts b/packages/core/src/agents/runtime/agent-headless.test.ts index e6705031945..62e8b6d812f 100644 --- a/packages/core/src/agents/runtime/agent-headless.test.ts +++ b/packages/core/src/agents/runtime/agent-headless.test.ts @@ -72,9 +72,7 @@ vi.mock('../../core/contentGenerator.js', async (importOriginal) => { createContentGenerator: vi.fn().mockResolvedValue({ generateContent: vi.fn(), generateContentStream: vi.fn(), - countTokens: vi.fn().mockResolvedValue({ totalTokens: 100 }), embedContent: vi.fn(), - useSummarizedThinking: vi.fn().mockReturnValue(false), }), createContentGeneratorConfig: vi.fn().mockReturnValue({ model: DEFAULT_QWEN_MODEL, diff --git a/packages/core/src/config/config.test.ts b/packages/core/src/config/config.test.ts index 5afa26d7021..602683ac8b7 100644 --- a/packages/core/src/config/config.test.ts +++ b/packages/core/src/config/config.test.ts @@ -5181,7 +5181,6 @@ describe('Server Config (config.ts)', () => { vi.mocked(createContentGenerator).mockResolvedValue({ generateContent: vi.fn(), generateContentStream: vi.fn(), - countTokens: vi.fn(), embedContent: vi.fn(), } as unknown as ContentGenerator); @@ -5232,7 +5231,6 @@ describe('Server Config (config.ts)', () => { vi.mocked(createContentGenerator).mockResolvedValue({ generateContent: vi.fn(), generateContentStream: vi.fn(), - countTokens: vi.fn(), embedContent: vi.fn(), } as unknown as ContentGenerator); @@ -5266,7 +5264,6 @@ describe('Server Config (config.ts)', () => { vi.mocked(createContentGenerator).mockResolvedValue({ generateContent: vi.fn(), generateContentStream: vi.fn(), - countTokens: vi.fn(), embedContent: vi.fn(), } as unknown as ContentGenerator); @@ -5823,7 +5820,6 @@ describe('Server Config (config.ts)', () => { vi.mocked(createContentGenerator).mockResolvedValue({ generateContent: vi.fn(), generateContentStream: vi.fn(), - countTokens: vi.fn(), embedContent: vi.fn(), } as unknown as ContentGenerator); diff --git a/packages/core/src/core/__tests__/openaiTimeoutHandling.test.ts b/packages/core/src/core/__tests__/openaiTimeoutHandling.test.ts index 86995d622e3..4d60ebc9915 100644 --- a/packages/core/src/core/__tests__/openaiTimeoutHandling.test.ts +++ b/packages/core/src/core/__tests__/openaiTimeoutHandling.test.ts @@ -363,24 +363,5 @@ describe('OpenAIContentGenerator Timeout Handling', () => { generator.generateContent(request, 'test-prompt-id'), ).rejects.toThrow(/Request timeout after \d+s/); }); - - it('should fall back to character-based estimation if countTokens fails', async () => { - const timeoutError = new Error('Request timeout'); - mockOpenAIClient.chat.completions.create.mockRejectedValue(timeoutError); - - // Mock countTokens to throw error - const mockCountTokens = vi.spyOn(generator, 'countTokens'); - mockCountTokens.mockRejectedValue(new Error('Count tokens failed')); - - const request = { - contents: [{ role: 'user' as const, parts: [{ text: 'Hello world' }] }], - model: 'gpt-4', - }; - - // Should not throw due to token counting failure - await expect( - generator.generateContent(request, 'test-prompt-id'), - ).rejects.toThrow(/Request timeout after \d+s/); - }); }); }); diff --git a/packages/core/src/core/anthropicContentGenerator/anthropicContentGenerator.test.ts b/packages/core/src/core/anthropicContentGenerator/anthropicContentGenerator.test.ts index 8d760c49eea..b03deeeaa57 100644 --- a/packages/core/src/core/anthropicContentGenerator/anthropicContentGenerator.test.ts +++ b/packages/core/src/core/anthropicContentGenerator/anthropicContentGenerator.test.ts @@ -6,10 +6,7 @@ import { getEventListeners } from 'node:events'; import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; -import type { - CountTokensParameters, - GenerateContentParameters, -} from '@google/genai'; +import type { GenerateContentParameters } from '@google/genai'; import { FinishReason, GenerateContentResponse } from '@google/genai'; import type { ContentGeneratorConfig } from '../contentGenerator.js'; import { @@ -17,19 +14,11 @@ import { DISABLED_REQUEST_TIMEOUT_MS, } from '../openaiContentGenerator/constants.js'; -// Mock the request tokenizer module BEFORE importing the class that uses it. -const mockTokenizer = { - calculateTokens: vi.fn(), - dispose: vi.fn(), -}; const mockReportAnthropicRequest = vi.hoisted(() => vi.fn()); const mockReportAnthropicFollowingRequest = vi.hoisted(() => vi.fn()); const mockReportAnthropicResponse = vi.hoisted(() => vi.fn()); const mockReportAnthropicEvent = vi.hoisted(() => vi.fn()); -vi.mock('../../utils/request-tokenizer/index.js', () => ({ - RequestTokenEstimator: vi.fn(() => mockTokenizer), -})); vi.mock('../../telemetry/gen-ai-request.js', () => ({ reportAnthropicRequest: mockReportAnthropicRequest, reportAnthropicFollowingRequest: mockReportAnthropicFollowingRequest, @@ -107,16 +96,6 @@ describe('AnthropicContentGenerator', () => { savedMaxOutputTokensEnv = process.env[MAX_OUTPUT_TOKENS_ENV]; delete process.env[MAX_OUTPUT_TOKENS_ENV]; - mockTokenizer.calculateTokens.mockResolvedValue({ - totalTokens: 50, - breakdown: { - textTokens: 50, - imageTokens: 0, - audioTokens: 0, - otherTokens: 0, - }, - processingTime: 1, - }); anthropicState = anthropicMockState; anthropicState.createImpl.mockReset(); @@ -3305,58 +3284,6 @@ describe('AnthropicContentGenerator', () => { }); }); - describe('countTokens', () => { - it('counts tokens using the request tokenizer', async () => { - const { AnthropicContentGenerator } = await importGenerator(); - const generator = new AnthropicContentGenerator( - { - model: 'claude-test', - apiKey: 'test-key', - timeout: 10_000, - maxRetries: 2, - samplingParams: {}, - schemaCompliance: 'auto', - }, - mockConfig, - ); - - const request: CountTokensParameters = { - contents: [{ role: 'user', parts: [{ text: 'Hello world' }] }], - model: 'claude-test', - }; - - const result = await generator.countTokens(request); - expect(mockTokenizer.calculateTokens).toHaveBeenCalledWith(request); - expect(result.totalTokens).toBe(50); - }); - - it('falls back to character approximation when tokenizer throws', async () => { - const { AnthropicContentGenerator } = await importGenerator(); - mockTokenizer.calculateTokens.mockRejectedValueOnce(new Error('boom')); - const generator = new AnthropicContentGenerator( - { - model: 'claude-test', - apiKey: 'test-key', - timeout: 10_000, - maxRetries: 2, - samplingParams: {}, - schemaCompliance: 'auto', - }, - mockConfig, - ); - - const request: CountTokensParameters = { - contents: [{ role: 'user', parts: [{ text: 'Hello' }] }], - model: 'claude-test', - }; - - const content = JSON.stringify(request.contents); - const expected = Math.ceil(content.length / 4); - const result = await generator.countTokens(request); - expect(result.totalTokens).toBe(expected); - }); - }); - describe('generateContentStream', () => { it('emits tool preparation metadata before the complete function call', async () => { const { AnthropicContentGenerator } = await importGenerator(); diff --git a/packages/core/src/core/anthropicContentGenerator/anthropicContentGenerator.ts b/packages/core/src/core/anthropicContentGenerator/anthropicContentGenerator.ts index bc4ad438154..5c252a311c0 100644 --- a/packages/core/src/core/anthropicContentGenerator/anthropicContentGenerator.ts +++ b/packages/core/src/core/anthropicContentGenerator/anthropicContentGenerator.ts @@ -6,8 +6,6 @@ import Anthropic from '@anthropic-ai/sdk'; import type { - CountTokensParameters, - CountTokensResponse, EmbedContentParameters, EmbedContentResponse, GenerateContentParameters, @@ -29,7 +27,6 @@ type MessageCreateParamsNonStreaming = Anthropic.MessageCreateParamsNonStreaming; type MessageCreateParamsStreaming = Anthropic.MessageCreateParamsStreaming; type RawMessageStreamEvent = Anthropic.RawMessageStreamEvent; -import { RequestTokenEstimator } from '../../utils/request-tokenizer/index.js'; import { safeJsonParse } from '../../utils/safeJsonParse.js'; import { AnthropicContentConverter } from './converter.js'; import { buildAnthropicUsageMetadata } from './usage.js'; @@ -445,41 +442,11 @@ export class AnthropicContentGenerator implements ContentGenerator { return drainThenCleanup(); } - async countTokens( - request: CountTokensParameters, - ): Promise { - try { - const estimator = new RequestTokenEstimator(); - const result = await estimator.calculateTokens(request); - - return { - totalTokens: result.totalTokens, - }; - } catch (error) { - debugLogger.warn( - 'Failed to calculate tokens with tokenizer, ' + - 'falling back to simple method:', - error, - ); - - const content = JSON.stringify(request.contents); - const totalTokens = Math.ceil(content.length / 4); - return { - totalTokens, - }; - } - } - async embedContent( _request: EmbedContentParameters, ): Promise { throw new Error('Anthropic does not support embeddings.'); } - - useSummarizedThinking(): boolean { - return false; - } - private buildHeaders(useProxyIdentity: boolean): Record { // Beta headers are computed per-request in buildPerRequestHeaders so they // stay in sync with what the request body actually carries — see #3788 diff --git a/packages/core/src/core/client.test.ts b/packages/core/src/core/client.test.ts index c35f8d7a967..17a47fd65f3 100644 --- a/packages/core/src/core/client.test.ts +++ b/packages/core/src/core/client.test.ts @@ -424,26 +424,6 @@ vi.mock( ); import { microcompactHistory } from '../services/microcompaction/microcompact.js'; -// Mock RequestTokenizer to use simple character-based estimation -vi.mock('../utils/request-tokenizer/requestTokenizer.js', () => ({ - RequestTokenizer: class { - async calculateTokens(request: { contents: unknown }) { - // Simple estimation: count characters in JSON and divide by 4 - const totalChars = JSON.stringify(request.contents).length; - return { - totalTokens: Math.floor(totalChars / 4), - breakdown: { - textTokens: Math.floor(totalChars / 4), - imageTokens: 0, - audioTokens: 0, - otherTokens: 0, - }, - processingTime: 0, - }; - } - }, -})); - /** * Array.fromAsync ponyfill, which will be available in es 2024. * @@ -562,7 +542,6 @@ describe('Gemini Client (client.ts)', () => { generateContent: mockGenerateContentFn, generateContentStream: vi.fn(), batchEmbedContents: vi.fn(), - countTokens: vi.fn().mockResolvedValue({ totalTokens: 100 }), } as unknown as ContentGenerator; // Because the GeminiClient constructor kicks off an async process (startChat) diff --git a/packages/core/src/core/contentGenerator.test.ts b/packages/core/src/core/contentGenerator.test.ts index 0a55801a844..11b5e0cd873 100644 --- a/packages/core/src/core/contentGenerator.test.ts +++ b/packages/core/src/core/contentGenerator.test.ts @@ -55,9 +55,7 @@ vi.mock('./openaiContentGenerator/index.js', () => ({ (async function* () { yield {}; })(), - countTokens: async () => ({ totalTokens: 1 }), embedContent: async () => ({ embeddings: [] }), - useSummarizedThinking: () => false, }); const gate = openaiMockState.constructionGates[attempt]; if (!gate) { @@ -88,12 +86,8 @@ vi.mock('../qwen/qwenContentGenerator.js', () => ({ qwenMockState.constructorModels.push(generatorConfig.model); } - async countTokens() { - return { totalTokens: 1 }; - } - - useSummarizedThinking() { - return false; + async embedContent() { + return { embeddings: [] }; } }, })); @@ -131,7 +125,7 @@ describe('createContentGenerator', () => { const mockGenerator = { models: { - countTokens: vi.fn().mockResolvedValue({ totalTokens: 1 }), + embedContent: vi.fn().mockResolvedValue({ embeddings: [] }), }, } as unknown as GoogleGenAI; vi.mocked(GoogleGenAI).mockImplementation(() => mockGenerator as never); @@ -144,9 +138,8 @@ describe('createContentGenerator', () => { mockConfig, ); expect(GoogleGenAI).not.toHaveBeenCalled(); - expect(generator.useSummarizedThinking()).toBe(true); - await generator.countTokens({ + await generator.embedContent({ model: 'test-model', contents: 'hello', }); @@ -173,7 +166,7 @@ describe('createContentGenerator', () => { } as unknown as Config; const mockGenerator = { models: { - countTokens: vi.fn().mockResolvedValue({ totalTokens: 1 }), + embedContent: vi.fn().mockResolvedValue({ embeddings: [] }), }, } as unknown as GoogleGenAI; vi.mocked(GoogleGenAI).mockImplementation(() => mockGenerator as never); @@ -186,7 +179,7 @@ describe('createContentGenerator', () => { mockConfig, ); expect(GoogleGenAI).not.toHaveBeenCalled(); - await generator.countTokens({ + await generator.embedContent({ model: 'test-model', contents: 'hello', }); @@ -219,10 +212,9 @@ describe('createContentGenerator', () => { ); expect(openaiMockState.createCount).toBe(0); - expect(generator.useSummarizedThinking()).toBe(false); await Promise.all([ - generator.countTokens({ model: 'test-model', contents: 'one' }), - generator.countTokens({ model: 'test-model', contents: 'two' }), + generator.embedContent({ model: 'test-model', contents: 'one' }), + generator.embedContent({ model: 'test-model', contents: 'two' }), ]); expect(openaiMockState.createCount).toBe(1); }); @@ -231,18 +223,14 @@ describe('createContentGenerator', () => { const generator: ContentGenerator = { generateContent: vi.fn(), generateContentStream: vi.fn(), - countTokens: vi.fn(), embedContent: vi.fn(), - useSummarizedThinking: vi.fn(), }; await expect(preloadContentGenerator(generator)).resolves.toBeUndefined(); resetPreloadedContentGenerator(generator); expect(generator.generateContent).not.toHaveBeenCalled(); expect(generator.generateContentStream).not.toHaveBeenCalled(); - expect(generator.countTokens).not.toHaveBeenCalled(); expect(generator.embedContent).not.toHaveBeenCalled(); - expect(generator.useSummarizedThinking).not.toHaveBeenCalled(); }); it('loads a provider once across concurrent preload and first use', async () => { @@ -264,7 +252,7 @@ describe('createContentGenerator', () => { await Promise.all([ preloadContentGenerator(generator), - generator.countTokens({ model: 'test-model', contents: 'hello' }), + generator.embedContent({ model: 'test-model', contents: 'hello' }), ]); expect(openaiMockState.createCount).toBe(1); @@ -289,7 +277,7 @@ describe('createContentGenerator', () => { await preloadContentGenerator(generator); resetPreloadedContentGenerator(generator); - await generator.countTokens({ + await generator.embedContent({ model: 'test-model', contents: 'hello', }); @@ -315,12 +303,12 @@ describe('createContentGenerator', () => { ); await preloadContentGenerator(generator); - await generator.countTokens({ + await generator.embedContent({ model: 'test-model', contents: 'first', }); resetPreloadedContentGenerator(generator); - await generator.countTokens({ + await generator.embedContent({ model: 'test-model', contents: 'second', }); @@ -353,7 +341,7 @@ describe('createContentGenerator', () => { const preload = preloadContentGenerator(generator); await vi.waitFor(() => expect(openaiMockState.createCount).toBe(1)); - const firstUse = generator.countTokens({ + const firstUse = generator.embedContent({ model: 'test-model', contents: 'hello', }); @@ -361,7 +349,7 @@ describe('createContentGenerator', () => { releaseConstruction(); await expect(preload).resolves.toBeUndefined(); - await expect(firstUse).resolves.toEqual({ totalTokens: 1 }); + await expect(firstUse).resolves.toEqual({ embeddings: [] }); expect(openaiMockState.createCount).toBe(1); }); @@ -399,22 +387,22 @@ describe('createContentGenerator', () => { ); await vi.waitFor(() => expect(openaiMockState.createCount).toBe(1)); resetPreloadedContentGenerator(generator); - const firstUse = generator.countTokens({ + const firstUse = generator.embedContent({ model: 'test-model', contents: 'hello', }); await vi.waitFor(() => expect(openaiMockState.createCount).toBe(2)); releaseFirstUse(); - await expect(firstUse).resolves.toEqual({ totalTokens: 1 }); + await expect(firstUse).resolves.toEqual({ embeddings: [] }); releasePreload(); await expect(discardedPreload).resolves.toBe(preloadError); await expect( - generator.countTokens({ + generator.embedContent({ model: 'test-model', contents: 'still uses the replacement', }), - ).resolves.toEqual({ totalTokens: 1 }); + ).resolves.toEqual({ embeddings: [] }); expect(openaiMockState.createCount).toBe(2); }); @@ -435,7 +423,7 @@ describe('createContentGenerator', () => { await preloadContentGenerator(generator); generatorConfig.model = 'coder-model'; resetPreloadedContentGenerator(generator); - await generator.countTokens({ + await generator.embedContent({ model: 'coder-model', contents: 'hello', }); @@ -470,7 +458,7 @@ describe('createContentGenerator', () => { await preloadContentGenerator(generator); workingDir = '/workspace/after'; resetPreloadedContentGenerator(generator); - await generator.countTokens({ + await generator.embedContent({ model: 'test-model', contents: 'hello', }); @@ -507,7 +495,7 @@ describe('createContentGenerator', () => { (error: unknown) => error, ); const firstUseError = await generator - .countTokens({ model: 'test-model', contents: 'hello' }) + .embedContent({ model: 'test-model', contents: 'hello' }) .catch((error: unknown) => error); expect(preloadError).toBeInstanceOf(Error); @@ -534,8 +522,7 @@ describe('createContentGenerator', () => { expect(qwenMockState.oauthCount).toBe(1); expect(qwenMockState.constructorCount).toBe(0); - expect(generator.useSummarizedThinking()).toBe(false); - await generator.countTokens({ model: 'test-model', contents: 'hello' }); + await generator.embedContent({ model: 'test-model', contents: 'hello' }); expect(qwenMockState.constructorCount).toBe(1); }); @@ -638,7 +625,7 @@ describe('createContentGenerator - ERR_MODULE_NOT_FOUND handling', () => { mockConfig, ); await expect( - generator.countTokens({ model: 'test-model', contents: 'hello' }), + generator.embedContent({ model: 'test-model', contents: 'hello' }), ).rejects.toThrow('network timeout'); }); @@ -688,7 +675,7 @@ describe('createContentGenerator - ERR_MODULE_NOT_FOUND handling', () => { }, mockConfig, ); - await generator.countTokens({ model: 'test-model', contents: 'hello' }); + await generator.embedContent({ model: 'test-model', contents: 'hello' }); expect.unreachable('should have thrown'); } catch (error) { expect(error).toBeInstanceOf(Error); diff --git a/packages/core/src/core/contentGenerator.ts b/packages/core/src/core/contentGenerator.ts index 7216c26e388..7d2e79f2f63 100644 --- a/packages/core/src/core/contentGenerator.ts +++ b/packages/core/src/core/contentGenerator.ts @@ -5,8 +5,6 @@ */ import type { - CountTokensParameters, - CountTokensResponse, EmbedContentParameters, EmbedContentResponse, GenerateContentParameters, @@ -33,7 +31,7 @@ import { preloadRuntimeFetchModule } from '../utils/runtimeFetchOptions.js'; import type { ReasoningEffort } from './reasoning-effort.js'; /** - * Interface abstracting the core functionalities for generating content and counting tokens. + * Interface abstracting the core content generation functionality. */ export interface ContentGenerator { generateContent( @@ -46,11 +44,7 @@ export interface ContentGenerator { userPromptId: string, ): Promise>; - countTokens(request: CountTokensParameters): Promise; - embedContent(request: EmbedContentParameters): Promise; - - useSummarizedThinking(): boolean; } export enum AuthType { @@ -403,10 +397,7 @@ class LazyContentGenerator implements ContentGenerator { private generatorPromise?: Promise; private preloadedOnly = false; - constructor( - private readonly loader: () => Promise, - private readonly summarizedThinking: boolean, - ) {} + constructor(private readonly loader: () => Promise) {} private getGenerator(): Promise { this.generatorPromise ??= this.loader(); @@ -451,21 +442,11 @@ class LazyContentGenerator implements ContentGenerator { ); } - async countTokens( - request: CountTokensParameters, - ): Promise { - return (await this.getGeneratorForUse()).countTokens(request); - } - async embedContent( request: EmbedContentParameters, ): Promise { return (await this.getGeneratorForUse()).embedContent(request); } - - useSummarizedThinking(): boolean { - return this.summarizedThinking; - } } /** @internal */ @@ -564,22 +545,19 @@ export async function createContentGenerator( throw wrapProviderLoadError(error, authType); } - return new LazyContentGenerator( - async () => { - try { - const [baseGenerator, { LoggingContentGenerator }] = await Promise.all([ - loadBaseGenerator(), - import('./loggingContentGenerator/index.js'), - ]); - return new LoggingContentGenerator( - baseGenerator, - config, - generatorConfig, - ); - } catch (error) { - throw wrapProviderLoadError(error, authType); - } - }, - authType === AuthType.USE_GEMINI || authType === AuthType.USE_VERTEX_AI, - ); + return new LazyContentGenerator(async () => { + try { + const [baseGenerator, { LoggingContentGenerator }] = await Promise.all([ + loadBaseGenerator(), + import('./loggingContentGenerator/index.js'), + ]); + return new LoggingContentGenerator( + baseGenerator, + config, + generatorConfig, + ); + } catch (error) { + throw wrapProviderLoadError(error, authType); + } + }); } diff --git a/packages/core/src/core/geminiChat.test.ts b/packages/core/src/core/geminiChat.test.ts index 1152224fabc..f452bfaeeb1 100644 --- a/packages/core/src/core/geminiChat.test.ts +++ b/packages/core/src/core/geminiChat.test.ts @@ -162,10 +162,8 @@ describe('GeminiChat', async () => { mockContentGenerator = { generateContent: vi.fn(), generateContentStream: vi.fn(), - countTokens: vi.fn(), embedContent: vi.fn(), batchEmbedContents: vi.fn(), - useSummarizedThinking: vi.fn().mockReturnValue(false), } as unknown as ContentGenerator; // Default mock implementation for tests that don't care about retry logic @@ -7195,10 +7193,8 @@ describe('GeminiChat', async () => { ({ generateContent: vi.fn(), generateContentStream, - countTokens: vi.fn(), embedContent: vi.fn(), batchEmbedContents: vi.fn(), - useSummarizedThinking: vi.fn().mockReturnValue(false), }) as unknown as ContentGenerator; const resolveForModel = vi .fn() @@ -7355,10 +7351,8 @@ describe('GeminiChat', async () => { ({ generateContent: vi.fn(), generateContentStream, - countTokens: vi.fn(), embedContent: vi.fn(), batchEmbedContents: vi.fn(), - useSummarizedThinking: vi.fn().mockReturnValue(false), }) as unknown as ContentGenerator; const resolveForModel = vi .fn() @@ -7447,10 +7441,8 @@ describe('GeminiChat', async () => { const fallbackBGenerator = { generateContent: vi.fn(), generateContentStream: fallbackBGenerateContentStream, - countTokens: vi.fn(), embedContent: vi.fn(), batchEmbedContents: vi.fn(), - useSummarizedThinking: vi.fn().mockReturnValue(false), } as unknown as ContentGenerator; const resolveError = new Error('unknown fallback alias'); const resolveForModel = vi @@ -7536,10 +7528,8 @@ describe('GeminiChat', async () => { ({ generateContent: vi.fn(), generateContentStream, - countTokens: vi.fn(), embedContent: vi.fn(), batchEmbedContents: vi.fn(), - useSummarizedThinking: vi.fn().mockReturnValue(false), }) as unknown as ContentGenerator; const resolveForModel = vi .fn() @@ -7700,10 +7690,8 @@ describe('GeminiChat', async () => { ({ generateContent: vi.fn(), generateContentStream, - countTokens: vi.fn(), embedContent: vi.fn(), batchEmbedContents: vi.fn(), - useSummarizedThinking: vi.fn().mockReturnValue(false), }) as unknown as ContentGenerator; const resolveForModel = vi .fn() @@ -9488,10 +9476,8 @@ describe('GeminiChat', async () => { contentGenerator: { generateContent: vi.fn(), generateContentStream: fallbackGenerateContentStream, - countTokens: vi.fn(), embedContent: vi.fn(), batchEmbedContents: vi.fn(), - useSummarizedThinking: vi.fn().mockReturnValue(false), } as unknown as ContentGenerator, retryAuthType: AuthType.USE_GEMINI, retryErrorCodes: undefined, diff --git a/packages/core/src/core/geminiContentGenerator/geminiContentGenerator.test.ts b/packages/core/src/core/geminiContentGenerator/geminiContentGenerator.test.ts index 0e3ef8a3280..ffa7653c403 100644 --- a/packages/core/src/core/geminiContentGenerator/geminiContentGenerator.test.ts +++ b/packages/core/src/core/geminiContentGenerator/geminiContentGenerator.test.ts @@ -15,7 +15,6 @@ const mockReportGeminiChunk = vi.hoisted(() => vi.fn()); vi.mock('@google/genai', () => { const mockGenerateContent = vi.fn(); const mockGenerateContentStream = vi.fn(); - const mockCountTokens = vi.fn(); const mockEmbedContent = vi.fn(); return { @@ -23,7 +22,6 @@ vi.mock('@google/genai', () => { models: { generateContent: mockGenerateContent, generateContentStream: mockGenerateContentStream, - countTokens: mockCountTokens, embedContent: mockEmbedContent, }, })), @@ -211,17 +209,6 @@ describe('GeminiContentGenerator', () => { expect(mockReportGeminiChunk).not.toHaveBeenCalled(); }); - it('should call countTokens on the underlying model', async () => { - const request = { model: 'gemini-1.5-flash', contents: [] }; - const expectedResponse = { totalTokens: 10 }; - mockGoogleGenAI.models.countTokens.mockResolvedValue(expectedResponse); - - const response = await generator.countTokens(request); - - expect(mockGoogleGenAI.models.countTokens).toHaveBeenCalledWith(request); - expect(response).toBe(expectedResponse); - }); - it('should call embedContent on the underlying model', async () => { const request = { model: 'embedding-model', contents: [] }; const expectedResponse = { embeddings: [] }; diff --git a/packages/core/src/core/geminiContentGenerator/geminiContentGenerator.ts b/packages/core/src/core/geminiContentGenerator/geminiContentGenerator.ts index b17cb6edbfc..d9e8e3241cd 100644 --- a/packages/core/src/core/geminiContentGenerator/geminiContentGenerator.ts +++ b/packages/core/src/core/geminiContentGenerator/geminiContentGenerator.ts @@ -5,8 +5,6 @@ */ import type { - CountTokensParameters, - CountTokensResponse, EmbedContentParameters, EmbedContentResponse, GenerateContentParameters, @@ -366,19 +364,9 @@ export class GeminiContentGenerator implements ContentGenerator { return part; } - async countTokens( - request: CountTokensParameters, - ): Promise { - return this.googleGenAI.models.countTokens(request); - } - async embedContent( request: EmbedContentParameters, ): Promise { return this.googleGenAI.models.embedContent(request); } - - useSummarizedThinking(): boolean { - return true; - } } diff --git a/packages/core/src/core/loggingContentGenerator/loggingContentGenerator.test.ts b/packages/core/src/core/loggingContentGenerator/loggingContentGenerator.test.ts index 02ca2c3facd..7a0867e46ce 100644 --- a/packages/core/src/core/loggingContentGenerator/loggingContentGenerator.test.ts +++ b/packages/core/src/core/loggingContentGenerator/loggingContentGenerator.test.ts @@ -339,9 +339,7 @@ const createWrappedGenerator = ( ({ generateContent, generateContentStream, - countTokens: vi.fn(), embedContent: vi.fn(), - useSummarizedThinking: vi.fn().mockReturnValue(false), }) as ContentGenerator; const createResponse = ( diff --git a/packages/core/src/core/loggingContentGenerator/loggingContentGenerator.ts b/packages/core/src/core/loggingContentGenerator/loggingContentGenerator.ts index 2f15dc9de28..41d7760b99a 100644 --- a/packages/core/src/core/loggingContentGenerator/loggingContentGenerator.ts +++ b/packages/core/src/core/loggingContentGenerator/loggingContentGenerator.ts @@ -7,8 +7,6 @@ import { GenerateContentResponse, type Content, - type CountTokensParameters, - type CountTokensResponse, type EmbedContentParameters, type EmbedContentResponse, type GenerateContentParameters, @@ -1245,20 +1243,12 @@ export class LoggingContentGenerator implements ContentGenerator { return areSensitiveSpanAttributesEnabled(this.config); } - async countTokens(req: CountTokensParameters): Promise { - return this.wrapped.countTokens(req); - } - async embedContent( req: EmbedContentParameters, ): Promise { return this.wrapped.embedContent(req); } - useSummarizedThinking(): boolean { - return this.wrapped.useSummarizedThinking(); - } - private toContents(contents: ContentListUnion): Content[] { if (Array.isArray(contents)) { // it's a Content[] or a PartsUnion[] diff --git a/packages/core/src/core/openaiContentGenerator/openaiContentGenerator.test.ts b/packages/core/src/core/openaiContentGenerator/openaiContentGenerator.test.ts index deab82034c2..3d314488d08 100644 --- a/packages/core/src/core/openaiContentGenerator/openaiContentGenerator.test.ts +++ b/packages/core/src/core/openaiContentGenerator/openaiContentGenerator.test.ts @@ -5,34 +5,10 @@ */ import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'; - -// Mock the request tokenizer module BEFORE importing the class that uses it -const mockTokenizer = { - calculateTokens: vi.fn().mockResolvedValue({ - totalTokens: 50, - breakdown: { - textTokens: 50, - imageTokens: 0, - audioTokens: 0, - otherTokens: 0, - }, - processingTime: 1, - }), - dispose: vi.fn(), -}; - -vi.mock('../../../utils/request-tokenizer/index.js', () => ({ - RequestTokenEstimator: vi.fn(() => mockTokenizer), -})); - -// Now import the modules that depend on the mocked modules import { OpenAIContentGenerator } from './openaiContentGenerator.js'; import type { Config } from '../../config/config.js'; import { AuthType } from '../contentGenerator.js'; -import type { - GenerateContentParameters, - CountTokensParameters, -} from '@google/genai'; +import type { GenerateContentParameters } from '@google/genai'; import type { OpenAICompatibleProvider } from './provider/index.js'; import type OpenAI from 'openai'; @@ -123,39 +99,6 @@ describe('OpenAIContentGenerator (Refactored)', () => { }); }); - describe('countTokens', () => { - it('should count tokens using character-based estimation', async () => { - const request: CountTokensParameters = { - contents: [{ role: 'user', parts: [{ text: 'Hello world' }] }], - model: 'gpt-4', - }; - - const result = await generator.countTokens(request); - - // 'Hello world' = 11 ASCII chars - // 11 / 4 = 2.75 -> ceil = 3 tokens - expect(result.totalTokens).toBe(3); - }); - - it('should handle multimodal content', async () => { - const request: CountTokensParameters = { - contents: [ - { - role: 'user', - parts: [{ text: 'Hello' }, { text: ' world' }], - }, - ], - model: 'gpt-4', - }; - - const result = await generator.countTokens(request); - - // Parts are combined for estimation: - // 'Hello world' = 11 ASCII chars -> 11/4 = 2.75 -> ceil = 3 tokens - expect(result.totalTokens).toBe(3); - }); - }); - describe('embedContent', () => { it('should delegate to pipeline.client.embeddings.create', async () => { // This test verifies the method exists and can be called diff --git a/packages/core/src/core/openaiContentGenerator/openaiContentGenerator.ts b/packages/core/src/core/openaiContentGenerator/openaiContentGenerator.ts index f4166d4acf8..78658fde44d 100644 --- a/packages/core/src/core/openaiContentGenerator/openaiContentGenerator.ts +++ b/packages/core/src/core/openaiContentGenerator/openaiContentGenerator.ts @@ -2,8 +2,6 @@ import type { ContentGenerator } from '../contentGenerator.js'; import type { Config } from '../../config/config.js'; import { type OpenAICompatibleProvider } from './provider/index.js'; import type { - CountTokensParameters, - CountTokensResponse, EmbedContentParameters, EmbedContentResponse, GenerateContentParameters, @@ -12,7 +10,6 @@ import type { import type { PipelineConfig } from './types.js'; import { ContentGenerationPipeline } from './pipeline.js'; import { EnhancedErrorHandler } from './errorHandler.js'; -import { RequestTokenEstimator } from '../../utils/request-tokenizer/index.js'; import type { ContentGeneratorConfig } from '../contentGenerator.js'; import { isAbortError } from '../../utils/errors.js'; import { createDebugLogger } from '../../utils/debugLogger.js'; @@ -80,33 +77,6 @@ export class OpenAIContentGenerator implements ContentGenerator { return this.pipeline.executeStream(request, userPromptId); } - async countTokens( - request: CountTokensParameters, - ): Promise { - try { - // Use the request token estimator (character-based). - const estimator = new RequestTokenEstimator(); - const result = await estimator.calculateTokens(request); - - return { - totalTokens: result.totalTokens, - }; - } catch (error) { - debugLogger.warn( - 'Failed to calculate tokens with new tokenizer, falling back to simple method:', - error, - ); - - // Fallback to original simple method - const content = JSON.stringify(request.contents); - const totalTokens = Math.ceil(content.length / 4); // Rough estimate: 1 token ≈ 4 characters - - return { - totalTokens, - }; - } - } - async embedContent( request: EmbedContentParameters, ): Promise { @@ -163,8 +133,4 @@ export class OpenAIContentGenerator implements ContentGenerator { ); } } - - useSummarizedThinking(): boolean { - return false; - } } diff --git a/packages/core/src/hooks/promptHookIntegration.test.ts b/packages/core/src/hooks/promptHookIntegration.test.ts index b8b8c904a32..5b88c5db7fa 100644 --- a/packages/core/src/hooks/promptHookIntegration.test.ts +++ b/packages/core/src/hooks/promptHookIntegration.test.ts @@ -39,9 +39,7 @@ describe('Prompt Hook Integration', () => { getContentGenerator: vi.fn().mockReturnValue({ generateContent: mockGenerateContent, generateContentStream: vi.fn(), - countTokens: vi.fn(), embedContent: vi.fn(), - useSummarizedThinking: vi.fn().mockReturnValue(false), }), getBaseLlmClient: vi.fn().mockReturnValue({ resolveForModel: vi.fn().mockResolvedValue({ diff --git a/packages/core/src/hooks/promptHookRunner.test.ts b/packages/core/src/hooks/promptHookRunner.test.ts index 3218a23d35b..b25937e8fa7 100644 --- a/packages/core/src/hooks/promptHookRunner.test.ts +++ b/packages/core/src/hooks/promptHookRunner.test.ts @@ -40,9 +40,7 @@ describe('PromptHookRunner', () => { getContentGenerator: vi.fn().mockReturnValue({ generateContent: mockGenerateContent, generateContentStream: vi.fn(), - countTokens: vi.fn(), embedContent: vi.fn(), - useSummarizedThinking: vi.fn().mockReturnValue(false), }), getBaseLlmClient: vi.fn().mockReturnValue({ resolveForModel: mockResolveForModel, diff --git a/packages/core/src/qwen/qwenContentGenerator.test.ts b/packages/core/src/qwen/qwenContentGenerator.test.ts index e52ba34caa1..40bba3b58dc 100644 --- a/packages/core/src/qwen/qwenContentGenerator.test.ts +++ b/packages/core/src/qwen/qwenContentGenerator.test.ts @@ -10,8 +10,6 @@ import { type QwenCredentials, type ErrorData } from './qwenOAuth2.js'; import type { GenerateContentParameters, GenerateContentResponse, - CountTokensParameters, - CountTokensResponse, EmbedContentParameters, EmbedContentResponse, } from '@google/genai'; @@ -96,12 +94,6 @@ vi.mock('../core/openaiContentGenerator/pipeline.js', () => ({ })(); } - async countTokens( - _request: CountTokensParameters, - ): Promise { - return { totalTokens: 15 }; - } - async embedContent( _request: EmbedContentParameters, ): Promise { @@ -251,12 +243,6 @@ vi.mock('../core/openaiContentGenerator/index.js', () => ({ })(); } - async countTokens( - _request: CountTokensParameters, - ): Promise { - return { totalTokens: 15 }; - } - async embedContent( _request: EmbedContentParameters, ): Promise { @@ -401,22 +387,6 @@ describe('QwenContentGenerator', () => { expect(mockQwenClient.getAccessToken).toHaveBeenCalled(); }); - it('should count tokens without requiring authentication', async () => { - // Clear any previous mock calls - vi.clearAllMocks(); - - const request: CountTokensParameters = { - model: 'qwen-turbo', - contents: [{ role: 'user', parts: [{ text: 'Count me' }] }], - }; - - const result = await qwenContentGenerator.countTokens(request); - - expect(result.totalTokens).toBe(15); - // countTokens is a local operation and should not require OAuth credentials - expect(mockQwenClient.getAccessToken).not.toHaveBeenCalled(); - }); - it('should embed content with valid token', async () => { vi.mocked(mockQwenClient.getAccessToken).mockResolvedValue({ token: 'valid-token', @@ -1650,7 +1620,7 @@ describe('QwenContentGenerator', () => { SharedTokenManager.getInstance = originalGetInstance; }); - it('should handle method types with token failure (except countTokens)', async () => { + it('should handle method types with token failure', async () => { const mockTokenManager = { getValidCredentials: vi .fn() @@ -1673,11 +1643,6 @@ describe('QwenContentGenerator', () => { contents: [{ role: 'user', parts: [{ text: 'Hello' }] }], }; - const countRequest: CountTokensParameters = { - model: 'qwen-turbo', - contents: [{ role: 'user', parts: [{ text: 'Count' }] }], - }; - const embedRequest: EmbedContentParameters = { model: 'qwen-turbo', contents: [{ parts: [{ text: 'Embed' }] }], @@ -1696,10 +1661,6 @@ describe('QwenContentGenerator', () => { 'Failed to obtain valid Qwen access token', ); - // countTokens should succeed as it's a local operation - const countResult = await newGenerator.countTokens(countRequest); - expect(countResult.totalTokens).toBe(15); - SharedTokenManager.getInstance = originalGetInstance; }); }); diff --git a/packages/core/src/qwen/qwenContentGenerator.ts b/packages/core/src/qwen/qwenContentGenerator.ts index 6d67fabf26d..25af0ef1c85 100644 --- a/packages/core/src/qwen/qwenContentGenerator.ts +++ b/packages/core/src/qwen/qwenContentGenerator.ts @@ -12,8 +12,6 @@ import { type Config } from '../config/config.js'; import type { GenerateContentParameters, GenerateContentResponse, - CountTokensParameters, - CountTokensResponse, EmbedContentParameters, EmbedContentResponse, } from '@google/genai'; @@ -173,15 +171,6 @@ export class QwenContentGenerator extends OpenAIContentGenerator { ); } - /** - * Override to use dynamic token and endpoint with automatic retry - */ - override async countTokens( - request: CountTokensParameters, - ): Promise { - return super.countTokens(request); - } - /** * Override to use dynamic token and endpoint with automatic retry */ diff --git a/packages/core/src/utils/nextSpeakerChecker.test.ts b/packages/core/src/utils/nextSpeakerChecker.test.ts index 451f38ee948..89f7ea66a90 100644 --- a/packages/core/src/utils/nextSpeakerChecker.test.ts +++ b/packages/core/src/utils/nextSpeakerChecker.test.ts @@ -59,9 +59,7 @@ describe('checkNextSpeaker', () => { { generateContent: vi.fn(), generateContentStream: vi.fn(), - countTokens: vi.fn(), embedContent: vi.fn(), - useSummarizedThinking: vi.fn().mockReturnValue(false), } as ContentGenerator, {} as Config, ); diff --git a/packages/core/src/utils/request-tokenizer/imageTokenizer.test.ts b/packages/core/src/utils/request-tokenizer/imageTokenizer.test.ts deleted file mode 100644 index 3e9724375eb..00000000000 --- a/packages/core/src/utils/request-tokenizer/imageTokenizer.test.ts +++ /dev/null @@ -1,337 +0,0 @@ -/** - * @license - * Copyright 2025 Qwen - * SPDX-License-Identifier: Apache-2.0 - */ - -import { describe, it, expect } from 'vitest'; -import { ImageTokenizer } from './imageTokenizer.js'; - -describe('ImageTokenizer', () => { - const tokenizer = new ImageTokenizer(); - - describe('token calculation', () => { - it('should calculate tokens based on image dimensions with reference logic', () => { - const metadata = { - width: 28, - height: 28, - mimeType: 'image/png', - dataSize: 1000, - }; - - const tokens = tokenizer.calculateTokens(metadata); - - // 28x28 = 784 pixels = 1 image token + 2 special tokens = 3 total - // But minimum scaling may apply for small images - expect(tokens).toBeGreaterThanOrEqual(6); // Minimum after scaling + special tokens - }); - - it('should calculate tokens for larger images', () => { - const metadata = { - width: 512, - height: 512, - mimeType: 'image/png', - dataSize: 10000, - }; - - const tokens = tokenizer.calculateTokens(metadata); - - // 512x512 with reference logic: rounded dimensions + scaling + special tokens - expect(tokens).toBeGreaterThan(300); - expect(tokens).toBeLessThan(400); // Should be reasonable for 512x512 - }); - - it('should enforce minimum tokens per image with scaling', () => { - const metadata = { - width: 1, - height: 1, - mimeType: 'image/png', - dataSize: 100, - }; - - const tokens = tokenizer.calculateTokens(metadata); - - // Tiny images get scaled up to minimum pixels + special tokens - expect(tokens).toBeGreaterThanOrEqual(6); // 4 image tokens + 2 special tokens - }); - - it('should handle very large images with scaling', () => { - const metadata = { - width: 8192, - height: 8192, - mimeType: 'image/png', - dataSize: 100000, - }; - - const tokens = tokenizer.calculateTokens(metadata); - - // Very large images should be scaled down to max limit + special tokens - expect(tokens).toBeLessThanOrEqual(16386); // 16384 max + 2 special tokens - expect(tokens).toBeGreaterThan(16000); // Should be close to the limit - }); - }); - - describe('PNG dimension extraction', () => { - it('should extract dimensions from valid PNG', async () => { - // 1x1 PNG image in base64 - const pngBase64 = - 'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNkYPhfDwAChAI9jU77yQAAAABJRU5ErkJggg=='; - - const metadata = await tokenizer.extractImageMetadata( - pngBase64, - 'image/png', - ); - - expect(metadata.width).toBe(1); - expect(metadata.height).toBe(1); - expect(metadata.mimeType).toBe('image/png'); - }); - - it('should handle invalid PNG gracefully', async () => { - const invalidBase64 = 'invalid-png-data'; - - const metadata = await tokenizer.extractImageMetadata( - invalidBase64, - 'image/png', - ); - - // Should return default dimensions - expect(metadata.width).toBe(512); - expect(metadata.height).toBe(512); - expect(metadata.mimeType).toBe('image/png'); - }); - }); - - describe('WebP dimension extraction', () => { - function buildWebp(format: string, totalLength: number): Buffer { - const buf = Buffer.alloc(totalLength); - buf.write('RIFF', 0, 'ascii'); - buf.writeUInt32LE(totalLength - 8, 4); - buf.write('WEBP', 8, 'ascii'); - buf.write(format, 12, 'ascii'); - buf.writeUInt32LE(Math.max(totalLength - 20, 0), 16); - return buf; - } - - function buildVp8lWebp( - width: number, - height: number, - signature = 0x2f, - ): Buffer { - const buf = buildWebp('VP8L', 26); - buf.writeUInt32LE(5, 16); - buf.writeUInt8(signature, 20); - buf.writeUInt32LE((width - 1) | ((height - 1) << 14), 21); - return buf; - } - - it('should extract dimensions from a short VP8L lossless WebP', async () => { - const width = 17; - const height = 13; - - const buf = buildVp8lWebp(width, height); - - const metadata = await tokenizer.extractImageMetadata( - buf.toString('base64'), - 'image/webp', - ); - - expect(metadata.width).toBe(width); - expect(metadata.height).toBe(height); - }); - - it('should fall back for VP8L with an invalid lossless signature', async () => { - const buf = buildVp8lWebp(17, 13, 0x00); - - const metadata = await tokenizer.extractImageMetadata( - buf.toString('base64'), - 'image/webp', - ); - - expect(metadata.width).toBe(512); - expect(metadata.height).toBe(512); - }); - - it('should still reject short VP8 and VP8X WebP files', async () => { - const formats = ['VP8 ', 'VP8X']; - - for (const format of formats) { - const metadata = await tokenizer.extractImageMetadata( - buildWebp(format, 26).toString('base64'), - 'image/webp', - ); - - expect(metadata.width).toBe(512); - expect(metadata.height).toBe(512); - } - }); - - it('should extract canvas dimensions from VP8X', async () => { - const width = 100; - const height = 80; - - const buf = Buffer.alloc(30); - buf.write('RIFF', 0, 'ascii'); - buf.writeUInt32LE(22, 4); - buf.write('WEBP', 8, 'ascii'); - buf.write('VP8X', 12, 'ascii'); - buf.writeUInt32LE(10, 16); // VP8X chunk size - buf.writeUInt8(0, 20); // flags - buf.writeUIntLE(width - 1, 24, 3); // canvas width minus one (24-bit LE) - buf.writeUIntLE(height - 1, 27, 3); // canvas height minus one (24-bit LE) - - const metadata = await tokenizer.extractImageMetadata( - buf.toString('base64'), - 'image/webp', - ); - - expect(metadata.width).toBe(width); - expect(metadata.height).toBe(height); - }); - }); - - describe('batch processing', () => { - it('should process multiple images serially', async () => { - const pngBase64 = - 'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNkYPhfDwAChAI9jU77yQAAAABJRU5ErkJggg=='; - - const images = [ - { data: pngBase64, mimeType: 'image/png' }, - { data: pngBase64, mimeType: 'image/png' }, - { data: pngBase64, mimeType: 'image/png' }, - ]; - - const tokens = await tokenizer.calculateTokensBatch(images); - - expect(tokens).toHaveLength(3); - expect(tokens.every((t) => t >= 4)).toBe(true); // All should have at least 4 tokens - }); - - it('should handle mixed valid and invalid images', async () => { - const validPng = - 'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNkYPhfDwAChAI9jU77yQAAAABJRU5ErkJggg=='; - const invalidPng = 'invalid-data'; - - const images = [ - { data: validPng, mimeType: 'image/png' }, - { data: invalidPng, mimeType: 'image/png' }, - ]; - - const tokens = await tokenizer.calculateTokensBatch(images); - - expect(tokens).toHaveLength(2); - expect(tokens.every((t) => t >= 4)).toBe(true); // All should have at least minimum tokens - }); - }); - - describe('different image formats', () => { - it('should handle different MIME types', async () => { - const pngBase64 = - 'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNkYPhfDwAChAI9jU77yQAAAABJRU5ErkJggg=='; - - const formats = ['image/png', 'image/jpeg', 'image/webp', 'image/gif']; - - for (const mimeType of formats) { - const metadata = await tokenizer.extractImageMetadata( - pngBase64, - mimeType, - ); - expect(metadata.mimeType).toBe(mimeType); - expect(metadata.width).toBeGreaterThan(0); - expect(metadata.height).toBeGreaterThan(0); - } - }); - - it('should extract dimensions from GIF images', async () => { - const buf = Buffer.alloc(10); - buf.write('GIF89a', 0, 'ascii'); - buf.writeUInt16LE(2, 6); - buf.writeUInt16LE(3, 8); - - const metadata = await tokenizer.extractImageMetadata( - buf.toString('base64'), - 'image/gif', - ); - - expect(metadata.width).toBe(2); - expect(metadata.height).toBe(3); - expect(metadata.mimeType).toBe('image/gif'); - }); - }); - - describe('TIFF dimension extraction', () => { - // Build a minimal single-IFD TIFF whose ImageWidth/ImageLength are stored - // as SHORT (type 3) -- the most common TIFF layout for small dimensions. - function buildShortTiff( - byteOrder: 'II' | 'MM', - width: number, - height: number, - ): string { - const buf = Buffer.alloc(38); - const le = byteOrder === 'II'; - buf.write(byteOrder, 0, 'ascii'); - const w16 = (off: number, v: number) => - le ? buf.writeUInt16LE(v, off) : buf.writeUInt16BE(v, off); - const w32 = (off: number, v: number) => - le ? buf.writeUInt32LE(v, off) : buf.writeUInt32BE(v, off); - w16(2, 42); // magic - w32(4, 8); // IFD starts at offset 8 - w16(8, 2); // two directory entries - // entry 0: ImageWidth (0x0100), SHORT, count 1, value left-justified - w16(10, 0x0100); - w16(12, 3); - w32(14, 1); - w16(18, width); - // entry 1: ImageLength (0x0101), SHORT, count 1 - w16(22, 0x0101); - w16(24, 3); - w32(26, 1); - w16(30, height); - w32(34, 0); // next-IFD offset - return buf.toString('base64'); - } - - it('reads SHORT dimensions from a big-endian (MM) TIFF', async () => { - const tiff = buildShortTiff('MM', 800, 600); - const metadata = await tokenizer.extractImageMetadata(tiff, 'image/tiff'); - expect(metadata.width).toBe(800); - expect(metadata.height).toBe(600); - }); - - it('reads SHORT dimensions from a little-endian (II) TIFF', async () => { - const tiff = buildShortTiff('II', 1024, 768); - const metadata = await tokenizer.extractImageMetadata(tiff, 'image/tiff'); - expect(metadata.width).toBe(1024); - expect(metadata.height).toBe(768); - }); - }); - - describe('BMP dimension extraction', () => { - function buildBmp(width: number, height: number): string { - // A negative height encodes a top-down BMP per the BITMAPINFOHEADER spec. - const buf = Buffer.alloc(26); - buf.write('BM', 0, 'ascii'); - buf.writeUInt32LE(26, 2); // file size - buf.writeUInt32LE(0, 6); // reserved - buf.writeUInt32LE(26, 10); // pixel data offset - buf.writeUInt32LE(40, 14); // BITMAPINFOHEADER size - buf.writeInt32LE(width, 18); - buf.writeInt32LE(height, 22); - return buf.toString('base64'); - } - - it('reads a bottom-up (positive height) BMP', async () => { - const bmp = buildBmp(120, 80); - const metadata = await tokenizer.extractImageMetadata(bmp, 'image/bmp'); - expect(metadata.width).toBe(120); - expect(metadata.height).toBe(80); - }); - - it('reads a top-down (negative height) BMP as its absolute height', async () => { - const bmp = buildBmp(100, -50); - const metadata = await tokenizer.extractImageMetadata(bmp, 'image/bmp'); - expect(metadata.width).toBe(100); - expect(metadata.height).toBe(50); - }); - }); -}); diff --git a/packages/core/src/utils/request-tokenizer/imageTokenizer.ts b/packages/core/src/utils/request-tokenizer/imageTokenizer.ts deleted file mode 100644 index 15c2f00b16e..00000000000 --- a/packages/core/src/utils/request-tokenizer/imageTokenizer.ts +++ /dev/null @@ -1,534 +0,0 @@ -/** - * @license - * Copyright 2025 Qwen - * SPDX-License-Identifier: Apache-2.0 - */ - -import type { ImageMetadata } from './types.js'; -import { isSupportedImageMimeType } from './supportedImageFormats.js'; -import { createDebugLogger } from '../debugLogger.js'; - -const debugLogger = createDebugLogger('IMAGE_TOKENIZER'); - -/** - * Image tokenizer for calculating image tokens based on dimensions - * - * Key rules: - * - 28x28 pixels = 1 token - * - Minimum: 4 tokens per image - * - Maximum: 16384 tokens per image - * - Additional: 2 special tokens (vision_bos + vision_eos) - * - Supports: PNG, JPEG, WebP, GIF, BMP, TIFF, HEIC formats - */ -export class ImageTokenizer { - /** 28x28 pixels = 1 token */ - private static readonly PIXELS_PER_TOKEN = 28 * 28; - - /** Minimum tokens per image */ - private static readonly MIN_TOKENS_PER_IMAGE = 4; - - /** Maximum tokens per image */ - private static readonly MAX_TOKENS_PER_IMAGE = 16384; - - /** Special tokens for vision markers */ - private static readonly VISION_SPECIAL_TOKENS = 2; - - /** - * Extract image metadata from base64 data - * - * @param base64Data Base64-encoded image data (with or without data URL prefix) - * @param mimeType MIME type of the image - * @returns Promise resolving to ImageMetadata with dimensions and format info - */ - async extractImageMetadata( - base64Data: string, - mimeType: string, - ): Promise { - try { - // Check if the MIME type is supported - if (!isSupportedImageMimeType(mimeType)) { - debugLogger.warn(`Unsupported image format: ${mimeType}`); - // Return default metadata for unsupported formats - return { - width: 512, - height: 512, - mimeType, - dataSize: Math.floor(base64Data.length * 0.75), - }; - } - - const cleanBase64 = base64Data.replace(/^data:[^;]+;base64,/, ''); - const buffer = Buffer.from(cleanBase64, 'base64'); - const dimensions = await this.extractDimensions(buffer, mimeType); - - return { - width: dimensions.width, - height: dimensions.height, - mimeType, - dataSize: buffer.length, - }; - } catch (error) { - debugLogger.warn('Failed to extract image metadata:', error); - // Return default metadata for fallback - return { - width: 512, - height: 512, - mimeType, - dataSize: Math.floor(base64Data.length * 0.75), - }; - } - } - - /** - * Extract image dimensions from buffer based on format - * - * @param buffer Binary image data buffer - * @param mimeType MIME type to determine parsing strategy - * @returns Promise resolving to width and height dimensions - */ - private async extractDimensions( - buffer: Buffer, - mimeType: string, - ): Promise<{ width: number; height: number }> { - if (mimeType.includes('png')) { - return this.extractPngDimensions(buffer); - } - - if (mimeType.includes('jpeg') || mimeType.includes('jpg')) { - return this.extractJpegDimensions(buffer); - } - - if (mimeType.includes('webp')) { - return this.extractWebpDimensions(buffer); - } - - if (mimeType.includes('gif')) { - return this.extractGifDimensions(buffer); - } - - if (mimeType.includes('bmp')) { - return this.extractBmpDimensions(buffer); - } - - if (mimeType.includes('tiff')) { - return this.extractTiffDimensions(buffer); - } - - if (mimeType.includes('heic')) { - return this.extractHeicDimensions(buffer); - } - - return { width: 512, height: 512 }; - } - - /** - * Extract PNG dimensions from IHDR chunk - * PNG signature: 89 50 4E 47 0D 0A 1A 0A - * Width/height at bytes 16-19 and 20-23 (big-endian) - */ - private extractPngDimensions(buffer: Buffer): { - width: number; - height: number; - } { - if (buffer.length < 24) { - throw new Error('Invalid PNG: buffer too short'); - } - - // Verify PNG signature - const signature = buffer.subarray(0, 8); - const expectedSignature = Buffer.from([ - 0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a, - ]); - if (!signature.equals(expectedSignature)) { - throw new Error('Invalid PNG signature'); - } - - const width = buffer.readUInt32BE(16); - const height = buffer.readUInt32BE(20); - - return { width, height }; - } - - /** - * Extract JPEG dimensions from SOF (Start of Frame) markers - * JPEG starts with FF D8, SOF markers: 0xC0-0xC3, 0xC5-0xC7, 0xC9-0xCB, 0xCD-0xCF - * Dimensions at offset +5 (height) and +7 (width) from SOF marker - */ - private extractJpegDimensions(buffer: Buffer): { - width: number; - height: number; - } { - if (buffer.length < 4 || buffer[0] !== 0xff || buffer[1] !== 0xd8) { - throw new Error('Invalid JPEG signature'); - } - - let offset = 2; - - while (offset < buffer.length - 8) { - if (buffer[offset] !== 0xff) { - offset++; - continue; - } - - const marker = buffer[offset + 1]; - - // SOF markers - if ( - (marker >= 0xc0 && marker <= 0xc3) || - (marker >= 0xc5 && marker <= 0xc7) || - (marker >= 0xc9 && marker <= 0xcb) || - (marker >= 0xcd && marker <= 0xcf) - ) { - const height = buffer.readUInt16BE(offset + 5); - const width = buffer.readUInt16BE(offset + 7); - return { width, height }; - } - - const segmentLength = buffer.readUInt16BE(offset + 2); - offset += 2 + segmentLength; - } - - throw new Error('Could not find JPEG dimensions'); - } - - /** - * Extract WebP dimensions from RIFF container - * Supports VP8, VP8L, and VP8X formats - */ - private extractWebpDimensions(buffer: Buffer): { - width: number; - height: number; - } { - if (buffer.length < 16) { - throw new Error('Invalid WebP: too short'); - } - - const riffSignature = buffer.subarray(0, 4).toString('ascii'); - const webpSignature = buffer.subarray(8, 12).toString('ascii'); - - if (riffSignature !== 'RIFF' || webpSignature !== 'WEBP') { - throw new Error('Invalid WebP signature'); - } - - const format = buffer.subarray(12, 16).toString('ascii'); - - if (format === 'VP8 ') { - if (buffer.length < 30) { - throw new Error('Invalid VP8 WebP: too short'); - } - // Lossy: 14-bit width/height at bytes 26-27 and 28-29 (little-endian) - const width = buffer.readUInt16LE(26) & 0x3fff; - const height = buffer.readUInt16LE(28) & 0x3fff; - return { width, height }; - } else if (format === 'VP8L') { - if (buffer.length < 25) { - throw new Error('Invalid VP8L WebP: too short'); - } - if (buffer[20] !== 0x2f) { - throw new Error('Invalid VP8L WebP signature'); - } - // Lossless: 14-bit (width-1) then (height-1) packed from byte 21 (little-endian) - const bits = buffer.readUInt32LE(21); - const width = (bits & 0x3fff) + 1; - const height = ((bits >> 14) & 0x3fff) + 1; - return { width, height }; - } else if (format === 'VP8X') { - if (buffer.length < 30) { - throw new Error('Invalid VP8X WebP: too short'); - } - // Extended: 24-bit (canvas width-1) at bytes 24-26 and (height-1) at bytes 27-29 (little-endian) - const width = buffer.readUIntLE(24, 3) + 1; - const height = buffer.readUIntLE(27, 3) + 1; - return { width, height }; - } - - throw new Error('Unsupported WebP format'); - } - - /** - * Extract GIF dimensions from header - * Supports GIF87a and GIF89a formats - */ - private extractGifDimensions(buffer: Buffer): { - width: number; - height: number; - } { - if (buffer.length < 10) { - throw new Error('Invalid GIF: too short'); - } - - const signature = buffer.subarray(0, 6).toString('ascii'); - if (signature !== 'GIF87a' && signature !== 'GIF89a') { - throw new Error('Invalid GIF signature'); - } - - const width = buffer.readUInt16LE(6); - const height = buffer.readUInt16LE(8); - - return { width, height }; - } - - /** - * Calculate tokens for an image based on its metadata - * - * @param metadata Image metadata containing width, height, and format info - * @returns Total token count including base image tokens and special tokens - */ - calculateTokens(metadata: ImageMetadata): number { - return this.calculateTokensWithScaling(metadata.width, metadata.height); - } - - /** - * Calculate tokens with scaling logic - * - * Steps: - * 1. Normalize to 28-pixel multiples - * 2. Scale large images down, small images up - * 3. Calculate tokens: pixels / 784 + 2 special tokens - * - * @param width Original image width in pixels - * @param height Original image height in pixels - * @returns Total token count for the image - */ - private calculateTokensWithScaling(width: number, height: number): number { - // Normalize to 28-pixel multiples - let hBar = Math.round(height / 28) * 28; - let wBar = Math.round(width / 28) * 28; - - // Define pixel boundaries - const minPixels = - ImageTokenizer.MIN_TOKENS_PER_IMAGE * ImageTokenizer.PIXELS_PER_TOKEN; - const maxPixels = - ImageTokenizer.MAX_TOKENS_PER_IMAGE * ImageTokenizer.PIXELS_PER_TOKEN; - - // Apply scaling - if (hBar * wBar > maxPixels) { - // Scale down large images - const beta = Math.sqrt((height * width) / maxPixels); - hBar = Math.floor(height / beta / 28) * 28; - wBar = Math.floor(width / beta / 28) * 28; - } else if (hBar * wBar < minPixels) { - // Scale up small images - const beta = Math.sqrt(minPixels / (height * width)); - hBar = Math.ceil((height * beta) / 28) * 28; - wBar = Math.ceil((width * beta) / 28) * 28; - } - - // Calculate tokens - const imageTokens = Math.floor( - (hBar * wBar) / ImageTokenizer.PIXELS_PER_TOKEN, - ); - - return imageTokens + ImageTokenizer.VISION_SPECIAL_TOKENS; - } - - /** - * Calculate tokens for multiple images serially - * - * @param base64DataArray Array of image data with MIME type information - * @returns Promise resolving to array of token counts in same order as input - */ - async calculateTokensBatch( - base64DataArray: Array<{ data: string; mimeType: string }>, - ): Promise { - const results: number[] = []; - - for (const { data, mimeType } of base64DataArray) { - try { - const metadata = await this.extractImageMetadata(data, mimeType); - results.push(this.calculateTokens(metadata)); - } catch (error) { - debugLogger.warn('Error calculating tokens for image:', error); - // Return minimum tokens as fallback - results.push( - ImageTokenizer.MIN_TOKENS_PER_IMAGE + - ImageTokenizer.VISION_SPECIAL_TOKENS, - ); - } - } - - return results; - } - - /** - * Extract BMP dimensions from header - * BMP signature: 42 4D (BM) - * Width/height at bytes 18-21 and 22-25 (little-endian) - */ - private extractBmpDimensions(buffer: Buffer): { - width: number; - height: number; - } { - if (buffer.length < 26) { - throw new Error('Invalid BMP: buffer too short'); - } - - // Verify BMP signature - if (buffer[0] !== 0x42 || buffer[1] !== 0x4d) { - throw new Error('Invalid BMP signature'); - } - - const width = buffer.readUInt32LE(18); - // Height is a signed int32: a negative value means a top-down BMP. Read it - // signed so Math.abs recovers the real height instead of a ~4-billion value. - const height = buffer.readInt32LE(22); - - return { width, height: Math.abs(height) }; // Height can be negative for top-down BMPs - } - - /** - * Extract TIFF dimensions from IFD (Image File Directory) - * TIFF can be little-endian (II) or big-endian (MM) - * Width/height are stored in IFD entries with tags 0x0100 and 0x0101 - */ - private extractTiffDimensions(buffer: Buffer): { - width: number; - height: number; - } { - if (buffer.length < 8) { - throw new Error('Invalid TIFF: buffer too short'); - } - - // Check byte order - const byteOrder = buffer.subarray(0, 2).toString('ascii'); - const isLittleEndian = byteOrder === 'II'; - const isBigEndian = byteOrder === 'MM'; - - if (!isLittleEndian && !isBigEndian) { - throw new Error('Invalid TIFF byte order'); - } - - // Read magic number (should be 42) - const magic = isLittleEndian - ? buffer.readUInt16LE(2) - : buffer.readUInt16BE(2); - if (magic !== 42) { - throw new Error('Invalid TIFF magic number'); - } - - // Read IFD offset - const ifdOffset = isLittleEndian - ? buffer.readUInt32LE(4) - : buffer.readUInt32BE(4); - - if (ifdOffset >= buffer.length) { - throw new Error('Invalid TIFF IFD offset'); - } - - // Read number of directory entries - const numEntries = isLittleEndian - ? buffer.readUInt16LE(ifdOffset) - : buffer.readUInt16BE(ifdOffset); - - let width = 0; - let height = 0; - - // Parse IFD entries - for (let i = 0; i < numEntries; i++) { - const entryOffset = ifdOffset + 2 + i * 12; - - if (entryOffset + 12 > buffer.length) break; - - const tag = isLittleEndian - ? buffer.readUInt16LE(entryOffset) - : buffer.readUInt16BE(entryOffset); - - const type = isLittleEndian - ? buffer.readUInt16LE(entryOffset + 2) - : buffer.readUInt16BE(entryOffset + 2); - - // A SHORT (type 3) value occupies only the first two bytes of the - // 4-byte value field. Reading it as a 32-bit int happens to work on - // little-endian (II) files but yields value << 16 on big-endian (MM) - // ones, so read it according to the field type. - const value = - type === 3 - ? isLittleEndian - ? buffer.readUInt16LE(entryOffset + 8) - : buffer.readUInt16BE(entryOffset + 8) - : isLittleEndian - ? buffer.readUInt32LE(entryOffset + 8) - : buffer.readUInt32BE(entryOffset + 8); - - if (tag === 0x0100) { - // ImageWidth - width = value; - } else if (tag === 0x0101) { - // ImageLength (height) - height = value; - } - - if (width > 0 && height > 0) break; - } - - if (width === 0 || height === 0) { - throw new Error('Could not find TIFF dimensions'); - } - - return { width, height }; - } - - /** - * Extract HEIC dimensions from meta box - * HEIC is based on ISO Base Media File Format - * This is a simplified implementation that looks for 'ispe' (Image Spatial Extents) box - */ - private extractHeicDimensions(buffer: Buffer): { - width: number; - height: number; - } { - if (buffer.length < 12) { - throw new Error('Invalid HEIC: buffer too short'); - } - - // Check for ftyp box with HEIC brand - const ftypBox = buffer.subarray(4, 8).toString('ascii'); - if (ftypBox !== 'ftyp') { - throw new Error('Invalid HEIC: missing ftyp box'); - } - - const brand = buffer.subarray(8, 12).toString('ascii'); - if (!['heic', 'heix', 'hevc', 'hevx'].includes(brand)) { - throw new Error('Invalid HEIC brand'); - } - - // Look for meta box and then ispe box - let offset = 0; - while (offset < buffer.length - 8) { - const boxSize = buffer.readUInt32BE(offset); - const boxType = buffer.subarray(offset + 4, offset + 8).toString('ascii'); - - if (boxType === 'meta') { - // Look for ispe box inside meta box - const metaOffset = offset + 8; - let innerOffset = metaOffset + 4; // Skip version and flags - - while (innerOffset < offset + boxSize - 8) { - const innerBoxSize = buffer.readUInt32BE(innerOffset); - const innerBoxType = buffer - .subarray(innerOffset + 4, innerOffset + 8) - .toString('ascii'); - - if (innerBoxType === 'ispe') { - // Found Image Spatial Extents box - if (innerOffset + 20 <= buffer.length) { - const width = buffer.readUInt32BE(innerOffset + 12); - const height = buffer.readUInt32BE(innerOffset + 16); - return { width, height }; - } - } - - if (innerBoxSize === 0) break; - innerOffset += innerBoxSize; - } - } - - if (boxSize === 0) break; - offset += boxSize; - } - - // Fallback: return default dimensions if we can't parse the structure - debugLogger.warn('Could not extract HEIC dimensions, using default'); - return { width: 512, height: 512 }; - } -} diff --git a/packages/core/src/utils/request-tokenizer/index.ts b/packages/core/src/utils/request-tokenizer/index.ts deleted file mode 100644 index dc16c65f641..00000000000 --- a/packages/core/src/utils/request-tokenizer/index.ts +++ /dev/null @@ -1,11 +0,0 @@ -/** - * @license - * Copyright 2025 Qwen - * SPDX-License-Identifier: Apache-2.0 - */ - -export { RequestTokenizer as RequestTokenEstimator } from './requestTokenizer.js'; -export { TextTokenizer } from './textTokenizer.js'; -export { ImageTokenizer } from './imageTokenizer.js'; - -export type { TokenCalculationResult, ImageMetadata } from './types.js'; diff --git a/packages/core/src/utils/request-tokenizer/requestTokenizer.test.ts b/packages/core/src/utils/request-tokenizer/requestTokenizer.test.ts deleted file mode 100644 index 83b4560391c..00000000000 --- a/packages/core/src/utils/request-tokenizer/requestTokenizer.test.ts +++ /dev/null @@ -1,271 +0,0 @@ -/** - * @license - * Copyright 2025 Qwen - * SPDX-License-Identifier: Apache-2.0 - */ - -import { describe, it, expect, beforeEach } from 'vitest'; -import { RequestTokenizer } from './requestTokenizer.js'; -import type { CountTokensParameters } from '@google/genai'; - -describe('RequestTokenEstimator', () => { - let tokenizer: RequestTokenizer; - - beforeEach(() => { - tokenizer = new RequestTokenizer(); - }); - - describe('text token calculation', () => { - it('should calculate tokens for simple text content', async () => { - const request: CountTokensParameters = { - model: 'test-model', - contents: [ - { - role: 'user', - parts: [{ text: 'Hello, world!' }], - }, - ], - }; - - const result = await tokenizer.calculateTokens(request); - - expect(result.totalTokens).toBeGreaterThan(0); - expect(result.breakdown.textTokens).toBeGreaterThan(0); - expect(result.breakdown.imageTokens).toBe(0); - expect(result.processingTime).toBeGreaterThan(0); - }); - - it('should handle multiple text parts', async () => { - const request: CountTokensParameters = { - model: 'test-model', - contents: [ - { - role: 'user', - parts: [ - { text: 'First part' }, - { text: 'Second part' }, - { text: 'Third part' }, - ], - }, - ], - }; - - const result = await tokenizer.calculateTokens(request); - - expect(result.totalTokens).toBeGreaterThan(0); - expect(result.breakdown.textTokens).toBeGreaterThan(0); - }); - - it('should handle string content', async () => { - const request: CountTokensParameters = { - model: 'test-model', - contents: ['Simple string content'], - }; - - const result = await tokenizer.calculateTokens(request); - - expect(result.totalTokens).toBeGreaterThan(0); - expect(result.breakdown.textTokens).toBeGreaterThan(0); - }); - }); - - describe('image token calculation', () => { - it('should calculate tokens for image content', async () => { - // Create a simple 1x1 PNG image in base64 - const pngBase64 = - 'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNkYPhfDwAChAI9jU77yQAAAABJRU5ErkJggg=='; - - const request: CountTokensParameters = { - model: 'test-model', - contents: [ - { - role: 'user', - parts: [ - { - inlineData: { - mimeType: 'image/png', - data: pngBase64, - }, - }, - ], - }, - ], - }; - - const result = await tokenizer.calculateTokens(request); - - expect(result.totalTokens).toBeGreaterThanOrEqual(4); // Minimum 4 tokens per image - expect(result.breakdown.imageTokens).toBeGreaterThanOrEqual(4); - expect(result.breakdown.textTokens).toBe(0); - }); - - it('should handle multiple images', async () => { - const pngBase64 = - 'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNkYPhfDwAChAI9jU77yQAAAABJRU5ErkJggg=='; - - const request: CountTokensParameters = { - model: 'test-model', - contents: [ - { - role: 'user', - parts: [ - { - inlineData: { - mimeType: 'image/png', - data: pngBase64, - }, - }, - { - inlineData: { - mimeType: 'image/png', - data: pngBase64, - }, - }, - ], - }, - ], - }; - - const result = await tokenizer.calculateTokens(request); - - expect(result.totalTokens).toBeGreaterThanOrEqual(8); // At least 4 tokens per image - expect(result.breakdown.imageTokens).toBeGreaterThanOrEqual(8); - }); - }); - - describe('mixed content', () => { - it('should handle text and image content together', async () => { - const pngBase64 = - 'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNkYPhfDwAChAI9jU77yQAAAABJRU5ErkJggg=='; - - const request: CountTokensParameters = { - model: 'test-model', - contents: [ - { - role: 'user', - parts: [ - { text: 'Here is an image:' }, - { - inlineData: { - mimeType: 'image/png', - data: pngBase64, - }, - }, - { text: 'What do you see?' }, - ], - }, - ], - }; - - const result = await tokenizer.calculateTokens(request); - - expect(result.totalTokens).toBeGreaterThan(4); - expect(result.breakdown.textTokens).toBeGreaterThan(0); - expect(result.breakdown.imageTokens).toBeGreaterThanOrEqual(4); - }); - }); - - describe('function content', () => { - it('should handle function calls', async () => { - const request: CountTokensParameters = { - model: 'test-model', - contents: [ - { - role: 'user', - parts: [ - { - functionCall: { - name: 'test_function', - args: { param1: 'value1', param2: 42 }, - }, - }, - ], - }, - ], - }; - - const result = await tokenizer.calculateTokens(request); - - expect(result.totalTokens).toBeGreaterThan(0); - expect(result.breakdown.otherTokens).toBeGreaterThan(0); - }); - }); - - describe('empty content', () => { - it('should handle empty request', async () => { - const request: CountTokensParameters = { - model: 'test-model', - contents: [], - }; - - const result = await tokenizer.calculateTokens(request); - - expect(result.totalTokens).toBe(0); - expect(result.breakdown.textTokens).toBe(0); - expect(result.breakdown.imageTokens).toBe(0); - }); - - it('should handle undefined contents', async () => { - const request: CountTokensParameters = { - model: 'test-model', - contents: [], - }; - - const result = await tokenizer.calculateTokens(request); - - expect(result.totalTokens).toBe(0); - }); - }); - - describe('images', () => { - it('should process multiple images serially', async () => { - const pngBase64 = - 'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNkYPhfDwAChAI9jU77yQAAAABJRU5ErkJggg=='; - - const request: CountTokensParameters = { - model: 'test-model', - contents: [ - { - role: 'user', - parts: Array(10).fill({ - inlineData: { - mimeType: 'image/png', - data: pngBase64, - }, - }), - }, - ], - }; - - const result = await tokenizer.calculateTokens(request); - - expect(result.totalTokens).toBeGreaterThanOrEqual(60); // At least 6 tokens per image * 10 images - }); - }); - - describe('error handling', () => { - it('should handle malformed image data gracefully', async () => { - const request: CountTokensParameters = { - model: 'test-model', - contents: [ - { - role: 'user', - parts: [ - { - inlineData: { - mimeType: 'image/png', - data: 'invalid-base64-data', - }, - }, - ], - }, - ], - }; - - const result = await tokenizer.calculateTokens(request); - - // Should still return some tokens (fallback to minimum) - expect(result.totalTokens).toBeGreaterThanOrEqual(4); - }); - }); -}); diff --git a/packages/core/src/utils/request-tokenizer/requestTokenizer.ts b/packages/core/src/utils/request-tokenizer/requestTokenizer.ts deleted file mode 100644 index 95ebd9a6709..00000000000 --- a/packages/core/src/utils/request-tokenizer/requestTokenizer.ts +++ /dev/null @@ -1,330 +0,0 @@ -/** - * @license - * Copyright 2025 Qwen - * SPDX-License-Identifier: Apache-2.0 - */ - -import type { - CountTokensParameters, - Content, - Part, - PartUnion, -} from '@google/genai'; -import type { TokenCalculationResult } from './types.js'; -import { TextTokenizer } from './textTokenizer.js'; -import { ImageTokenizer } from './imageTokenizer.js'; -import { createDebugLogger } from '../debugLogger.js'; - -const debugLogger = createDebugLogger('TOKENIZER'); - -/** - * Simple request token estimator that handles text and image content serially - */ -export class RequestTokenizer { - private textTokenizer: TextTokenizer; - private imageTokenizer: ImageTokenizer; - - constructor() { - this.textTokenizer = new TextTokenizer(); - this.imageTokenizer = new ImageTokenizer(); - } - - /** - * Calculate tokens for a request using serial processing - */ - async calculateTokens( - request: CountTokensParameters, - ): Promise { - const startTime = performance.now(); - - try { - // Process request content and group by type - const { textContents, imageContents, audioContents, otherContents } = - this.processAndGroupContents(request); - - if ( - textContents.length === 0 && - imageContents.length === 0 && - audioContents.length === 0 && - otherContents.length === 0 - ) { - return { - totalTokens: 0, - breakdown: { - textTokens: 0, - imageTokens: 0, - audioTokens: 0, - otherTokens: 0, - }, - processingTime: performance.now() - startTime, - }; - } - - // Calculate tokens for each content type serially - const textTokens = await this.calculateTextTokens(textContents); - const imageTokens = await this.calculateImageTokens(imageContents); - const audioTokens = await this.calculateAudioTokens(audioContents); - const otherTokens = await this.calculateOtherTokens(otherContents); - - const totalTokens = textTokens + imageTokens + audioTokens + otherTokens; - const processingTime = performance.now() - startTime; - - return { - totalTokens, - breakdown: { - textTokens, - imageTokens, - audioTokens, - otherTokens, - }, - processingTime, - }; - } catch (error) { - debugLogger.error('Error calculating tokens:', error); - - // Fallback calculation - const fallbackTokens = this.calculateFallbackTokens(request); - - return { - totalTokens: fallbackTokens, - breakdown: { - textTokens: fallbackTokens, - imageTokens: 0, - audioTokens: 0, - otherTokens: 0, - }, - processingTime: performance.now() - startTime, - }; - } - } - - /** - * Calculate tokens for text contents - */ - private async calculateTextTokens(textContents: string[]): Promise { - if (textContents.length === 0) return 0; - - try { - // Avoid per-part rounding inflation by estimating once on the combined text. - return await this.textTokenizer.calculateTokens(textContents.join('')); - } catch (error) { - debugLogger.warn('Error calculating text tokens:', error); - // Fallback: character-based estimation - const totalChars = textContents.join('').length; - return Math.ceil(totalChars / 4); - } - } - - /** - * Calculate tokens for image contents using serial processing - */ - private async calculateImageTokens( - imageContents: Array<{ data: string; mimeType: string }>, - ): Promise { - if (imageContents.length === 0) return 0; - - try { - const tokenCounts = - await this.imageTokenizer.calculateTokensBatch(imageContents); - return tokenCounts.reduce((sum, count) => sum + count, 0); - } catch (error) { - debugLogger.warn('Error calculating image tokens:', error); - // Fallback: minimum tokens per image - return imageContents.length * 6; // 4 image tokens + 2 special tokens as minimum - } - } - - /** - * Calculate tokens for audio contents - * TODO: Implement proper audio token calculation - */ - private async calculateAudioTokens( - audioContents: Array<{ data: string; mimeType: string }>, - ): Promise { - if (audioContents.length === 0) return 0; - - // Placeholder implementation - audio token calculation would depend on - // the specific model's audio processing capabilities - // For now, estimate based on data size - let totalTokens = 0; - - for (const audioContent of audioContents) { - try { - const dataSize = Math.floor(audioContent.data.length * 0.75); // Approximate binary size - // Rough estimate: 1 token per 100 bytes of audio data - totalTokens += Math.max(Math.ceil(dataSize / 100), 10); // Minimum 10 tokens per audio - } catch (error) { - debugLogger.warn('Error calculating audio tokens:', error); - totalTokens += 10; // Fallback minimum - } - } - - return totalTokens; - } - - /** - * Calculate tokens for other content types (functions, files, etc.) - */ - private async calculateOtherTokens(otherContents: string[]): Promise { - if (otherContents.length === 0) return 0; - - try { - // Treat other content as text, and avoid per-item rounding inflation. - return await this.textTokenizer.calculateTokens(otherContents.join('')); - } catch (error) { - debugLogger.warn('Error calculating other content tokens:', error); - // Fallback: character-based estimation - const totalChars = otherContents.join('').length; - return Math.ceil(totalChars / 4); - } - } - - /** - * Fallback token calculation using simple string serialization - */ - private calculateFallbackTokens(request: CountTokensParameters): number { - try { - const content = JSON.stringify(request.contents); - return Math.ceil(content.length / 4); // Rough estimate: 1 token ≈ 4 characters - } catch (error) { - debugLogger.warn('Error in fallback token calculation:', error); - return 100; // Conservative fallback - } - } - - /** - * Process request contents and group by type - */ - private processAndGroupContents(request: CountTokensParameters): { - textContents: string[]; - imageContents: Array<{ data: string; mimeType: string }>; - audioContents: Array<{ data: string; mimeType: string }>; - otherContents: string[]; - } { - const textContents: string[] = []; - const imageContents: Array<{ data: string; mimeType: string }> = []; - const audioContents: Array<{ data: string; mimeType: string }> = []; - const otherContents: string[] = []; - - if (!request.contents) { - return { textContents, imageContents, audioContents, otherContents }; - } - - const contents = Array.isArray(request.contents) - ? request.contents - : [request.contents]; - - for (const content of contents) { - this.processContent( - content, - textContents, - imageContents, - audioContents, - otherContents, - ); - } - - return { textContents, imageContents, audioContents, otherContents }; - } - - /** - * Process a single content item and add to appropriate arrays - */ - private processContent( - content: Content | string | PartUnion, - textContents: string[], - imageContents: Array<{ data: string; mimeType: string }>, - audioContents: Array<{ data: string; mimeType: string }>, - otherContents: string[], - ): void { - if (typeof content === 'string') { - if (content.trim()) { - textContents.push(content); - } - return; - } - - if ('parts' in content && content.parts) { - for (const part of content.parts) { - this.processPart( - part, - textContents, - imageContents, - audioContents, - otherContents, - ); - } - return; - } - - // Some request shapes (e.g. CountTokensParameters) allow passing parts directly - // instead of wrapping them in a { parts: [...] } Content object. - this.processPart( - content as Part | string, - textContents, - imageContents, - audioContents, - otherContents, - ); - } - - /** - * Process a single part and add to appropriate arrays - */ - private processPart( - part: Part | string, - textContents: string[], - imageContents: Array<{ data: string; mimeType: string }>, - audioContents: Array<{ data: string; mimeType: string }>, - otherContents: string[], - ): void { - if (typeof part === 'string') { - if (part.trim()) { - textContents.push(part); - } - return; - } - - if ('text' in part && part.text) { - textContents.push(part.text); - return; - } - - if ('inlineData' in part && part.inlineData) { - const { data, mimeType } = part.inlineData; - if (mimeType && mimeType.startsWith('image/')) { - imageContents.push({ data: data || '', mimeType }); - return; - } - if (mimeType && mimeType.startsWith('audio/')) { - audioContents.push({ data: data || '', mimeType }); - return; - } - } - - if ('fileData' in part && part.fileData) { - otherContents.push(JSON.stringify(part.fileData)); - return; - } - - if ('functionCall' in part && part.functionCall) { - otherContents.push(JSON.stringify(part.functionCall)); - return; - } - - if ('functionResponse' in part && part.functionResponse) { - otherContents.push(JSON.stringify(part.functionResponse)); - return; - } - - // Unknown part type - try to serialize - try { - const serialized = JSON.stringify(part); - if (serialized && serialized !== '{}') { - otherContents.push(serialized); - } - } catch (error) { - debugLogger.warn('Failed to serialize unknown part type:', error); - } - } -} diff --git a/packages/core/src/utils/request-tokenizer/supportedImageFormats.ts b/packages/core/src/utils/request-tokenizer/supportedImageFormats.ts index 3c7d0ede39d..9ae749c241a 100644 --- a/packages/core/src/utils/request-tokenizer/supportedImageFormats.ts +++ b/packages/core/src/utils/request-tokenizer/supportedImageFormats.ts @@ -5,11 +5,15 @@ */ /** - * Image MIME types the image tokenizer can decode for metadata extraction. - * This is a capability list, not an acceptance contract: the file-read path - * only forwards a narrower set to model endpoints (see - * PROVIDER_SAFE_IMAGE_MIME_TYPES in fileUtils.ts and #9291), so some types - * here are omitted from requests before they ever reach the tokenizer. + * Image MIME types accepted for vision input (attachment/thumbnail paths). + * This is an acceptance list for inputs, not a decode-capability list: + * token accounting uses the flat DEFAULT_IMAGE_TOKEN_ESTIMATE in + * compactionInputSlimming.ts — the former request-tokenizer estimator + * cluster, including ImageTokenizer and its dimension parsing, was removed + * as orphaned in PR #9676. The file-read path forwards only the narrower + * PIPELINE_IMAGE_MIME_TYPES subset to model endpoints (see + * PROVIDER_SAFE_IMAGE_MIME_TYPES in fileUtils.ts and #9291); anything else + * is omitted from requests with an in-band notice. */ export const SUPPORTED_IMAGE_MIME_TYPES = [ 'image/bmp', @@ -58,7 +62,7 @@ export function isSupportedImageMimeType( /** * Get a human-readable list of image formats the pipeline forwards to the - * model (not the tokenizer's wider decode capability). + * model (the narrower pipeline subset, not the full acceptance list above). * @returns Comma-separated string of forwarded formats */ export function getSupportedImageFormatsString(): string { diff --git a/packages/core/src/utils/request-tokenizer/types.ts b/packages/core/src/utils/request-tokenizer/types.ts deleted file mode 100644 index 21fc3ff9b13..00000000000 --- a/packages/core/src/utils/request-tokenizer/types.ts +++ /dev/null @@ -1,36 +0,0 @@ -/** - * @license - * Copyright 2025 Qwen - * SPDX-License-Identifier: Apache-2.0 - */ - -/** - * Token calculation result for different content types - */ -export interface TokenCalculationResult { - /** Total tokens calculated */ - totalTokens: number; - /** Breakdown by content type */ - breakdown: { - textTokens: number; - imageTokens: number; - audioTokens: number; - otherTokens: number; - }; - /** Processing time in milliseconds */ - processingTime: number; -} - -/** - * Image metadata extracted from base64 data - */ -export interface ImageMetadata { - /** Image width in pixels */ - width: number; - /** Image height in pixels */ - height: number; - /** MIME type of the image */ - mimeType: string; - /** Size of the base64 data in bytes */ - dataSize: number; -}