Skip to content

perf(core): add pure-ASCII fast path to text token estimation #6551

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
wenshao merged 1 commit into from
Jul 8, 2026
Merged

perf(core): add pure-ASCII fast path to text token estimation #6551

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions packages/core/src/utils/request-tokenizer/textTokenizer.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -262,6 +262,37 @@ describe('TextTokenizer', () => {
});
});

describe('ASCII/non-ASCII boundary', () => {
it('should treat DEL (U+007F) as ASCII and U+0080 as non-ASCII', async () => {
// '\x7F' = 1 ASCII char: 1 / 4 = 0.25 -> ceil = 1
expect(await tokenizer.calculateTokens('\x7F')).toBe(1);
// '\u0080' = 1 non-ASCII char: 1 * 1.1 = 1.1 -> ceil = 2
expect(await tokenizer.calculateTokens('\u0080')).toBe(2);
});

it('should count pure-ASCII text of any length as ceil(length / 4)', async () => {
for (const len of [1, 3, 4, 5, 4096, 4097]) {
const text = 'a'.repeat(len);
expect(await tokenizer.calculateTokens(text)).toBe(Math.ceil(len / 4));
}
});

it('should stay consistent when a single non-ASCII char joins long ASCII text', async () => {
const ascii = 'x'.repeat(1000);
// 1000 / 4 = 250
expect(await tokenizer.calculateTokens(ascii)).toBe(250);
// 1000 / 4 + 1 * 1.1 = 251.1 -> ceil = 252, wherever the char sits
expect(await tokenizer.calculateTokens(ascii + '中')).toBe(252);
expect(await tokenizer.calculateTokens('中' + ascii)).toBe(252);
});

it('should count surrogate pairs as two non-ASCII units within mixed text', async () => {
const text = 'abcd🚀'; // 4 ASCII + 2 UTF-16 units
// 4 / 4 + 2 * 1.1 = 3.2 -> ceil = 4
expect(await tokenizer.calculateTokens(text)).toBe(4);
});
});

describe('large inputs', () => {
it('should handle very long text', async () => {
const longText = 'a'.repeat(200000); // 200k characters
Expand Down
16 changes: 10 additions & 6 deletions packages/core/src/utils/request-tokenizer/textTokenizer.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@
* SPDX-License-Identifier: Apache-2.0
*/

const NON_ASCII_RE = /[\u0080-\uffff]/;

/**
* Text tokenizer for calculating text tokens using character-based estimation.
*
Expand All @@ -19,17 +21,19 @@ export function estimateTextTokens(text: string): number {
return 0;
}

let asciiChars = 0;
let nonAsciiChars = 0;
// Fast path: pure-ASCII text (code, English prose). A single regex scan
// uses V8's optimized string search instead of a per-character JS loop.
if (!NON_ASCII_RE.test(text)) {
return Math.ceil(text.length / 4);
}

let nonAsciiChars = 0;
for (let i = 0; i < text.length; i++) {
Comment on lines +26 to 31

@wenshao wenshao Jul 8, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] The fast path helps pure ASCII, but mixed strings with a late non-ASCII unit now scan almost the entire text twice: once in NON_ASCII_RE.test(text), then again from index 0 in the loop. RequestTokenizer and PDF token estimation can pass large mostly-ASCII text with an occasional emoji/CJK character, so this regresses a plausible hot path. Reuse the first match position and continue counting after it.

Suggested change
if (!NON_ASCII_RE.test(text)) {
return Math.ceil(text.length / 4);
}
let nonAsciiChars = 0;
for (let i = 0; i < text.length; i++) {
const firstNonAscii = text.search(NON_ASCII_RE);
if (firstNonAscii === -1) {
return Math.ceil(text.length / 4);
}
let nonAsciiChars = 1;
for (let i = firstNonAscii + 1; i < text.length; i++) {

— GPT-5 via Qwen Code /review

const charCode = text.charCodeAt(i);
if (charCode < 128) {
asciiChars++;
} else {
if (text.charCodeAt(i) >= 128) {
nonAsciiChars++;
}
}
const asciiChars = text.length - nonAsciiChars;

const tokens = asciiChars / 4 + nonAsciiChars * 1.1;
return Math.ceil(tokens);
Expand Down
Loading