Skip to content

fix: replace unit test with integration test for proactive context compression #1378

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
diegosouzapw merged 8 commits into from
Apr 18, 2026
Merged

fix: replace unit test with integration test for proactive context compression #1378

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
a3d1dc6
fix(api): support image-only models and authless search providers
diegosouzapw Apr 17, 2026
0afd304
fix(routes): require prompts for media generation requests
diegosouzapw Apr 17, 2026
447c135
refactor(audit): align audit dashboard with compliance log entries
diegosouzapw Apr 17, 2026
d963c2b
fix: replace unit test with integration test for proactive context co…
oyi77 Apr 17, 2026
fceb9d4
fix(core): resolve runtime edge cases and TypeScript regressions
diegosouzapw Apr 17, 2026
eff2c0b
fix(services): pass provider to refreshWithRetry to avoid tripping ge…
diegosouzapw Apr 17, 2026
e4e7bde
fix(context): scale reserved tokens for smaller model windows
diegosouzapw Apr 17, 2026
9e8b29d
chore: apply review suggestions and fix merge conflict in contextMana…
diegosouzapw Apr 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions open-sse/handlers/chatCore.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1452,12 +1452,20 @@ export async function handleChatCore({
}
}

// ── Proactive Context Compression (Phase 4) ──
// Check if context exceeds 85% of limit and compress proactively before sending to provider.
// This prevents "prompt too long" errors for large-but-not-full contexts.
if (translatedBody && translatedBody.messages && Array.isArray(translatedBody.messages)) {
const estimatedTokens = estimateTokens(JSON.stringify(translatedBody.messages));
const contextLimit = getTokenLimit(provider, effectiveModel);
const COMPRESSION_THRESHOLD = 0.85;
const threshold = Math.floor(contextLimit * COMPRESSION_THRESHOLD);

log?.debug?.(
"CONTEXT",
`Checking compression: ${estimatedTokens} tokens vs ${threshold} threshold (${contextLimit} limit)`
);

if (estimatedTokens > threshold) {
log?.info?.(
"CONTEXT",
Expand All @@ -1468,6 +1476,7 @@ export async function handleChatCore({
provider,
model: effectiveModel,
maxTokens: contextLimit,
reserveTokens: 0,

@gemini-code-assist gemini-code-assist Bot Apr 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Setting reserveTokens to 0 here effectively disables the proactive compression for contexts that fall between the 85% threshold and the 100% limit.

In compressContext, the target token count is calculated as maxTokens - reserveTokens. If reserveTokens is 0, the target is the full contextLimit. This means that if a request is triggered at 90% of the limit, compressContext will see that it already fits within the 100% target and will not perform any compression.

Furthermore, passing 0 here overrides the environment variable CONTEXT_RESERVE_TOKENS and the default value of 16000 defined in contextManager.ts. To allow the service to use its configured default or environment override, you should avoid passing 0 explicitly.

Suggested change
reserveTokens: 0,
reserveTokens: undefined,

});

if (compressionResult.compressed) {
Expand Down Expand Up @@ -1495,8 +1504,15 @@ export async function handleChatCore({
layers: "layers" in stats ? stats.layers : undefined,
},
});
} else {
log?.debug?.("CONTEXT", `Compression not applied: context already fits within target`);
}
}
} else {
log?.debug?.(
"CONTEXT",
`Skipping compression check: translatedBody=${!!translatedBody}, messages=${!!translatedBody?.messages}, isArray=${Array.isArray(translatedBody?.messages)}`
);
}

// Resolve executor with optional upstream proxy (CLIProxyAPI) routing.
Expand Down
14 changes: 12 additions & 2 deletions open-sse/services/contextManager.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
*/

import { REGISTRY } from "../config/providerRegistry.ts";
import { getModelContextLimit } from "../../src/lib/modelCapabilities";
import { getModelContextLimit } from "../../src/lib/modelCapabilities.ts";

// Default token limits per provider (fallbacks when not in registry)
const DEFAULT_LIMITS: Record<string, number> = {
Expand Down Expand Up @@ -34,6 +34,16 @@ function getEnvOverride(provider: string): number | null {
return null;
}

// Reserve tokens override from environment variable
function getReserveTokensOverride(): number | null {
const envValue = process.env.CONTEXT_RESERVE_TOKENS;
if (envValue) {
const parsed = parseInt(envValue, 10);
if (!isNaN(parsed) && parsed > 0) return parsed;
}
return null;
}

// Rough chars-per-token ratio for quick estimation
const CHARS_PER_TOKEN = 4;

Expand Down Expand Up @@ -109,7 +119,7 @@ export function compressContext(
const provider = options.provider || "default";
const maxTokens =
options.maxTokens || getTokenLimit(provider, (body.model as string) || options.model || null);
const reserveTokens = options.reserveTokens || 16000; // Reserve for response
const reserveTokens = options.reserveTokens ?? getReserveTokensOverride() ?? 16000;
const targetTokens = maxTokens - reserveTokens;

let messages = [...body.messages];
Expand Down
Loading