Skip to content

feat(compression): risk-gate pre-pass — shield sensitive spans from lossy compression - #5243

Merged
diegosouzapw merged 11 commits into
release/v3.8.40from
feat/compression-risk-gate
Jun 28, 2026
Merged

diegosouzapw merged 11 commits into
release/v3.8.40from
feat/compression-risk-gate

Conversation

@diegosouzapw

Copy link
Copy Markdown
Owner

What

Adds an opt-in, fail-open "risk-gate" pre-pass to the compression subsystem that surgically shields sensitive spans from lossy compression while letting the rest of the text compress normally. 7th feature of the compression feature-extraction roadmap (bench: #5080, fidelity gate: #5127, fuzzy: #5143, ionizer: #5148, TOON: #5163, CCR ranged: #5187).

Today the compressor has zero awareness of content risk — aggressive/ultra/ionizer-sampling modes can mangle or drop a secret, a private key, a stack trace, a k8s Secret, a DB migration, or license text. The fidelity gate (#5127) only catches corruption after the fact and a lossy-by-design engine bypasses it. The risk-gate prevents the damage up front. Default off — flipping it on is per-operator (config/env), never silent.

How

A new riskGate/ module (peer to the fidelity gate), wired as an outer mask→run→restore wrapper around the three compression entry points — a single, universal integration point:

  • riskGate/riskPatterns.ts — RiskCategory catalog + per-category bounded regexes (every variable-length pattern uses {0,N} quantifiers → no ReDoS) + the self-evident set.
  • riskGate/riskGate.ts — detectRiskSpans(text, cfg): structural detection with a multi-signal guard (self-evident categories private_key/k8s_secret/db_migration(≥2 DDL) promote alone; secret_assignment/stack_trace/legal need ≥2 corroborating signals or a short <200-char section) + a commit-log/diff guard (drops DDL that only appears inside a diff --git hunk; strong VCS markers only). Pure, fail-open (never throws).
  • riskGate/riskGateStep.ts — applyRiskMask(body, cfg) masks detected spans across message content (string and {type:"text"} multimodal parts) into SENTINEL placeholders, restoreRiskBlocks restores them byte-identically.
  • riskGate/strategyWrap.ts — resolveRiskGate + withRiskGate/withRiskGateAsync (kept out of strategySelector to minimize its growth).
  • preservation.ts — one new exported helper preserveSpans(text, spans) that wraps literal offsets into the exact OMNI_CAVEMAN SENTINEL family every engine already treats as opaque — so the secret survives caveman/ultra/etc. with zero per-engine changes (proven by a real-engine integration test).
  • strategySelector.ts — the three exported entry points (applyCompression, applyStackedCompression, applyStackedCompressionAsync) become thin wrappers over pure-extracted private bodies (runCompression/runStackedCompression/runStackedCompressionAsync — identical logic, guarded by a byte-identical parity test). The gate sits strictly outside the per-step loop, so fidelity/gateAdvance runs on the masked body unchanged.
  • types.ts — CompressionConfig.riskGate? + CompressionStats.riskGate? (additive, optional; DEFAULT_COMPRESSION_CONFIG unchanged → default off).
  • Preview route + studio — /api/compression/preview accepts riskGate: { enabled } and returns riskGate stats; the studio gets a toggle checkbox + a 🛡️ N risky spans protected badge.

Security: the patterns are ours (not agent-supplied) and bounded; the gate only reduces risk surface (shields secrets from lossy transforms) and telemetry is counts + category names only (never the secret content). Error responses on the route stay routed through the untouched sanitizeErrorMessage path (Hard Rule #12).

Tests (TDD, both runners)

  • Node runner (tests/unit/compression/): riskGateDetect (11 — every category + the multi-signal/short-section/commit-log guards + bounded-regex adversarial input), riskGateStep (mask/restore byte-identical round-trip incl. multimodal + config defaults), riskGateIntegration (PEM stays byte-identical through real caveman while prose compresses; byte-identical to baseline when disabled; preview-route stats). 19/19 + 13 existing preservation/caveman suites stay green.
  • Vitest jsdom (tests/unit/ui/riskGateBadge.test.tsx): badge render/empty-state — 2/2.
  • TDD-hardened mid-flight: a code-review pass on the detection core fixed 4 issues (ReDoS-bounded whitespace, strong-only VCS markers, enabled short-circuit, dead import) — all with tests intact.

Gates (local)

lint 0 errors · typecheck:core clean · check:cycles OK · check:complexity 1980/1980 (baseline) · check:cognitive-complexity 841/841 (baseline) · next build compiles 596/596 static pages · no new dependency (no check:deps change).

One expected red — file-size on strategySelector.ts (own growth, please rebaseline at merge)

open-sse/services/compression/strategySelector.ts: 899 > frozen 854. This is genuine own-growth from the wrapper wiring, already minimized (helpers extracted to strategyWrap.ts; 929→898 lines). The residual is irreducible for the extract-to-runX pattern on a file already slated for structural decomposition (#3501). I did not edit config/quality/file-size-baseline.json (baseline is yours) — please --admin/rebaseline at merge, same as #5143/#5148/#5163/#5187.

Ready for review & merge.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a 'risk-gate' pre-pass feature designed to shield sensitive data (such as private keys, secrets, stack traces, and database migrations) from lossy compression, restoring them verbatim afterward. It includes pattern matching, masking/restoring utilities, integration with the compression pipeline, and UI updates in the compression studio. The review feedback highlights several important improvements: resolving redundant double-wrapping of withRiskGate during stacked compression, moving PreviewRequestSchema to the shared validation directory to comply with Repository Rule 8, adding defensive guards against potential runtime TypeErrors when parsing messages, refining DDL span promotion to avoid shielding excessive intermediate text, and fixing the secret_assignment regex to properly match closing quotes.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment on lines +272 to +274
): CompressionResult {
return withRiskGate(body, resolveRiskGate(options), (b) => runCompression(b, mode, options));
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

When applyCompression is called with mode === "stacked", it wraps the execution with withRiskGate. However, runCompression then delegates to applyStackedCompression, which also wraps itself with withRiskGate. This results in redundant double-wrapping and executes the risk-gate regex matching twice on the same content, impacting performance. We should strip the riskGate option from the options passed to runCompression to prevent this double-wrapping.

): CompressionResult {
  const gate = resolveRiskGate(options);
  if (!gate) return runCompression(body, mode, options);
  const { riskGate, ...rest } = options ?? {};
  const cleanedConfig = options?.config ? { ...options.config, riskGate: undefined } : undefined;
  const cleanedOptions = { ...rest, ...(cleanedConfig ? { config: cleanedConfig } : {}) };
  return withRiskGate(body, gate, (b) => runCompression(b, mode, cleanedOptions));
}

Comment on lines +43 to +46
// Playground risk-gate toggle → masks high-risk spans (secrets/keys) before compression and
// restores them verbatim after, so they pass through byte-identical. Reported via
// result.stats.riskGate (spansProtected + per-category counts).
riskGate: z.object({ enabled: z.boolean() }).optional(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Defining PreviewRequestSchema locally in the route file violates Repository Rule 8, which states: 'Always validate inputs with Zod schemas from src/shared/validation/schemas.ts'. To adhere to the repository style guide, this schema should be moved to the shared validation schemas directory and imported here.

References
  1. Always validate inputs with Zod schemas from src/shared/validation/schemas.ts.

Comment on lines +50 to +52
const maskedMessages = messages.map((msg) => {
const m = msg as { role?: unknown; content?: unknown };
if (typeof m.content === "string") {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If msg is null or not an object at runtime, accessing m.content will throw a TypeError. We should add a defensive guard to ensure msg is a valid object before accessing its properties.

Suggested change
const maskedMessages = messages.map((msg) => {
const m = msg as { role?: unknown; content?: unknown };
if (typeof m.content === "string") {
const maskedMessages = messages.map((msg) => {
if (!msg || typeof msg !== "object") return msg;
const m = msg as { role?: unknown; content?: unknown };
if (typeof m.content === "string") {

Comment on lines +107 to +109
messages: messages.map((msg) => {
const m = msg as { content?: unknown };
if (typeof m.content === "string" || Array.isArray(m.content)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If msg is null or not an object at runtime, accessing m.content will throw a TypeError. We should add a defensive guard to ensure msg is a valid object before accessing its properties.

Suggested change
messages: messages.map((msg) => {
const m = msg as { content?: unknown };
if (typeof m.content === "string" || Array.isArray(m.content)) {
messages: messages.map((msg) => {
if (!msg || typeof msg !== "object") return msg;
const m = msg as { content?: unknown };
if (typeof m.content === "string" || Array.isArray(m.content)) {

Comment on lines +109 to +110
const ddlPromoted: RiskSpan[] =
ddl.length >= MIN_DDL ? [{ start: ddl[0].start, end: ddl[ddl.length - 1].end, category: "db_migration" }] : [];

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Promoting a single giant span from the first DDL statement to the last DDL statement (ddl[0].start to ddl[ddl.length - 1].end) is extremely blunt. If a document contains two DDL statements separated by a large amount of normal prose, the entire intermediate text will be shielded from compression. Instead, we should promote each DDL hit individually as its own span. mergeSpans will automatically merge them if they actually overlap or are adjacent.

Suggested change
const ddlPromoted: RiskSpan[] =
ddl.length >= MIN_DDL ? [{ start: ddl[0].start, end: ddl[ddl.length - 1].end, category: "db_migration" }] : [];
const ddlPromoted: RiskSpan[] =
ddl.length >= MIN_DDL ? ddl.map((h) => ({ start: h.start, end: h.end, category: "db_migration" })) : [];

Comment on lines +43 to +44
regex:
/\b(?:api[_-]?key|secret|token|password|passwd|bearer|authorization|client[_-]?secret)\b[ \t]{0,20}[:=][ \t]{0,20}["']?[A-Za-z0-9._\-+/]{8,200}/gi,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

security-medium medium

The secret_assignment regex matches an optional opening quote ["']? but does not match the corresponding closing quote. This leaves the closing quote exposed to lossy compression, which can lead to syntax errors or broken quotes in the restored text if the lossy compressor decides to strip or modify the unmatched trailing quote. We should use a capture group and backreference to match matching quotes cleanly.

Suggested change
regex:
/\b(?:api[_-]?key|secret|token|password|passwd|bearer|authorization|client[_-]?secret)\b[ \t]{0,20}[:=][ \t]{0,20}["']?[A-Za-z0-9._\-+/]{8,200}/gi,
regex:
/\b(?:api[_-]?key|secret|token|password|passwd|bearer|authorization|client[_-]?secret)\b[ \t]{0,20}[:=][ \t]{0,20}(["']?)[A-Za-z0-9._\-+/]{8,200}\1/gi,

@diegosouzapw

Copy link
Copy Markdown
Owner Author

Babysit summary — risk-gate (#5)

CI verdict: all checks green except the one expected, owner-handled red.

Check Result
Unit Tests fast-path (1/2) ✅ pass
Unit Tests fast-path (2/2) ✅ pass
Vitest (fast-path) ✅ pass
dast-smoke ✅ pass
semgrep / semgrep-cloud ✅ pass
Fast Quality Gates ❌ fail — only step: npm run check:file-size

The single red is check:file-size on open-sse/services/compression/strategySelector.ts (899 > frozen 854). This is genuine own-growth from the risk-gate wrapper wiring, already minimized (wrapper helpers extracted to riskGate/strategyWrap.ts; 929 → 898 lines). The residual is irreducible for the extract-to-runX pattern on a file already slated for structural decomposition (#3501). I deliberately did not edit config/quality/file-size-baseline.json (baseline is the maintainer's) — please --admin / rebaseline at merge, same as #5143 / #5148 / #5163 / #5187.

Verified the failing step is file-size and nothing else via actions/jobs step inspection (step 11 = Run npm run check:file-size, the only failure).

Local full battery (all green): lint 0 · typecheck:core · check:cycles · check:complexity 1980/1980 · check:cognitive-complexity 841/841 · 19 node tests + 2 vitest UI · byte-identical parity when gate disabled · next build compiled 596/596 static pages (the wreq-js/rust EINVAL in standalone assembly is a non-fatal symlinked-worktree artifact, build exit 0).

Ready for human review & merge. Not auto-merging.

…e-pass wiring)

The risk-gate mask->run->restore wrapper extracts the 3 entry points into thin
wrappers over pure private bodies so the gate sits outside the per-step loop;
the +45 residual is dispatch-boundary wiring guarded by the byte-identical
parity test. Default off. Structural shrink tracked in #3501.
@diegosouzapw
diegosouzapw merged commit 462bca6 into release/v3.8.40 Jun 28, 2026
7 checks passed
@diegosouzapw
diegosouzapw deleted the feat/compression-risk-gate branch June 28, 2026 21:48
tkgo11 pushed a commit to tkgo11/OmniRoute that referenced this pull request Sep 23, 2026
…ossy compression (diegosouzapw#5243)

Risk-gate pre-pass — shields sensitive spans (PEM/secret/stack/k8s/migration/legal) from lossy compression via SENTINEL preserveSpans. Default off, fail-open, ReDoS-bounded patterns. strategySelector baseline rebaselined for the wrapper extraction.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant