Skip to content

fix(compression): make memo key model-independent for non-vision engines - #8196

Merged
diegosouzapw merged 3 commits into
cheaper-inference:release/v3.8.49from
rafaumeu:fix/combo-redundant-compression-8137
Jul 23, 2026
Merged

diegosouzapw merged 3 commits into
cheaper-inference:release/v3.8.49from
rafaumeu:fix/combo-redundant-compression-8137

Conversation

@rafaumeu

Copy link
Copy Markdown
Contributor

Summary

Fixes #8137 — combo retry redundantly recompresses identical bodies across targets because the memo cache key includes model, which changes every retry attempt.

Problem

makeMemoKey() always included model and supportsVision in the cache key. This is correct for the lite engine (which strips data:image URLs based on vision support), but wrong for model-independent engines like rtk, caveman, and stacked pipelines without lite.

In the combo retry loop, the body and config are identical across targets — only the model changes. Including model in the key produced a fresh cache miss on every retry, re-running the full compression pipeline 5-8x per request instead of serving the cached result.

Fix

makeMemoKey() now conditionally includes model + supportsVision only when the compression pipeline actually uses a vision-dependent engine:

  • lite — yes (image-URL stripping)
  • standard — yes (lite → caveman pipeline)
  • stacked with a lite step — yes
  • rtk, caveman, stacked without lite — no (model-independent)

New helper: usesVisionDependentEngine(mode, config) classifies modes.

Test plan

  • All 32 existing result-memo tests pass
  • 5 new tests:
    • rtk mode: same key across different models
    • caveman mode: same key across different models
    • stacked-without-lite: same key across different models
    • stacked-with-lite: different keys across different models (still vision-dependent)
    • lite mode: model + supportsVision still affect key (existing regression test preserved)
  • No LSP diagnostics introduced

@rafaumeu
rafaumeu requested a review from diegosouzapw as a code owner July 22, 2026 17:46
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@rafaumeu
rafaumeu force-pushed the fix/combo-redundant-compression-8137 branch from 2de8766 to 7441966 Compare July 22, 2026 22:32
@mergify

mergify Bot commented Jul 22, 2026

Copy link
Copy Markdown

⚠️ The sha of the head commit of this PR conflicts with #8179. Mergify cannot evaluate rules on this PR. Once #8179 is merged or closed, Mergify will resume processing this PR. ⚠️

@rafaumeu
rafaumeu force-pushed the fix/combo-redundant-compression-8137 branch 2 times, most recently from bafaf3a to b3f8612 Compare July 22, 2026 23:19
@diegosouzapw
diegosouzapw changed the base branch from main to release/v3.8.49 July 23, 2026 00:25
@rafaumeu
rafaumeu force-pushed the fix/combo-redundant-compression-8137 branch 3 times, most recently from bc8f3b6 to 8d634c1 Compare July 23, 2026 02:10
rafaumeu added 2 commits July 22, 2026 23:44
…nes (cheaper-inference#8137)

The compression result memo included `model` and `supportsVision` in the
cache key for ALL deterministic modes. This was correct for the `lite` engine
(which strips data:image URLs based on vision support) but unnecessary for
model-independent engines like `rtk`, `caveman`, and stacked pipelines
without a `lite` step.

In the combo retry loop, the body and config are identical across targets —
only the model changes each attempt. Including model in the key forced a fresh
cache miss on every retry, re-running the full compression pipeline 5-8x per
request instead of serving the cached result.

Fix: `makeMemoKey` now only includes model + supportsVision when the
compression pipeline actually uses a vision-dependent engine (lite, standard,
or stacked containing lite). All other deterministic engines use a
model-independent key.

- Add `usesVisionDependentEngine()` helper to classify modes
- `makeMemoKey` conditionally includes model/vision fields
- 5 new tests covering rtk, caveman, stacked-with-lite, stacked-without-lite

Closes cheaper-inference#8137
@rafaumeu
rafaumeu force-pushed the fix/combo-redundant-compression-8137 branch from 8d634c1 to 29bb369 Compare July 23, 2026 02:47
@diegosouzapw

Copy link
Copy Markdown
Collaborator

Thanks, @rafaumeu! 🙏 Merged — compression memo key is now model-independent for non-vision engines. result-memo tests green.

@diegosouzapw

Copy link
Copy Markdown
Collaborator

Thanks, @rafaumeu! 🙏 Merged — compression memo key is now model-independent for non-vision engines. Rebased onto the current release tip (dropped the stale bundled .env.example/README noise, kept only the resultMemo change). 32/32 tests green.

@diegosouzapw
diegosouzapw merged commit 5e8b130 into cheaper-inference:release/v3.8.49 Jul 23, 2026
0 of 3 checks passed
@diegosouzapw diegosouzapw mentioned this pull request Jul 23, 2026
HouMinXi pushed a commit to HouMinXi/OmniRoute that referenced this pull request Aug 2, 2026
…nes (cheaper-inference#8196)

* fix(ci): resolve upstream-inherited check failures

* fix(compression): make memo key model-independent for non-vision engines (cheaper-inference#8137)

The compression result memo included `model` and `supportsVision` in the
cache key for ALL deterministic modes. This was correct for the `lite` engine
(which strips data:image URLs based on vision support) but unnecessary for
model-independent engines like `rtk`, `caveman`, and stacked pipelines
without a `lite` step.

In the combo retry loop, the body and config are identical across targets —
only the model changes each attempt. Including model in the key forced a fresh
cache miss on every retry, re-running the full compression pipeline 5-8x per
request instead of serving the cached result.

Fix: `makeMemoKey` now only includes model + supportsVision when the
compression pipeline actually uses a vision-dependent engine (lite, standard,
or stacked containing lite). All other deterministic engines use a
model-independent key.

- Add `usesVisionDependentEngine()` helper to classify modes
- `makeMemoKey` conditionally includes model/vision fields
- 5 new tests covering rtk, caveman, stacked-with-lite, stacked-without-lite

Closes cheaper-inference#8137

---------

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
muhamadgalihsaputra pushed a commit to niyatna/NiyatnaRoute that referenced this pull request Sep 27, 2026
…nes (cheaper-inference#8196)

* fix(ci): resolve upstream-inherited check failures

* fix(compression): make memo key model-independent for non-vision engines (cheaper-inference#8137)

The compression result memo included `model` and `supportsVision` in the
cache key for ALL deterministic modes. This was correct for the `lite` engine
(which strips data:image URLs based on vision support) but unnecessary for
model-independent engines like `rtk`, `caveman`, and stacked pipelines
without a `lite` step.

In the combo retry loop, the body and config are identical across targets —
only the model changes each attempt. Including model in the key forced a fresh
cache miss on every retry, re-running the full compression pipeline 5-8x per
request instead of serving the cached result.

Fix: `makeMemoKey` now only includes model + supportsVision when the
compression pipeline actually uses a vision-dependent engine (lite, standard,
or stacked containing lite). All other deterministic engines use a
model-independent key.

- Add `usesVisionDependentEngine()` helper to classify modes
- `makeMemoKey` conditionally includes model/vision fields
- 5 new tests covering rtk, caveman, stacked-with-lite, stacked-without-lite

Closes cheaper-inference#8137

---------

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(resilience): redundant compression on every combo retry wastes CPU (memo keyed by model)

2 participants