Repository navigation
fix(compression): make memo key model-independent for non-vision engines - #8196
Merged
diegosouzapw merged 3 commits intoJul 23, 2026
Conversation
Contributor
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
rafaumeu
force-pushed
the
fix/combo-redundant-compression-8137
branch
from
July 22, 2026 22:32
2de8766 to
7441966
Compare
rafaumeu
force-pushed
the
fix/combo-redundant-compression-8137
branch
2 times, most recently
from
July 22, 2026 23:19
bafaf3a to
b3f8612
Compare
rafaumeu
force-pushed
the
fix/combo-redundant-compression-8137
branch
3 times, most recently
from
July 23, 2026 02:10
bc8f3b6 to
8d634c1
Compare
…nes (cheaper-inference#8137) The compression result memo included `model` and `supportsVision` in the cache key for ALL deterministic modes. This was correct for the `lite` engine (which strips data:image URLs based on vision support) but unnecessary for model-independent engines like `rtk`, `caveman`, and stacked pipelines without a `lite` step. In the combo retry loop, the body and config are identical across targets — only the model changes each attempt. Including model in the key forced a fresh cache miss on every retry, re-running the full compression pipeline 5-8x per request instead of serving the cached result. Fix: `makeMemoKey` now only includes model + supportsVision when the compression pipeline actually uses a vision-dependent engine (lite, standard, or stacked containing lite). All other deterministic engines use a model-independent key. - Add `usesVisionDependentEngine()` helper to classify modes - `makeMemoKey` conditionally includes model/vision fields - 5 new tests covering rtk, caveman, stacked-with-lite, stacked-without-lite Closes cheaper-inference#8137
rafaumeu
force-pushed
the
fix/combo-redundant-compression-8137
branch
from
July 23, 2026 02:47
8d634c1 to
29bb369
Compare
Collaborator
|
Thanks, @rafaumeu! 🙏 Merged — compression memo key is now model-independent for non-vision engines. result-memo tests green. |
# Conflicts: # .env.example # README.md
Collaborator
|
Thanks, @rafaumeu! 🙏 Merged — compression memo key is now model-independent for non-vision engines. Rebased onto the current release tip (dropped the stale bundled .env.example/README noise, kept only the resultMemo change). 32/32 tests green. |
diegosouzapw
merged commit Jul 23, 2026
5e8b130
into
cheaper-inference:release/v3.8.49
0 of 3 checks passed
Merged
HouMinXi
pushed a commit
to HouMinXi/OmniRoute
that referenced
this pull request
Aug 2, 2026
…nes (cheaper-inference#8196) * fix(ci): resolve upstream-inherited check failures * fix(compression): make memo key model-independent for non-vision engines (cheaper-inference#8137) The compression result memo included `model` and `supportsVision` in the cache key for ALL deterministic modes. This was correct for the `lite` engine (which strips data:image URLs based on vision support) but unnecessary for model-independent engines like `rtk`, `caveman`, and stacked pipelines without a `lite` step. In the combo retry loop, the body and config are identical across targets — only the model changes each attempt. Including model in the key forced a fresh cache miss on every retry, re-running the full compression pipeline 5-8x per request instead of serving the cached result. Fix: `makeMemoKey` now only includes model + supportsVision when the compression pipeline actually uses a vision-dependent engine (lite, standard, or stacked containing lite). All other deterministic engines use a model-independent key. - Add `usesVisionDependentEngine()` helper to classify modes - `makeMemoKey` conditionally includes model/vision fields - 5 new tests covering rtk, caveman, stacked-with-lite, stacked-without-lite Closes cheaper-inference#8137 --------- Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
muhamadgalihsaputra
pushed a commit
to niyatna/NiyatnaRoute
that referenced
this pull request
Sep 27, 2026
…nes (cheaper-inference#8196) * fix(ci): resolve upstream-inherited check failures * fix(compression): make memo key model-independent for non-vision engines (cheaper-inference#8137) The compression result memo included `model` and `supportsVision` in the cache key for ALL deterministic modes. This was correct for the `lite` engine (which strips data:image URLs based on vision support) but unnecessary for model-independent engines like `rtk`, `caveman`, and stacked pipelines without a `lite` step. In the combo retry loop, the body and config are identical across targets — only the model changes each attempt. Including model in the key forced a fresh cache miss on every retry, re-running the full compression pipeline 5-8x per request instead of serving the cached result. Fix: `makeMemoKey` now only includes model + supportsVision when the compression pipeline actually uses a vision-dependent engine (lite, standard, or stacked containing lite). All other deterministic engines use a model-independent key. - Add `usesVisionDependentEngine()` helper to classify modes - `makeMemoKey` conditionally includes model/vision fields - 5 new tests covering rtk, caveman, stacked-with-lite, stacked-without-lite Closes cheaper-inference#8137 --------- Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #8137 — combo retry redundantly recompresses identical bodies across targets because the memo cache key includes
model, which changes every retry attempt.Problem
makeMemoKey()always includedmodelandsupportsVisionin the cache key. This is correct for theliteengine (which stripsdata:imageURLs based on vision support), but wrong for model-independent engines likertk,caveman, and stacked pipelines without lite.In the combo retry loop, the body and config are identical across targets — only the model changes. Including model in the key produced a fresh cache miss on every retry, re-running the full compression pipeline 5-8x per request instead of serving the cached result.
Fix
makeMemoKey()now conditionally includesmodel+supportsVisiononly when the compression pipeline actually uses a vision-dependent engine:lite— yes (image-URL stripping)standard— yes (lite → caveman pipeline)stackedwith alitestep — yesrtk,caveman,stackedwithout lite — no (model-independent)New helper:
usesVisionDependentEngine(mode, config)classifies modes.Test plan