QVAC-18420 fix[mod]: Bergamot vocab re-downloaded on every loadModel for shared-vocab pairs by simon-iribarren · Pull Request #1892 · tetherto/qvac

simon-iribarren · 2026-05-04T19:34:46Z

Note: be concise and prefer bullet points.

🎯 What problem does this PR solve?

QVAC-18420 — reported by Dominic in #qvac-sdk: the Bergamot FR→EN vocab.fren.spm was being deleted and re-downloaded from the registry on every loadModel call, even though it was already cached. Investigation confirmed 7 bidirectional Bergamot pairs reproduce: FR_EN, EN_DE, EN_CS, ET_EN, FI_EN, PL_EN, PT_EN.
Shipping-hot: translation is not "offline after first use" for these pairs; every load costs ~600 KB – 1 MB of bandwidth and extra latency (originally caught on Keet/iOS).

📝 How does it solve it?

Root cause: deduplicateModels (sdk-pod models/update-models/registry.ts) drops the second occurrence of two registry entries that share a sha256Checksum. For the 7 affected pairs the two translation directions ship the same vocab blob under different paths (e.g. bergamot-fren/.../vocab.fren.spm ≡ bergamot-enfr/.../vocab.enfr.spm, both sha 783abf3a…c002a, 814404 bytes). Alphabetical dedup kept the bergamot-enfr path and dropped the bergamot-fren one — leaving bergamot-fren/vocab.fren.spm with no standalone RegistryItem. When the Bergamot plugin then called ctx.resolveModelPath("registry://…/vocab.fren.spm"), getModelByPath() returned undefined → expectedSize = 0 → the cache validator saw 814404 ≠ 0 and unlinked + re-downloaded every load.
Fix A — dedup preserves companion-referenced duplicates (packages/sdk/models/update-models/registry.ts): before dropping on sha256 collision, check whether the candidate's registryPath is referenced as a file in any other entry's companionSet. Companion-referenced duplicates survive. Regenerated packages/sdk/models/registry/models.ts — the 7 previously-missing BERGAMOT_*_VOCAB entries are back. Remaining non-companion duplicates (6) are still deduped.
Fix C — skip redundant vocab resolve for registry:// Bergamot loads (packages/sdk/server/bare/plugins/nmtcpp-translation/plugin.ts): when the primary model is a registry:// source with auto-derived vocabs, skip the separate ctx.resolveModelPath calls entirely. downloadCompanionSetFromRegistry already places the vocabs next to the model under sets/<setKey>/, and createModel derives the colocated paths via the existing deriveColocatedBergamotVocabPaths helper. Eliminates the redundant flat-cache copy on every load (bandwidth + latency win even on unaffected pairs) and narrows the blast radius of this class of dedup hole. Same optimization applied to the pivot branch.
Unit tests: new packages/sdk/test/unit/update-models-dedup.test.ts covers the QVAC-18420 regression plus general dedup semantics (plain duplicates still drop, empty-sha entries untouched, cross-set companion references all survive).

🧪 How was it tested?

bun lint (sdk) — clean.
bun run build (sdk) — clean.
bun run test:unit (sdk) — all green, including 5 new dedup tests.
bun run update-models (sdk) against the live registry — diff is purely the 7 BERGAMOT_*_VOCAB entries added plus index-shifts on subsequent export consts; no other registry items changed. 🧹 Removed 6 duplicate model(s) (was 13) confirms exactly the 7 affected vocabs are now preserved.
Reproduction script from the bug report (packages/sdk/examples/translation/translation-bergamot-cache-bug.ts) is expected to return "0 of 15 reproduce" after this lands. Recommend the reporter re-run as an end-to-end check on a fresh ~/.qvac/models cache.

📦 Models

Added models

BERGAMOT_EN_CS_VOCAB
BERGAMOT_EN_DE_VOCAB
BERGAMOT_ET_EN_VOCAB
BERGAMOT_FI_EN_VOCAB
BERGAMOT_FR_EN_VOCAB
BERGAMOT_PL_EN_VOCAB
BERGAMOT_PT_EN_VOCAB

These are companion-only entries (no export const); they only appear inside the models array so that getModelByPath() resolves the companion vocab paths that are shared across both directions of each bidirectional pair.

…for shared-vocab pairs - Dedup preserves registry entries referenced as companion files in any companion set, so shared-vocab blobs (identical sha256 under two paths) keep their standalone RegistryItem. This adds back 7 previously-dropped BERGAMOT_<pair>_VOCAB entries (FR_EN, EN_DE, EN_CS, ET_EN, FI_EN, PL_EN, PT_EN) and restores correct expectedSize/sha lookups for the Bergamot plugin's separate vocab resolve. - For registry:// Bergamot loads with auto-derived vocabs (non-pivot and pivot), skip the separate per-vocab resolveModelPath entirely. The companion-set download already colocates vocabs under sets/<setKey>/ and createModel derives those paths via deriveColocatedBergamotVocabPaths — no redundant flat-cache download, no dedup-hole exposure.

Reviewer ask on PR tetherto#1892: lock the Fix-C registry-source vocab optimization behind unit tests so the contract doesn't drift. - Extract resolveBergamotVocab and its derivation helpers into resolve-vocab.ts. The plugin module imports @qvac/translation-nmtcpp at module-load time (Bare-only native addon), so the helpers were not importable from a bun-driven test runner. The new module has zero native deps; plugin.ts re-imports from it. - Add test/unit/nmtcpp-resolve-vocab.test.ts covering: - registry:// + auto-derived vocabs (non-pivot) → ctx.resolveModelPath is never called and no vocab artifacts are emitted - registry:// + auto-derived vocabs (pivot) → exactly one resolveModelPath call (the pivot model itself), no vocab artifacts - registry:// + user-supplied srcVocabSrc OR dstVocabSrc → optimization is skipped and full per-vocab resolution runs (sanity: keeps the optimization narrow) - pear:// source still resolves vocabs explicitly (companion-set semantics don't apply) - mixed registry/pear pivot triggers full resolution path - unsupported sources raise ModelLoadFailedError - pure derivation helpers (buildBergamotVocabSources, deriveBergamotVocabSources, deriveBergamotRegistryVocabSources) for both shared-vocab and CJK split-vocab pairs

opaninakuffo · 2026-05-05T11:53:12Z

review

github-actions · 2026-05-05T11:53:35Z

Tier-based Approval Status

**PR Tier:** TIER1

**Current Status:** ✅ APPROVED

**Requirements:**
- 1 Team Member approval ✅ (1/1)
- 1 Team Lead OR Management approval ✅ (1/1)



---
*This comment is automatically updated when reviews change.*

…for shared-vocab pairs (#1892) * QVAC-18420 fix[mod]: Bergamot vocab re-downloaded on every loadModel for shared-vocab pairs - Dedup preserves registry entries referenced as companion files in any companion set, so shared-vocab blobs (identical sha256 under two paths) keep their standalone RegistryItem. This adds back 7 previously-dropped BERGAMOT_<pair>_VOCAB entries (FR_EN, EN_DE, EN_CS, ET_EN, FI_EN, PL_EN, PT_EN) and restores correct expectedSize/sha lookups for the Bergamot plugin's separate vocab resolve. - For registry:// Bergamot loads with auto-derived vocabs (non-pivot and pivot), skip the separate per-vocab resolveModelPath entirely. The companion-set download already colocates vocabs under sets/<setKey>/ and createModel derives those paths via deriveColocatedBergamotVocabPaths — no redundant flat-cache download, no dedup-hole exposure. * QVAC-18420 test: cover Bergamot vocab resolver + extract pure helpers Reviewer ask on PR #1892: lock the Fix-C registry-source vocab optimization behind unit tests so the contract doesn't drift. - Extract resolveBergamotVocab and its derivation helpers into resolve-vocab.ts. The plugin module imports @qvac/translation-nmtcpp at module-load time (Bare-only native addon), so the helpers were not importable from a bun-driven test runner. The new module has zero native deps; plugin.ts re-imports from it. - Add test/unit/nmtcpp-resolve-vocab.test.ts covering: - registry:// + auto-derived vocabs (non-pivot) → ctx.resolveModelPath is never called and no vocab artifacts are emitted - registry:// + auto-derived vocabs (pivot) → exactly one resolveModelPath call (the pivot model itself), no vocab artifacts - registry:// + user-supplied srcVocabSrc OR dstVocabSrc → optimization is skipped and full per-vocab resolution runs (sanity: keeps the optimization narrow) - pear:// source still resolves vocabs explicitly (companion-set semantics don't apply) - mixed registry/pear pivot triggers full resolution path - unsupported sources raise ModelLoadFailedError - pure derivation helpers (buildBergamotVocabSources, deriveBergamotVocabSources, deriveBergamotRegistryVocabSources) for both shared-vocab and CJK split-vocab pairs --------- Co-authored-by: Opanin Akuffo <46673050+opaninakuffo@users.noreply.github.com>

…for shared-vocab pairs (tetherto#1892) * QVAC-18420 fix[mod]: Bergamot vocab re-downloaded on every loadModel for shared-vocab pairs - Dedup preserves registry entries referenced as companion files in any companion set, so shared-vocab blobs (identical sha256 under two paths) keep their standalone RegistryItem. This adds back 7 previously-dropped BERGAMOT_<pair>_VOCAB entries (FR_EN, EN_DE, EN_CS, ET_EN, FI_EN, PL_EN, PT_EN) and restores correct expectedSize/sha lookups for the Bergamot plugin's separate vocab resolve. - For registry:// Bergamot loads with auto-derived vocabs (non-pivot and pivot), skip the separate per-vocab resolveModelPath entirely. The companion-set download already colocates vocabs under sets/<setKey>/ and createModel derives those paths via deriveColocatedBergamotVocabPaths — no redundant flat-cache download, no dedup-hole exposure. * QVAC-18420 test: cover Bergamot vocab resolver + extract pure helpers Reviewer ask on PR tetherto#1892: lock the Fix-C registry-source vocab optimization behind unit tests so the contract doesn't drift. - Extract resolveBergamotVocab and its derivation helpers into resolve-vocab.ts. The plugin module imports @qvac/translation-nmtcpp at module-load time (Bare-only native addon), so the helpers were not importable from a bun-driven test runner. The new module has zero native deps; plugin.ts re-imports from it. - Add test/unit/nmtcpp-resolve-vocab.test.ts covering: - registry:// + auto-derived vocabs (non-pivot) → ctx.resolveModelPath is never called and no vocab artifacts are emitted - registry:// + auto-derived vocabs (pivot) → exactly one resolveModelPath call (the pivot model itself), no vocab artifacts - registry:// + user-supplied srcVocabSrc OR dstVocabSrc → optimization is skipped and full per-vocab resolution runs (sanity: keeps the optimization narrow) - pear:// source still resolves vocabs explicitly (companion-set semantics don't apply) - mixed registry/pear pivot triggers full resolution path - unsupported sources raise ModelLoadFailedError - pure derivation helpers (buildBergamotVocabSources, deriveBergamotVocabSources, deriveBergamotRegistryVocabSources) for both shared-vocab and CJK split-vocab pairs --------- Co-authored-by: Opanin Akuffo <46673050+opaninakuffo@users.noreply.github.com>

…for shared-vocab pairs (#1892) * QVAC-18420 fix[mod]: Bergamot vocab re-downloaded on every loadModel for shared-vocab pairs - Dedup preserves registry entries referenced as companion files in any companion set, so shared-vocab blobs (identical sha256 under two paths) keep their standalone RegistryItem. This adds back 7 previously-dropped BERGAMOT_<pair>_VOCAB entries (FR_EN, EN_DE, EN_CS, ET_EN, FI_EN, PL_EN, PT_EN) and restores correct expectedSize/sha lookups for the Bergamot plugin's separate vocab resolve. - For registry:// Bergamot loads with auto-derived vocabs (non-pivot and pivot), skip the separate per-vocab resolveModelPath entirely. The companion-set download already colocates vocabs under sets/<setKey>/ and createModel derives those paths via deriveColocatedBergamotVocabPaths — no redundant flat-cache download, no dedup-hole exposure. * QVAC-18420 test: cover Bergamot vocab resolver + extract pure helpers Reviewer ask on PR #1892: lock the Fix-C registry-source vocab optimization behind unit tests so the contract doesn't drift. - Extract resolveBergamotVocab and its derivation helpers into resolve-vocab.ts. The plugin module imports @qvac/translation-nmtcpp at module-load time (Bare-only native addon), so the helpers were not importable from a bun-driven test runner. The new module has zero native deps; plugin.ts re-imports from it. - Add test/unit/nmtcpp-resolve-vocab.test.ts covering: - registry:// + auto-derived vocabs (non-pivot) → ctx.resolveModelPath is never called and no vocab artifacts are emitted - registry:// + auto-derived vocabs (pivot) → exactly one resolveModelPath call (the pivot model itself), no vocab artifacts - registry:// + user-supplied srcVocabSrc OR dstVocabSrc → optimization is skipped and full per-vocab resolution runs (sanity: keeps the optimization narrow) - pear:// source still resolves vocabs explicitly (companion-set semantics don't apply) - mixed registry/pear pivot triggers full resolution path - unsupported sources raise ModelLoadFailedError - pure derivation helpers (buildBergamotVocabSources, deriveBergamotVocabSources, deriveBergamotRegistryVocabSources) for both shared-vocab and CJK split-vocab pairs --------- Co-authored-by: Opanin Akuffo <46673050+opaninakuffo@users.noreply.github.com>

* QVAC-18420 fix[mod]: Bergamot vocab re-downloaded on every loadModel for shared-vocab pairs (#1892) * QVAC-18420 fix[mod]: Bergamot vocab re-downloaded on every loadModel for shared-vocab pairs - Dedup preserves registry entries referenced as companion files in any companion set, so shared-vocab blobs (identical sha256 under two paths) keep their standalone RegistryItem. This adds back 7 previously-dropped BERGAMOT_<pair>_VOCAB entries (FR_EN, EN_DE, EN_CS, ET_EN, FI_EN, PL_EN, PT_EN) and restores correct expectedSize/sha lookups for the Bergamot plugin's separate vocab resolve. - For registry:// Bergamot loads with auto-derived vocabs (non-pivot and pivot), skip the separate per-vocab resolveModelPath entirely. The companion-set download already colocates vocabs under sets/<setKey>/ and createModel derives those paths via deriveColocatedBergamotVocabPaths — no redundant flat-cache download, no dedup-hole exposure. * QVAC-18420 test: cover Bergamot vocab resolver + extract pure helpers Reviewer ask on PR #1892: lock the Fix-C registry-source vocab optimization behind unit tests so the contract doesn't drift. - Extract resolveBergamotVocab and its derivation helpers into resolve-vocab.ts. The plugin module imports @qvac/translation-nmtcpp at module-load time (Bare-only native addon), so the helpers were not importable from a bun-driven test runner. The new module has zero native deps; plugin.ts re-imports from it. - Add test/unit/nmtcpp-resolve-vocab.test.ts covering: - registry:// + auto-derived vocabs (non-pivot) → ctx.resolveModelPath is never called and no vocab artifacts are emitted - registry:// + auto-derived vocabs (pivot) → exactly one resolveModelPath call (the pivot model itself), no vocab artifacts - registry:// + user-supplied srcVocabSrc OR dstVocabSrc → optimization is skipped and full per-vocab resolution runs (sanity: keeps the optimization narrow) - pear:// source still resolves vocabs explicitly (companion-set semantics don't apply) - mixed registry/pear pivot triggers full resolution path - unsupported sources raise ModelLoadFailedError - pure derivation helpers (buildBergamotVocabSources, deriveBergamotVocabSources, deriveBergamotRegistryVocabSources) for both shared-vocab and CJK split-vocab pairs --------- Co-authored-by: Opanin Akuffo <46673050+opaninakuffo@users.noreply.github.com> * QVAC-17324 feat[api]: add harmony tool-call dialect (gpt-oss) (#1878) * chore: bump @qvac/llm-llamacpp to 0.17.2 for harmony EOG fix - Picks up #1812 which stops the addon from suppressing the `<|call|>` end-of-generation token, unblocking harmony tool-call parsing on the SDK side. * feat: expose toolDialect override and add harmony to the dialect enum * feat: add harmony tool-call parser for gpt-oss models * feat: stream harmony frames and decouple dialect from tool-call activation * test: add harmony unit coverage for parser and normalizer * chore: consolidate tool examples under examples/tools/ * doc: clarify emitThinking capture-gating contract * chore: remove conflict markers * feat[mod]: sync sdk model registry to bergamot base-memory and drop deprecated marian opus (#1903) - Bumps BERGAMOT_EN_IT and BERGAMOT_ES_EN to the base-memory variant at bergamot-{enit,esen}/2026-04-28/. Fixes leading "- " hallucinations on short inputs and en->it quality regression (registry seed updated in #1785, synced to DHT 2026-05-05). - Drops 32 deprecated Marian Opus entries (NMT_Q0F16*, NMT_Q4_0*) auto-deprecated by #1680. - Auto-generated by `bun update-models`. * chore[skiplog]: release sdk 0.10.1 --------- Co-authored-by: Simon Iribarren <simon.ig13@gmail.com>

…for shared-vocab pairs (#1892) * QVAC-18420 fix[mod]: Bergamot vocab re-downloaded on every loadModel for shared-vocab pairs - Dedup preserves registry entries referenced as companion files in any companion set, so shared-vocab blobs (identical sha256 under two paths) keep their standalone RegistryItem. This adds back 7 previously-dropped BERGAMOT_<pair>_VOCAB entries (FR_EN, EN_DE, EN_CS, ET_EN, FI_EN, PL_EN, PT_EN) and restores correct expectedSize/sha lookups for the Bergamot plugin's separate vocab resolve. - For registry:// Bergamot loads with auto-derived vocabs (non-pivot and pivot), skip the separate per-vocab resolveModelPath entirely. The companion-set download already colocates vocabs under sets/<setKey>/ and createModel derives those paths via deriveColocatedBergamotVocabPaths — no redundant flat-cache download, no dedup-hole exposure. * QVAC-18420 test: cover Bergamot vocab resolver + extract pure helpers Reviewer ask on PR #1892: lock the Fix-C registry-source vocab optimization behind unit tests so the contract doesn't drift. - Extract resolveBergamotVocab and its derivation helpers into resolve-vocab.ts. The plugin module imports @qvac/translation-nmtcpp at module-load time (Bare-only native addon), so the helpers were not importable from a bun-driven test runner. The new module has zero native deps; plugin.ts re-imports from it. - Add test/unit/nmtcpp-resolve-vocab.test.ts covering: - registry:// + auto-derived vocabs (non-pivot) → ctx.resolveModelPath is never called and no vocab artifacts are emitted - registry:// + auto-derived vocabs (pivot) → exactly one resolveModelPath call (the pivot model itself), no vocab artifacts - registry:// + user-supplied srcVocabSrc OR dstVocabSrc → optimization is skipped and full per-vocab resolution runs (sanity: keeps the optimization narrow) - pear:// source still resolves vocabs explicitly (companion-set semantics don't apply) - mixed registry/pear pivot triggers full resolution path - unsupported sources raise ModelLoadFailedError - pure derivation helpers (buildBergamotVocabSources, deriveBergamotVocabSources, deriveBergamotRegistryVocabSources) for both shared-vocab and CJK split-vocab pairs --------- Co-authored-by: Opanin Akuffo <46673050+opaninakuffo@users.noreply.github.com>

simon-iribarren requested a review from a team as a code owner May 4, 2026 19:34

simon-iribarren added verify tier1 labels May 4, 2026

simon-iribarren had a problem deploying to release May 4, 2026 19:35 — with GitHub Actions Failure

Merge branch 'main' into fix/qvac-18420-bergamot-vocab-dedup

e69fc6b

simon-iribarren had a problem deploying to release May 5, 2026 08:18 — with GitHub Actions Failure

opaninakuffo previously approved these changes May 5, 2026

View reviewed changes

Merge branch 'main' into fix/qvac-18420-bergamot-vocab-dedup

1ab657a

simon-iribarren had a problem deploying to release May 5, 2026 09:16 — with GitHub Actions Failure

lauripiisang reviewed May 5, 2026

View reviewed changes

Comment thread packages/sdk/server/bare/plugins/nmtcpp-translation/plugin.ts Outdated

NamelsKing previously approved these changes May 5, 2026

View reviewed changes

simon-iribarren dismissed stale reviews from NamelsKing and opaninakuffo via e2c47ed May 5, 2026 09:39

simon-iribarren had a problem deploying to release May 5, 2026 09:40 — with GitHub Actions Failure

NamelsKing approved these changes May 5, 2026

View reviewed changes

lauripiisang approved these changes May 5, 2026

View reviewed changes

Merge branch 'main' into fix/qvac-18420-bergamot-vocab-dedup

40cd14b

opaninakuffo had a problem deploying to release May 5, 2026 11:53 — with GitHub Actions Failure

simon-iribarren merged commit b3c751b into tetherto:main May 5, 2026
12 of 13 checks passed

simon-iribarren had a problem deploying to release May 5, 2026 12:14 — with GitHub Actions Failure

This was referenced May 5, 2026

QVAC-18121 chore[skiplog]: release sdk 0.10.1 #1909

Merged

QVAC-18121 chore[skiplog]: backmerge release-sdk-0.10.1 — version bump, changelog, NOTICE #1910

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

QVAC-18420 fix[mod]: Bergamot vocab re-downloaded on every loadModel for shared-vocab pairs#1892

QVAC-18420 fix[mod]: Bergamot vocab re-downloaded on every loadModel for shared-vocab pairs#1892
simon-iribarren merged 5 commits into
tetherto:mainfrom
simon-iribarren:fix/qvac-18420-bergamot-vocab-dedup

simon-iribarren commented May 4, 2026

Uh oh!

Uh oh!

opaninakuffo commented May 5, 2026

Uh oh!

github-actions Bot commented May 5, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

simon-iribarren commented May 4, 2026

🎯 What problem does this PR solve?

📝 How does it solve it?

🧪 How was it tested?

📦 Models

Added models

Uh oh!

Uh oh!

opaninakuffo commented May 5, 2026

Uh oh!

github-actions Bot commented May 5, 2026

Tier-based Approval Status

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants