Skip to content

fix(i18n): reject translated chunks that leak reasoning or lose table rows - #14731

Merged
diegosouzapw merged 2 commits into
release/v3.8.51from
fix/i18n-translation-output-guard
Sep 24, 2026
Merged

diegosouzapw merged 2 commits into
release/v3.8.51from
fix/i18n-translation-output-guard

Conversation

@diegosouzapw

Copy link
Copy Markdown
Owner

Summary

run-translation.mjs accepted whatever the model returned. The drift check only compares hashes, so broken mirrors still passed it. In the two refreshes on 2026-09-23/24 (#14676, #14724), fallback models:

  • leaked their reasoning into about 90 mirrors, as <think>…</think> blocks and English meta-prose such as "I'll keep the table header…" and "My plan became clear";
  • dropped table rows or code fences in about 60 mirrors. For example, es PROVIDER_REFERENCE.md lost 23 rows, and bs REMOVED_PROVIDERS.md collapsed from 67 lines to 32.

Both PRs had to be repaired by hand before merge.

Change

validateTranslatedChunk(source, output) (exported, pure):

  • strips <think>…</think> blocks and a ```markdown fence that wraps the whole answer;
  • rejects the chunk when the code-fence count or the table-row count differs from the source chunk;
  • rejects a stray <think> tag;
  • rejects English meta phrases (I'll, I need to, Okay,, My plan, …) that appear more often than in the source.

translateBody() retries a rejected chunk up to 3 times. If the chunk still fails, the doc fails and the mirror is not written. The runner already reports failed docs and retries them.

Validation

  • New tests/unit/i18n-translation-output-guard.test.ts (9 cases, built from the real refresh-5/6 failures). It failed before the change (no export) and passes after.
  • Existing i18n suites pass: chunking 6/6, sections 17/17, state-adopt 12/12, language-bar 10/10.
  • I could not run a live check: the translation backend was down at the time (cinf 503, auto/gemini → dva 502).

⚠️ base-red inherited: #14547

… rows

The drift check only compares hashes, so broken mirrors passed it. In the
2026-09-23/24 refreshes, fallback models leaked their <think> block and
English meta-prose into about 90 mirrors, and dropped table rows in about
60 (es PROVIDER_REFERENCE lost 23 rows).

validateTranslatedChunk() strips <think> blocks and a wrapping ```markdown
fence. It then rejects the chunk when the code-fence count or the
table-row count differs from the source chunk, or when English meta
phrases appear more often than in the source. translateBody() retries a
rejected chunk up to 3 times and then fails the doc instead of writing it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant