docs(investigation): LLM model quality audit for DatasetUpdater (#299) - #466
Merged
Conversation
Grounded inventory of the 47 DatasetUpdater task-configs: model per task/language (33 gpt-5.5, 10 gpt-5.4-mini, 4 gpt-5.4), no BaseUrl set (OpenAI direct), and the prompt set. Correlates #411 cosmetic defects to producing model/prompt and flags concrete gaps (ES/EN Cat A uncovered, pt-PT vs pt-BR register conflict, cosmetic JSON-vs-function-calling mismatch). Actionable recommendations for #192 multi-pass. DOC-ONLY: no code/CSV/prompt changes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This was referenced Jun 13, 2026
jsboige
added a commit
that referenced
this pull request
Jun 13, 2026
…, #299 follow-up) (#467) Enforce the standing gpt-5.5-only translation mandate: the eco-tier gpt-5.4-mini is banned for translations. Flips every gpt-5.4-mini occurrence (12 translation tasks) to gpt-5.5 in DatasetUpdaterRootConfig.cs. Follows the #466 DOC-ONLY investigation (docs/investigations/2026-06-14-llm-model-quality-datasetupdater.md), which flagged the gpt-5.4-mini residue as drift from the mandate (note: the working-tree file actually carries 12 such tasks, not the 10 stated in the doc's pre-edit table — the doc undercounted; all are translations and all move up). The 4 gpt-5.4 (non-mini) tasks are intentionally left untouched: they are FR-native generation/refinement (Virtues taxonomy FR, Fallacies FR description, Fallacies FR example, multi-lang cleanup review), not translation. The mandate concerns translation only — flagged as a separate decision for jsboige. All 47 tasks remain Enabled=false, so there is no runtime or cost effect until a deliberate run is enabled. Validation: dotnet test green (159 pass / 0 fail / 5 skip); EOL attribute preserved (i/lf w/crlf); git diff is exactly 12 Model= line changes, file otherwise byte-identical. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
jsboige
added a commit
that referenced
this pull request
Jun 14, 2026
…t 155->159 (#469) The v0.9.0 validation brief (PR #462) was written from a pre-#460 reading and flagged doc discrepancies that #460 had already resolved, plus it under-reported the test count. Corrections (3 files, docs-only): 1. release-v0.9.0-validation-brief.md - reframe §3.1/§3.2 as RESOLVED by PR #460 (36124be): MindMap SVGs correctly state FR/EN/RU/PT committed (21) + ES/AR/FA/ZH pending (CHANGELOG L16, RELEASE-NOTES L25/L35/L92); Generated Assets counts reconciled to 64 (Tarot 24 + Poker 16 + Web 24). Simplifies jsboige's residual decision to: regenerate MindMaps 8-lang (Track 1a, option A) vs status quo (option B, already applied by #460). Also fixes a structural duplication in the prior commit (stale §3.1 RED / §3.2 YELLOW sections that should have been removed) and adds §3.4. 2. §3.4 (new) - test-count drift: CHANGELOG L70 + RELEASE-NOTES L46 said 155 tests, but dashboard reports 159 after #465/#28. Verified by grep [Fact]/[Theory]: Issue28TargetDissociationTests.cs = 4 tests; #466/#467 /#468 are doc-only / Enabled=false config -> 0 runtime tests. 155+4=159. 3. CHANGELOG.md L70 + RELEASE-NOTES-v0.9.0.md L46 - 155 -> 159 tests ("+4 from #465/#28 front/back dissociation"). Only §3.3 trivial date placeholders remain (normal, pre-tag). Gate release unchanged - no tag/launch/merge. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
DOC-ONLY investigation for #299 — a grounded audit of LLM model quality in the
DatasetUpdatertranslation pipeline. New file:docs/investigations/2026-06-14-llm-model-quality-datasetupdater.md.Every technical claim cites a
file:lineand is tagged VÉRIFIÉ (read in code/git) / RAPPORTÉ (issue/PR) / SUPPOSÉ (deduction). No code/CSV/prompt changes.Key findings (VÉRIFIÉ)
DatasetUpdaterRootConfig.cs, allEnabled=false. Model split: 33 ×gpt-5.5, 10 ×gpt-5.4-mini, 4 ×gpt-5.4. The Investigate LLM model quality for DatasetUpdater translations #299 hint ("uses gpt-5.4-mini") is stale — code migrated to gpt-5.5 (PR feat(dataset-updater): multi-provider API + gpt-5.5 for EN translations #302).BaseUrl→ all run on the default OpenAI endpoint despite the multi-provider support added in feat(dataset-updater): multi-provider API + gpt-5.5 for EN translations #302. Memory says the key file may hold an OpenRouter key+URL, but the code fixes no BaseUrl — flagged as an unresolved inconsistency.gpt-5.4-miniwhile EN moved togpt-5.5— matches the EN-good / RU-PT-weak asymmetry in feat(translations): multi-pass quality improvement for EN/RU/PT (beyond initial translation) #192.text_es/text_enappear in no cosmeticFieldsToUpdate.UseFunctionCalling=truewith a system prompt that forbids free text.Doc sections
Top 3 recommendations for #192
gpt-5.4-minitogpt-5.5(tasks images famille 1 #5,Bump acorn from 6.1.1 to 6.4.1 in /DNNPlatform/Portals/0-System/Skins/Bootstrap 4 Instant #6,Bump node-sass from 4.12.0 to 4.13.1 in /DNNPlatform/Portals/0-System/Skins/Bootstrap 4 Instant #9,Bump dot-prop from 4.2.0 to 4.2.1 in /DNNPlatform/Portals/0-System/Skins/Bootstrap 4 Instant #10,Bump System.Net.Http from 4.3.0 to 4.3.4 in /DNNPlatform/DesktopModules/ToSIC_SexyContent #15,Bump System.Net.Http from 4.3.0 to 4.3.4 in /DNNPlatform/DesktopModules/ToSIC_SexyContent/2sxc Dnn #16), as EN already is (feat(dataset-updater): multi-provider API + gpt-5.5 for EN translations #302); the highest-value 2nd pass is PT cell-by-cell via the maturePromptScenariiPtRefineUserpattern.AutoCompare, or (safer for the 2 known PKs) a deterministic CSV fix; no LLM needed for 2 cells.Note
Created amid heavy concurrent-worker activity on the shared working tree; the branch was assembled via git plumbing onto
c9197f15so it contains only the doc (verifiedgit diff --stat c9197f15..HEAD= 1 file, 234 insertions). CRLF preserved to match existing investigation docs.Closes #299 (investigation deliverable; live benchmark already in the #299 comment).
🤖 Generated with Claude Code