docs(taxonomy): #299 LLM quality delta investigation (read-only, 0 run) — gpt-5.5 pivot + dormant /v1/responses

[jsboige]

What

#299 LLM translation quality — delta investigation (dispatched by ai-01 msg-gvpl31, tick 88, SECONDARY). Read-only — 0 run live (all 48 task configs Enabled=false, T&A freeze). 0 prod-CSV. INPUT for the post-T&A activation decision.

#299 is already closed — this is a delta, not a redo

#299 is CLOSED with a complete provider benchmark by jsboige (2026-05-17): OpenRouter claude-sonnet-4 (best quality + fastest 7s), OpenAI gpt-4.1-mini (reliable fallback), Myia qwen + ZAI glm-5.1 (reasoning-overhead, too slow for batch). jsboige's comment already measured the reasoning-overhead failure mode:

"Reasoning model uses tokens for thinking before generating content. Even with 4000 max_tokens, the reasoning consumed all tokens before producing output."

That is the same root cause later formalized for gpt-5.5 in [[gpt55-responses-api-effort-low]] — jsboige observed it empirically 2 months before the gpt-5.5 pivot. This doc does not redo the benchmark (it is closed and valid for its era); it characterizes what changed since.

The delta (two changes stale the benchmark)

1. The gpt-5.5 pivot

The benchmark tested gpt-4.1-mini / claude-sonnet-4 / glm-5.1 / qwen. Today (DatasetUpdaterRootConfig.cs, f70b20dc):

Model	n tasks
gpt-5.5	44
gpt-5.4	4

Neither benchmark winner is configured (gpt-4.1-mini = 0, claude-sonnet-4 = 0). The fleet pivoted to gpt-5.5 — a reasoning model, the class jsboige flagged over-budget on Chat Completions.

2. The /v1/responses plumbing is dormant

PR #852 (merged f70b20dc) added the Responses-API path (reversible, default off). Empirical scan: UseResponsesApi set on 0/48 tasks, ReasoningEffort on 0/48. All 44 gpt-5.5 tasks are wired to legacy Chat Completions → the empty-Content failure mode is latent (would fire if Enabled flipped today).

Decision space for post-T&A activation

Option	Path	Model	Verdict
A	Chat Completions	gpt-5.5 (current)	❌ empty-Content — don't activate as-is
B	Chat Completions	revert to gpt-4.1-mini / claude-sonnet-4 (#299 winners)	proven by #299, discards gpt-5.5 reasoning
C	/v1/responses + reasoning.effort=low (#852, OFF today)	gpt-5.5	intended per dashboard metric; untested (didn't exist when #299 ran)

Recommendation (for ai-01 synthesis, not a verdict): Option C is the intended config (dashboard metric: gpt-5.5 /v1/responses, reasoning:{effort:"low"}, max_output_tokens=7000). Before any post-T&A campaign, flip UseResponsesApi=true + ReasoningEffort="low" on gpt-5.5 tasks + run a 5-record pilot (not full Phase 1) to confirm usable Content — Option C couldn't be tested by #299.

Governance (what this did NOT do)

• 0 run live (Enabled=false × 48, T&A freeze). Config + code + doc read only.
• 0 prod-CSV. Did not change config (UseResponsesApi/ReasoningEffort stay 0 — activation is gated jsboige post-T&A).
• Did not reopen Investigate LLM model quality for DatasetUpdater translations #299 (benchmark stands). Did not redo the benchmark.

Refs

• Investigate LLM model quality for DatasetUpdater translations #299 (closed benchmark, 2026-05-17). feat(dataset-updater): route gpt-5.x via Responses API + reasoning effort (#141) #852 (/v1/responses plumbing, f70b20dc). Memory [[gpt55-responses-api-effort-low]]. Dispatch ai-01 → po-2024 msg-gvpl31 (tick 88).

🤖 Generated with Claude Code