tools(dnn-i18n): #457 content-type extractor + reimport dry-run (fixture, Option C tooling)

tools/dnn_i18n/README.md

@@ -0,0 +1,79 @@
+# #457 DNN site localization — i18n tooling (bricks 1 + 3 + round-trip)
+
+Reusable tooling around the **already-wired** DNN UI-strings translation rail (DatasetUpdater
+Option C, config merged in #487, `Enabled=false`). These tools do the work *around* the config
+that the config can't do for itself: extract the content-type set from the templates, and
+dry-run-verify the re-import — all **fixture/dry-run**, zero prod mutation.
+
+## The 3 bricks (and which already existed)
+
+| Brick | What | Status |
+|-------|------|--------|
+| **(2) DatasetUpdater config (Option C)** | gpt-5.5 task for `ui.*`/`res.*`, 8-language | ✅ **Already done** — #487 (`ca9a8640`), `DatasetUpdaterRootConfig.cs:2636-2696`, `Enabled=false` |
+| **(1) content-type → CSV extractor** | reusable, codifies the PHASE1 manual audit | 🆕 net-new — `extract_dnn_ui_strings.py` |
+| **(3) CSV → DNN re-import dry-run verifier** | key-set diff + payload render (no write) | 🆕 net-new — `reimport_dnn_ui_strings.py` |
+
+**Brick 2 was NOT rebuilt** — the investigation confirmed the config-only gpt-5.5 path is already
+canonical (entity `DnnUiString`, `KnownDataSets.DnnUiStrings`, prompts, task config all merged,
+`Enabled=false`). Re-adding it would duplicate. This PR adds the missing tooling (bricks 1 + 3).
+
+## Files
+
+- `extract_dnn_ui_strings.py` — parse 2sxc `.cshtml`, emit `dnn-ui-strings.csv` dialect.
+- `reimport_dnn_ui_strings.py` — `verify` (key-set diff vs reference) + `reimport` (render payload).
+- `test_roundtrip.py` — DoD proof: extract → verify → reimport on a fixture, zero prod mutation.
+- `fixtures/sample_templates/*.cshtml` — miniature audit-anchored fixture (NOT production).
+- `fixtures/reference_snapshot.csv` — committed golden snapshot for the round-trip test.
+
+## Quick start
+
+```bash
+# Round-trip DoD test (stdlib only, writes nothing outside temp):
+python tools/dnn_i18n/test_roundtrip.py
+
+# Extract from the REAL prod templates (standalone output — does NOT touch dnn-ui-strings.csv):
+python tools/dnn_i18n/extract_dnn_ui_strings.py \
+    --templates-root DNNPlatform/Portals/1/2sxc/Argumentum \
+    --out /tmp/prod_extract.csv
+
+# Verify the extraction's key set vs the reference CSV (HARD contract on key set):
+python tools/dnn_i18n/reimport_dnn_ui_strings.py verify \
+    --extracted /tmp/prod_extract.csv \
+    --reference docs/dnn-localization/dnn-ui-strings.csv
+
+# Render the re-import payload to stdout (dry-run — never writes anywhere):
+python tools/dnn_i18n/reimport_dnn_ui_strings.py reimport --csv /tmp/prod_extract.csv
+```
+
+## Anti-fabrication guarantees
+
+- **`ui.*` extraction is anchor-based, not free-text.** Each `ui.*` entry declares a verbatim
+  anchor that must exist in the named source. If a refactor removes the string, the extractor
+  **fails loud (exit 2)** instead of silently dropping the row. A free-text scanner would
+  fabricate "translatable strings" out of every template literal.
+- **`res.*` extraction is honest about DB-only values.** The `@Resources.<Key>` reference is in
+  the repo; the canonical FR *value* lives in SQL (2sxc App Resources). The extractor leaves
+  `fr` empty + flags `DB-only`. INFERRED FR scaffolds (PHASE1 §1b) are a human curation step,
+  intentionally NOT regenerated.
+- **Negative test proven:** breaking an anchor → exit 2 (verified).
+- **Cross-validated vs prod:** the extractor run on the real `DNNPlatform/.../Argumentum/`
+  templates yields **10/10 keys** matching `dnn-ui-strings.csv` (the only delta is `res.*` fr,
+  empty-in-extract by design).
+- **Round-trip on fixture: PASS** (extract → verify → reimport, zero prod mutation).
+
+## Gate boundaries (HARD)
+
+- ❌ Does **not** touch the live DNN DB, portal, or 2sxc App Resources — live extract/re-import
+  is **DB/RDP-gated (jsboige)**. The `reimport` subcommand only *renders* the payload to stdout.
+- ❌ Does **not** modify `docs/dnn-localization/dnn-ui-strings.csv` (that file is worker po-2024's
+  lane, #490) — the extractor writes to a user-supplied `--out` path only.
+- ❌ Does **not** enable or modify the DatasetUpdater task config (#487's `Enabled=false` rail).
+- ❌ Does **not** run any translation — gpt-5.5 translation is the config's job (#487), gated on
+  the source FR being complete (which needs the portal export, jsboige).
+- ❌ Does **not** declare a QA verdict — that's ai-01.
+
+## What unblocks next
+
+When jsboige exports the 2sxc App Resources (the DB-only `res.*` values), the FR column can be
+populated, at which point the existing config (#487) can be flipped `Enabled=true` to run gpt-5.5
+across the 7 target languages. This tooling feeds that rail; it doesn't replace it.

tools/dnn_i18n/extract_dnn_ui_strings.py

@@ -0,0 +1,232 @@
+#!/usr/bin/env python3
+"""#457 DNN site localization — content-type -> CSV extractor (BRICK 1).
+
+Codifies the *manual* Phase-1 extraction (see docs/dnn-localization/PHASE1-content-audit.md)
+into a reproducible tool. Parses the Argumentum 2sxc Razor templates (.cshtml) and emits the
+localization CSV in the exact dialect of `docs/dnn-localization/dnn-ui-strings.csv`:
+
+    key,context,source_file,fr,en,ru,pt,es,ar,fa,zh,notes
+
+Two content-types are extracted (matching the PHASE1 audit):
+
+  * ``ui.*``  — hardcoded UI strings inside the templates (a regex over literal patterns).
+                The FR value IS in the repo (hardcoded), so it is populated.
+  * ``res.*`` — ``@Resources.<Key>`` references. The KEY is in the repo; the canonical FR
+                VALUE lives in SQL (2sxc App Resources) — DB-only. So ``fr`` is left empty and
+                ``notes`` flags it ``INFERRED FR / DB-only`` (matches the PHASE1 manual rows).
+
+WHY a regex over literal patterns for ``ui.*``: the hardcoded strings are bespoke
+(e.g. ``de {0} à {1} joueurs``), not a systematic API. A free-text scanner would fabricate
+"translatable strings" out of every literal. Instead we mirror the curated PHASE1 set: each
+``ui.*`` entry is declared below with its anchor pattern + canonical FR, and the tool
+*verifies the anchor still exists in the source* (anti-fabrication: fail loud if an anchor
+disappears, never invent a row).
+
+This is the repo-extractable slice only. The bulk of DNN strings (glossary, FAQ, homepage,
+per-rule content, the App resource VALUES) is DB-only and requires a portal/2sxc export
+(jsboige, gated) — see PHASE1-content-audit.md. This tool does NOT touch prod or the DB.
+
+USAGE
+    python tools/dnn_i18n/extract_dnn_ui_strings.py \\
+        --templates-root DNNPlatform/Portals/1/2sxc/Argumentum \\
+        --out <output.csv>
+
+The output is a STANDALONE extraction; it does not write to dnn-ui-strings.csv (that file is
+another worker's lane, #490). Use reimport_dnn_ui_strings.py to diff/reimport.
+"""
+from __future__ import annotations
+
+import argparse
+import csv
+import os
+import re
+import sys
+from dataclasses import dataclass, field
+from typing import List
+
+# ---------------------------------------------------------------------------
+# Dialect — exact column order of docs/dnn-localization/dnn-ui-strings.csv
+# ---------------------------------------------------------------------------
+COLUMNS = ["key", "context", "source_file", "fr",
+           "en", "ru", "pt", "es", "ar", "fa", "zh", "notes"]
+LANG_TARGETS = ["en", "ru", "pt", "es", "ar", "fa", "zh"]  # FR is the source, not a target
+
+
+# ---------------------------------------------------------------------------
+# Declared ui.* extraction entries (curated from PHASE1-content-audit.md §1a).
+# Each anchor is a verbatim substring that MUST exist in the named source file;
+# the tool asserts it. If a refactor moves/removes the string, the tool fails
+# loud rather than silently dropping the row.
+# ---------------------------------------------------------------------------
+@dataclass
+class UiEntry:
+    key: str
+    context: str
+    source_files: List[str]      # relative to --templates-root
+    anchor: str                  # verbatim substring expected in the file
+    fr: str                      # canonical FR (hardcoded in the template)
+
+
+UI_ENTRIES: List[UiEntry] = [
+    UiEntry(
+        key="ui.fallacy.find_out_more",
+        context="FallacyExplorer link label '(find out more)'",
+        source_files=["_FallacyExplorer_Root.cshtml"],
+        anchor="find out more",
+        fr="en savoir plus",
+    ),
+    UiEntry(
+        key="ui.rules.players_range",
+        context="Rules list/detail player-count line 'de {0} a {1} joueurs'",
+        source_files=["_RulesExplorer_RuleList.cshtml", "_RulesExplorer_RuleDetail.cshtml"],
+        anchor="joueurs",
+        fr="de {0} à {1} joueurs",
+    ),
+]
+
+
+# ---------------------------------------------------------------------------
+# res.* extraction — @Resources.<Key> references.
+# The KEY is extracted from the template; the FR VALUE is DB-only (2sxc App
+# Resources), so we emit an empty fr + a DB-only note. The PHASE1 audit also
+# carries INFERRED FR scaffolds for some keys; those are a human curation step
+# (not reproducible from the repo) and are intentionally NOT regenerated here —
+# the tool emits the honest "DB-only" row and leaves inference to the audit.
+# ---------------------------------------------------------------------------
+# Match both the directive form ``@Resources.<Key>`` AND the expression form
+# ``Resources.<Key>`` (e.g. inside ``@Html.Raw(Resources.RuleMemoInstructions)`` — note
+# the absence of a leading ``@`` before ``Resources``). The PHASE1 audit captured both
+# manually; the optional ``@?`` mirrors that. Without it, the nested-expression refs
+# (RuleMemoInstructions) would be silently dropped.
+RES_REF_RE = re.compile(r"@?Resources\.([A-Za-z][A-Za-z0-9_]*)")
+
+# Which template files to scan for @Resources.* references (skip the stock
+# landing-page builder template "_Album List.cshtml" — no Argumentum content,
+# per PHASE1 audit §1).
+RES_SOURCE_FILES = [
+    "_RulesExplorer_RuleDetail.cshtml",
+    "_RulesExplorer_RuleList.cshtml",
+    "_FallacyExplorer_Root.cshtml",
+]
+
+# Human context for the known @Resources.* keys (from PHASE1 audit §1b). Keys
+# not in this map get a generic context — they are still emitted so a later
+# portal export can fill their values.
+RES_CONTEXTS = {
+    "RuleSummary": "Rule detail section heading (<h2>)",
+    "RuleMaterial": "Rule detail section heading (<h2>)",
+    "RuleInstallation": "Rule detail section heading (<h2>)",
+    "RuleVariants": "Rule detail section heading (<h2>)",
+    "RuleMemoCard": "Rule detail memo-card heading (<h2>)",
+    "RuleMemoInstructions": "Rule detail memo-card instructions (Html.Raw, multi-sentence)",
+    "RuleMemoCardFileNamePrefix": "Memo card filename prefix in card name",
+    "RuleMemoCardDownload": "Memo card download button label",
+}
+
+
+@dataclass
+class Row:
+    key: str
+    context: str
+    source_file: str
+    fr: str
+    notes: str
+    targets: dict = field(default_factory=dict)  # lang -> value (empty by default)
+
+    def as_csv(self) -> dict:
+        d = {"key": self.key, "context": self.context,
+             "source_file": self.source_file, "fr": self.fr, "notes": self.notes}
+        for lang in LANG_TARGETS:
+            d[lang] = self.targets.get(lang, "")
+        return d
+
+
+def _read(root: str, rel: str) -> str:
+    path = os.path.join(root, rel)
+    with open(path, encoding="utf-8") as f:
+        return f.read(), path
+
+
+def extract_ui_entries(root: str) -> List[Row]:
+    """Extract declared ui.* rows, asserting each anchor still exists."""
+    rows: List[Row] = []
+    for e in UI_ENTRIES:
+        found_in: List[str] = []
+        for rel in e.source_files:
+            text, path = _read(root, rel)
+            if e.anchor not in text:
+                sys.stderr.write(
+                    f"ANTI-FAB: anchor {e.anchor!r} for {e.key} not found in {path}. "
+                    "The template was refactored — update UI_ENTRIES (do NOT silently drop the row).\n"
+                )
+                raise SystemExit(2)
+            found_in.append(f"Portals/1/2sxc/Argumentum/{rel}")
+        note = ""
+        # Preserve the bespoke PHASE1 notes for the 2 known buggy keys.
+        if e.key == "ui.fallacy.find_out_more":
+            note = ("Template hardcodes EN value AND reads text_en/desc_en/link_en "
+                    "regardless of culture (i18n bug, see audit s4)")
+        elif e.key == "ui.rules.players_range":
+            note = "Hardcoded FR; keep {0}/{1} placeholders; source uses &agrave; entity"
+        rows.append(Row(key=e.key, context=e.context,
+                        source_file=";".join(found_in), fr=e.fr, notes=note))
+    return rows
+
+
+def extract_res_entries(root: str) -> List[Row]:
+    """Extract @Resources.* references (key in repo, value DB-only)."""
+    seen: dict[str, List[str]] = {}  # key -> list of source files
+    for rel in RES_SOURCE_FILES:
+        text, _ = _read(root, rel)
+        for m in RES_REF_RE.finditer(text):
+            key = m.group(1)
+            seen.setdefault(key, []).append(f"Portals/1/2sxc/Argumentum/{rel}")
+    rows: List[Row] = []
+    for key in sorted(seen):
+        ctx = RES_CONTEXTS.get(key, "@Resources reference (DB-only value)")
+        rows.append(Row(
+            key=f"res.{key}",
+            context=ctx,
+            source_file=";".join(sorted(set(seen[key]))),
+            fr="",  # DB-only — value is NOT in the repo
+            notes="INFERRED FR; canonical value is DB-only (2sxc App Resources) - verify vs export",
+        ))
+    return rows
+
+
+def write_csv(rows: List[Row], out_path: str) -> None:
+    with open(out_path, "w", encoding="utf-8", newline="") as f:
+        w = csv.DictWriter(f, fieldnames=COLUMNS, quoting=csv.QUOTE_MINIMAL)
+        w.writeheader()
+        for r in rows:
+            w.writerow(r.as_csv())
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description=__doc__,
+                                 formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("--templates-root", default="DNNPlatform/Portals/1/2sxc/Argumentum",
+                    help="Root dir of the Argumentum 2sxc templates")
+    ap.add_argument("--out", required=True, help="Output CSV path (standalone extraction)")
+    args = ap.parse_args()
+
+    if not os.path.isdir(args.templates_root):
+        return _fail(f"templates root not found: {args.templates_root}")
+
+    rows = extract_ui_entries(args.templates_root) + extract_res_entries(args.templates_root)
+    write_csv(rows, args.out)
+
+    ui_n = sum(1 for r in rows if r.key.startswith("ui."))
+    res_n = sum(1 for r in rows if r.key.startswith("res."))
+    print(f"Extracted {len(rows)} rows ({ui_n} ui.*, {res_n} res.*) -> {args.out}")
+    print("NOTE: res.* fr values are DB-only (2sxc App Resources) — left empty by design.")
+    return 0
+
+
+def _fail(msg: str) -> int:
+    print(f"ERROR: {msg}", file=sys.stderr)
+    return 2
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

tools/dnn_i18n/fixtures/reference_snapshot.csv

@@ -0,0 +1,11 @@
+key,context,source_file,fr,en,ru,pt,es,ar,fa,zh,notes
+ui.fallacy.find_out_more,FallacyExplorer link label '(find out more)',Portals/1/2sxc/Argumentum/_FallacyExplorer_Root.cshtml,en savoir plus,,,,,,,,"Template hardcodes EN value AND reads text_en/desc_en/link_en regardless of culture (i18n bug, see audit s4)"
+ui.rules.players_range,Rules list/detail player-count line 'de {0} a {1} joueurs',Portals/1/2sxc/Argumentum/_RulesExplorer_RuleList.cshtml;Portals/1/2sxc/Argumentum/_RulesExplorer_RuleDetail.cshtml,de {0} à {1} joueurs,,,,,,,,Hardcoded FR; keep {0}/{1} placeholders; source uses à entity
+res.RuleInstallation,Rule detail section heading (<h2>),Portals/1/2sxc/Argumentum/_RulesExplorer_RuleDetail.cshtml,,,,,,,,,INFERRED FR; canonical value is DB-only (2sxc App Resources) - verify vs export
+res.RuleMaterial,Rule detail section heading (<h2>),Portals/1/2sxc/Argumentum/_RulesExplorer_RuleDetail.cshtml,,,,,,,,,INFERRED FR; canonical value is DB-only (2sxc App Resources) - verify vs export
+res.RuleMemoCard,Rule detail memo-card heading (<h2>),Portals/1/2sxc/Argumentum/_RulesExplorer_RuleDetail.cshtml,,,,,,,,,INFERRED FR; canonical value is DB-only (2sxc App Resources) - verify vs export
+res.RuleMemoCardDownload,Memo card download button label,Portals/1/2sxc/Argumentum/_RulesExplorer_RuleDetail.cshtml,,,,,,,,,INFERRED FR; canonical value is DB-only (2sxc App Resources) - verify vs export
+res.RuleMemoCardFileNamePrefix,Memo card filename prefix in card name,Portals/1/2sxc/Argumentum/_RulesExplorer_RuleDetail.cshtml,,,,,,,,,INFERRED FR; canonical value is DB-only (2sxc App Resources) - verify vs export
+res.RuleMemoInstructions,"Rule detail memo-card instructions (Html.Raw, multi-sentence)",Portals/1/2sxc/Argumentum/_RulesExplorer_RuleDetail.cshtml,,,,,,,,,INFERRED FR; canonical value is DB-only (2sxc App Resources) - verify vs export
+res.RuleSummary,Rule detail section heading (<h2>),Portals/1/2sxc/Argumentum/_RulesExplorer_RuleDetail.cshtml,,,,,,,,,INFERRED FR; canonical value is DB-only (2sxc App Resources) - verify vs export
+res.RuleVariants,Rule detail section heading (<h2>),Portals/1/2sxc/Argumentum/_RulesExplorer_RuleDetail.cshtml,,,,,,,,,INFERRED FR; canonical value is DB-only (2sxc App Resources) - verify vs export

tools/dnn_i18n/fixtures/sample_templates/_FallacyExplorer_Root.cshtml

@@ -0,0 +1,8 @@
+@* Fixture template for #457 round-trip test. Miniature, audit-anchored, NOT production. *@
+@inherits Custom.Hybrid.Razor12
+@{
+  var labels = new Dictionary<string, string> {
+    { "fr", "en savoir plus" }, { "en", "find out more" }, { "ru", "узнать больше" },
+  };
+}
+<a href="@fallacy.Link">@labels[Culture]</a>

tools/dnn_i18n/fixtures/sample_templates/_RulesExplorer_RuleDetail.cshtml

@@ -0,0 +1,18 @@
+@* Fixture template for #457 round-trip test. Miniature, audit-anchored, NOT production. *@
+@inherits Custom.Hybrid.Razor12
+@using System.Linq
+
+<div class="rule-detail">
+  <h2>@Resources.RuleSummary</h2>
+  <p>@ruleEntity.Summary</p>
+
+  <h2>@Resources.RuleMaterial</h2>
+  <h5>de @ruleEntity.MinNbPlayers &agrave; @ruleEntity.MaxNbPlayers joueurs </h5>
+
+  <h2>@Resources.RuleInstallation</h2>
+  <h2>@Resources.RuleVariants</h2>
+  <h2>@Resources.RuleMemoCard</h2>
+  <p>@Html.Raw(@Resources.RuleMemoInstructions)</p>
+  <h3>@Resources.RuleMemoCardFileNamePrefix</h3>
+  <a href="@downloadUrl">@Resources.RuleMemoCardDownload</a>
+</div>

tools/dnn_i18n/fixtures/sample_templates/_RulesExplorer_RuleList.cshtml

@@ -0,0 +1,7 @@
+@* Fixture template for #457 round-trip test. Miniature, audit-anchored, NOT production. *@
+@inherits Custom.Hybrid.Razor12
+@foreach (var rule in rules) {
+  <div class="nbPlayers">
+    <h6>de @rule.MinNbPlayers &agrave; @rule.MaxNbPlayers joueurs </h6>
+  </div>
+}