Skip to content

fix(csv): #202 repair Memo JSON %C3->A13 URL corruption + prod hygiene audit - #581

Merged
jsboige merged 2 commits into
masterfrom
fix/memo-url-encoding-corruption
Jun 23, 2026
Merged

fix(csv): #202 repair Memo JSON %C3->A13 URL corruption + prod hygiene audit#581
jsboige merged 2 commits into
masterfrom
fix/memo-url-encoding-corruption

Conversation

@jsboige

@jsboige jsboige commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

What

Data-hygiene audit of the other prod CSVs (Scenarii/Virtues/Rules/Memo) + the one real defect found, fixed: the Memo templates carry the same %C3A13 URL corruption as #579 (6 occurrences per file × 2 active files = 12 restorations).

Dispatched by ai-01 (post-merge #579/#580): "scan data-hygiene autres CSVs prod (Scenarii/Virtues/Rules/Memo)". This closes that dispatch.

The fix — Memo JSON %C3A13 (same bug as #579)

The two active prod Memo templates (Argumentum_Memo_Back_fr.json, Argumentum_Memo_Face_fr.json) embed a French-wiki fallacy-URL catalog on line 30. That catalog has the identical UTF-8 lead-byte percent-encoding corruption #579 fixed in Fallacies: %C3 mangled to A13.

Twin-proof: each file carries 28 correct %C3% sequences alongside the 6 corrupted A13% twins — if %C3 weren't the intended encoding, those 28 wouldn't exist. Every A13 is A13% in a wiki-URL context (0 bare) → replacing A13%%C3% is zero-risk.

corrupted restored word
autoritA13%A9 autorit%C3%A9 autorité
(dA13%A9cision) (d%C3%A9cision) décision
PA13%A9tition P%C3%A9tition Pétition
consA13%A9quent cons%C3%A9quent conséquent
vrai_A13%89cossais vrai_%C3%89cossais Écossais
A13%89quivoque %C3%89quivoque Équivoque

Scope: only the 2 active prod templates. The Archive/2022 Memo variants carry the same corruption but are frozen historical snapshots — left untouched (No-Pendulum: don't rewrite history).

Why the three CSVs need no fix — they're clean

Applied the full #579 defect lens (percent-encoding corruption, scheme-less URLs, leaked dev-notes, HTML entities, mojibake) cell-level + raw-byte across all active prod CSVs:

Asset %C3→A13 mojibake entities scheme-less real dev-notes
Scenarii (167×70) 0 0 0 0 0
Virtues (223×66) 0 0 0 0 0
Rules (15×10) 0 0 0 0 0
RulesPP (6×10) 0 0 0 0 0

The %C3A13 corruption is Fallacies + Memo specific — both embed the same French-wiki URL catalog that hit the same encoding mishap. The tabular CSVs don't carry that catalog.

⚠️ The false-positive trap — 59 todo matches are NOT placeholders

The dev-note scan surfaced 59 todo across the three CSVs (Scenarii 24, Virtues 31, Rules 4). A naive reader flags these as leaked placeholder strings (like #579's XXXptet inverser). They are not — every one is the Spanish/Portuguese word todo (= "all/every/whole") inside a complete, correct translation:

  • Virtues (the bulk): syllogism forms "Todo M es P; todo S es M; por lo tanto todo S es P" (= "All M is P…"), "todo o raciocínio", "compreender um todo".
  • Scenarii: "todo va bien", "todo el poder", "todo o resto".
  • Rules: "Todos os outros jogadores", "por todo o júri".

The CLAUDE.md claim "Virtues i18n 100% coverage" HOLDS — cross-checked: for every affected Virtue PK, the fr/en/ru remarks are also populated (no half-translated row).

Lesson recorded in the audit doc: a case-insensitive \bTODO\b regex is a trap on multilingual corpora. Bulk-deleting these 59 cells would have been a catastrophic regression — exactly the "vibecodé" failure mode this project suffered pre-recovery. Future hygiene scans must read full cell values (or scope to isolated ^todo$ / uppercase TODO / XXX markers) before flagging.

Verification

  • Memo fix: A13 6→0 per file, %C3% 28→34, bytes unchanged (4-char→4-char ASCII), CRLF preserved, ConvertFrom-Json VALID.
  • git diff --stat: 2 files / 2 ins / 2 del — surgical (1 line/file; all 6 restorations sit on line 30). Filter for non-A13 deletions returns empty.
  • CSVs: read-only scan, no writes.

Context

Cluster: ai-01 + po-2023 (#568 in flight) + po-2024 (this audit). Follows #579 (Fallacies %C3→A13, merged ed66f800) and #580 (#499 prod-write spec, merged 9db76d91). Same verify-before-code + anti-fab discipline: scanned exhaustively, found one real defect, refused to fabricate 59 placeholder-leaks that were correct translations.

🤖 Worker po-2024 · prod-CSV hygiene audit · Memo %C3 fix (12 URLs) · Scenarii/Virtues/Rules clean · 59 false-positive todo documented

Your and others added 2 commits June 23, 2026 07:05
…e audit

Extends the #579 %C3->A13 fix from Fallacies to the two active prod Memo
templates (Argumentum_Memo_Back_fr.json, Argumentum_Memo_Face_fr.json).
The embedded French-wiki fallacy-URL catalog had the same UTF-8 lead-byte
percent-encoding corruption: 6 occurrences per file (12 total), every
"A13" = "A13%" in wiki-URL context (0 bare). Twin-proof: each file carries
28 correct %C3% sequences alongside the 6 corrupted A13% twins.

Restored URLs decode to valid French (autorite, decision, Petition,
consequent, Ecossais, Equivoque). Byte-exact 4-char->4-char ASCII
substitution: A13 6->0, %C3% 28->34, bytes unchanged, CRLF preserved, JSON
parses valid, git diff surgical (1 line/file).

Scope: only the 2 active prod Memo templates. Archive/2022 variants carry
the same corruption but are frozen historical snapshots (untouched).

Audit (docs/investigations/2026-06-23-prod-csv-hygiene-audit.md): applied
the full #579 defect lens (percent-encoding, scheme-less URLs, dev-notes,
HTML entities, mojibake) cell-level + raw-byte across all active prod
CSVs. Result: Scenarii/Virtues/Rules(RulesPP) CLEAN on every class; the
%C3->A13 corruption is Fallacies+Memo specific (same embedded URL catalog).

CRITICAL false-positive doc: 59 "todo" matches across the three CSVs
(Scenarii 24, Virtues 31, Rules 4) are the Spanish/Portuguese word "todo"
(=all/every) in fully-correct translations (syllogism "Todo M es P", "todo
o raciocinio", rule "Todos os outros jogadores") - NOT placeholder leaks.
Virtues i18n 100% claim HOLDS (cross-checked: fr/en/ru remarks all
populated for affected PKs). Lesson: case-insensitive \bTODO\b is a trap
on multilingual corpora; bulk-deleting these would be a catastrophic
regression.

Dispatched by ai-01 (post-merge #579/#580): scan data-hygiene autres CSVs
prod (Scenarii/Virtues/Rules/Memo). Closes that dispatch.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…"Valid results" gap

Closes the 4th defect class from ai-01's dispatch (verbatim duplicates
between columns). (c1) URL-in-name-col: 0 across Scenarii/Virtues/Rules.
(c2) cross-lang identical: 207 hits but ALL in short fields (proper nouns,
Ibero-Romance cognates, shared wiki links) -- long-prose test (len>=30)
returns 0, ruling out #216-style contamination.

One content-level finding flagged for jsboige (not auto-fixed): the
"Mathematical accuracy" family (pk 71-78) has subfamily_ru = untranslated
English "Valid results" (FR/ES/PT are translated) + pk 71 title_ru.
9 RU cells -- isolated oversight, not mass contamination. Correct RU
term is a content judgment (Достоверные/Правильные/Корректные результаты).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

@clusterManager-Myia clusterManager-Myia left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[NanoClaw]

Deep-reviewed the corruption repair and the bundled audit doc. Verified the byte-level fix independently — it is correct and surgical.

Fix verification (Memo %C3A13) — CONFIRMED CORRECT

I fetched both files at base (ed66f80) and head (a750f72) from raw and compared:

A13 count A13% count %C3% count file size
base Back 6 6 28 216064
head Back 0 0 34 216064
base Face 6 6 28 213927
head Face 0 0 34 213927
  • Diff is exactly 30c30 in both files — only the embedded CSV-string line changes. Nothing else in the JSON touched.
  • Byte sizes identical base→head (4-char A13%→4-char %C3% in-place substitution).
  • CRLF preserved (\r\n at line 30 boundary identical on both sides).
  • JSON parses valid at head.
  • Decoded the 6 restored sequences: autorité, décision, Pétition, conséquent, Écossais, Équivoque — all valid French, matching the surrounding link_fr wiki-URL context.
  • Twin-proof holds: the 28 untouched %C3% sequences on the same line are the control — confirms %C3 is the intended encoding, so restoring the 6 corrupted twins is not a guess.
  • All A13 are A13% (0 bare A13) — zero false-positive risk from the substitution.

Scope discipline is sound: Archive/2022 Memo variants left frozen (verified — only the 2 active templates modified).

Prod hygiene audit doc — solid, one caution

The audit doc (docs/investigations/2026-06-23-prod-csv-hygiene-audit.md) is genuinely valuable: the 59 todo false-positive analysis (Spanish/Portuguese "todo" = "all", not placeholders) is the right call and prevents a catastrophic bulk-delete. Documenting the regex trap is good project hygiene.

Caution — the 9-cell RU "Valid results" gap (§4) is flagged for jsboige but not fixed. That's the correct handling for a translation-judgment item, but worth tracking: pk 71–78 subfamily_ru + pk 71 title_ru hold untranslated English. Consider opening a follow-up issue so it isn't lost.

Security / scope scan — clean

  • No secrets, tokens, keys, or credentials in the diff.
  • No executable code changes — pure data repair (CSV-in-JSON string).
  • No auth, SQL, path-traversal, or external-URL-fetch changes.
  • No scope creep into logic, build, or config files.
  • Audit scripts correctly kept in gitignored tmp/.

Minor observations (non-blocking)

  • The embedded CSV lives on a single JSON line (30c30) — the line is ~200KB. Pre-existing structure, not introduced here, but makes review by patch effectively impossible without raw fetch (which is why the patch returned null). A future refactor splitting this string out would aid maintainability.
  • No automated test coverage for the repair. Given this is data-hygiene on prod CSVs, a smoke test asserting A13 count == 0 on these templates would guard against regression of the exact corruption class.

Base master (correct default), mergeable_state clean, 3 files / +111/-2. The fix is correct, targeted, and well-documented; the audit doc is a genuine contribution. Approve from a data-correctness standpoint — translation judgment on the 9 RU cells is the only open thread.

@jsboige
jsboige merged commit 7745bf3 into master Jun 23, 2026
3 checks passed
@jsboige
jsboige deleted the fix/memo-url-encoding-corruption branch June 23, 2026 07:13
jsboige added a commit that referenced this pull request Jun 23, 2026
…s templates (#584)

The %C3->A13 UTF-8 lead-byte corruption (defect #579, fixed in Memo templates
via PR #581) was also present in the 4 LIVE Fallacies face templates loaded by
the pipeline: Cards/Fallacies/Argumentum_Fallacies_Face_{,_2_,_3_,Web_}fr.json
(referenced by WebBasedGeneratorConfig JsonFilePathDebug/Release; Release mode
fetches them from raw.githubusercontent.com/.../master, so the corruption hit
generated cards directly).

Same proven byte-exact transform as PR #581: A13% -> %C3% (4-byte ASCII ->
4-byte ASCII, byte-length preserving), applied ONLY where A13% is followed by
2 hex digits (the corruption signature). Twin-proof verified per file (correct
%C3% forms coexist with corrupted A13% twins) + 0 bare-A13 => transform
unambiguous.

Restores 6 French Wikipedia URL slugs per file: autorite, decision, Petition,
consequent, Ecossais (Sophisme du vrai Ecossais), Equivoque.

Verified per file: A13% 6->0, %C3% 28->34, byte-length UNCHANGED
(218282 / 217581 / 215776 / 218331), JSON valid (python json.load), CRLF
preserved, UTF-8 no-BOM. Same defect class + same fix as merged PR #581.

Co-authored-by: Your <your.email@example.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
jsboige added a commit that referenced this pull request Jul 4, 2026
…tability fixes) (#134) (#689)

The v0.9.0 entry predating bundle v3 missed the print-production and
late-cycle stability work. This refresh brings it in line with master
`21e2c666`, cross-checked against the release-validation dossier v4
(docs/RELEASE-VALIDATION-v0.9.0.md §3.3) and dashboard decisions #26-69.

Added:
- "Print Production (CMYK & Print&Play)" section: Ghostscript post-process
  (#632), `--pdf-cmyk` entry-point (#652), bundle v3 = 80 PDFs DeviceCMYK +
  SWOP OutputIntent (6.18 GB RGB -> 5.30 GB CMYK), P&P Standard/Light
  (#645/#648-650, 64 -> 80 PDFs), GS timeout 180->900s (#670)
- "OWL Ontology (Bilingual EN/FR)" section: Fallacies OWL 5.07 MB (#634
  regen), Virtues OWL (#592/#499 Phase 2), honest scope note (EN+FR only,
  not 8-lang)
- "Fixed - Pipeline Stability (Jun-Jul 2026)" section: harvest deadlock
  #651, serial retry #613/#676, logger Spectre #630/#655, CMYK oxymore,
  Johnny 6.1.3 #653, Rules i18n refonte #640, CSV hygiene #579/#581/#584,
  OWL staleness #634

Corrected (code=truth):
- Magick.NET 14.13.1 -> 14.14.0 (verified in .csproj)
- Test count 548 -> 578 pass / 1 known-fail #133 / 5 skip / 584 total
  (empirical `dotnet test` on Argumentum.AssetConverter.Tests, 2026-07-04,
  .NET 9). Previous counts in docs (548/549/570) were all stale

No existing content regressed; all prior sections preserved.

Co-authored-by: Your <your.email@example.com>
Co-authored-by: Claude-Code <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants