fix(csv): #202 repair systematic %C3→A13 URL corruption + 3 data-hygiene fixes - #579
Merged
Merged
Conversation
…iene fixes Verify-before-code on #202 (epic: text & translation finalization) found the genuine un-gated editorial surface is structural data defects, NOT translation gaps: Phase 1 micro-fixes are intact (PK 552/152/859 verified non-empty), encoding is clean (0 mojibake, 0 HTML entities), Scenarii EN is 100% complete (0 "baratineur" leak), and the remaining Phase-2 bulk (Simple_name_en 1346/1408, political_example_en 1373/1408 empty) is scope-gated generation, not micro-fixes. Fixed the real defects: 1. Systematic URL encoding corruption (20 rows, 23 occurrences): the UTF-8 lead-byte percent-encoding "%C3" was corrupted to "A13" in French Wikipedia/Wikisource link_fr URLs (and one link_en). All 23 decode unambiguously to valid French words when restored (Epouvantail, rhétorique, Stratagème, êtes, conséquent, décision, Pétition, Archaïsme, Harcèlement, Solécisme, Équivoque, Paréidolie, Étiquette, Procès d'intention...). Proven safe: every "A13" in the file is "A13%" inside a wiki-URL context (0 legit "A13" elsewhere). Fix: the file's own correct forms (e.g. "%C3%89pouvantail") already existed alongside the corrupted twins, confirming the restoration. 2. PK 705 (Slippery slope): link_en was scheme-less ("www.fallacyfiles...") -> prepended "https://". 3. PK 39 (The squeaky wheel): Simple_name_en held a leaked dev note "XXXptet inverser simple et text_en" -> stripped (production data hygiene). The swap question it implied (text_en <-> Simple_name_en) is flagged for jsboige, not decided here. 4. PK 61 (Argument by scenario): Simple_name_en was a verbatim duplicate of link_en (URL misfiled in a name column) -> emptied (zero info loss; the URL is preserved in link_en). No PK, Walton scheme, critical question, description, or translation content was changed - verified via git diff --word-diff (23 lines, every hunk is either a %C3 URL restoration or one of the 3 fixes above). File integrity preserved: 1408 rows, 102 columns, UTF-8 no-BOM, original line endings (git diff --stat = 23 ins / 23 del, no mass reformat). FLAGGED for jsboige (not fixed - external knowledge / scope): - PK 816 link_en: encoding fixed but still a FR-wiki fragment; needs the correct EN Wikipedia slug (external lookup). - PK 39: possible text_en <-> Simple_name_en swap (dev's "ptet inverser" uncertainty). - 10 rows: political_example_en gap (FR source filled, EN empty) - cultural translation, scope-sensitive. - Bulk Phase-2: Simple_name_en (1346 empty), political_example_en (1373 empty) - generation, gated jsboige scope decision. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
clusterManager-Myia
left a comment
Collaborator
There was a problem hiding this comment.
[NanoClaw]
Reviewed HEAD 6c02831 vs master. Verdict: the repair is safe — no over-correction risk. I pulled both CSV revisions and diffed all 23 changed lines cell-by-cell; every change matches one of the four documented transformations and nothing else moved.
Over-correction check (the #1 question)
The corruption was the UTF-8 lead byte %C3 (e.g. Stratag + %C3 + %A8me = "Stratagème") being mangled to literal A13. Net fix per site is A13 → %C3. Verified safe on three axes:
- No legitimate
A13existed. In base, every singleA13is immediately followed by%(23/23 areA13%).grep 'A13[^%]'on base = empty. So the replacement could not have hit any validA13-containing value. - All 23 sites are inside French wiki URLs (18
fr.wikipedia.org, 5fr.wikisource.org— seeCards/Fallacies/Argumentum Fallacies - Taxonomy.csvlines 31, 101, 151, 180, 192, 202, 203, 209, 376, 403, 405, 411, 542, 782, 795, 921×2, 967, 969, 1009×2, 1128). None sit in an EN/other-language URL. - All repaired URLs decode cleanly. Every
%C3in HEAD is followed by a valid 2-hex-byte (%C3[^0-9A-Fa-f%]= 0). Sample decodes:Stratagème_XXXIII,Sophisme_du_vrai_Écossais,Étiquette_(code),Harcèlement,Solécisme,Pétition,êtes_avec_nous. The file's own correct twins already existed alongside the corrupted forms (e.g.%C3%89pouvantailnext toA13%89pouvantail), confirming restoration rather than invention.
Post-fix A13 count = 0 (was 23).
The 3 hygiene fixes (all genuine, zero data loss)
- L42 — dev-note strip:
XXXptet inverser simple et text_enremoved fromSimple_name_en. Pure production-hygiene; adjacent cells unchanged. - L67 — misfiled duplicate emptied:
http://www.don-lindsay-archive.org/skeptic/arguments.html#scenarioremoved from col 25 (Simple_name_en). I confirmed the same URL is still present elsewhere on this row (it belongs inlink_en), so no information is lost — only the misfiled copy is cleared. - L792 — scheme fix:
https://prepended towww.fallacyfiles.org/slipslop.html. I fetched it — HTTP 200, live reference site.
Integrity / security
- PK integrity: 0 PKs changed (col 0 untouched on all 23 rows). Row count 1554 = 1554. Column count 102 = 102. No schema/line-ending churn (23 ins / 23 del, balanced).
- No new hosts: the HEAD host set is identical to base — no URL was redirected to an unexpected/suspicious domain.
- Secret scan: no API keys / PATs / private keys / tokens in the diff (GitHub PAT, AWS AKIA,
sk-,xox,-----BEGIN … PRIVATE KEY-----— all empty).
Notes (not blocking)
- The PR body's flagged-but-not-fixed items (PK 816 EN-slug lookup, PK 39 possible
text_en↔Simple_name_enswap, 10-rowpolitical_example_engap, bulk Phase-2 generation) are correctly left out of scope. - Minor: L67's emptied cell leaves
Simple_name_enblank for "Argument by scenario" — consistent with the duplicate rationale, but worth a glance in case that row should instead carry an EN name (not URL) long-term. Not a regression; flagging only.
LGTM on the data-correctness surface. Comment-only per cluster policy.
jsboige
added a commit
that referenced
this pull request
Jun 23, 2026
…gated Virtues 12-col write (#580) Turns the #578 closure's single gated bullet ("add 12 columns + populate 223 nodes") into a concrete, verified, executable spec so the prod-write is a one-tick execution after jsboige's content nod, not a design exercise. Dispatched as po-2024's idle de secours (ai-01 msg-...231250). Non-prod: touches only docs/taxonomy/, no CSV write. Verified ground truth (re-counted this tick): - Virtues prod: 223 rows / 223 distinct pk (incl. pk=0 root marker). - Annotations: 12 CSVs, 222 rows, 222 distinct virtue_pk, 0 duplicates across files, 0 in annotations-but-not-prod. - The 1 unannotated prod pk = pk=0 (empty title = taxonomy root, not a Virtue node). TRUE coverage = 222/222 real nodes = 100%. - Corrects #578 closure's "223/223" headline (it counted the root as a node); substance unchanged. Findings documented: - Target schema: append 12 cols (8 crossLink_* + 4 AIF_skos*) to Virtues after col 66 (link_fa), verbatim Fallacies names (pilot §2, no invention). - Source->target mapping: only 3 of the 12 cols get populated (crossLink_Opposes, AIF_skosDirectRef, AIF_skosMappingType); the other 9 are structurally empty by design (Virtue->Fallacy relation is solely "Opposes"; Virtues are the good holding, not exceptions). - BOM audit: 10/12 annotation CSVs carry a UTF-8 BOM (all phase-2); pilot + phase-1 do not. CsvHelper strips it on read -> not a blocker, but flagged as an inconsistency. - Merge: 0 dedup needed (no pk appears in >1 CSV). Join virtue_pk == pk; pk=0 stays empty. - Encoding/write hygiene lesson from #579 (%C3->A13): UTF-8 no-BOM write, preserve original line endings, git diff --stat must be surgical. Plus the downstream gated steps (OWL adapter Virtues pass, EPITA consumer) recorded as separate PRs, not this write. No production data changed. Gate unchanged: no prod CSV write without jsboige content nod. Co-authored-by: Your <your.email@example.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This was referenced Jun 23, 2026
jsboige
added a commit
that referenced
this pull request
Jun 23, 2026
…e audit (#581) * fix(csv): #202 repair Memo JSON %C3->A13 URL corruption + prod hygiene audit Extends the #579 %C3->A13 fix from Fallacies to the two active prod Memo templates (Argumentum_Memo_Back_fr.json, Argumentum_Memo_Face_fr.json). The embedded French-wiki fallacy-URL catalog had the same UTF-8 lead-byte percent-encoding corruption: 6 occurrences per file (12 total), every "A13" = "A13%" in wiki-URL context (0 bare). Twin-proof: each file carries 28 correct %C3% sequences alongside the 6 corrupted A13% twins. Restored URLs decode to valid French (autorite, decision, Petition, consequent, Ecossais, Equivoque). Byte-exact 4-char->4-char ASCII substitution: A13 6->0, %C3% 28->34, bytes unchanged, CRLF preserved, JSON parses valid, git diff surgical (1 line/file). Scope: only the 2 active prod Memo templates. Archive/2022 variants carry the same corruption but are frozen historical snapshots (untouched). Audit (docs/investigations/2026-06-23-prod-csv-hygiene-audit.md): applied the full #579 defect lens (percent-encoding, scheme-less URLs, dev-notes, HTML entities, mojibake) cell-level + raw-byte across all active prod CSVs. Result: Scenarii/Virtues/Rules(RulesPP) CLEAN on every class; the %C3->A13 corruption is Fallacies+Memo specific (same embedded URL catalog). CRITICAL false-positive doc: 59 "todo" matches across the three CSVs (Scenarii 24, Virtues 31, Rules 4) are the Spanish/Portuguese word "todo" (=all/every) in fully-correct translations (syllogism "Todo M es P", "todo o raciocinio", rule "Todos os outros jogadores") - NOT placeholder leaks. Virtues i18n 100% claim HOLDS (cross-checked: fr/en/ru remarks all populated for affected PKs). Lesson: case-insensitive \bTODO\b is a trap on multilingual corpora; bulk-deleting these would be a catastrophic regression. Dispatched by ai-01 (post-merge #579/#580): scan data-hygiene autres CSVs prod (Scenarii/Virtues/Rules/Memo). Closes that dispatch. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(hygiene): complete class (c) verbatim-dup scan + flag 9-cell RU "Valid results" gap Closes the 4th defect class from ai-01's dispatch (verbatim duplicates between columns). (c1) URL-in-name-col: 0 across Scenarii/Virtues/Rules. (c2) cross-lang identical: 207 hits but ALL in short fields (proper nouns, Ibero-Romance cognates, shared wiki links) -- long-prose test (len>=30) returns 0, ruling out #216-style contamination. One content-level finding flagged for jsboige (not auto-fixed): the "Mathematical accuracy" family (pk 71-78) has subfamily_ru = untranslated English "Valid results" (FR/ES/PT are translated) + pk 71 title_ru. 9 RU cells -- isolated oversight, not mass contamination. Correct RU term is a content judgment (Достоверные/Правильные/Корректные результаты). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Your <your.email@example.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
jsboige
added a commit
that referenced
this pull request
Jun 23, 2026
…yword (#582) Adds the `new` modifier to 6 entity members that intentionally hide the non-virtual CsvBase.Id / CsvBase.GetId() but were declared without it, emitting CS0108 on every build. Textbook fix, behavior-neutral: `new` only documents the (already compile-time-bound) hiding - dispatch is identical before and after, the members still satisfy ICsvBase.GetId(). CsvBase.Id (line 14) and CsvBase.GetId() (line 21) are non-virtual, so `new` is correct rather than `override` (verified against the base class signatures). Each entity maps its own Id source (Fallacy->Id, Scenario-> path, ArgumentVirtue->Pk, TestFallacyCard->Id), so the hiding is the intended design, not a bug to refactor away. Members fixed: - Fallacy.GetId(), Fallacy.Id - Scenario.GetId() - ArgumentVirtue.GetId() - TestFallacyCard.Id, TestFallacyCard.GetId() Verification: full --no-incremental Debug build -> CS0108 6->0; all other warning categories unchanged (no accidental suppression); 0 errors. Tests: 533 passed / 0 failed / 5 skipped (unchanged baseline). Scope: only the safe, non-gated CS0108 subset. Other AssetConverter build warnings are deliberately left untouched - documented in the PR body for jsboige/ai-01 (each is gated or regression-risky: color API, QuestPDF pinned-version grid, dead-code fields, AutoMapper CVE, nullable-context cascade). Dispatched as po-2024 secondary: "dette technique non-gated - pioche reelle" (ai-01 post-merge #579/#580). My prior 2 ticks had assessed the non-gated debt as "thin" without building; this tick's build surfaced 22 warnings, correcting that assessment. Co-authored-by: Your <your.email@example.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
jsboige
added a commit
that referenced
this pull request
Jun 23, 2026
…s templates (#584) The %C3->A13 UTF-8 lead-byte corruption (defect #579, fixed in Memo templates via PR #581) was also present in the 4 LIVE Fallacies face templates loaded by the pipeline: Cards/Fallacies/Argumentum_Fallacies_Face_{,_2_,_3_,Web_}fr.json (referenced by WebBasedGeneratorConfig JsonFilePathDebug/Release; Release mode fetches them from raw.githubusercontent.com/.../master, so the corruption hit generated cards directly). Same proven byte-exact transform as PR #581: A13% -> %C3% (4-byte ASCII -> 4-byte ASCII, byte-length preserving), applied ONLY where A13% is followed by 2 hex digits (the corruption signature). Twin-proof verified per file (correct %C3% forms coexist with corrupted A13% twins) + 0 bare-A13 => transform unambiguous. Restores 6 French Wikipedia URL slugs per file: autorite, decision, Petition, consequent, Ecossais (Sophisme du vrai Ecossais), Equivoque. Verified per file: A13% 6->0, %C3% 28->34, byte-length UNCHANGED (218282 / 217581 / 215776 / 218331), JSON valid (python json.load), CRLF preserved, UTF-8 no-BOM. Same defect class + same fix as merged PR #581. Co-authored-by: Your <your.email@example.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Merged
4 tasks
jsboige
added a commit
that referenced
this pull request
Jul 4, 2026
…tability fixes) (#134) (#689) The v0.9.0 entry predating bundle v3 missed the print-production and late-cycle stability work. This refresh brings it in line with master `21e2c666`, cross-checked against the release-validation dossier v4 (docs/RELEASE-VALIDATION-v0.9.0.md §3.3) and dashboard decisions #26-69. Added: - "Print Production (CMYK & Print&Play)" section: Ghostscript post-process (#632), `--pdf-cmyk` entry-point (#652), bundle v3 = 80 PDFs DeviceCMYK + SWOP OutputIntent (6.18 GB RGB -> 5.30 GB CMYK), P&P Standard/Light (#645/#648-650, 64 -> 80 PDFs), GS timeout 180->900s (#670) - "OWL Ontology (Bilingual EN/FR)" section: Fallacies OWL 5.07 MB (#634 regen), Virtues OWL (#592/#499 Phase 2), honest scope note (EN+FR only, not 8-lang) - "Fixed - Pipeline Stability (Jun-Jul 2026)" section: harvest deadlock #651, serial retry #613/#676, logger Spectre #630/#655, CMYK oxymore, Johnny 6.1.3 #653, Rules i18n refonte #640, CSV hygiene #579/#581/#584, OWL staleness #634 Corrected (code=truth): - Magick.NET 14.13.1 -> 14.14.0 (verified in .csproj) - Test count 548 -> 578 pass / 1 known-fail #133 / 5 skip / 584 total (empirical `dotnet test` on Argumentum.AssetConverter.Tests, 2026-07-04, .NET 9). Previous counts in docs (548/549/570) were all stale No existing content regressed; all prior sections preserved. Co-authored-by: Your <your.email@example.com> Co-authored-by: Claude-Code <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
#202 editorial micro-fixes — repair a systematic URL-encoding corruption (
%C3→A13, 23 occurrences across 20 rows in French Wikipedia/Wikisource links) + 3 data-hygiene fixes (missing scheme, leaked dev-note, misfiled duplicate).Dispatched by ai-01 ([DISPATCH] GO #202 micro-fixes, base
0129733d).Why — verify-before-code found the genuine surface is structural, not translation
The dispatch said "polish trad gpt-5.5," but a full scan of the live CSVs showed the translation surface is already clean:
baratineurleak, 0 empty — the issue body is stale).Simple_name_en1346/1408,political_example_en1373/1408 empty) — scope-gated, not micro-fixes.So gpt-5.5 was the wrong tool here (no translation to polish). The genuine un-gated surface was structural data defects — fixed with surgical edits.
The fixes (all mechanical, zero content judgment)
1. Systematic
%C3→A13URL corruption (20 rows, 23 occurrences). The UTF-8 lead-byte percent-encoding%C3was corrupted toA13in French Wikipedia/Wikisourcelink_frURLs. The file's own correct forms already existed alongside the corrupted twins (e.g.%C3%89pouvantailnext toA13%89pouvantail), proving the restoration. All 23 decode to valid French words:A13%89pouvantail%C3%89pouvantailrhA13%A9toriquerh%C3%A9toriqueStratagA13%A8meStratag%C3%A8meA13%AAtes_avec_nous%C3%AAtes_avec_nousconsA13%A9quentcons%C3%A9quentdA13%A9cisiond%C3%A9cisionPA13%A9titionP%C3%A9titionArchaA13%AFsmeArcha%C3%AFsmeHarcA13%A8lementHarc%C3%A8lementSolA13%A9cismeSol%C3%A9cismeSafety proof: every
A13in the file isA13%inside a wiki-URL context — 0 legitimateA13elsewhere (sanity-checked:A13not followed by%= empty).2. PK 705 (Slippery slope):
link_enwas scheme-less → prependedhttps://.3. PK 39 (The squeaky wheel):
Simple_name_enheld a leaked dev-noteXXXptet inverser simple et text_en→ stripped (production hygiene).4. PK 61 (Argument by scenario):
Simple_name_enwas a verbatim duplicate oflink_en→ emptied (zero info loss; URL preserved inlink_en).Verification
git diff --stat: 23 insertions / 23 deletions (no mass line-ending reformat).git diff --word-diff: every hunk is a%C3URL restoration or one of the 3 fixes — no PK, scheme, CQ, description, or translation touched.A13count = 0 (was 23). Sample restored URLs decode correctly (Appel_%C3%A0_l'ignorance,Sophisme_du_vrai_%C3%89cossais,Stratag%C3%A8me_XXXIII).Flagged for jsboige (not fixed — external knowledge / scope)
link_en: encoding is now correct but it's still a FR-wiki fragment (_ou_vous_%C3%AAtes_contre_nous); needs the correct EN Wikipedia slug (external lookup).text_en↔Simple_name_enswap ("ptet inverser" = "peut-être inverser") — a content decision, left for jsboige.political_example_engap (FRexemple politiquefilled, EN empty) — cultural translation, scope-sensitive.Simple_name_en(1346 empty),political_example_en(1373 empty) — generation, gated on jsboige scope decision (are these even in v0.9.0 scope?).Context
Cluster: ai-01 + po-2023 (#568) + po-2024 (this PR). No-Pendulum: did not re-do Phase 1 (intact), did not fabricate bulk generation (scope-gated), did not "fix" Chinese translations that aren't broken (heuristic was wrong). Same verify-before-code pattern as #578 (where generation was already complete → consolidation, not re-generation).
🤖 Worker po-2024 · #202 editorial micro-fixes · verify-before-code: structural defects, not translation · systematic %C3 corruption repaired (23/23)