Skip to content

fix(csv): #202 repair systematic %C3→A13 URL corruption + 3 data-hygiene fixes - #579

Merged
jsboige merged 1 commit into
masterfrom
fix/202-csv-url-corruption-and-hygiene
Jun 23, 2026
Merged

fix(csv): #202 repair systematic %C3→A13 URL corruption + 3 data-hygiene fixes#579
jsboige merged 1 commit into
masterfrom
fix/202-csv-url-corruption-and-hygiene

Conversation

@jsboige

@jsboige jsboige commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

What

#202 editorial micro-fixes — repair a systematic URL-encoding corruption (%C3A13, 23 occurrences across 20 rows in French Wikipedia/Wikisource links) + 3 data-hygiene fixes (missing scheme, leaked dev-note, misfiled duplicate).

Dispatched by ai-01 ([DISPATCH] GO #202 micro-fixes, base 0129733d).

Why — verify-before-code found the genuine surface is structural, not translation

The dispatch said "polish trad gpt-5.5," but a full scan of the live CSVs showed the translation surface is already clean:

So gpt-5.5 was the wrong tool here (no translation to polish). The genuine un-gated surface was structural data defects — fixed with surgical edits.

The fixes (all mechanical, zero content judgment)

1. Systematic %C3A13 URL corruption (20 rows, 23 occurrences). The UTF-8 lead-byte percent-encoding %C3 was corrupted to A13 in French Wikipedia/Wikisource link_fr URLs. The file's own correct forms already existed alongside the corrupted twins (e.g. %C3%89pouvantail next to A13%89pouvantail), proving the restoration. All 23 decode to valid French words:

corrupted restored word
A13%89pouvantail %C3%89pouvantail Épouvantail
rhA13%A9torique rh%C3%A9torique rhétorique
StratagA13%A8me Stratag%C3%A8me Stratagème
A13%AAtes_avec_nous %C3%AAtes_avec_nous êtes
consA13%A9quent cons%C3%A9quent conséquent
dA13%A9cision d%C3%A9cision décision
PA13%A9tition P%C3%A9tition Pétition
ArchaA13%AFsme Archa%C3%AFsme Archaïsme
HarcA13%A8lement Harc%C3%A8lement Harcèlement
SolA13%A9cisme Sol%C3%A9cisme Solécisme
... ... (Équivoque, Paréidolie, Étiquette, Procès d'intention, charité, autorité)

Safety proof: every A13 in the file is A13% inside a wiki-URL context — 0 legitimate A13 elsewhere (sanity-checked: A13 not followed by % = empty).

2. PK 705 (Slippery slope): link_en was scheme-less → prepended https://.
3. PK 39 (The squeaky wheel): Simple_name_en held a leaked dev-note XXXptet inverser simple et text_en → stripped (production hygiene).
4. PK 61 (Argument by scenario): Simple_name_en was a verbatim duplicate of link_en → emptied (zero info loss; URL preserved in link_en).

Verification

  • git diff --stat: 23 insertions / 23 deletions (no mass line-ending reformat).
  • git diff --word-diff: every hunk is a %C3 URL restoration or one of the 3 fixes — no PK, scheme, CQ, description, or translation touched.
  • Post-fix scan: A13 count = 0 (was 23). Sample restored URLs decode correctly (Appel_%C3%A0_l'ignorance, Sophisme_du_vrai_%C3%89cossais, Stratag%C3%A8me_XXXIII).
  • File integrity: 1408 rows, 102 columns (Import-Csv parse clean), UTF-8 no-BOM, original line endings preserved.

Flagged for jsboige (not fixed — external knowledge / scope)

  • PK 816 link_en: encoding is now correct but it's still a FR-wiki fragment (_ou_vous_%C3%AAtes_contre_nous); needs the correct EN Wikipedia slug (external lookup).
  • PK 39: the dev-note implied a possible text_enSimple_name_en swap ("ptet inverser" = "peut-être inverser") — a content decision, left for jsboige.
  • 10 rows: political_example_en gap (FR exemple politique filled, EN empty) — cultural translation, scope-sensitive.
  • Bulk Phase-2: Simple_name_en (1346 empty), political_example_en (1373 empty) — generation, gated on jsboige scope decision (are these even in v0.9.0 scope?).

Context

Cluster: ai-01 + po-2023 (#568) + po-2024 (this PR). No-Pendulum: did not re-do Phase 1 (intact), did not fabricate bulk generation (scope-gated), did not "fix" Chinese translations that aren't broken (heuristic was wrong). Same verify-before-code pattern as #578 (where generation was already complete → consolidation, not re-generation).

🤖 Worker po-2024 · #202 editorial micro-fixes · verify-before-code: structural defects, not translation · systematic %C3 corruption repaired (23/23)

…iene fixes

Verify-before-code on #202 (epic: text & translation finalization) found
the genuine un-gated editorial surface is structural data defects, NOT
translation gaps: Phase 1 micro-fixes are intact (PK 552/152/859 verified
non-empty), encoding is clean (0 mojibake, 0 HTML entities), Scenarii EN
is 100% complete (0 "baratineur" leak), and the remaining Phase-2 bulk
(Simple_name_en 1346/1408, political_example_en 1373/1408 empty) is
scope-gated generation, not micro-fixes. Fixed the real defects:

1. Systematic URL encoding corruption (20 rows, 23 occurrences): the
   UTF-8 lead-byte percent-encoding "%C3" was corrupted to "A13" in
   French Wikipedia/Wikisource link_fr URLs (and one link_en). All 23
   decode unambiguously to valid French words when restored (Epouvantail,
   rhétorique, Stratagème, êtes, conséquent, décision, Pétition,
   Archaïsme, Harcèlement, Solécisme, Équivoque, Paréidolie, Étiquette,
   Procès d'intention...). Proven safe: every "A13" in the file is
   "A13%" inside a wiki-URL context (0 legit "A13" elsewhere). Fix: the
   file's own correct forms (e.g. "%C3%89pouvantail") already existed
   alongside the corrupted twins, confirming the restoration.

2. PK 705 (Slippery slope): link_en was scheme-less ("www.fallacyfiles...")
   -> prepended "https://".
3. PK 39 (The squeaky wheel): Simple_name_en held a leaked dev note
   "XXXptet inverser simple et text_en" -> stripped (production data
   hygiene). The swap question it implied (text_en <-> Simple_name_en)
   is flagged for jsboige, not decided here.
4. PK 61 (Argument by scenario): Simple_name_en was a verbatim duplicate
   of link_en (URL misfiled in a name column) -> emptied (zero info loss;
   the URL is preserved in link_en).

No PK, Walton scheme, critical question, description, or translation
content was changed - verified via git diff --word-diff (23 lines, every
hunk is either a %C3 URL restoration or one of the 3 fixes above). File
integrity preserved: 1408 rows, 102 columns, UTF-8 no-BOM, original line
endings (git diff --stat = 23 ins / 23 del, no mass reformat).

FLAGGED for jsboige (not fixed - external knowledge / scope):
- PK 816 link_en: encoding fixed but still a FR-wiki fragment; needs the
  correct EN Wikipedia slug (external lookup).
- PK 39: possible text_en <-> Simple_name_en swap (dev's "ptet inverser"
  uncertainty).
- 10 rows: political_example_en gap (FR source filled, EN empty) -
  cultural translation, scope-sensitive.
- Bulk Phase-2: Simple_name_en (1346 empty), political_example_en
  (1373 empty) - generation, gated jsboige scope decision.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

@clusterManager-Myia clusterManager-Myia left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[NanoClaw]

Reviewed HEAD 6c02831 vs master. Verdict: the repair is safe — no over-correction risk. I pulled both CSV revisions and diffed all 23 changed lines cell-by-cell; every change matches one of the four documented transformations and nothing else moved.

Over-correction check (the #1 question)

The corruption was the UTF-8 lead byte %C3 (e.g. Stratag + %C3 + %A8me = "Stratagème") being mangled to literal A13. Net fix per site is A13%C3. Verified safe on three axes:

  • No legitimate A13 existed. In base, every single A13 is immediately followed by % (23/23 are A13%). grep 'A13[^%]' on base = empty. So the replacement could not have hit any valid A13-containing value.
  • All 23 sites are inside French wiki URLs (18 fr.wikipedia.org, 5 fr.wikisource.org — see Cards/Fallacies/Argumentum Fallacies - Taxonomy.csv lines 31, 101, 151, 180, 192, 202, 203, 209, 376, 403, 405, 411, 542, 782, 795, 921×2, 967, 969, 1009×2, 1128). None sit in an EN/other-language URL.
  • All repaired URLs decode cleanly. Every %C3 in HEAD is followed by a valid 2-hex-byte (%C3[^0-9A-Fa-f%] = 0). Sample decodes: Stratagème_XXXIII, Sophisme_du_vrai_Écossais, Étiquette_(code), Harcèlement, Solécisme, Pétition, êtes_avec_nous. The file's own correct twins already existed alongside the corrupted forms (e.g. %C3%89pouvantail next to A13%89pouvantail), confirming restoration rather than invention.

Post-fix A13 count = 0 (was 23).

The 3 hygiene fixes (all genuine, zero data loss)

  • L42 — dev-note strip: XXXptet inverser simple et text_en removed from Simple_name_en. Pure production-hygiene; adjacent cells unchanged.
  • L67 — misfiled duplicate emptied: http://www.don-lindsay-archive.org/skeptic/arguments.html#scenario removed from col 25 (Simple_name_en). I confirmed the same URL is still present elsewhere on this row (it belongs in link_en), so no information is lost — only the misfiled copy is cleared.
  • L792 — scheme fix: https:// prepended to www.fallacyfiles.org/slipslop.html. I fetched it — HTTP 200, live reference site.

Integrity / security

  • PK integrity: 0 PKs changed (col 0 untouched on all 23 rows). Row count 1554 = 1554. Column count 102 = 102. No schema/line-ending churn (23 ins / 23 del, balanced).
  • No new hosts: the HEAD host set is identical to base — no URL was redirected to an unexpected/suspicious domain.
  • Secret scan: no API keys / PATs / private keys / tokens in the diff (GitHub PAT, AWS AKIA, sk-, xox, -----BEGIN … PRIVATE KEY----- — all empty).

Notes (not blocking)

  • The PR body's flagged-but-not-fixed items (PK 816 EN-slug lookup, PK 39 possible text_enSimple_name_en swap, 10-row political_example_en gap, bulk Phase-2 generation) are correctly left out of scope.
  • Minor: L67's emptied cell leaves Simple_name_en blank for "Argument by scenario" — consistent with the duplicate rationale, but worth a glance in case that row should instead carry an EN name (not URL) long-term. Not a regression; flagging only.

LGTM on the data-correctness surface. Comment-only per cluster policy.

jsboige added a commit that referenced this pull request Jun 23, 2026
…gated Virtues 12-col write (#580)

Turns the #578 closure's single gated bullet ("add 12 columns + populate
223 nodes") into a concrete, verified, executable spec so the prod-write
is a one-tick execution after jsboige's content nod, not a design
exercise. Dispatched as po-2024's idle de secours (ai-01 msg-...231250).
Non-prod: touches only docs/taxonomy/, no CSV write.

Verified ground truth (re-counted this tick):
- Virtues prod: 223 rows / 223 distinct pk (incl. pk=0 root marker).
- Annotations: 12 CSVs, 222 rows, 222 distinct virtue_pk, 0 duplicates
  across files, 0 in annotations-but-not-prod.
- The 1 unannotated prod pk = pk=0 (empty title = taxonomy root, not a
  Virtue node). TRUE coverage = 222/222 real nodes = 100%.
- Corrects #578 closure's "223/223" headline (it counted the root as a
  node); substance unchanged.

Findings documented:
- Target schema: append 12 cols (8 crossLink_* + 4 AIF_skos*) to Virtues
  after col 66 (link_fa), verbatim Fallacies names (pilot §2, no invention).
- Source->target mapping: only 3 of the 12 cols get populated
  (crossLink_Opposes, AIF_skosDirectRef, AIF_skosMappingType); the other
  9 are structurally empty by design (Virtue->Fallacy relation is solely
  "Opposes"; Virtues are the good holding, not exceptions).
- BOM audit: 10/12 annotation CSVs carry a UTF-8 BOM (all phase-2); pilot
  + phase-1 do not. CsvHelper strips it on read -> not a blocker, but
  flagged as an inconsistency.
- Merge: 0 dedup needed (no pk appears in >1 CSV). Join virtue_pk == pk;
  pk=0 stays empty.
- Encoding/write hygiene lesson from #579 (%C3->A13): UTF-8 no-BOM write,
  preserve original line endings, git diff --stat must be surgical.

Plus the downstream gated steps (OWL adapter Virtues pass, EPITA consumer)
recorded as separate PRs, not this write.

No production data changed. Gate unchanged: no prod CSV write without
jsboige content nod.

Co-authored-by: Your <your.email@example.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
@jsboige
jsboige merged commit ed66f80 into master Jun 23, 2026
3 checks passed
@jsboige
jsboige deleted the fix/202-csv-url-corruption-and-hygiene branch June 23, 2026 03:12
jsboige added a commit that referenced this pull request Jun 23, 2026
…e audit (#581)

* fix(csv): #202 repair Memo JSON %C3->A13 URL corruption + prod hygiene audit

Extends the #579 %C3->A13 fix from Fallacies to the two active prod Memo
templates (Argumentum_Memo_Back_fr.json, Argumentum_Memo_Face_fr.json).
The embedded French-wiki fallacy-URL catalog had the same UTF-8 lead-byte
percent-encoding corruption: 6 occurrences per file (12 total), every
"A13" = "A13%" in wiki-URL context (0 bare). Twin-proof: each file carries
28 correct %C3% sequences alongside the 6 corrupted A13% twins.

Restored URLs decode to valid French (autorite, decision, Petition,
consequent, Ecossais, Equivoque). Byte-exact 4-char->4-char ASCII
substitution: A13 6->0, %C3% 28->34, bytes unchanged, CRLF preserved, JSON
parses valid, git diff surgical (1 line/file).

Scope: only the 2 active prod Memo templates. Archive/2022 variants carry
the same corruption but are frozen historical snapshots (untouched).

Audit (docs/investigations/2026-06-23-prod-csv-hygiene-audit.md): applied
the full #579 defect lens (percent-encoding, scheme-less URLs, dev-notes,
HTML entities, mojibake) cell-level + raw-byte across all active prod
CSVs. Result: Scenarii/Virtues/Rules(RulesPP) CLEAN on every class; the
%C3->A13 corruption is Fallacies+Memo specific (same embedded URL catalog).

CRITICAL false-positive doc: 59 "todo" matches across the three CSVs
(Scenarii 24, Virtues 31, Rules 4) are the Spanish/Portuguese word "todo"
(=all/every) in fully-correct translations (syllogism "Todo M es P", "todo
o raciocinio", rule "Todos os outros jogadores") - NOT placeholder leaks.
Virtues i18n 100% claim HOLDS (cross-checked: fr/en/ru remarks all
populated for affected PKs). Lesson: case-insensitive \bTODO\b is a trap
on multilingual corpora; bulk-deleting these would be a catastrophic
regression.

Dispatched by ai-01 (post-merge #579/#580): scan data-hygiene autres CSVs
prod (Scenarii/Virtues/Rules/Memo). Closes that dispatch.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs(hygiene): complete class (c) verbatim-dup scan + flag 9-cell RU "Valid results" gap

Closes the 4th defect class from ai-01's dispatch (verbatim duplicates
between columns). (c1) URL-in-name-col: 0 across Scenarii/Virtues/Rules.
(c2) cross-lang identical: 207 hits but ALL in short fields (proper nouns,
Ibero-Romance cognates, shared wiki links) -- long-prose test (len>=30)
returns 0, ruling out #216-style contamination.

One content-level finding flagged for jsboige (not auto-fixed): the
"Mathematical accuracy" family (pk 71-78) has subfamily_ru = untranslated
English "Valid results" (FR/ES/PT are translated) + pk 71 title_ru.
9 RU cells -- isolated oversight, not mass contamination. Correct RU
term is a content judgment (Достоверные/Правильные/Корректные результаты).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Your <your.email@example.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
jsboige added a commit that referenced this pull request Jun 23, 2026
…yword (#582)

Adds the `new` modifier to 6 entity members that intentionally hide the
non-virtual CsvBase.Id / CsvBase.GetId() but were declared without it,
emitting CS0108 on every build. Textbook fix, behavior-neutral: `new`
only documents the (already compile-time-bound) hiding - dispatch is
identical before and after, the members still satisfy ICsvBase.GetId().

CsvBase.Id (line 14) and CsvBase.GetId() (line 21) are non-virtual, so
`new` is correct rather than `override` (verified against the base class
signatures). Each entity maps its own Id source (Fallacy->Id, Scenario->
path, ArgumentVirtue->Pk, TestFallacyCard->Id), so the hiding is the
intended design, not a bug to refactor away.

Members fixed:
- Fallacy.GetId(), Fallacy.Id
- Scenario.GetId()
- ArgumentVirtue.GetId()
- TestFallacyCard.Id, TestFallacyCard.GetId()

Verification: full --no-incremental Debug build -> CS0108 6->0; all other
warning categories unchanged (no accidental suppression); 0 errors.
Tests: 533 passed / 0 failed / 5 skipped (unchanged baseline).

Scope: only the safe, non-gated CS0108 subset. Other AssetConverter build
warnings are deliberately left untouched - documented in the PR body for
jsboige/ai-01 (each is gated or regression-risky: color API, QuestPDF
pinned-version grid, dead-code fields, AutoMapper CVE, nullable-context
cascade).

Dispatched as po-2024 secondary: "dette technique non-gated - pioche
reelle" (ai-01 post-merge #579/#580). My prior 2 ticks had assessed the
non-gated debt as "thin" without building; this tick's build surfaced 22
warnings, correcting that assessment.

Co-authored-by: Your <your.email@example.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
jsboige added a commit that referenced this pull request Jun 23, 2026
…s templates (#584)

The %C3->A13 UTF-8 lead-byte corruption (defect #579, fixed in Memo templates
via PR #581) was also present in the 4 LIVE Fallacies face templates loaded by
the pipeline: Cards/Fallacies/Argumentum_Fallacies_Face_{,_2_,_3_,Web_}fr.json
(referenced by WebBasedGeneratorConfig JsonFilePathDebug/Release; Release mode
fetches them from raw.githubusercontent.com/.../master, so the corruption hit
generated cards directly).

Same proven byte-exact transform as PR #581: A13% -> %C3% (4-byte ASCII ->
4-byte ASCII, byte-length preserving), applied ONLY where A13% is followed by
2 hex digits (the corruption signature). Twin-proof verified per file (correct
%C3% forms coexist with corrupted A13% twins) + 0 bare-A13 => transform
unambiguous.

Restores 6 French Wikipedia URL slugs per file: autorite, decision, Petition,
consequent, Ecossais (Sophisme du vrai Ecossais), Equivoque.

Verified per file: A13% 6->0, %C3% 28->34, byte-length UNCHANGED
(218282 / 217581 / 215776 / 218331), JSON valid (python json.load), CRLF
preserved, UTF-8 no-BOM. Same defect class + same fix as merged PR #581.

Co-authored-by: Your <your.email@example.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
jsboige added a commit that referenced this pull request Jul 4, 2026
…tability fixes) (#134) (#689)

The v0.9.0 entry predating bundle v3 missed the print-production and
late-cycle stability work. This refresh brings it in line with master
`21e2c666`, cross-checked against the release-validation dossier v4
(docs/RELEASE-VALIDATION-v0.9.0.md §3.3) and dashboard decisions #26-69.

Added:
- "Print Production (CMYK & Print&Play)" section: Ghostscript post-process
  (#632), `--pdf-cmyk` entry-point (#652), bundle v3 = 80 PDFs DeviceCMYK +
  SWOP OutputIntent (6.18 GB RGB -> 5.30 GB CMYK), P&P Standard/Light
  (#645/#648-650, 64 -> 80 PDFs), GS timeout 180->900s (#670)
- "OWL Ontology (Bilingual EN/FR)" section: Fallacies OWL 5.07 MB (#634
  regen), Virtues OWL (#592/#499 Phase 2), honest scope note (EN+FR only,
  not 8-lang)
- "Fixed - Pipeline Stability (Jun-Jul 2026)" section: harvest deadlock
  #651, serial retry #613/#676, logger Spectre #630/#655, CMYK oxymore,
  Johnny 6.1.3 #653, Rules i18n refonte #640, CSV hygiene #579/#581/#584,
  OWL staleness #634

Corrected (code=truth):
- Magick.NET 14.13.1 -> 14.14.0 (verified in .csproj)
- Test count 548 -> 578 pass / 1 known-fail #133 / 5 skip / 584 total
  (empirical `dotnet test` on Argumentum.AssetConverter.Tests, 2026-07-04,
  .NET 9). Previous counts in docs (548/549/570) were all stale

No existing content regressed; all prior sections preserved.

Co-authored-by: Your <your.email@example.com>
Co-authored-by: Claude-Code <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants