docs(tools): fix bytecheck EOL bug + extend to Fallacies/Virtues taxonomy (read-only, 0 write) - #818
Merged
Merged
Conversation
…nomy (read-only, 0 write)
Correction + extension of the tick-24 byte-check audit (report + tool),
following jsboige's request for a work session. Still read-only: no CSV
write, no template edit, no encoding change on prod data.
Tool bug fixed (tools/bytecheck-datasets.py):
- detect_encoding() read only the first 64 bytes and tested `b"\r\n" in raw`.
The first line break of every production CSV sits past byte 64 (header is
wide: first CRLF at bytes 82..1235), so it falsely reported LF for all of
Rules/Rules PnP/Scenarii. Now reads the whole file and separates the row
terminator (CRLF) from bare LF inside quoted multi-line Markdown cells.
Coverage extended to the 2 core production taxonomies never scanned on the
encoding dimension:
- Fallacies Taxonomy (1408 x 104) and Virtues Taxonomy (223 x 81) added to
TARGETS. Runtime ~1.5s, stdlib only, idempotent.
Findings (report rewritten):
- EOL is CRLF for all 5 render CSVs (tick-24 "LF" was wrong).
- BOM is the MAJORITY convention (4/5: Fallacies + Virtues + Rules + Rules
PnP); Scenarii is the lone no-BOM file. Tick-24 framing ("Rules anomalous")
was inverted.
- Fallacies 1408 rows: 0 FR contamination. Virtues: 2 hits both in link_*
URL columns = the known #192 link-URL research gap (not text MT).
- Read-path is BOM-safe (pipeline reads BOM files fine; CsvBase decodes
upstream, UtilityExtensions.TrimStart(BOM), dedicated diff-engine BOM test).
- Latent churn trap: GSheetSyncRunner.cs:98 writes UTF-8 no-BOM (no encoding
arg). If #193 GSheet-sync (Enabled=false) is ever run in pull on the 4 BOM
files, it rewrites them no-BOM -> whole-file diff. Decision A/B/C surfaced
for jsboige before enabling #193.
Out of scope: CSV write (#202), BOM normalization decision (gated jsboige),
GSheetSyncRunner edit (needs #193 mandate), link_* research (#192, post-tag).
Re-run: python tools/bytecheck-datasets.py
Refs: #202, #193, #192, dispatch 07kpoq (tick 24), tick 28.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
jsboige
added a commit
that referenced
this pull request
Jul 16, 2026
…ppers (read-only, 0 write) (#821) Read-only check on the 29 integrated HTML mindmap wrappers regenerated by #819 (the T&A review vehicle for gate (b)), extending the #818 encoding thread. Answers a concrete de-risking question: did the #804 residual #3 (.content.svg declare encoding="utf-16" but are UTF-8+BOM, ai-01 Cycle H) propagate into the file Thomas & Adeline actually open? Result: NO. 0/29 wrappers carry the latent. - 29/29 are really UTF-8, declare charset=UTF-8 (meta). - 0 utf-16-declared-but-utf8 mismatches. - Cause: MindMapHtmlWrapper.FormatWrapper strips the <?xml?> prologue and injects the SVG into an HTML shell with its own charset=UTF-8, so the erroneous utf-16 declaration is eliminated, not inherited. Independently corroborates ai-01's Cycle H localization certification (SHA256 #636 + script composition) on the freshest artifact (#819). Tracked-not-fixed observation: 4/29 wrappers carry a BOM (Fallacies_{ar,es,fa,zh}.html) — harmless (meta charset + BOM precedence) but inconsistent. Fix = po-2023 regen lane (mindmaps), not po-2024 (Cards). Folded into po-2023's post-tag mindmap regen pass (#804 #1/#2/#3). Out of scope: wrapper/.content.svg regen, MindMapHtmlWrapper edit, BOM strip — all lane po-2023. No write prod this PR. Refs: #819, #804 residu #3, #818, #458. Co-authored-by: Your <your.email@example.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Correction and extension of the tick-24 byte-check audit (
docs/investigations/2026-07-16-bytecheck-datasets-non-couverts.md+tools/bytecheck-datasets.py), on jsboige's request for a work session while the Cards lane is otherwise drained. Read-only: no CSV write, no template edit, no prod encoding change.Why
The committed tick-24 report had a factual error and a coverage gap:
detect_encoding()read only the first 64 bytes → falsely reported LF for Rules/Rules PnP/Scenarii (the first line break sits past byte 64 — headers are wide).A committed report that is wrong on a fact + blind to the biggest datasets is worth fixing for accuracy, not left as-is.
Findings (corrected)
Encoding — BOM is the majority convention, EOL is CRLF everywhere
Tick-24 implied "Rules is the BOM anomaly". Reality once the taxonomies are included: 4/5 carry BOM; Scenarii is the lone no-BOM outlier. All 5 are CRLF-terminated (bare LF counts are in-cell Markdown newlines — normal).
i18n — clean
link_*URL columns = the known feat(translations): multi-pass quality improvement for EN/RU/PT (beyond initial translation) #192 link-URL research gap (URLs to research by hand, not MT-translatable text) — not contamination.trompe-l'œil,Caméléa,Obélix).BOM impact on the pipeline (verified in C#)
CsvBase.LoadFromContentworks on already-decoded content (BOM stripped upstream),UtilityExtensions.cs:268does an explicit.TrimStart(''), and the diff engine has a dedicatedBOM_Prefixed_Header_Does_Not_Break_Primary_Key_Lookuptest.GSheetSyncRunner.cs:98writesFile.WriteAllTextAsync(localPath, downloadedCsv)with no encoding arg → .NET default UTF-8 no-BOM. If feat(tooling): bidirectional Google Spreadsheet sync for source CSVs with diff safeguards #193 GSheet-sync (currentlyEnabled=false) is ever run in pull on the 4 BOM files, it rewrites them no-BOM → whole-file diff. Decision point before enabling feat(tooling): bidirectional Google Spreadsheet sync for source CSVs with diff safeguards #193.Decision surfaced for jsboige (decision-ready, gated)
GSheetSyncRunnerBOM-preserving (keeps the majority convention);No action taken — this PR only surfaces the picture.
Out of scope
GSheetSyncRunner.csedit (needs feat(tooling): bidirectional Google Spreadsheet sync for source CSVs with diff safeguards #193 mandate; po-2024 lane = Cards/data)link_*localized-URL research (feat(translations): multi-pass quality improvement for EN/RU/PT (beyond initial translation) #192, human research, post-tag)Verification
python tools/bytecheck-datasets.pyre-runs idempotently, ~1.5s, stdlib only.Refs
07kpoq(tick 24 idle secours) · corrected + extended tick 28— po-2024 (tick 28, jsboige work-session request)