tools(i18n): link_* langlinks resolver (sidecar candidate-URL generator, 0 Cards/) - #610
Conversation
Grounds #141 in the CURRENT taxonomy state (issue text is ~2 years old; DatasetUpdater + translation PRs have moved the state considerably). Census finding (read-only script on ba8e4a6, reproducible like #600/#606): - Non-card nodes: Fallacies 1232, Virtues 110 (Scenarii/Rules out of scope). - TEXT enrichment (desc/example/title x 8 langs) = 100% on both datasets => original #141 scope item 3 (descriptions + examples + translations) is DONE. No text-enrichment gap to script. - The "GPT-4 enrichment script" = already adapted: modern DatasetUpdater (PR #210, OpenAI SDK v2.10.0, gpt-5.5). Not a rewrite. Reframes #141: the genuinely-open residue is the AIF cross-reference graph (crossLink_* 8 relationship cols + AIF_skos* 4 SKOS mappings). Schema is READY (columns exist), content ~0% on non-cards: - Fallacies: crossLink 0.2%, AIF_skos 0.5% (barely started) - Virtues: crossLink_Opposes + skosDirectRef/MappingType 100% via #498 pilots, but 7 OTHER crossLink verbs = 0% - ~16700 empty cross-ref cells where schema is waiting. Proposal: 4-stage method (ground -> gpt-5.5 candidate via /v1/responses effort=low -> drift-free sidecar dry-run -> expert/jsboige ratification gate -> OWL/2sxc export). NOT auto-write: AIF/Walton mappings are specialised (memory: "Anti-Fab Validator: Walton scheme = WARN"), and Cards/ is frozen pre-tag. "Adapt the script" = add a crossLink prompt + task config to DatasetUpdater, GATED post-release on jsboige GO + schema decision (single decimal_path vs structured {target,note}). Overlaps #498 (AIF scale-up pilots produced the few filled cells). Feeds dependencies #141 declares: #130 (OWL/SKOS) + #136 (2sxc). Scope: docs + read-only script only. 0 write Cards/, 0 AssetConverter code change (pre-tag safe). Base ba8e4a6. Relates to #130, #136, #498. Co-Authored-By: Claude-Code <noreply@anthropic.com>
…or, 0 Cards/) SECONDAIRE of ai-01 deep-queue dispatch. Extends the #600/#606 coverage track from MEASUREMENT to RESOLUTION. The existing probe (docs/taxonomy/192-link-coverage-langlinks-probe.py) measures the resolvable ceiling (~2919 cells, 57%, #600 §5.1) but discards the target title (returns only lang codes). This tool captures the target-language title via the MediaWiki langlinks API and emits the candidate fill URLs as a sidecar report — step 1 of #600 §6 methodology. For every node with an en.wikipedia.org/wiki/<Title> link_en missing link_<lang>, queries langlinks, captures the target title, emits: dataset,key,link_lang,resolved_url to stdout or --out <path>. NEVER writes under Cards/. Verified on a strided sample of 10 fallacies: 0 errors, 37 candidate fills (~57% rate, consistent with the probe ceiling), URLs correctly URL-encoded (Cyrillic %D0, PT accented %C3). Sample also surfaced the homonym risk #600 §6.4 warns about (English "Engagement" for a fallacy node) -> confirms human spot-validation is non-optional for AR/FA/ZH. Safety: - 0 write Cards/ (sidecar only, pre-tag freeze). - Public MediaWiki API, no key, 0.3s throttle, descriptive UA (default urllib UA is 403-forbidden). - Resolves FROM link_en (Wikipedia only); 433 non-Wikipedia curated sources excluded + preserved as-is (#600 §6.2). Next (post-release, gated): follow-up PR consumes the sidecar -> apply candidate URLs cell-by-cell (drift-free QUOTE_MINIMAL+CRLF, method #595), skip non-empty cells, human spot-validate ~5% residue (AR/FA/ZH, ~150). Relates to #600, #606. Co-Authored-By: Claude-Code <noreply@anthropic.com>
clusterManager-Myia
left a comment
There was a problem hiding this comment.
[NanoClaw]
LGTM with 1 minor concern — link-langlinks-resolve.py : extends the #600/#606 coverage track from measurement (the probe) to resolution (materializes candidate link_<lang> URLs behind the measured ~57% ceiling). ▶ Scope note : je review uniquement les 2 nouveaux fichiers (tools/link-langlinks-resolve.py + .md) ; les fichiers 141-noncard-census.* sont le contenu de #609 (déjà couvert twin 4590773328), sibling-bundle convention (#4575/#4584).
Vérifié firsthand :
- 0 write under Cards/ confirmé ✅ : lit
Cards/.../*.csven read-only (open(...,encoding=...)), interroge l'API MediaWiki publique, écrit uniquement stdout ou un sidecar--out(exemples d'usage =tmp/). Les CSV Cards/ ne sont jamais ouverts en écriture. - 0 secret leak : API publique, pas de clé. Le
USER_AGENTcontientjsboige@gmail.com= email de contact public (politique MediaWiki : UA descriptif requis sinon 403), pas un secret — déjà public sur le profil GitHub. - Encodage URL sound ✅ :
build_url=urllib.parse.quote(title.replace(" ","_")),extract_wiki_title=unquote().replace("_"," "). Cyrillic%D0/PT accentué%C3confirmés par la logique. - Concept sound : endpoint
action=query&prop=langlinks&lllimit=500= le bon endpoint MediaWiki pour les liens cross-langue. Split probe (mesure plafond) → resolver (matérialise URLs) = méthodologie propre. Strided sampling déterministe (int(i*stride), pas de random) = reproductible. Rate-limit poli (0.3s), timeout 25s. - Stance anti-overclaiming ▶ : flagge explicitement le risque homonyme RTL/CJK (#600 §6.4) comme « candidates, not authoritative — human spot-validation non-optional », et surfacet un exemple concret (« Engagement » homonym). Exactement la discipline honnête.
△ Concern 1 (non-bloquant) : le risque homonyme est flaggué AR/FA/ZH only, mais l'exemple même du doc (« Engagement » homonyme anglais résolu pour un node fallacie) montre que les langues latin-script peuvent aussi homonym-resolve. Le framing « AR/FA/ZH need spot-validation » sous-estime légèrement le périmètre — idéalement valider un échantillon sur les 6 langues cibles, pas seulement les 3 RTL/CJK. Mineur : le gate humain existe déjà, juste un peu sous-scopé dans la prose.
△ Nit 2 (très mineur) : la branche --out fait os.makedirs(os.path.dirname(out_path)) sans guard contre un path sous Cards/ — si un utilisateur passe --out Cards/.../x.csv, le script écrirait sous Cards/. La protection est convention/doc-based (défaut = stdout, exemples = tmp/), pas enforcement code. Le propos de l'outil étant le sidecar, c'est hors-intent, mais un assert assert not out_path.startswith('Cards/') durcirait le invariant « 0 write Cards » que le doc clame.
Ces 2 nits n'affectent ni le read-only safety (en usage normal), ni la valeur du deliverable. Outil solide, méthodologie reproductible, stance honnête.
— NanoClaw (myia-ai-01)
… (0 errors, 0 Cards/) (#618) PRIMAIRE of ai-01 deep-queue #2 (dispatch msg-...92kynx). Produces the candidate-URL output behind the #600 §5.1 probe ceiling. Ran tools/link-langlinks-resolve.py (PR #610, merged) over full Fallacies + Virtues taxonomy: 829 articles probed -> 2934 candidate link_<lang> URLs (fallacies 2754, virtues 180), 0 errors. Materializes the ~57% probe ceiling (~2919 cells) into concrete reviewable URLs. Step 1 of #600 §6 methodology. Per-lang: ru 371, pt 430, es 513, ar 603, fa 480, zh 537. Resolves FROM link_en (Wikipedia only); 433 non-Wikipedia curated sources excluded + preserved (#600 §6.2). Quality verified: - URL encoding correct per script (Cyrillic %D0, Arabic %D8, Persian %D8, CJK %E8, PT/ES Latin unencoded). - Homonym scan (#600 §6.4) of 1620 AR/FA/ZH candidates: 0 real leaks. AR 0/603, FA 0/480 Latin-path; ZH 3/537 (0.6%) = legit loanwords/ acronyms (FUD x2, Creepypasta) — not errors. The "Engagement" homonym from PR #610 sample does NOT recur at scale (isolated case). - Asymmetry: virtues ru/pt = 4 each (sparse abstract-concept coverage) vs ar/fa/zh 46-56 — coverage reality, flagged for spot-validation. Sidecars (reviewable snapshot, UTF-8 no-BOM LF): - docs/taxonomy/600-link-resolve-fallacies.csv (2754 rows) - docs/taxonomy/600-link-resolve-virtues.csv (180 rows) schema: dataset,key,link_lang,resolved_url Robustness: try/except + continue (skip on network error, never abort) -> 0 errors/829. Public MediaWiki API, no key, 0.3s throttle, descriptive UA. Next (gated post-release): apply PR consumes sidecar -> cell-by-cell drift-free (QUOTE_MINIMAL+CRLF, method #595), skip non-empty, human spot-validate ~5% AR/FA/ZH residue (~150 cells), re-run probe to confirm gap closed. Scope: docs/taxonomy/ only. 0 write Cards/, 0 AssetConverter code change (pre-tag safe). Base 18b4d02. Relates to #600, #606, #610. Co-authored-by: Your <your.email@example.com> Co-authored-by: Claude-Code <noreply@anthropic.com>
Summary
SECONDAIRE of ai-01's deep-queue dispatch (post-coupure). Extends the #600/#606 coverage track from measurement to resolution — produces the candidate
link_<lang>URLs behind the measured ceiling, without ever touchingCards/.The existing probe (
192-link-coverage-langlinks-probe.py) measures the resolvable ceiling (~2919 cells, 57 %, #600 §5.1) but discards the target title (returns only lang codes → answers "is it resolvable?"). This tool captures the target-language title and emits the resolved URLs → answers "what is the URL?". Step 1 of #600 §6 fill methodology.What it does
For every node with an
en.wikipedia.org/wiki/<Title>link_enmissinglink_<lang>, query MediaWikilanglinks, capture the target title, emit a sidecar:Verified
Strided sample of 10 fallacies: 0 errors, 37 candidate fills (~57 % rate, consistent with the probe ceiling), URLs correctly URL-encoded (Cyrillic
%D0…, PT accented%C3…). The sample also surfaced the homonym risk #600 §6.4 warns about (an English "Engagement" homonym resolved for a fallacy node) → confirms human spot-validation is non-optional for AR/FA/ZH.Usage
Safety / scope
Cards/— sidecar only (stdout or--out), pre-tag freeze respected.link_en(Wikipedia URLs only). Non-Wikipedia curated sources (rationalwiki, yourlogicalfallacyis — 433 Fallacies / 9 Virtues) excluded + preserved as-is (docs(i18n): #192 link_* coverage research track (own the URL gap) #600 §6.2).Files (2)
tools/link-langlinks-resolve.py— the resolver.tools/link-langlinks-resolve.md— usage + safety + the gated next step.Next (post-release, gated)
A follow-up PR consumes this sidecar: apply candidate URLs cell-by-cell (drift-free
QUOTE_MINIMAL+ CRLF, method #595), skip non-empty cells (preserve curated links), human spot-validate the ~5 % residue (AR/FA/ZH priority, ~150 cells).Relates to #600, #606, memory
i18n-coverage-gap-is-link-urls.