Skip to content

tools(i18n): link_* langlinks resolver (sidecar candidate-URL generator, 0 Cards/) - #610

Merged
jsboige merged 2 commits into
masterfrom
tools/link-langlinks-resolver
Jun 29, 2026
Merged

tools(i18n): link_* langlinks resolver (sidecar candidate-URL generator, 0 Cards/)#610
jsboige merged 2 commits into
masterfrom
tools/link-langlinks-resolver

Conversation

@jsboige

@jsboige jsboige commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Summary

SECONDAIRE of ai-01's deep-queue dispatch (post-coupure). Extends the #600/#606 coverage track from measurement to resolution — produces the candidate link_<lang> URLs behind the measured ceiling, without ever touching Cards/.

The existing probe (192-link-coverage-langlinks-probe.py) measures the resolvable ceiling (~2919 cells, 57 %, #600 §5.1) but discards the target title (returns only lang codes → answers "is it resolvable?"). This tool captures the target-language title and emits the resolved URLs → answers "what is the URL?". Step 1 of #600 §6 fill methodology.

What it does

For every node with an en.wikipedia.org/wiki/<Title> link_en missing link_<lang>, query MediaWiki langlinks, capture the target title, emit a sidecar:

dataset,key,link_lang,resolved_url
fallacies,52,pt,https://pt.wikipedia.org/wiki/Fal%C3%A1cia_da_proje%C3%A7%C3%A3o_mental

Verified

Strided sample of 10 fallacies: 0 errors, 37 candidate fills (~57 % rate, consistent with the probe ceiling), URLs correctly URL-encoded (Cyrillic %D0…, PT accented %C3…). The sample also surfaced the homonym risk #600 §6.4 warns about (an English "Engagement" homonym resolved for a fallacy node) → confirms human spot-validation is non-optional for AR/FA/ZH.

Usage

python tools/link-langlinks-resolve.py                       # full, fallacies, stdout
python tools/link-langlinks-resolve.py 50 virtues             # strided sample, virtues
python tools/link-langlinks-resolve.py 0 fallacies --out tmp/link-resolve-fallacies.csv

Safety / scope

Files (2)

  • tools/link-langlinks-resolve.py — the resolver.
  • tools/link-langlinks-resolve.md — usage + safety + the gated next step.

Next (post-release, gated)

A follow-up PR consumes this sidecar: apply candidate URLs cell-by-cell (drift-free QUOTE_MINIMAL + CRLF, method #595), skip non-empty cells (preserve curated links), human spot-validate the ~5 % residue (AR/FA/ZH priority, ~150 cells).

Relates to #600, #606, memory i18n-coverage-gap-is-link-urls.

Your and others added 2 commits June 29, 2026 10:45
Grounds #141 in the CURRENT taxonomy state (issue text is ~2 years old;
DatasetUpdater + translation PRs have moved the state considerably).

Census finding (read-only script on ba8e4a6, reproducible like #600/#606):
  - Non-card nodes: Fallacies 1232, Virtues 110 (Scenarii/Rules out of scope).
  - TEXT enrichment (desc/example/title x 8 langs) = 100% on both datasets
    => original #141 scope item 3 (descriptions + examples + translations)
    is DONE. No text-enrichment gap to script.
  - The "GPT-4 enrichment script" = already adapted: modern DatasetUpdater
    (PR #210, OpenAI SDK v2.10.0, gpt-5.5). Not a rewrite.

Reframes #141: the genuinely-open residue is the AIF cross-reference graph
(crossLink_* 8 relationship cols + AIF_skos* 4 SKOS mappings). Schema is
READY (columns exist), content ~0% on non-cards:
  - Fallacies: crossLink 0.2%, AIF_skos 0.5% (barely started)
  - Virtues: crossLink_Opposes + skosDirectRef/MappingType 100% via #498
    pilots, but 7 OTHER crossLink verbs = 0%
  - ~16700 empty cross-ref cells where schema is waiting.

Proposal: 4-stage method (ground -> gpt-5.5 candidate via /v1/responses
effort=low -> drift-free sidecar dry-run -> expert/jsboige ratification
gate -> OWL/2sxc export). NOT auto-write: AIF/Walton mappings are
specialised (memory: "Anti-Fab Validator: Walton scheme = WARN"), and
Cards/ is frozen pre-tag. "Adapt the script" = add a crossLink prompt +
task config to DatasetUpdater, GATED post-release on jsboige GO + schema
decision (single decimal_path vs structured {target,note}).

Overlaps #498 (AIF scale-up pilots produced the few filled cells).
Feeds dependencies #141 declares: #130 (OWL/SKOS) + #136 (2sxc).

Scope: docs + read-only script only. 0 write Cards/, 0 AssetConverter code
change (pre-tag safe). Base ba8e4a6.

Relates to #130, #136, #498.

Co-Authored-By: Claude-Code <noreply@anthropic.com>
…or, 0 Cards/)

SECONDAIRE of ai-01 deep-queue dispatch. Extends the #600/#606 coverage
track from MEASUREMENT to RESOLUTION.

The existing probe (docs/taxonomy/192-link-coverage-langlinks-probe.py)
measures the resolvable ceiling (~2919 cells, 57%, #600 §5.1) but
discards the target title (returns only lang codes). This tool captures
the target-language title via the MediaWiki langlinks API and emits the
candidate fill URLs as a sidecar report — step 1 of #600 §6 methodology.

For every node with an en.wikipedia.org/wiki/<Title> link_en missing
link_<lang>, queries langlinks, captures the target title, emits:
  dataset,key,link_lang,resolved_url
to stdout or --out <path>. NEVER writes under Cards/.

Verified on a strided sample of 10 fallacies: 0 errors, 37 candidate
fills (~57% rate, consistent with the probe ceiling), URLs correctly
URL-encoded (Cyrillic %D0, PT accented %C3). Sample also surfaced the
homonym risk #600 §6.4 warns about (English "Engagement" for a fallacy
node) -> confirms human spot-validation is non-optional for AR/FA/ZH.

Safety:
  - 0 write Cards/ (sidecar only, pre-tag freeze).
  - Public MediaWiki API, no key, 0.3s throttle, descriptive UA
    (default urllib UA is 403-forbidden).
  - Resolves FROM link_en (Wikipedia only); 433 non-Wikipedia curated
    sources excluded + preserved as-is (#600 §6.2).

Next (post-release, gated): follow-up PR consumes the sidecar -> apply
candidate URLs cell-by-cell (drift-free QUOTE_MINIMAL+CRLF, method #595),
skip non-empty cells, human spot-validate ~5% residue (AR/FA/ZH, ~150).

Relates to #600, #606.

Co-Authored-By: Claude-Code <noreply@anthropic.com>

@clusterManager-Myia clusterManager-Myia left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[NanoClaw]

LGTM with 1 minor concern — link-langlinks-resolve.py : extends the #600/#606 coverage track from measurement (the probe) to resolution (materializes candidate link_<lang> URLs behind the measured ~57% ceiling). ▶ Scope note : je review uniquement les 2 nouveaux fichiers (tools/link-langlinks-resolve.py + .md) ; les fichiers 141-noncard-census.* sont le contenu de #609 (déjà couvert twin 4590773328), sibling-bundle convention (#4575/#4584).

Vérifié firsthand :

  • 0 write under Cards/ confirmé ✅ : lit Cards/.../*.csv en read-only (open(...,encoding=...)), interroge l'API MediaWiki publique, écrit uniquement stdout ou un sidecar --out (exemples d'usage = tmp/). Les CSV Cards/ ne sont jamais ouverts en écriture.
  • 0 secret leak : API publique, pas de clé. Le USER_AGENT contient jsboige@gmail.com = email de contact public (politique MediaWiki : UA descriptif requis sinon 403), pas un secret — déjà public sur le profil GitHub.
  • Encodage URL sound ✅ : build_url = urllib.parse.quote(title.replace(" ","_")), extract_wiki_title = unquote().replace("_"," "). Cyrillic %D0/PT accentué %C3 confirmés par la logique.
  • Concept sound : endpoint action=query&prop=langlinks&lllimit=500 = le bon endpoint MediaWiki pour les liens cross-langue. Split probe (mesure plafond) → resolver (matérialise URLs) = méthodologie propre. Strided sampling déterministe (int(i*stride), pas de random) = reproductible. Rate-limit poli (0.3s), timeout 25s.
  • Stance anti-overclaiming ▶ : flagge explicitement le risque homonyme RTL/CJK (#600 §6.4) comme « candidates, not authoritative — human spot-validation non-optional », et surfacet un exemple concret (« Engagement » homonym). Exactement la discipline honnête.

△ Concern 1 (non-bloquant) : le risque homonyme est flaggué AR/FA/ZH only, mais l'exemple même du doc (« Engagement » homonyme anglais résolu pour un node fallacie) montre que les langues latin-script peuvent aussi homonym-resolve. Le framing « AR/FA/ZH need spot-validation » sous-estime légèrement le périmètre — idéalement valider un échantillon sur les 6 langues cibles, pas seulement les 3 RTL/CJK. Mineur : le gate humain existe déjà, juste un peu sous-scopé dans la prose.

△ Nit 2 (très mineur) : la branche --out fait os.makedirs(os.path.dirname(out_path)) sans guard contre un path sous Cards/ — si un utilisateur passe --out Cards/.../x.csv, le script écrirait sous Cards/. La protection est convention/doc-based (défaut = stdout, exemples = tmp/), pas enforcement code. Le propos de l'outil étant le sidecar, c'est hors-intent, mais un assert assert not out_path.startswith('Cards/') durcirait le invariant « 0 write Cards » que le doc clame.

Ces 2 nits n'affectent ni le read-only safety (en usage normal), ni la valeur du deliverable. Outil solide, méthodologie reproductible, stance honnête.

— NanoClaw (myia-ai-01)

@jsboige
jsboige merged commit 3c4579d into master Jun 29, 2026
3 checks passed
@jsboige
jsboige deleted the tools/link-langlinks-resolver branch June 29, 2026 10:56
jsboige added a commit that referenced this pull request Jul 1, 2026
… (0 errors, 0 Cards/) (#618)

PRIMAIRE of ai-01 deep-queue #2 (dispatch msg-...92kynx). Produces the
candidate-URL output behind the #600 §5.1 probe ceiling.

Ran tools/link-langlinks-resolve.py (PR #610, merged) over full Fallacies +
Virtues taxonomy: 829 articles probed -> 2934 candidate link_<lang> URLs
(fallacies 2754, virtues 180), 0 errors. Materializes the ~57% probe ceiling
(~2919 cells) into concrete reviewable URLs. Step 1 of #600 §6 methodology.

Per-lang: ru 371, pt 430, es 513, ar 603, fa 480, zh 537. Resolves FROM
link_en (Wikipedia only); 433 non-Wikipedia curated sources excluded +
preserved (#600 §6.2).

Quality verified:
  - URL encoding correct per script (Cyrillic %D0, Arabic %D8, Persian %D8,
    CJK %E8, PT/ES Latin unencoded).
  - Homonym scan (#600 §6.4) of 1620 AR/FA/ZH candidates: 0 real leaks.
    AR 0/603, FA 0/480 Latin-path; ZH 3/537 (0.6%) = legit loanwords/
    acronyms (FUD x2, Creepypasta) — not errors. The "Engagement" homonym
    from PR #610 sample does NOT recur at scale (isolated case).
  - Asymmetry: virtues ru/pt = 4 each (sparse abstract-concept coverage)
    vs ar/fa/zh 46-56 — coverage reality, flagged for spot-validation.

Sidecars (reviewable snapshot, UTF-8 no-BOM LF):
  - docs/taxonomy/600-link-resolve-fallacies.csv (2754 rows)
  - docs/taxonomy/600-link-resolve-virtues.csv (180 rows)
  schema: dataset,key,link_lang,resolved_url

Robustness: try/except + continue (skip on network error, never abort) ->
0 errors/829. Public MediaWiki API, no key, 0.3s throttle, descriptive UA.

Next (gated post-release): apply PR consumes sidecar -> cell-by-cell drift-free
(QUOTE_MINIMAL+CRLF, method #595), skip non-empty, human spot-validate ~5%
AR/FA/ZH residue (~150 cells), re-run probe to confirm gap closed.

Scope: docs/taxonomy/ only. 0 write Cards/, 0 AssetConverter code change
(pre-tag safe). Base 18b4d02.

Relates to #600, #606, #610.

Co-authored-by: Your <your.email@example.com>
Co-authored-by: Claude-Code <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants