Skip to content

test(taxonomy): #518 contract tests for the anti-fab validator (38 tests) - #549

Merged
jsboige merged 1 commit into
masterfrom
test/taxonomy-validator-contract
Jun 19, 2026
Merged

test(taxonomy): #518 contract tests for the anti-fab validator (38 tests)#549
jsboige merged 1 commit into
masterfrom
test/taxonomy-validator-contract

Conversation

@jsboige

@jsboige jsboige commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

What

Adds tools/test_validate_taxonomy_annotations.py — a stdlib-only (unittest) contract test suite for the reusable anti-fab validator introduced in #518 (tools/validate_taxonomy_annotations.py).

This is output-neutral: the validator itself is unchanged. Only a new test file is added.

Why

The taxonomy scale-up lane (#498 AIF, #499 Virtues) depends on the validator's hard-violation detection being correct. Before this PR the validator had zero tests — a regression that silently turned a HARD violation into a pass (or vice-versa) would land undetected. This pins the contract.

What it pins (38 tests)

Group What it locks down
TestConstants CROSSLINK_TYPES (8), ATTACK_TYPES (3), ATTACK_COMPONENT_OK (3 ASPIC+ coherent pairs)
TestDetectKind auto-detection of the 4 schemas + unknown + scaleup-takes-precedence-over-pilot
TestSplitPks ; / , / mixed-whitespace / empty / single PK-cell parsing
TestValidateVirtues virtue_pk / family_pk / opposed-pk corpus membership HARD, family depth≠1 HARD, bad link_type HARD, novel-scheme WARN-not-hard
TestValidateAifScaleup fallacy_pk membership HARD, bad attack_type HARD, coherent pairs clean, incoherent pair WARN, name-drift WARN
TestValidateAifPilot legacy pilot PK membership HARD
TestValidateCrosslinks source/target PK membership HARD, bad link_type HARD, symmetry reciprocity HARD (symmetric edge w/o reverse → HARD)
TestBuildWaltonCatalog collects capitalized multi-word scheme names from *-annotations.csv, skips non-annotation files, empty on missing dir

Design

  • No real-corpus dependency. The tests build synthetic rows (dicts) + gt (ground-truth map shaped like load_ground_truth's output) in-memory. This exercises the logic, not the data — a detection regression surfaces as a failing assertion regardless of corpus state.
  • Import via path injection (sys.path.insert(0, HERE)), matching the tools/dnn_i18n/test_roundtrip.py convention.
  • Runnable two ways:
    • python tools/test_validate_taxonomy_annotations.py (exit 0 = all pass)
    • python -m unittest tools.test_validate_taxonomy_annotations (via repo root)
  • HARD vs WARN discipline: every validator group has at least one CLEAN (no hard, no/all warn) assertion AND at least one HARD-trigger assertion, so both the "passes valid input" and "rejects invalid input" contracts are pinned.

Verification

Ran 38 tests in 0.027s
OK

All 38 pass locally (Python 3, stdlib only — no pip install needed).

Related

🤖 Generated with Claude Code

…sts, stdlib-only)

Pins the contract the taxonomy scale-up lane depends on, with no dependency on the
real corpus — synthetic in-memory rows + ground-truth exercise the LOGIC, so a
HARD-violation detection regression surfaces as a failing assertion.

Covers: detect_kind (4 kinds + unknown + precedence), split_pks, validate_virtues,
validate_aif_scaleup, validate_aif_pilot, validate_crosslinks (symmetry HARD),
build_walton_catalog, and the CROSSLINK_TYPES/ATTACK_TYPES/ATTACK_COMPONENT_OK
constants. Output-neutral: no change to the tool under test.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

@clusterManager-Myia clusterManager-Myia left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[NanoClaw] — COMMENT (LGTM, code) · verified at head 3c861b607d

Real-code PR: new stdlib-only unittest contract suite (tools/test_validate_taxonomy_annotations.py, +333/-0) for the unchanged anti-fab validator (#518). 38 tests (Constants 4 · detect_kind 6 · split_pks 5 · virtues 7 · aif-scaleup 6 · aif-pilot 2 · crosslinks 5 · walton_catalog 3 = 38 ✓).

I cross-checked the suite against the live validator on master (tools/validate_taxonomy_annotations.py), not just the diff:

  • All API names matchdetect_kind, split_pks, the 5 validators, build_walton_catalog, constants CROSSLINK_TYPES/ATTACK_TYPES/ATTACK_COMPONENT_OK, all present and called correctly.
  • All 7 asserted error-substrings exist in the validator — including "NO reverse edge", which looks absent in source only because the validator splits it across two adjacent f-string literals ("...has NO reverse " "edge (...)"); Python concatenates them at compile time, so the substring IS contiguous at runtime. The test is correct.
  • Drift-gating matches: validator guards novelty WARNs on scheme and walton_catalog and … (lines 192/233/253), so test_clean_row's warn==[] with an empty catalog holds, and test_novel_scheme_is_WARN_not_hard (non-empty catalog) correctly expects the WARN.
  • split_pks normalizes ;/,/whitespace as pinned; TRUTHY set matches the symmetric-flag tests.
  • Domain accuracy solid (ASPIC+ attack types undermine/undercut/rebut + their 3 coherent component pairs).
  • Synthetic in-memory builders exercise LOGIC not real data — a HARD-detection regression surfaces as a failing assert. Stdlib-only, no subprocess/eval/network, safe tempfile I/O, exit code propagates.

One verification I couldn't complete: no Python interpreter in my container, so I didn't execute the suite — recommend confirming python tools/test_validate_taxonomy_annotations.py exits 0 on the branch before merge. Static cross-check is clean.

— NanoClaw (myia-ai-01)

@jsboige
jsboige merged commit 2b942ec into master Jun 19, 2026
3 checks passed
@jsboige
jsboige deleted the test/taxonomy-validator-contract branch June 19, 2026 17:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants