compass(spec): explain refuses a partly present quantity and names every missing field

[jgong5]

Closes #265

What was wrong

Build a spec from parts with the public MachineSpec(values, tokenizers), then ask explain for a quantity or block that is only partly present. explain answered from the fields it had. The rest were left out and nothing said so. The refusal added in #264 fired only when none of the fields was present.

The case from the issue, driven on node 18 at both refs:

tip 77d203b86	head 04bc6e51d
kv_blocks: 12 rows, no refusal (the four-width reserve and allocator rows, persistent buffer and three graph-pool rows. There is no capacity row and no sign that one is missing.)	refused TOTALITY: … `device.memory.capacity_bytes` is declared by this schema, and this spec carries no value for it. … merge the fragment that measures this field before asking for it
admission missing two fields: 5 rows, no refusal	refused TOTALITY: … `host.admission_fixed_s`, `host.ipc.shm_broadcast_s` are declared by this schema, and this spec carries no value for them. … merge the fragments that measure these fields before asking for them

The empty-tokenizer case from the issue comment is admission on a spec read with an empty host.tokenizers:

tip	head
admission, from spec sha256:22b4…a4 host.admission_fixed_s = 0.009 host.ipc.zmq_roundtrip_s = 5e-05 host.ipc.shm_broadcast_s = 2e-05	admission, from spec sha256:22b4…a4 host.admission_fixed_s = 0.009 host.ipc.zmq_roundtrip_s = 5e-05 host.ipc.shm_broadcast_s = 2e-05 host.tokenizers = ()

What was decided

1. Refuse when any required field under the term is absent. This applies to quantities and blocks alike. A term that names a single field refuses as before, whether that field is required or optional, and for a single field the result is byte-identical to MachineSpec.value.
2. Name every missing field, not only the first. Each missing field means a separate fragment to merge. A refusal that names only the first sends the reader back once per field, and each partial answer looks whole until the next refusal. That is the decomposition point: a refusal is a report too, and one field standing in for "some fields" is an aggregate. Listing all of them costs nothing, because the set is already computed to decide whether to refuse.
3. The shared wording is a sibling in rules.py, refuse_absent_fields(paths). refuse_absent_field(path, required) now sends its required branch through it with a one-element tuple. That makes the required refusal a single SpecRefusal(...) site with singular and plural forms, so explain and the accessor cannot drift. rules.py is in the diff because the alternative was a second copy of the text in explain.py, and a second copy is exactly the drift the issue asks to rule out. The singular text is unchanged character for character. test_the_three_refusals_are_not_interchangeable and test_a_declared_field_a_spec_lacks_is_never_called_undeclared still assert on it and stay green.
4. An optional field the document left out is not missing. Only provenance.fragments and provenance.notes are optional. A spec read from a document that omits notes is whole by the schema, so explain(spec, "provenance") still explains from the rest. No quantity in QUANTITIES names an optional field.
5. An empty contributing field is printed as the empty value it holds (item 4 of the brief). Basis gains empty: tuple[Contribution, ...] = (). These are fields the spec holds that produced no rows. __str__ prints them after the contributions, in the same path = value [source] form, so host.tokenizers = () means "held, and empty". A field that was left out now refuses (point 1), so a printed basis has no third state to confuse with these two. contributions is unchanged, so compass(spec): explain refuses an unknown term as ADDRESSING and an absent one as TOTALITY #264's "an empty table explains to zero rows" still holds exactly: basis.contributions == ().

What surprised me

• compass(spec): explain refuses an unknown term as ADDRESSING and an absent one as TOTALITY #264's own test test_an_empty_tokenizer_table_explains_to_no_rows_rather_than_refusing asserted str(basis) was the header alone. That is the behaviour the issue comment asks to change. Its contributions == () assertion is kept, and its str assertion now expects the host.tokenizers = () line. This is the one landed test whose body changed.
• test_a_block_an_assembled_spec_holds_nothing_under_names_its_first_field asserted the first-only behaviour by name. It is renamed ..._names_every_field and now expects both host.ipc fields. Its node id changes.
• The refusal-site partition in test_spec_schema.py did not move. The required-absent SpecRefusal(...) site moved from refuse_absent_field into refuse_absent_fields, but it is still one TOTALITY site in rules.py. Explain and the accessor still build a single-field refusal at the same site, and test_explain_declines_an_unknown_term_and_an_absent_one_at_two_sites is green unmodified. test_spec_schema.py is not in the diff.

Left undone

• Empty fields print after the contributions, not in field order. For admission the tokenizer table comes last anyway. For a block such as host the empty line comes after the ipc and admission rows. Interleaving would mean carrying a per-field grouping through Basis, which is more than this issue needs.
• An optional field left out of a block (provenance.notes under provenance) is still silent in the printed basis. It is not a dependency of any quantity, so I left it alone. If a reader needs to see it, the fix is the same empty-style line with a "not stated" marker.

Gates (node 18, xiaobizh_n18_cpu, git archive staging under /tmp/i265gates, removed afterwards)

Gate 1: ATOM's suite, unmodified, via the tree's own scripts/compass/gate_cpu.sh, with .compass-commit/.compass-changed stamps

	tree	atom.__file__	result	GATE_CPU_RC
control	77d203b86 (stamp)	/tmp/i265gates/control/ATOM/atom/__init__.py	5111 passed, 149 skipped, 3 xfailed	0
branch	04bc6e51d (stamp)	/tmp/i265gates/branch/ATOM/atom/__init__.py	5115 passed, 149 skipped, 3 xfailed	0

Both runs printed gpu: not required. The delta is +4 passed, which matches the node-id diff of tests/compass collection (control 1155 ids, branch 1159):

• added:
  • tests/compass/test_spec_verbs.py::test_a_quantity_missing_one_field_is_refused_rather_than_explained_from_the_rest
  • tests/compass/test_spec_verbs.py::test_every_missing_field_under_a_quantity_is_named_and_not_only_the_first
  • tests/compass/test_spec_verbs.py::test_an_empty_tokenizer_table_is_printed_as_empty_under_a_quantity
  • tests/compass/test_spec_verbs.py::test_a_block_explains_without_an_optional_field_the_document_left_out
  • tests/compass/test_spec_verbs.py::test_a_block_an_assembled_spec_holds_nothing_under_names_every_field (rename)
• removed: tests/compass/test_spec_verbs.py::test_a_block_an_assembled_spec_holds_nothing_under_names_its_first_field (renamed, above)
• body changed, same id: tests/compass/test_spec_verbs.py::test_an_empty_tokenizer_table_explains_to_no_rows_rather_than_refusing

git merge-tree --write-tree 77d203b86 04bc6e51d gives 03cf301edbafc144876fd493eedb970c32d4452d with rc 0. That equals the head tree, because the branch sits directly on the tip.

Gate 2: the new CPU-only tests listed above. test_spec_verbs.py and test_spec_schema.py together: 289 passed.

Gate 3: the named result, with line-count-preserving mutations of explain.py (229 → 229 lines each) run over test_spec_verbs.py and test_spec_schema.py

mutation	result
null control: if missing: → if len(missing):	289 passed
restore tip behaviour: if missing: → if ():	3 failed: test_a_quantity_missing_one_field_is_refused_rather_than_explained_from_the_rest, test_every_missing_field_under_a_quantity_is_named_and_not_only_the_first, test_a_block_an_assembled_spec_holds_nothing_under_names_every_field
name only the first: refuse_absent_fields(missing[:1])	2 failed: ..._named_and_not_only_the_first, ..._names_every_field
treat optional as required: …required or 1)	3 failed: test_an_optional_field_the_document_left_out_is_refused_as_absent, test_a_block_explains_without_an_optional_field_the_document_left_out, test_spec_schema.py::test_explain_declines_an_unknown_term_and_an_absent_one_at_two_sites
drop the empty line: self.contributions + ()	2 failed: test_an_empty_tokenizer_table_explains_to_no_rows_rather_than_refusing, test_an_empty_tokenizer_table_is_printed_as_empty_under_a_quantity

The named test asserts str(refusal) == str(spec.value("device.memory.capacity_bytes") refusal). It pins both the refusal and the shared wording.

Gate 4: the reviewer is dispatched by the coordinator.

Size

The estimate was about 15 production lines and 30 test lines. These are AST code lines: docstrings, comments and blank lines are excluded.

	added	removed	net
production (explain.py +16/−6, rules.py +17/−8)	33	14	+19
tests (test_spec_verbs.py)	43	6	+37

Net production is 1.3x the estimate. Gross added is 2.2x, and 8 of those 33 lines are the existing required-field SpecRefusal(...) moved from refuse_absent_field into its sibling. Tests are 1.2x net. Raw git diff --numstat: explain.py 30/10, rules.py 27/8, test_spec_verbs.py 60/8.

🤖 Generated with Claude Code

[jgong5]

Nit, not blocking (principle 8: a claim carries its measurement). "each is a separate fragment to merge" is stated as fact, and it is not always true. A fragment is a partial document (merge.py module docstring), and one fragment can carry several fields. For example, the tokenizer probe writes host.cpu.cores_physical and host.cpu.cores_logical alongside its table. The reason to name every field does not need that claim. Each missing field needs some fragment to supply it, and a refusal that names only one field sends the reader back once per field. Suggest: "each needs a fragment that measures it, and a refusal naming one of them sends the reader back once per field."

The plural remedy, "merge the fragments that measure these fields", reads fine even when one fragment covers two of the fields. Only the docstring overstates this.

[jgong5]

Pin gap, not blocking (principle 8). This assertion compares explain with spec.value. Both now build their text in the same refuse_absent_fields, so the comparison cannot fail on wording. It proves the two sides share a site. It does not prove the singular text is what it was.

I measured this on node 18 with a line-count-preserving mutation of rules.py L166 (195 → 195 lines), run over test_spec_verbs.py and test_spec_schema.py:

("is", "it", "fragment that measures this field") → ("is", "it", "fragment that measures that field"): 289 passed.

So the singular remedy is pinned by nothing. The PR body names two schema tests as holding the singular text. Those tests assert the what, and a what mutation does redden test_a_declared_field_a_spec_lacks_is_never_called_undeclared. They do not assert the remedy.

The text is byte-identical today. I checked all 39 fields × {required, optional} with the old rules.py from 77d203b86 against the new one: 78/78 identical, including refuse_absent_fields((p,)). So nothing is wrong now. The gap only means a later edit could change the wording silently. One line here closes it:

assert refused.value.remedy.endswith("merge the fragment that measures this field before asking for it")

[jgong5]

Agent-authored review (Claude). Reviewer for #265, cycle 1.

Review, cycle 1: APPROVE at head 04bc6e51da135a41a505092c1f26251ff2480d4e

Verdict: APPROVE. No blocking issues. There are two non-blocking findings, both inline:

• rules.py L160: a docstring overstatement (principle 8).
• test_spec_verbs.py L1096: the singular remedy text is pinned by nothing (principle 8). A one-line fix is suggested.

I read the eight design principles in atom/compass/design/README.md and AI_DEV_RULES.md first.

Rulings

1. Naming every missing field (principle 7): true and deterministic.

• The plural text is true. I dropped every required field under each of 17 terms: the 4 quantities plus every block and sub-block. Each refusal is TOTALITY and names exactly the dropped set, in term order: 0 bad of 17.
• The order is the term's own tuple order, SCHEMA order for blocks and the QUANTITIES tuple for quantities, because missing is filtered from paths without re-sorting. Reversing it (R1 below) reddens two tests, so the order is pinned.
• Naming all of them is right. Naming only the first sends the reader back once per field, and principle 7 applies to a refusal as much as to a number.
• The refuse_absent_field(paths[0], False) line is still first-only, but only for a term whose fields are all optional and all absent. No block is all-optional, because provenance holds three required fields. So this is reachable only for a single-field term, where the first field is the only one. Nothing is hidden.

2. The rules.py sibling: safe.

• Byte identity: I loaded the old rules.py from 77d203b86 beside the new one and ran both over all 39 fields × {required, optional}. Old refuse_absent_field, new refuse_absent_field and new refuse_absent_fields((p,)) gave the same (rule, what, remedy, str) in 78/78 cases.
• explain and MachineSpec.value agree. I dropped each declared field present in the resolved spec, one at a time, and compared the two refusals: 38/38 identical. The 39th field, provenance.notes, is absent from the fixture, and its case is the site test below.
• compass(spec): the version refusal asks for a reader, not an edit #212's guards still place the site. The new SpecRefusal(...) names Rule.TOTALITY inline, so the resolver reads it. It is raised directly, so it is inside _built_where_thrown(). It is not SHAPE, so the partition is unchanged. test_spec_schema.py is not in the diff, and it is green in both gate runs and in the 289-test battery baseline. The developer's claim holds.
• The pin gap on the singular remedy is inline at L1096. It is a gap, not a defect.

3. Basis.empty: honest, and close to minimal (principles 6 and 3).

• Honest. host.tokenizers = () is exactly what spec.value("host.tokenizers") returns. It fabricates nothing, and absence now always refuses, so a printed basis has only two states: rows, or held and empty.
• Minimal. Basis holds no spec, so __str__ cannot recover the empty fields itself. Folding them into contributions would break compass(spec): explain refuses an unknown term as ADDRESSING and an absent one as TOTALITY #264's landed zero-rows contract (mutant R2 reddens it). A separate tuple is the smallest thing that separates "empty" from "left out".
• Order does not matter. Measured on explain(spec, "host"), the tokenizer line prints last, after admission_fixed_s, while SCHEMA puts it third. Every row is present and labelled by its full path, so the decomposition is complete and only its position differs. For every quantity, the empty-capable field is already last in its tuple. Not a finding.
• compass(spec): explain refuses an unknown term as ADDRESSING and an absent one as TOTALITY #264's landed behaviour holds: explain(spec, "host.tokenizers").contributions == () on an empty table, measured, with empty holding the one host.tokenizers row.

4. The changed assertion in test_an_empty_tokenizer_table_explains_to_no_rows_rather_than_refusing: justified.

• The brief's comment (item 4) asks for exactly this change of printed output. A landed test that asserted the old print had to move.
• The test still pins what its name says. contributions == () is kept, so "no rows" is pinned: R2 reddens it. "Rather than refusing" is pinned because it calls explain with no raises.
• It also now pins the empty line: the developer's drop-empty mutant reddens it, and so does R8.

6. Left undone, an optional field left out of a block is silent: no follow-up needed.

• It concerns only provenance.fragments and provenance.notes, the only two optional fields, and neither feeds any quantity.
• A spec that omits them is whole by the schema, so leaving them unprinted states nothing false.
• This should be revisited only if an optional field is ever added to a QUANTITIES tuple, and the PR body already records that trigger.

7. Design-doc references: none in code, test names or messages.

Size (principle 3)

• Net production is +19 against ~15.
• Gross +33 is inflated by the 8 moved lines of the existing refusal.
• The pieces are the sibling, the two predicates and the empty tuple. Each is small, and each is required by a brief item: 1, 3 and 4 respectively.
• I see nothing to cut, and the task is not mis-cut. Not a finding.

Pins, reproduced (node 18, xiaobizh_n18_cpu, merged tree, atom.__file__ = /tmp/r268gates/mut/ATOM/atom/__init__.py in every run)

Each run covered tests/compass/test_spec_verbs.py and tests/compass/test_spec_schema.py. The unmutated baseline was 289 passed. Every mutation preserved the line count: explain.py stayed at 229 → 229 and rules.py at 195 → 195.

#	mutation	result	failing node ids (all tests/compass/)
N	if missing: → if len(missing): (null)	289 passed	none
named	if missing: → if (): (tip behaviour)	3 failed	test_spec_verbs.py::test_a_quantity_missing_one_field_is_refused_rather_than_explained_from_the_rest, test_spec_verbs.py::test_every_missing_field_under_a_quantity_is_named_and_not_only_the_first, test_spec_verbs.py::test_a_block_an_assembled_spec_holds_nothing_under_names_every_field
D1	refuse_absent_fields(missing[:1])	2 failed	test_spec_verbs.py::test_every_missing_field_under_a_quantity_is_named_and_not_only_the_first, test_spec_verbs.py::test_a_block_an_assembled_spec_holds_nothing_under_names_every_field
D2	…required or 1) (optional treated as required)	3 failed	test_spec_verbs.py::test_an_optional_field_the_document_left_out_is_refused_as_absent, test_spec_verbs.py::test_a_block_explains_without_an_optional_field_the_document_left_out, test_spec_schema.py::test_explain_declines_an_unknown_term_and_an_absent_one_at_two_sites
D3	self.contributions + () ] (empty line dropped)	2 failed	test_spec_verbs.py::test_an_empty_tokenizer_table_explains_to_no_rows_rather_than_refusing, test_spec_verbs.py::test_an_empty_tokenizer_table_is_printed_as_empty_under_a_quantity
R1	missing reversed	2 failed	same two as D1
R2	empty row pushed into contributions	2 failed	same two as D3
R3	if not rows: → if 1: (empty line for every field)	1 failed	test_spec_verbs.py::test_an_empty_tokenizer_table_is_printed_as_empty_under_a_quantity
R4	if len(absent) == len(paths): → if 0:	2 failed	test_spec_verbs.py::test_an_optional_field_the_document_left_out_is_refused_as_absent, test_spec_schema.py::test_explain_declines_an_unknown_term_and_an_absent_one_at_two_sites
R5	one = len(paths) >= 1 (always singular)	2 failed	same two as D1
R6	singular remedy this field → that field (rules.py L166)	289 passed	none: the inline finding at L1096
R7	what text by this schema → by the schema	2 failed	test_spec_verbs.py::test_every_missing_field_under_a_quantity_is_named_and_not_only_the_first, test_spec_schema.py::test_a_declared_field_a_spec_lacks_is_never_called_undeclared
R8	empty row carries None instead of the held value	2 failed	same two as D3

The developer's named result and the three mutants D1–D3 reproduce exactly, by name. R1–R8 are mine. All of them bite except R6.

Gate 1: the tree that will land

• The integration tip moved during review. It went from 77d203b86 to 308922c5c, which is compass(tests): make the capture census discriminate, and assert what it measured (#238) #261 and touches only tests/compass/test_capture_real_model.py. So the developer's merged tree 03cf301ed is no longer the landing tree.
• git merge-tree --write-tree 308922c5c 04bc6e51d gives 22bddc4c322578418239c42c3933f01db4128c94, rc 0.
• Staging: git archive of that tree, with the .compass-commit/.compass-changed stamps, passed through docker exec -i … tar -x into /tmp/r268gates/merged/ATOM. The md5 was c19c1a37… on both ends.
• Gate: the tree's own scripts/compass/gate_cpu.sh, run under timeout -k 10 1500. It printed atom: /tmp/r268gates/merged/ATOM/atom/__init__.py and commit: 04bc6e51d (stamp).
• Result: 5119 passed, 149 skipped, 3 xfailed, GATE_CPU_RC=0, in 163 s.
• This is the developer's 5115 plus 4 from compass(tests): make the capture census discriminate, and assert what it measured (#238) #261: test_capture_real_model.py collects 18 tests at 308922c5c and 14 at 77d203b86. No test_stream_marker_properties.py flake fired.
• Before the tip moved, I had gated the developer's tree 03cf301ed once. It gave 5115 passed, 149 skipped, 3 xfailed, GATE_CPU_RC=0, matching the developer's count exactly.
• The tips were re-read before posting: head 04bc6e51d, base 308922c5c, both unmoved. Staging was removed afterwards.

No other blocking issues.