Repository navigation
compass(tests): pin the geometry refusal for every field it reads - #303
Conversation
The refusal test deleted `hidden_size` only, while the memory model reads four geometry fields off the config. A message that hardcoded the field name, or a field read with a silent default, stayed green. The test now runs once per field, deletes that field, and asserts the refusal names it. Closes #301 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Review: APPROVE at head
|
| mutant | edit | tip b3078336d |
head b736fd8ba |
|---|---|---|---|
| none | 47 passed | 50 passed | |
| null | line 71, docstring wording only | 47 passed | 50 passed |
| M2 | line 76, `{name}` → `hidden_size` |
47 passed | 3 failed, 47 passed |
| M4 | line 137, int(getattr(text, "intermediate_size", 0)) |
47 passed | 1 failed, 49 passed |
| M5 (reviewer) | line 138, int(getattr(text, "head_dim", 128)) |
47 passed | 1 failed, 49 passed |
| M6 (reviewer) | line 139, int(getattr(text, "max_position_embeddings", 40960)) |
47 passed | 1 failed, 49 passed |
| M7 (reviewer) | line 74, if value is None and name == "hidden_size":, so the refusal fires only for the field the old test deleted |
47 passed | 3 failed, 47 passed |
Failing node ids at the head, all tests/compass/test_memory_readings.py::test_a_config_missing_a_geometry_field_refuses_naming_that_field[...]:
- M2:
[intermediate_size],[head_dim],[max_position_embeddings]. Each is anAssertionErroron.what, readingstates nohidden_size`` against the deleted field.[hidden_size]stays green, as it must. - M4:
[intermediate_size], withFailed: DID NOT RAISE <class 'atom.compass.memory.readings.MemoryRefusal'>. - M5:
[head_dim], withDID NOT RAISE MemoryRefusal. - M6:
[max_position_embeddings], withDID NOT RAISE MemoryRefusal. - M7:
[intermediate_size],[head_dim],[max_position_embeddings], eachTypeError: int() argument must be ... not 'NoneType'.
Every mutant is green on the tip, so the old single-field test saw none of them. On the head, each field's case fires on its own. The pin is not inert.
3. Tree that will land
git merge-tree --write-tree b3078336d b736fd8bagives17e76b25c472beca4c0c65e47646fe56944614bd. That equalsb736fd8ba^{tree}, so the head is already based on the tip.- For the stamp I built a commit object with
git commit-tree 17e76b25c -p b3078336d -p b736fd8ba, which gives1ffa6fc39fb1c0124fe7dd86aa0ef655e885a7ca. .compass-commitand.compass-changedwere written from that commit. The changed set istests/compass/test_memory_readings.py.
The gate was the tree's own scripts/compass/gate_cpu.sh, run once on node 18 from /tmp/rv303b/merged/ATOM:
atom: /tmp/rv303b/merged/ATOM/atom/__init__.py
commit: 1ffa6fc39 (stamp)
gpu: not required (.compass-changed stamp)
5209 passed, 155 skipped, 3 xfailed, 16 warnings in 189.07s
GATE_CPU_RC=0 PASSED
That is 5209 passed, identical to the dev record's head figure. It is +3 over the dev record's control of 5206 at b3078336d: one test was removed and four parametrized cases added. Nothing failed, including the known timing-flaky classes in tests/entrypoints/test_stream_marker_properties.py. ruff check and ruff format --check (0.16.7) are clean on the changed file.
4. No design-doc references
I scanned tests/compass/test_memory_readings.py in full at the head for D<n>, P<n>.<n>, T<n>, W<n>, "principle", "Gate N", doc numbers and design/. There were no hits.
5. ponytail-review (principle 3)
The diff is +6/−3 and replaces a single-field test with a four-value parametrize and an f-string. Nothing is dead or speculative, and no shorter form of the same check exists.
Lean already. Ship.
Notes (non-blocking)
- Hand-kept field list (principle 3). A fifth
_geometrycall added later would stay uncovered until someone extends this list. Deriving the list from the source would be machinery for a four-item constant, so I accept the list as written. The next task that adds a geometry field should extend this parametrize list in the same PR. - Dev-record wording (principle 8). The record says "the fixture's
text_configcarries all four fields". Theqwenfixture isPretrainedConfig.from_dict(raw["text_config"]), a flat config with notext_configattribute, so_geometryreads the config itself. The substance holds, as M4-M6 show:delattrremoves the attribute_geometryactually reads.
No blocking issues.
Closes #301
What changed
test_a_config_with_no_hidden_size_refuses_naming_the_fieldbecomestest_a_config_missing_a_geometry_field_refuses_naming_that_field, parametrized over the four fields_geometryreads inModelTerms.declared_for_m1(hidden_size,intermediate_size,head_dim,max_position_embeddings). Each case deletes its field and assertsMemoryRefusalwith.whatnaming that field. The remedy assertion is unchanged.File set:
tests/compass/test_memory_readings.pyonly. Production lines: 0. Test lines: +6 / -3 (net +3), inside the +2-5 estimate.Named result (node 18,
xiaobizh_n18_cpu)Mutations are line-count-preserving swaps of
atom/compass/memory/readings.py(366 lines each), run on the tip (b3078336d) and the head (b736fd8ba),tests/compass/test_memory_readings.py:_geometryonly`{name}`->`hidden_size`in the messageint(_geometry(config, "intermediate_size"))->int(getattr(text, "intermediate_size", 0))Failing node ids at the head:
tests/compass/test_memory_readings.py::test_a_config_missing_a_geometry_field_refuses_naming_that_field[intermediate_size],[head_dim],[max_position_embeddings], eachAssertionErroron.what(states nohidden_size`` against the deleted field).[hidden_size]stays green, as it must....::test_a_config_missing_a_geometry_field_refuses_naming_that_field[intermediate_size],Failed: DID NOT RAISE MemoryRefusal.Original
readings.pyrestored after each run and md5-checked (438f6578b550on both trees).Gates
Gate 1: ATOM suite, unmodified, CPU tier (
scripts/compass/gate_cpu.shfrom each tree,git archivesnapshots with.compass-commit/.compass-changedstamps,atom.__file__asserted under the staged root):b3078336d/tmp/i301gates/ctl/ATOM/atom/__init__.pyb736fd8ba/tmp/i301gates/head/ATOM/atom/__init__.pyDelta +3 passed, 0 failed. Node-id delta (collect-only on the changed file, 47 -> 50): removed
test_a_config_with_no_hidden_size_refuses_naming_the_field; addedtest_a_config_missing_a_geometry_field_refuses_naming_that_field[hidden_size|intermediate_size|head_dim|max_position_embeddings]. No flaky-class failures on either side.git merge-tree --write-tree b3078336d b736fd8ba=17e76b25c472beca4c0c65e47646fe56944614bd=b736fd8ba^{tree}.Gate 2: the change is itself a CPU-only test in
tests/compass/; it exercises_geometryandModelTerms.declared_for_m1, which this PR does not touch.Gate 3: the named result above.
Gate 4: reviewer to be dispatched by the coordinator.
Dev record
text_configcarries all four fields, sodelattrremoves a real attribute in every case.declared_for_m1rename noted in the brief has no open PR; nothing to coordinate.🤖 Generated with Claude Code