Skip to content

compass(memory): drive a decode shape against a prefill one at both refusal sites #283

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jgong5 merged 1 commit into from
Sep 23, 2026
Merged

compass(memory): drive a decode shape against a prefill one at both refusal sites #283

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions tests/compass/test_memory_compare.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -611,6 +611,39 @@ def test_two_shapes_and_no_side_naming_a_shaped_term_refuses_outright():
assert "neither names a term it took at one" in str(refusal.value)


@pytest.mark.parametrize(
"at_shape", [AT_SHAPE, frozenset()], ids=["per-term", "outright"]
)
def test_a_decode_side_refuses_a_prefill_side_at_the_same_token_count(at_shape):
# The token counts agree, so only the phase separates the two sides. Both
# places that read shape agreement are driven: the per-term refusal when a
# shaped term is named, and the outright one when neither side names one.
predicted = Predicted(
label="a prefill prediction",
shape=HISTORICAL_SHAPE,
terms=HISTORICAL_PREDICTED.terms,
at_shape=at_shape,
)
decode = Recorded(
run="a decode step at the prediction's token count",
shape=Shape(tokens=HISTORICAL_SHAPE.tokens, phase="decode"),

@jgong5 jgong5 Sep 23, 2026

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking (principle 8). This line pins prefill differs from decode. It does not pin the other half of the claim in Shape's docstring: "Equality is exact, so two spellings of one phase refuse each other." #176's audit row states that claim as its property.

Measured on node 18, against the merged tree 66614e4bd and the tip 37824643b. I compared phase case-insensitively at both sites, (tokens, phase.lower()) at compare.py:587 and at :607. The mutation keeps the file at 847 lines.

  • merged tree: 53 passed, which is what the unmutated file gives;
  • tip: 51 passed, also unchanged.

So a comparator that treated "Prefill" and "prefill" as one phase stays green. The docstring calls that direction unsafe. "prefill" against "decode" still differs after casefolding, so this test cannot see the change, and no other test can either.

This does not block the PR. #217's named result is decode against prefill, and that is met: every mutation that removes the phase distinction goes red. Adding a third parameter would close it in about five lines: the same token count, with phase="Prefill" on the recorded side. That could be done here, or in whichever follow-up next touches Shape, with #176's tokens < 1 and blank-phase row.

high_water_reset=True,
terms=HISTORICAL_RECORDED.terms,
at_shape=at_shape,
)
both = "predicted at 4096 tokens, prefill, recorded at 4096 tokens, decode"
if not at_shape:
with pytest.raises(MemoryRefusal) as refusal:
compare(predicted, decode)
assert f"taken at different shapes -- {both} -- " in str(refusal.value)
return
comparison = compare(predicted, decode)
refused = {r.name: r for r in comparison.refused}
assert set(refused) == {"activations"}
assert f"the two disagree -- {both}" in refused["activations"].what
assert {t.name for t in comparison.compared} == {"weights", UNATTRIBUTED}


def test_a_side_that_claims_a_shaped_term_it_does_not_carry_is_rejected():
with pytest.raises(ValueError) as bad:
Recorded(
Expand Down