bench(worldmodel): the control failure is an interaction, not a component - #63
Merged
Conversation
…nent
Ledger node 22. Node 21's named test: replace _cem_plan's latent rollout with a
rollout through the TRUE env, same episodes, same CEM structure, same
per-episode generators.
THE FORWARD MODEL IS NOT THE BOTTLENECK. Perfect dynamics with the incumbent
latent cost: 35.0% -> 37.5%, against zero-action's 37.0%. Nothing.
THE COST IS. Perfect dynamics with a true-position cost: 93.5%, log-ratio
-4.2251. The executed trajectory can reach the goal; minimising
||enc(s_H) - z_goal||^2 does not correspond to minimising |pos_H - goal|.
AND IT IS AN INTERACTION -- neither fix does anything alone:
goal via TARGET goal via ONLINE
predictor rollout 35.5% 35.0%
true-env rollout 37.5% 76.0%
(predictor rows 3 seeds; true-env rows seed 0, n=200 episodes.) Frame alone:
nothing. Dynamics alone: nothing. Both: 76.0%, two-thirds of the way to the
93.5% ceiling.
So the control failure has THREE necessary conditions and no sufficient one: a
representation carrying the task and the action (node 21), a goal latent from
the SAME encoder as the start, and a forward model accurate over the horizon.
Fixing any one leaves control at chance -- which is exactly why nodes 6-21
produced null after null while each individual diagnosis was correct.
THIS RESOLVES NODE 14 RETROSPECTIVELY. Its one-frame remedy did nothing, which
was correctly measured and the right conclusion on what was then known. It did
nothing because the representation was broken AND the rollout was inadequate.
The remedy was necessary all along and could not show it alone. No
single-variable experiment could have found this, and the ledger ran seventeen.
Method note: the fast batched encoder used here was verified against
_encode_states to 9.5e-07 before being relied on, since ~38k encodes through the
JSONL path was not viable.
Next: the deployable configuration still fails -- under the predictor rollout the
one-frame goal changes nothing, so shipping the frame fix alone would be
pointless. The predictor's multi-step accuracy is now the ONLY untreated
condition rather than one suspect among several, and the honest gate is that
predictor+one-frame must approach 76%, not merely beat chance.
Ledger-only. Verified: smoke 2 runs, ledger and ADR gates green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DJJ8y1zo7WSqEmGNW8DQap
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Node 21's named test: replace
_cem_plan's latent rollout with a rollout through the true env, same episodes, same CEM structure, same per-episode generators.The forward model is not the bottleneck
Perfect dynamics buy nothing (35.0 → 37.5, against zero-action's 37.0). Swapping the cost buys everything.
So the executed trajectory can reach the goal — minimising
||enc(s_H) − z_goal||²simply does not correspond to minimising|pos_H − goal|. Node 14's frame ratio is still 4.25, above its own registered >1.0 threshold.And it is an interaction — neither fix does anything alone
(predictor rows 3 seeds; true-env rows seed 0, n=200 episodes.)
Frame alone: nothing. Dynamics alone: nothing. Both: 76.0% — two-thirds of the way to the 93.5% ceiling.
Three necessary conditions, no sufficient one
Fixing any one leaves control at chance. That is exactly why nodes 6–21 produced null after null while each individual diagnosis was correct.
This resolves node 14 retrospectively
Its one-frame remedy did nothing — correctly measured, and the right conclusion given what was then known. It did nothing because the representation was broken and the rollout was inadequate. The remedy was necessary all along and could not show it alone.
No single-variable experiment could have found this, and the ledger ran seventeen of them.
Method note
The fast batched encoder used here was verified against
_encode_statesto 9.5e-07 before being relied on — ~38k encodes through the JSONL path was not viable, and an unverified fast path would have invalidated every number above.Next
The deployable configuration still fails: under the predictor rollout the one-frame goal changes nothing (35.5% → 35.0%), so shipping the frame fix alone would be pointless.
The predictor's multi-step accuracy is now the only untreated condition rather than one suspect among several. Ordered:
Ledger-only. Verified: smoke 2 runs, ledger and ADR gates green.