Skip to content

bench(worldmodel): the control failure is an interaction, not a component - #63

Merged
aurascoper merged 1 commit into
mainfrom
bench/true-env-rollout-test
Aug 3, 2026
Merged

bench(worldmodel): the control failure is an interaction, not a component#63
aurascoper merged 1 commit into
mainfrom
bench/true-env-rollout-test

Conversation

@aurascoper

Copy link
Copy Markdown
Owner

Node 21's named test: replace _cem_plan's latent rollout with a rollout through the true env, same episodes, same CEM structure, same per-episode generators.

The forward model is not the bottleneck

arm success log ratio
predictor rollout + latent cost 35.0% +0.0861
true-env rollout + latent cost 37.5% +0.1059
true-env rollout + true-pos cost 93.5% −4.2251
zero 37.0% +0.0341

Perfect dynamics buy nothing (35.0 → 37.5, against zero-action's 37.0). Swapping the cost buys everything.

So the executed trajectory can reach the goal — minimising ||enc(s_H) − z_goal||² simply does not correspond to minimising |pos_H − goal|. Node 14's frame ratio is still 4.25, above its own registered >1.0 threshold.

And it is an interaction — neither fix does anything alone

goal via TARGET encoder goal via ONLINE encoder
predictor rollout 35.5% 35.0%
true-env rollout 37.5% 76.0%

(predictor rows 3 seeds; true-env rows seed 0, n=200 episodes.)

Frame alone: nothing. Dynamics alone: nothing. Both: 76.0% — two-thirds of the way to the 93.5% ceiling.

Three necessary conditions, no sufficient one

  1. a representation carrying the task and the action (node 21)
  2. a goal latent from the same encoder as the start
  3. a forward model accurate over the planning horizon

Fixing any one leaves control at chance. That is exactly why nodes 6–21 produced null after null while each individual diagnosis was correct.

This resolves node 14 retrospectively

Its one-frame remedy did nothing — correctly measured, and the right conclusion given what was then known. It did nothing because the representation was broken and the rollout was inadequate. The remedy was necessary all along and could not show it alone.

No single-variable experiment could have found this, and the ledger ran seventeen of them.

Method note

The fast batched encoder used here was verified against _encode_states to 9.5e-07 before being relied on — ~38k encodes through the JSONL path was not viable, and an unverified fast path would have invalidated every number above.

Next

The deployable configuration still fails: under the predictor rollout the one-frame goal changes nothing (35.5% → 35.0%), so shipping the frame fix alone would be pointless.

The predictor's multi-step accuracy is now the only untreated condition rather than one suspect among several. Ordered:

  1. train the predictor on multi-step rollouts so h=6 error is optimised rather than inherited from h=1
  2. re-measure the 2×2 — the honest gate is that predictor+one-frame must approach 76%, not merely beat chance
  3. only if (1) fails, revisit whether latent-space MPC is the right architecture — a design decision for the user per the node-13 stop rule, not an autonomous build

Ledger-only. Verified: smoke 2 runs, ledger and ADR gates green.

…nent

Ledger node 22. Node 21's named test: replace _cem_plan's latent rollout with a
rollout through the TRUE env, same episodes, same CEM structure, same
per-episode generators.

THE FORWARD MODEL IS NOT THE BOTTLENECK. Perfect dynamics with the incumbent
latent cost: 35.0% -> 37.5%, against zero-action's 37.0%. Nothing.

THE COST IS. Perfect dynamics with a true-position cost: 93.5%, log-ratio
-4.2251. The executed trajectory can reach the goal; minimising
||enc(s_H) - z_goal||^2 does not correspond to minimising |pos_H - goal|.

AND IT IS AN INTERACTION -- neither fix does anything alone:

                        goal via TARGET    goal via ONLINE
    predictor rollout        35.5%              35.0%
    true-env rollout         37.5%              76.0%

(predictor rows 3 seeds; true-env rows seed 0, n=200 episodes.) Frame alone:
nothing. Dynamics alone: nothing. Both: 76.0%, two-thirds of the way to the
93.5% ceiling.

So the control failure has THREE necessary conditions and no sufficient one: a
representation carrying the task and the action (node 21), a goal latent from
the SAME encoder as the start, and a forward model accurate over the horizon.
Fixing any one leaves control at chance -- which is exactly why nodes 6-21
produced null after null while each individual diagnosis was correct.

THIS RESOLVES NODE 14 RETROSPECTIVELY. Its one-frame remedy did nothing, which
was correctly measured and the right conclusion on what was then known. It did
nothing because the representation was broken AND the rollout was inadequate.
The remedy was necessary all along and could not show it alone. No
single-variable experiment could have found this, and the ledger ran seventeen.

Method note: the fast batched encoder used here was verified against
_encode_states to 9.5e-07 before being relied on, since ~38k encodes through the
JSONL path was not viable.

Next: the deployable configuration still fails -- under the predictor rollout the
one-frame goal changes nothing, so shipping the frame fix alone would be
pointless. The predictor's multi-step accuracy is now the ONLY untreated
condition rather than one suspect among several, and the honest gate is that
predictor+one-frame must approach 76%, not merely beat chance.

Ledger-only. Verified: smoke 2 runs, ledger and ADR gates green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DJJ8y1zo7WSqEmGNW8DQap
@aurascoper
aurascoper merged commit 3f7f35c into main Aug 3, 2026
2 checks passed
@aurascoper
aurascoper deleted the bench/true-env-rollout-test branch August 3, 2026 00:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant