diff --git a/WorldModel/EXPERIMENT_LEDGER.md b/WorldModel/EXPERIMENT_LEDGER.md index 5c3eb3d..6be61fd 100644 --- a/WorldModel/EXPERIMENT_LEDGER.md +++ b/WorldModel/EXPERIMENT_LEDGER.md @@ -413,3 +413,24 @@ Two reviewers verified the delivered synthetic A/Bs are sound (symlog threaded i - Decision: **KEEP the repair.** It passed every pre-registered gate, fixed a real and measured defect, and made the benchmark's task factors observable as its own comment always claimed. **But nodes 17-19's diagnosis is now NECESSARY BUT NOT SUFFICIENT.** The action channel was genuinely broken and is genuinely fixed; control did not follow. The bottleneck is downstream of the representation. - **This is the third defect found, fixed, and followed by no change in control** — the frame offset (node 14, remedied, no effect), the benchmark's resolving power (node 15, repaired, still underpowered), and now the action channel (nodes 17-19, repaired, no effect). Each was real. None was binding. That pattern is itself the finding: the control failure is not a single broken component. - Next question: the remaining untested link is the PREDICTOR'S MULTI-STEP ROLLOUT, and it now has evidence rather than suspicion. Measured after the repair, rollout error compounds **11.3x to 17.4x** from h=1 to h=8 (1-step 0.0010-0.0016 -> h=8 0.0173-0.0290) while `horizon=6`. A CEM plan feeds the predictor its own output for six steps, so the cost being minimised is computed on a state that may not correspond to the trajectory actually executed. Test it directly and cheaply: score the CEM against a rollout that uses the TRUE env instead of the predictor, on the same episodes and plans. If control jumps, the predictor's rollout is the bottleneck; if it does not, the fault is in the cost or the goal latent, not the forward model. + +## 22 — The control failure is an INTERACTION: neither the forward model nor the frame fixes anything alone (2026-08-02, commit ) +- Category: Benchmark +- Hypothesis: (node 21) the representation is now fully repaired — `pos` 0.98, `vel` 0.97, `ax` 0.83 — and control is still at chance, so the bottleneck is downstream. Rollout error compounds 11.3x-17.4x from h=1 to h=8 while `horizon=6`, so the predictor's multi-step rollout was the named suspect: a CEM plan may be scored on a state that does not match the trajectory executed. +- Prediction: replace `_cem_plan`'s latent rollout with a rollout through the TRUE env, same episodes, same CEM structure, same per-episode generators. If control jumps, the forward model is the bottleneck; if not, the fault is in the cost or the goal latent. +- Implementation: a fast batched encoder replacing `_encode_states` (which writes JSONL per call, far too slow for ~38k encodes) — **verified against `_encode_states` to 9.5e-07 before use**. `episode_seed=2`, n=200, NS=64, IT=3, EF=0.2, horizon 6. +- Evidence: + - **The forward model is NOT the bottleneck.** Perfect dynamics with the incumbent latent cost: **35.0% -> 37.5%**, against zero-action's 37.0%. Replacing the predictor with the true environment buys nothing. + - **The COST is where the gap lives.** Perfect dynamics with a true-position cost: **93.5%**, log-ratio **-4.2251**. So the executed trajectory can reach the goal; minimising `||enc(s_H) - z_goal||^2` simply does not correspond to minimising `|pos_H - goal|`. + - **The frame offset is why** — node 14's A/B ratio is still **4.25** post-repair, above its own registered >1.0 threshold, so the offset still exceeds the goal signal fourfold. + - **AND IT IS AN INTERACTION. Neither fix does anything alone:** + + | | goal via TARGET encoder | goal via ONLINE encoder | + |---|---|---| + | **predictor rollout** | 35.5% | **35.0%** | + | **true-env rollout** | 37.5% | **76.0%** | + + (predictor rows 3 seeds; true-env rows seed 0, n=200 episodes.) Fixing the frame alone: 35.5% -> 35.0%, nothing. Fixing the dynamics alone: 35.5% -> 37.5%, nothing. **Both together: 76.0%**, two-thirds of the way to the 93.5% ceiling. +- Decision: **The control failure has THREE necessary conditions and no sufficient one.** A representation that carries the task and the action (node 21), a goal latent produced by the SAME encoder as the start, and a forward model accurate enough over the planning horizon. Fixing any one leaves control at chance, which is precisely why nodes 6-21 produced null after null while each individual diagnosis was correct. +- **This resolves node 14 retrospectively.** Its one-frame remedy was tested and found to do nothing — correctly measured, and the right conclusion given what was then known. It did nothing because the representation was broken (`pos` 0.65, `vel` 0.033, action unrecoverable) AND the predictor rollout was inadequate. The remedy was necessary all along and could not show it alone. **No single-variable experiment could have found this**, and the ledger ran seventeen of them. +- Next question: the deployable configuration still fails — under the predictor rollout the one-frame goal changes nothing (35.5% -> 35.0%), so shipping the frame fix alone would be pointless. The remaining gap is the predictor's multi-step accuracy, now the ONLY untreated condition and no longer a suspect among several. Bounded and ordered: (a) train the predictor on multi-step rollouts rather than single transitions, so h=6 error is optimised rather than inherited from h=1; (b) re-measure the 2x2 — the honest gate is that predictor+one-frame must approach the 76% that true-env+one-frame achieves, not merely beat chance; (c) only if (a) fails, revisit whether latent-space MPC is the right architecture, which is a design decision for the user rather than an autonomous build, per the node-13 stop rule.