Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions WorldModel/EXPERIMENT_LEDGER.md
Original file line number Diff line number Diff line change
Expand Up @@ -413,3 +413,24 @@ Two reviewers verified the delivered synthetic A/Bs are sound (symlog threaded i
- Decision: **KEEP the repair.** It passed every pre-registered gate, fixed a real and measured defect, and made the benchmark's task factors observable as its own comment always claimed. **But nodes 17-19's diagnosis is now NECESSARY BUT NOT SUFFICIENT.** The action channel was genuinely broken and is genuinely fixed; control did not follow. The bottleneck is downstream of the representation.
- **This is the third defect found, fixed, and followed by no change in control** — the frame offset (node 14, remedied, no effect), the benchmark's resolving power (node 15, repaired, still underpowered), and now the action channel (nodes 17-19, repaired, no effect). Each was real. None was binding. That pattern is itself the finding: the control failure is not a single broken component.
- Next question: the remaining untested link is the PREDICTOR'S MULTI-STEP ROLLOUT, and it now has evidence rather than suspicion. Measured after the repair, rollout error compounds **11.3x to 17.4x** from h=1 to h=8 (1-step 0.0010-0.0016 -> h=8 0.0173-0.0290) while `horizon=6`. A CEM plan feeds the predictor its own output for six steps, so the cost being minimised is computed on a state that may not correspond to the trajectory actually executed. Test it directly and cheaply: score the CEM against a rollout that uses the TRUE env instead of the predictor, on the same episodes and plans. If control jumps, the predictor's rollout is the bottleneck; if it does not, the fault is in the cost or the goal latent, not the forward model.

## 22 — The control failure is an INTERACTION: neither the forward model nor the frame fixes anything alone (2026-08-02, commit <pending>)
- Category: Benchmark
- Hypothesis: (node 21) the representation is now fully repaired — `pos` 0.98, `vel` 0.97, `ax` 0.83 — and control is still at chance, so the bottleneck is downstream. Rollout error compounds 11.3x-17.4x from h=1 to h=8 while `horizon=6`, so the predictor's multi-step rollout was the named suspect: a CEM plan may be scored on a state that does not match the trajectory executed.
- Prediction: replace `_cem_plan`'s latent rollout with a rollout through the TRUE env, same episodes, same CEM structure, same per-episode generators. If control jumps, the forward model is the bottleneck; if not, the fault is in the cost or the goal latent.
- Implementation: a fast batched encoder replacing `_encode_states` (which writes JSONL per call, far too slow for ~38k encodes) — **verified against `_encode_states` to 9.5e-07 before use**. `episode_seed=2`, n=200, NS=64, IT=3, EF=0.2, horizon 6.
- Evidence:
- **The forward model is NOT the bottleneck.** Perfect dynamics with the incumbent latent cost: **35.0% -> 37.5%**, against zero-action's 37.0%. Replacing the predictor with the true environment buys nothing.
- **The COST is where the gap lives.** Perfect dynamics with a true-position cost: **93.5%**, log-ratio **-4.2251**. So the executed trajectory can reach the goal; minimising `||enc(s_H) - z_goal||^2` simply does not correspond to minimising `|pos_H - goal|`.
- **The frame offset is why** — node 14's A/B ratio is still **4.25** post-repair, above its own registered >1.0 threshold, so the offset still exceeds the goal signal fourfold.
- **AND IT IS AN INTERACTION. Neither fix does anything alone:**

| | goal via TARGET encoder | goal via ONLINE encoder |
|---|---|---|
| **predictor rollout** | 35.5% | **35.0%** |
| **true-env rollout** | 37.5% | **76.0%** |

(predictor rows 3 seeds; true-env rows seed 0, n=200 episodes.) Fixing the frame alone: 35.5% -> 35.0%, nothing. Fixing the dynamics alone: 35.5% -> 37.5%, nothing. **Both together: 76.0%**, two-thirds of the way to the 93.5% ceiling.
- Decision: **The control failure has THREE necessary conditions and no sufficient one.** A representation that carries the task and the action (node 21), a goal latent produced by the SAME encoder as the start, and a forward model accurate enough over the planning horizon. Fixing any one leaves control at chance, which is precisely why nodes 6-21 produced null after null while each individual diagnosis was correct.
- **This resolves node 14 retrospectively.** Its one-frame remedy was tested and found to do nothing — correctly measured, and the right conclusion given what was then known. It did nothing because the representation was broken (`pos` 0.65, `vel` 0.033, action unrecoverable) AND the predictor rollout was inadequate. The remedy was necessary all along and could not show it alone. **No single-variable experiment could have found this**, and the ledger ran seventeen of them.
- Next question: the deployable configuration still fails — under the predictor rollout the one-frame goal changes nothing (35.5% -> 35.0%), so shipping the frame fix alone would be pointless. The remaining gap is the predictor's multi-step accuracy, now the ONLY untreated condition and no longer a suspect among several. Bounded and ordered: (a) train the predictor on multi-step rollouts rather than single transitions, so h=6 error is optimised rather than inherited from h=1; (b) re-measure the 2x2 — the honest gate is that predictor+one-frame must approach the 76% that true-env+one-frame achieves, not merely beat chance; (c) only if (a) fails, revisit whether latent-space MPC is the right architecture, which is a design decision for the user rather than an autonomous build, per the node-13 stop rule.
Loading