Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions WorldModel/EXPERIMENT_LEDGER.md
Original file line number Diff line number Diff line change
Expand Up @@ -336,3 +336,30 @@ Two reviewers verified the delivered synthetic A/Bs are sound (symlog threaded i
- Decision: **ADOPT `action_recovery` into the panel permanently**, with the `mode_bit` negative control asserted in `smoke_test` — mutation-checked: leaking the target makes `mode_bit` score 1.000 and the assertion fires. **REPLACE node 17's "under-represents by ~3.5x" with "does not represent at all".** Node 17 measured the PREDICTOR's response to an action it receives as an explicit input; this measures whether the ENCODED TRANSITION retains which action was taken. The second is the load-bearing quantity and it is zero, which also means node 17's 6.1% predictor response is fitting noise rather than a weak signal.
- This closes the nodes 6-17 arc. Control was never reachable from any representation-side change, because the encoder discards the action's effect and encodes the frozen per-trajectory factors nearly perfectly instead. Every panel metric that improved was measuring the constants. `vel` at R^2 = 0.033 was visible in every run since node 3 and was never read as a control result.
- Next question: this is now a `_render_state` / encoder-capacity question, not a planner, objective, benchmark or predictor question. Ordered and bounded: (a) check whether `vel` is even present in the rendered observation — if the spectral window at a fixed `t` does not carry velocity, no encoder can recover it and the task is unobservable rather than hard, which would be a generator defect and would retire the entire control line as specified; (b) if it IS present, raise the encoder's capacity for the evolving factors, e.g. by feeding a two-window stack so a difference is representable; (c) only then revisit multi-step training targets. Test (a) FIRST — it is a read of `_render_state` plus one probe, and it decides whether the remaining candidates are worth anything.

## 19 — Channel ablation: ax is inert, ay is harmful, and the generator explains both (2026-08-02, commit <pending>)
- Category: Benchmark
- Hypothesis: (node 18) the representation retains no recoverable action information. Ablating the CEM action space should separate the channels. **H1** (velocity is the missing variable): zeroing `ax` costs nothing and `ax` alone performs like random. **H2** (horizon/authority is binding): both ablations degrade and neither matches full.
- Prediction: REGISTERED BEFORE RUNNING — discriminator is |A - FULL| against |B - FULL|; H1 predicts the first small and the second large, H2 predicts both comparable.
- Implementation: in `_cem_plan`'s returned plan, zero column 0 (`ay_only`) or column 1 (`ax_only`). 200 fixed episodes, `episode_seed=2`, per-episode generators, no retraining. Plus an EXACT optimal controller — the dynamics are linear in the actions, so the optimum is a clamped linear solve — in full / ay-only / ax-only variants, as a reachability ceiling.
- Evidence:

| arm | log(d_f/d_0) | vs FULL | success |
|---|---|---|---|
| MPC full | +0.0886 | — | 35.0% |
| MPC ay_only (ax=0) | +0.0971 | **+0.0086** | 36.5% |
| MPC ax_only (ay=0) | +0.0374 | **-0.0512** | 33.5% |
| random | +0.0402 | -0.0483 | 34.5% |
| zero | +0.0341 | -0.0545 | 37.0% |
| ORACLE full | -11.11 | | **100.0%** |
| ORACLE ay_only | -8.72 | | **91.5%** |
| ORACLE ax_only | -4.52 | | **73.5%** |

- **H1 CONFIRMED, H2 REFUTED.** |A - FULL| = 0.0086 against |B - FULL| = 0.0512, a six-fold separation. Removing `ax` costs nothing, so `ax` was already inert; `ax` alone lands on random/zero. The channels separate, so horizon/authority is not the binding constraint.
- **THE DEFICIT IS REPRESENTATIONAL, NOT REACHABILITY — the pre-registered discriminator.** `ax` alone is worth **73.5%** success to an exact controller. `ax` alone is worth **33.5%** to MPC, which is below the 37.0% of doing nothing. The authority exists and the planner cannot use it.
- **NEW: all of MPC's harm is in `ay`, the channel it CAN partly see.** `ay_only` reproduces full MPC (+0.0971 vs +0.0886); `ax_only` collapses to the zero policy (+0.0374 vs +0.0341). So MPC only ever acts through `ay`, and those actions are uncorrelated with need (node 17: r = +0.04), which is strictly worse than not acting. Both channels fail, for different reasons: `ax` is unrepresentable (`vel` R^2 = 0.033), `ay` is representable but at SNR 0.35 so the choices are noise.
- **Zero-action beats MPC on BOTH endpoints** — 37.0% vs 35.0% success, +0.0341 vs +0.0886 log-ratio.
- **THE GENERATOR EXPLAINS IT** (finding contributed by review; verified here independently). `_render_state` does carry velocity — `c1 = beta * (1.0 + 0.10 * z["vel"])` — but as a 10% modulation on `beta = 10^offset * 21^(-chi)`. Variance decomposition over 4000 samples: `c0` task share **0.845%**, `c1` task share **0.0106%** — `pos` gets an ~80x larger share than `vel`. (Review reported 1.15% and 0.05% by a different freeze-point; magnitudes differ, conclusion identical.) And `chi`/`peak_amp`/`offset` are held constant per trajectory, so predicting them at t+1 is exact and free. The JEPA loss is therefore minimised by encoding the nuisance, and VICReg's variance term rewards the directions carrying the most batch variance — which are the held factors. **Node 18's `vel` R^2 = 0.033 is not a training failure; it is the correct behaviour of a working objective on a generator that made the task variable 0.01% of the signal.**
- **Velocity is not destroyed.** `vel = 10 * (c1 / betaPower - 1)` exactly — max error 2e-15 over 4000 samples. It is a ratio of two observed channels: nonlinear, hence invisible to the linear probe that caps at R^2 = 0.033, but trivially within an MLP's reach. Nothing prevents the encoder learning it; nothing rewards it.
- Decision: **CLOSE the representation-objective line as specified.** No representation objective can encode a variable the generator made 0.01% of the observed variance while paying a variance penalty for ignoring the other 99.99%. This is a generator design fault, not an encoder, planner, objective or benchmark fault. Nodes 6-18 were measuring the wrong thing correctly.
- Next question: repair the generator, under node-14 blind discipline. Two candidates, NOT equivalent: **(1)** raise the velocity modulation depth — cheapest, but it is a knob, and tuning it until an arm wins is unfalsifiable; **(2)** let `chi`/`offset` vary within-trajectory — removes the free-prediction subsidy that makes the nuisance attractive without touching velocity's coefficient, and changes what the objective is REWARDED for rather than what it is SHOWN. Prefer (2). Either way: specify using only oracle/random/zero, never a candidate arm, and require the repaired generator to reproduce the existing ladder ordering (oracle > ay-oracle > ax-oracle > zero ~ random ~ MPC) before any arm is rerun. Add to the acceptance gate an assertion that a probe on the RAW observation recovers `vel` above a floor, so a future generator edit cannot silently reintroduce this.
Loading