diff --git a/WorldModel/EXPERIMENT_LEDGER.md b/WorldModel/EXPERIMENT_LEDGER.md index 8cad6e9..00be826 100644 --- a/WorldModel/EXPERIMENT_LEDGER.md +++ b/WorldModel/EXPERIMENT_LEDGER.md @@ -336,3 +336,30 @@ Two reviewers verified the delivered synthetic A/Bs are sound (symlog threaded i - Decision: **ADOPT `action_recovery` into the panel permanently**, with the `mode_bit` negative control asserted in `smoke_test` — mutation-checked: leaking the target makes `mode_bit` score 1.000 and the assertion fires. **REPLACE node 17's "under-represents by ~3.5x" with "does not represent at all".** Node 17 measured the PREDICTOR's response to an action it receives as an explicit input; this measures whether the ENCODED TRANSITION retains which action was taken. The second is the load-bearing quantity and it is zero, which also means node 17's 6.1% predictor response is fitting noise rather than a weak signal. - This closes the nodes 6-17 arc. Control was never reachable from any representation-side change, because the encoder discards the action's effect and encodes the frozen per-trajectory factors nearly perfectly instead. Every panel metric that improved was measuring the constants. `vel` at R^2 = 0.033 was visible in every run since node 3 and was never read as a control result. - Next question: this is now a `_render_state` / encoder-capacity question, not a planner, objective, benchmark or predictor question. Ordered and bounded: (a) check whether `vel` is even present in the rendered observation — if the spectral window at a fixed `t` does not carry velocity, no encoder can recover it and the task is unobservable rather than hard, which would be a generator defect and would retire the entire control line as specified; (b) if it IS present, raise the encoder's capacity for the evolving factors, e.g. by feeding a two-window stack so a difference is representable; (c) only then revisit multi-step training targets. Test (a) FIRST — it is a read of `_render_state` plus one probe, and it decides whether the remaining candidates are worth anything. + +## 19 — Channel ablation: ax is inert, ay is harmful, and the generator explains both (2026-08-02, commit ) +- Category: Benchmark +- Hypothesis: (node 18) the representation retains no recoverable action information. Ablating the CEM action space should separate the channels. **H1** (velocity is the missing variable): zeroing `ax` costs nothing and `ax` alone performs like random. **H2** (horizon/authority is binding): both ablations degrade and neither matches full. +- Prediction: REGISTERED BEFORE RUNNING — discriminator is |A - FULL| against |B - FULL|; H1 predicts the first small and the second large, H2 predicts both comparable. +- Implementation: in `_cem_plan`'s returned plan, zero column 0 (`ay_only`) or column 1 (`ax_only`). 200 fixed episodes, `episode_seed=2`, per-episode generators, no retraining. Plus an EXACT optimal controller — the dynamics are linear in the actions, so the optimum is a clamped linear solve — in full / ay-only / ax-only variants, as a reachability ceiling. +- Evidence: + + | arm | log(d_f/d_0) | vs FULL | success | + |---|---|---|---| + | MPC full | +0.0886 | — | 35.0% | + | MPC ay_only (ax=0) | +0.0971 | **+0.0086** | 36.5% | + | MPC ax_only (ay=0) | +0.0374 | **-0.0512** | 33.5% | + | random | +0.0402 | -0.0483 | 34.5% | + | zero | +0.0341 | -0.0545 | 37.0% | + | ORACLE full | -11.11 | | **100.0%** | + | ORACLE ay_only | -8.72 | | **91.5%** | + | ORACLE ax_only | -4.52 | | **73.5%** | + + - **H1 CONFIRMED, H2 REFUTED.** |A - FULL| = 0.0086 against |B - FULL| = 0.0512, a six-fold separation. Removing `ax` costs nothing, so `ax` was already inert; `ax` alone lands on random/zero. The channels separate, so horizon/authority is not the binding constraint. + - **THE DEFICIT IS REPRESENTATIONAL, NOT REACHABILITY — the pre-registered discriminator.** `ax` alone is worth **73.5%** success to an exact controller. `ax` alone is worth **33.5%** to MPC, which is below the 37.0% of doing nothing. The authority exists and the planner cannot use it. + - **NEW: all of MPC's harm is in `ay`, the channel it CAN partly see.** `ay_only` reproduces full MPC (+0.0971 vs +0.0886); `ax_only` collapses to the zero policy (+0.0374 vs +0.0341). So MPC only ever acts through `ay`, and those actions are uncorrelated with need (node 17: r = +0.04), which is strictly worse than not acting. Both channels fail, for different reasons: `ax` is unrepresentable (`vel` R^2 = 0.033), `ay` is representable but at SNR 0.35 so the choices are noise. + - **Zero-action beats MPC on BOTH endpoints** — 37.0% vs 35.0% success, +0.0341 vs +0.0886 log-ratio. +- **THE GENERATOR EXPLAINS IT** (finding contributed by review; verified here independently). `_render_state` does carry velocity — `c1 = beta * (1.0 + 0.10 * z["vel"])` — but as a 10% modulation on `beta = 10^offset * 21^(-chi)`. Variance decomposition over 4000 samples: `c0` task share **0.845%**, `c1` task share **0.0106%** — `pos` gets an ~80x larger share than `vel`. (Review reported 1.15% and 0.05% by a different freeze-point; magnitudes differ, conclusion identical.) And `chi`/`peak_amp`/`offset` are held constant per trajectory, so predicting them at t+1 is exact and free. The JEPA loss is therefore minimised by encoding the nuisance, and VICReg's variance term rewards the directions carrying the most batch variance — which are the held factors. **Node 18's `vel` R^2 = 0.033 is not a training failure; it is the correct behaviour of a working objective on a generator that made the task variable 0.01% of the signal.** +- **Velocity is not destroyed.** `vel = 10 * (c1 / betaPower - 1)` exactly — max error 2e-15 over 4000 samples. It is a ratio of two observed channels: nonlinear, hence invisible to the linear probe that caps at R^2 = 0.033, but trivially within an MLP's reach. Nothing prevents the encoder learning it; nothing rewards it. +- Decision: **CLOSE the representation-objective line as specified.** No representation objective can encode a variable the generator made 0.01% of the observed variance while paying a variance penalty for ignoring the other 99.99%. This is a generator design fault, not an encoder, planner, objective or benchmark fault. Nodes 6-18 were measuring the wrong thing correctly. +- Next question: repair the generator, under node-14 blind discipline. Two candidates, NOT equivalent: **(1)** raise the velocity modulation depth — cheapest, but it is a knob, and tuning it until an arm wins is unfalsifiable; **(2)** let `chi`/`offset` vary within-trajectory — removes the free-prediction subsidy that makes the nuisance attractive without touching velocity's coefficient, and changes what the objective is REWARDED for rather than what it is SHOWN. Prefer (2). Either way: specify using only oracle/random/zero, never a candidate arm, and require the repaired generator to reproduce the existing ladder ordering (oracle > ay-oracle > ax-oracle > zero ~ random ~ MPC) before any arm is rerun. Add to the acceptance gate an assertion that a probe on the RAW observation recovers `vel` above a floor, so a future generator edit cannot silently reintroduce this.