diff --git a/WorldModel/EXPERIMENT_LEDGER.md b/WorldModel/EXPERIMENT_LEDGER.md index 209d448..e685caf 100644 --- a/WorldModel/EXPERIMENT_LEDGER.md +++ b/WorldModel/EXPERIMENT_LEDGER.md @@ -280,3 +280,17 @@ Two reviewers verified the delivered synthetic A/Bs are sound (symlog threaded i - **The per-episode CEM generator is not cosmetic.** A single generator consumed sequentially made the planning noise on episode i depend on the draws taken by episodes 0..i-1, i.e. on `n_samples * cem_iters * horizon`. Any arm touching a CEM knob got different noise on the same episode. Nodes 10 and 11 swept exactly those knobs. That sweep was therefore not the paired comparison it appeared to be — it is not necessarily wrong, but its "within seed noise" reading rested on an assumption that did not hold. - **The continuous endpoint helps, by less than predicted.** Measured paired, ld32 vs ld64 on identical episodes and identical planner noise: mean |t| of 0.62 for the per-episode binary against 0.88 for per-episode `log(d_final/d_start)`, a factor of **1.43**, implying median n for |t|=2 of ~248 versus ~154. A power simulation had suggested nearer 3-4x. Only 3-8 of 20 pairs are discordant, which is why the binary endpoint is weak and is the mechanism the simulation got roughly right while overstating the size. Both endpoints are now returned per episode so this can be recomputed rather than re-derived. - **`log_distance_ratio_mean` is reported, NOT adopted as the adjudicator.** Switching the primary endpoint in the same change that repins the episode set would confound the two. Adoption is a separate pre-registered decision, and it still has to clear the blind criterion: separate zero from random from MSE on existing data. + +## 15 — Benchmark repair: what it bought, and the power claim it refuted (2026-08-02, commit ) +- Category: Benchmark +- Hypothesis: (node 14) the control benchmark cannot resolve a 0.07 effect because its chance baseline swings 0.15-0.45 across five seeds. Repairing the noise sources — persist the EMA target, pin the baseline, fix the episode set, give each episode its own planner stream, adopt a paired continuous endpoint, raise n — should make it able to adjudicate. +- Prediction: each repair removes an identified noise source, and |t| for a real effect then grows as sqrt(n). +- Implementation: `eeg_jepa.py` checkpoint format v2; `forward_eval.py` `baseline_reps` (100), `episode_seed` (2, replacing a `seed` that tracked the model seed), a `torch.Generator` per episode, per-episode `log(d_final/d_start)` and `final_distance_per_episode`, `n_episodes` 20 -> 200. +- Evidence: + - **Noise sources removed, measured.** Baseline sampling sd **0.0750 -> 0.0054** at reps=100. Baseline across 10 runs (2 latent dims x 5 seeds) **0.254-0.483 -> 0.303 for every run**. Episode sets across seeds **5 distinct -> 1, byte-identical**. `random_baseline_stderr` **0.0761 -> 0.0250** at n=200, against a sqrt(10)=3.16 prediction and a measured factor of 3.04. + - **Cost is nil.** Wall clock is flat at ~1.1s from n=20 to n=200; the run is dominated by training, not by the MPC block. n was never a compute decision. + - **THE POWER CLAIM IN THE PREVIOUS ENTRY IS REFUTED.** That entry reported "median n for |t|=2: binary ~248, log-ratio ~154", extrapolated from a paired ld32-vs-ld64 comparison at n=20. Scaling n directly shows |t| does NOT grow — 0.88, 0.21, 0.81, 0.54 at n = 20, 50, 100, 200. That is the null distribution: ld32 vs ld64 has no real effect, so no n reaches |t|=2 and a required-n extrapolated from it is meaningless. The 1.43x sensitivity RATIO at fixed n stands (it compares two endpoints on identical data); the required-n figures do not. + - **Against a KNOWN effect (2 epochs vs 22), the endpoints separate sharply.** log-ratio |t| = 0.43, 0.80, 0.76, **1.48** at n = 20, 50, 100, 200, tracking its sqrt(n) prediction of 0.43, 0.69, 0.97, 1.38. Binary |t| = 0.94, 0.63, 0.62, **0.68** — flat. Thresholding at `goal_tol` destroys the effect, so **more episodes buy the binary endpoint nothing at any n**. + - **The repaired benchmark is still underpowered.** At n=200, against a deliberately large effect (an eleven-fold difference in training), the continuous endpoint reaches only |t| = 1.48. Extrapolating on the sqrt(n) behaviour it does exhibit, |t|=2 needs n ~ 365; on 3 seeds that estimate is itself noisy. +- Decision: **KEEP all six repairs** — every one removes a measured noise source and n=200 is free. **DO NOT declare the benchmark repaired.** It can now adjudicate a large effect at n~400; it cannot adjudicate the 0.07-scale effects the objective A/B line cares about, and raising n alone will not get there because the binary endpoint does not respond to n. Adopting `log_distance_ratio_mean` as the adjudicator is now a prerequisite rather than an option, and remains a separate pre-registered decision that must still clear the blind criterion. +- Next question: the ceiling is now `goal_tol` and the horizon, not the sample size. A success criterion that discards the distance it is computed from cannot be fixed by more episodes. Before any regularizer A/B: decide the adjudicating endpoint under the blind criterion, and re-examine whether `goal_tol=0.15` at `horizon=6` leaves any resolvable signal at all. Also outstanding — nodes 10 and 11 swept CEM knobs under the shared-generator defect, so their paired reading did not hold; re-running them is cheap and would either confirm "the planner is not the bottleneck" on sound footing or overturn it. diff --git a/WorldModel/forward_eval.py b/WorldModel/forward_eval.py index e48a2b6..f52be39 100644 --- a/WorldModel/forward_eval.py +++ b/WorldModel/forward_eval.py @@ -318,7 +318,7 @@ def _cem_plan(model, z0_lat, zg_lat, device, *, horizon, cem_iters, n_samples, e @torch.no_grad() -def _mpc_success(model, train_mean, train_std, device, *, n_episodes=20, horizon=6, +def _mpc_success(model, train_mean, train_std, device, *, n_episodes=200, horizon=6, cem_iters=3, n_samples=64, elite_frac=0.2, mode="signal", episode_seed=2, goal_offset=0.4, goal_tol=0.15, baseline_reps=100, log_features: bool = False, @@ -458,7 +458,7 @@ def evaluate( batch_size: int = 64, rollout_traj: int = 32, rollout_len: int = 8, - mpc_episodes: int = 20, + mpc_episodes: int = 200, mpc_horizon: int = 6, mpc_cem_iters: int = 3, mpc_n_samples: int = 64,