Skip to content

fix(worldmodel): pin the episode set, per-episode CEM noise, paired endpoint - #55

Merged
aurascoper merged 1 commit into
mainfrom
fix/pin-episodes-and-paired-endpoint
Aug 2, 2026
Merged

fix(worldmodel): pin the episode set, per-episode CEM noise, paired endpoint#55
aurascoper merged 1 commit into
mainfrom
fix/pin-episodes-and-paired-endpoint

Conversation

@aurascoper

Copy link
Copy Markdown
Owner

Items 3–5 of the benchmark repair.

1. The episode set is a property of the benchmark, not the run

evaluate() passed seed = model_seed + 2 into _mpc_success, so every model seed drew a different episode set — "which episodes" was confounded with "which model". Between-set difficulty carried sd 0.0853, about half the observed baseline swing, and was being read as model variance.

_mpc_success now takes episode_seed (default 2) and drops seed entirely. Episodes, baseline draws and planner noise all derive from it and nothing else.

across 10 runs (2 latent dims × 5 seeds) before after
random_baseline_success 0.254 – 0.483 0.303 for every run
episode sets 5 distinct 1, byte-identical

2. A generator per episode

One torch.Generator consumed sequentially made the planning noise on episode i depend on how many draws episodes 0…i−1 had taken — a function of n_samples × cem_iters × horizon. Any arm touching a CEM knob got different noise on the same episode, silently defeating pairing even with the episode set fixed. Nothing in the panel would have shown it.

Nodes 10 and 11 swept exactly those knobs. That sweep was not the paired comparison it appeared to be — not necessarily wrong, but its "within seed noise" reading rested on an assumption that did not hold.

3. Paired continuous endpoint — log(d_final / d_start)

The ratio, not the difference: episodes differ enormously in how far the goal starts, so a raw difference is dominated by episode difficulty. The log makes "halved the distance" one effect size wherever it happens.

Measured, not simulated — paired ld32 vs ld64 on identical episodes and identical planner noise:

endpoint mean |t| median n for |t|=2
per-episode binary 0.62 ~248
per-episode log-ratio 0.88 ~154

1.43×, where a power simulation suggested nearer 3–4×. The mechanism the simulation identified is right — only 3–8 of 20 pairs are discordant, which is why binary is weak — but the size was overstated, and five seeds is a noisy estimate of it.

An earlier version of this measurement said 1.82×; that used an approximation of the paired binary sd from the two marginal rates. final_distance_per_episode is now returned so either endpoint can be computed exactly downstream, and the exact figure is 1.43×.

log_distance_ratio_mean is reported, NOT adopted as the adjudicator. Switching the primary endpoint in the same change that repins the episode set would confound the two. Adoption is a separate pre-registered decision that still has to clear the blind criterion.

Node 14's success_rate values no longer reproduce — by design

Per-episode generators change planner noise. The episode set for model-seed 0 is unchanged (the old effective seed was 2; the new fixed episode_seed is 2), but every seed now scores that same set.

Node 14's conclusions are untouched. The frame ratio, the Procrustes residual and the one-frame remedy were all measured within a run, never across seeds. Recorded in a dated Corrections to node 14, part 2 block rather than by editing node 14.

Verification

smoke test   3 consecutive runs, and at 1 and 12 threads
ledger gate  PASS (0 warnings)     ledger tests  OK
ADR gate     PASS                  ADR tests     OK
frame tests  OK (2 skipped under the shim, as designed)

Remaining

Raise n to ~200 — the last item, and now a config change rather than a design one.

…ndpoint

Items 3-5 of the benchmark repair.

1. EPISODE SET IS A PROPERTY OF THE BENCHMARK, NOT THE RUN.

evaluate() passed seed = model_seed + 2 into _mpc_success, so every model seed
drew a DIFFERENT episode set: "which episodes" was confounded with "which
model". Measured previously, between-set difficulty carried sd 0.0853 -- about
half the observed baseline swing -- and was being read as model variance.

_mpc_success now takes `episode_seed` (default 2) and drops `seed` entirely.
Episodes, baseline draws and planner noise all derive from it and nothing else.
Result across 10 runs (2 latent dims x 5 seeds): random_baseline_success is
0.303 for EVERY run, against 0.254-0.483 before, and the episode sets are
byte-identical.

2. A GENERATOR PER EPISODE.

One torch.Generator consumed sequentially made the planning noise on episode i
depend on how many draws episodes 0..i-1 had taken -- a function of
n_samples * cem_iters * horizon. Any arm touching a CEM knob got different noise
on the SAME episode, silently defeating pairing even with the episode set fixed.
Nothing in the panel would have shown it. Nodes 10 and 11 swept exactly those
knobs, so that sweep was not the paired comparison it appeared to be.

3. PAIRED CONTINUOUS ENDPOINT: per-episode log(d_final / d_start).

The ratio rather than the difference, because episodes differ enormously in how
far the goal starts and a raw difference is dominated by that. The log makes
"halved the distance" one effect size wherever it happens.

MEASURED, not simulated -- paired ld32 vs ld64 on identical episodes and
identical planner noise:

  mean |t|   binary 0.62   log-ratio 0.88   -> 1.43x
  median n for |t|=2:  binary ~248   log-ratio ~154

A power simulation had suggested nearer 3-4x. The mechanism it identified is
right -- only 3-8 of 20 pairs are discordant, which is why the binary endpoint
is weak -- but the size was overstated, and 5 seeds is a noisy estimate of it.
Reported here rather than repeated from the simulation.

log_distance_ratio_mean is REPORTED, NOT ADOPTED as the adjudicator. Switching
the primary endpoint in the same change that repins the episode set would
confound the two; adoption is a separate pre-registered decision that still has
to clear the blind criterion. final_distance_per_episode is also returned so a
downstream paired analysis can compute either endpoint exactly instead of
approximating the binary one from two marginal rates -- an approximation that
overstated the advantage as 1.82x before it was replaced with the exact figure.

NODE 14's success_rate VALUES NO LONGER REPRODUCE, by design: per-episode
generators change planner noise. The episode set for model-seed 0 is unchanged
(the old effective seed was 2 and the new fixed episode_seed is 2). Node 14's
conclusions are untouched -- the frame ratio, Procrustes residual and one-frame
remedy were all measured within a run, never across seeds. Recorded in a dated
"Corrections to node 14, part 2" block rather than by editing node 14.

Verified: smoke test passes on 3 consecutive runs and at 1 and 12 threads;
ledger gate, ADR gate and both suites green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DJJ8y1zo7WSqEmGNW8DQap
@aurascoper
aurascoper merged commit 3ef0a3d into main Aug 2, 2026
2 checks passed
@aurascoper
aurascoper deleted the fix/pin-episodes-and-paired-endpoint branch August 2, 2026 04:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant