Skip to content

bench(worldmodel): raise n to 200, and refute the power claim behind it - #56

Merged
aurascoper merged 1 commit into
mainfrom
fix/raise-mpc-episodes
Aug 2, 2026
Merged

bench(worldmodel): raise n to 200, and refute the power claim behind it#56
aurascoper merged 1 commit into
mainfrom
fix/raise-mpc-episodes

Conversation

@aurascoper

Copy link
Copy Markdown
Owner

Last item of the benchmark repair — plus ledger node 15, which refutes the power claim I made in #55.

n_episodes 20 → 200

Cost is nil: wall clock is flat at ~1.1 s from n=20 to n=200, because the run is dominated by training, not the MPC block. random_baseline_stderr falls 0.0761 → 0.0250 (√10 = 3.16 predicted, 3.04 measured). n was never a compute decision.

The power claim in #55 is refuted, by its own method

#55 reported "median n for |t|=2: binary ~248, log-ratio ~154", extrapolated from a paired ld32-vs-ld64 comparison at n=20. Scaling n directly:

n 20 50 100 200
log |t| 0.88 0.21 0.81 0.54

It does not grow. That is the null distribution — ld32 vs ld64 has no real effect, so no n reaches |t|=2 and a required-n extrapolated from it is meaningless. I computed a sample size from noise and put it in the ledger.

The 1.43× sensitivity ratio at fixed n stands — it compares two endpoints on identical data. The required-n figures do not, and node 15 says so.

Against a known effect, the endpoints separate sharply

2 epochs vs 22 — a real, large difference:

n 20 50 100 200 √n prediction (log)
log |t| 0.43 0.80 0.76 1.48 0.43 / 0.69 / 0.97 / 1.38
binary |t| 0.94 0.63 0.62 0.68 — flat

The continuous endpoint tracks √n. The binary endpoint does not respond to n at all — thresholding at goal_tol destroys the effect, so more episodes buy it nothing.

So raising n is nearly useless unless the continuous endpoint is adopted, not merely reported. That coupling was invisible before this measurement.

And the repaired benchmark is still underpowered

At n=200, against an eleven-fold difference in training, the continuous endpoint reaches only |t| = 1.48. On its own √n behaviour |t|=2 needs n ≈ 365 — and on 3 seeds that estimate is itself noisy.

Keep all six repairs — each removes a measured noise source and n=200 is free. Do not declare the benchmark repaired. The ceiling is now goal_tol and the horizon, not the sample size. A success criterion that discards the distance it is computed from cannot be fixed by more episodes.

Repair scorecard

before after
target encoder on restore untrained exact
baseline sampling sd 0.0750 0.0054
baseline across seeds 0.254–0.483 0.303 constant
episode sets across seeds 5 distinct 1, identical
planner noise position-dependent per-episode
n 20 200
can it adjudicate 0.07? no still no

Also outstanding

Nodes 10 and 11 swept CEM knobs under the shared-generator defect, so their paired reading did not hold. Re-running them is cheap and would either confirm "the planner is not the bottleneck" on sound footing or overturn it.

Verification

smoke test   3 consecutive runs, exit 0
ledger gate  PASS (0 warnings)    ADR gate  PASS
tests        OK

Last item of the benchmark repair, plus ledger node 15 recording what the whole
repair bought — which is less than the previous entry claimed.

n_episodes 20 -> 200 in both _mpc_success and evaluate. Cost is nil: wall clock
is flat at ~1.1s from n=20 to n=200 because the run is dominated by training,
not by the MPC block. n was never a compute decision.

THE POWER CLAIM IN THE PREVIOUS COMMIT IS REFUTED, BY ITS OWN METHOD.

That commit reported "median n for |t|=2: binary ~248, log-ratio ~154",
extrapolated from a paired ld32-vs-ld64 comparison at n=20. Scaling n directly:

  n_ep      20    50   100   200
  log |t| 0.88  0.21  0.81  0.54     <- does not grow

That is the null distribution. ld32 vs ld64 has no real effect, so no n reaches
|t|=2 and a required-n extrapolated from it is meaningless. I computed a sample
size from noise and put it in the ledger. The 1.43x sensitivity RATIO at fixed n
stands — it compares two endpoints on identical data — but the required-n
figures do not, and node 15 says so.

AGAINST A KNOWN EFFECT (2 epochs vs 22) THE ENDPOINTS SEPARATE SHARPLY.

  n_ep         20    50   100   200     sqrt(n) prediction
  log |t|    0.43  0.80  0.76  1.48     0.43 0.69 0.97 1.38
  binary |t| 0.94  0.63  0.62  0.68     flat

The continuous endpoint tracks sqrt(n). The binary endpoint does not respond to
n AT ALL: thresholding at goal_tol destroys the effect, so more episodes buy it
nothing. Raising n is therefore nearly useless unless the continuous endpoint is
ADOPTED rather than merely reported — a coupling that was not visible before
this measurement.

AND THE REPAIRED BENCHMARK IS STILL UNDERPOWERED. At n=200, against an
eleven-fold difference in training, the continuous endpoint reaches only
|t| = 1.48. On its own sqrt(n) behaviour, |t|=2 needs n ~ 365, and on 3 seeds
that estimate is itself noisy.

So: keep all six repairs, every one removes a measured noise source and n=200 is
free — but DO NOT declare the benchmark repaired. The ceiling is now goal_tol
and the horizon, not the sample size. A success criterion that discards the
distance it is computed from cannot be fixed by more episodes.

Node 15 also records the outstanding item from the previous commit: nodes 10 and
11 swept CEM knobs under the shared-generator defect, so their paired reading did
not hold. Re-running them is cheap and would either confirm "the planner is not
the bottleneck" on sound footing or overturn it.

Verified: smoke test 3 consecutive runs; ledger gate, ADR gate and tests green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DJJ8y1zo7WSqEmGNW8DQap
@aurascoper
aurascoper merged commit c911ea1 into main Aug 2, 2026
2 checks passed
@aurascoper
aurascoper deleted the fix/raise-mpc-episodes branch August 2, 2026 04:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant