bench(worldmodel): raise n to 200, and refute the power claim behind it - #56
Merged
Conversation
Last item of the benchmark repair, plus ledger node 15 recording what the whole repair bought — which is less than the previous entry claimed. n_episodes 20 -> 200 in both _mpc_success and evaluate. Cost is nil: wall clock is flat at ~1.1s from n=20 to n=200 because the run is dominated by training, not by the MPC block. n was never a compute decision. THE POWER CLAIM IN THE PREVIOUS COMMIT IS REFUTED, BY ITS OWN METHOD. That commit reported "median n for |t|=2: binary ~248, log-ratio ~154", extrapolated from a paired ld32-vs-ld64 comparison at n=20. Scaling n directly: n_ep 20 50 100 200 log |t| 0.88 0.21 0.81 0.54 <- does not grow That is the null distribution. ld32 vs ld64 has no real effect, so no n reaches |t|=2 and a required-n extrapolated from it is meaningless. I computed a sample size from noise and put it in the ledger. The 1.43x sensitivity RATIO at fixed n stands — it compares two endpoints on identical data — but the required-n figures do not, and node 15 says so. AGAINST A KNOWN EFFECT (2 epochs vs 22) THE ENDPOINTS SEPARATE SHARPLY. n_ep 20 50 100 200 sqrt(n) prediction log |t| 0.43 0.80 0.76 1.48 0.43 0.69 0.97 1.38 binary |t| 0.94 0.63 0.62 0.68 flat The continuous endpoint tracks sqrt(n). The binary endpoint does not respond to n AT ALL: thresholding at goal_tol destroys the effect, so more episodes buy it nothing. Raising n is therefore nearly useless unless the continuous endpoint is ADOPTED rather than merely reported — a coupling that was not visible before this measurement. AND THE REPAIRED BENCHMARK IS STILL UNDERPOWERED. At n=200, against an eleven-fold difference in training, the continuous endpoint reaches only |t| = 1.48. On its own sqrt(n) behaviour, |t|=2 needs n ~ 365, and on 3 seeds that estimate is itself noisy. So: keep all six repairs, every one removes a measured noise source and n=200 is free — but DO NOT declare the benchmark repaired. The ceiling is now goal_tol and the horizon, not the sample size. A success criterion that discards the distance it is computed from cannot be fixed by more episodes. Node 15 also records the outstanding item from the previous commit: nodes 10 and 11 swept CEM knobs under the shared-generator defect, so their paired reading did not hold. Re-running them is cheap and would either confirm "the planner is not the bottleneck" on sound footing or overturn it. Verified: smoke test 3 consecutive runs; ledger gate, ADR gate and tests green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJJ8y1zo7WSqEmGNW8DQap
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Last item of the benchmark repair — plus ledger node 15, which refutes the power claim I made in #55.
n_episodes20 → 200Cost is nil: wall clock is flat at ~1.1 s from n=20 to n=200, because the run is dominated by training, not the MPC block.
random_baseline_stderrfalls 0.0761 → 0.0250 (√10 = 3.16 predicted, 3.04 measured). n was never a compute decision.The power claim in #55 is refuted, by its own method
#55 reported "median n for |t|=2: binary ~248, log-ratio ~154", extrapolated from a paired ld32-vs-ld64 comparison at n=20. Scaling n directly:
It does not grow. That is the null distribution — ld32 vs ld64 has no real effect, so no n reaches |t|=2 and a required-n extrapolated from it is meaningless. I computed a sample size from noise and put it in the ledger.
The 1.43× sensitivity ratio at fixed n stands — it compares two endpoints on identical data. The required-n figures do not, and node 15 says so.
Against a known effect, the endpoints separate sharply
2 epochs vs 22 — a real, large difference:
The continuous endpoint tracks √n. The binary endpoint does not respond to n at all — thresholding at
goal_toldestroys the effect, so more episodes buy it nothing.So raising n is nearly useless unless the continuous endpoint is adopted, not merely reported. That coupling was invisible before this measurement.
And the repaired benchmark is still underpowered
At n=200, against an eleven-fold difference in training, the continuous endpoint reaches only |t| = 1.48. On its own √n behaviour |t|=2 needs n ≈ 365 — and on 3 seeds that estimate is itself noisy.
Keep all six repairs — each removes a measured noise source and n=200 is free. Do not declare the benchmark repaired. The ceiling is now
goal_toland the horizon, not the sample size. A success criterion that discards the distance it is computed from cannot be fixed by more episodes.Repair scorecard
Also outstanding
Nodes 10 and 11 swept CEM knobs under the shared-generator defect, so their paired reading did not hold. Re-running them is cheap and would either confirm "the planner is not the bottleneck" on sound footing or overturn it.
Verification