From 4939f456d87a5329e09ac9cbf2fe0c8f99335706 Mon Sep 17 00:00:00 2001 From: aurascoper Date: Sun, 2 Aug 2026 23:48:23 -0500 Subject: [PATCH] bench(worldmodel): the fourth condition is rank preservation, not accuracy Ledger node 24. Node 23's adjudicating sweep: inject latent error into an otherwise-perfect rollout and measure control against magnitude. READING (a) IS REFUTED -- control degrades gracefully, not discontinuously. noise sd equiv MSE error/signal success 0.000 0.00000 0.00 80.5% 0.074 0.00548 0.79 77.0% <- multi-step trained level 0.152 0.02311 1.62 65.0% <- incumbent level 0.250 0.06250 2.67 59.0% Median latent goal signal is 0.5303, so the incumbent's rollout error is 1.62x the ENTIRE goal signal -- and random error at that magnitude still yields 65.0% where the real predictor yields 35.0%. Magnitude is not the explanation. THE FOURTH CONDITION IS RANK PRESERVATION. Over 120 episodes x 64 candidates: Spearman(predictor cost, true LATENT cost) +0.389 median Spearman(predictor cost, true POSITION cost) -0.018 median The predictor ranks candidates acceptably in latent space, and its ranking is uncorrelated with the objective that decides success. Why injected noise is benign and the predictor is not: random noise perturbs each candidate INDEPENDENTLY so the ordering survives and the elite set stays informative, while a predictor whose error is structured -- concentrated in the 31 directions that do not encode position while the one that does is swamped -- destroys the ordering at ANY MSE. The two are not comparable by magnitude, which is exactly why five nodes of accuracy-improving repairs changed nothing. REJECT "improve the forward model's accuracy" as a route to control, and with it the framing driving nodes 19-23. MSE over a 32-d latent rewards average accuracy across dimensions dominated by the frozen factors; the planner needs correct ORDERING along the one direction encoding position. 65% from noisy-but-rank-preserving vs 35% from accurate-but-rank-destroying, same error magnitude. THIS VINDICATES NODE 13's PARKED CONCLUSION. Node 13 named "a control-aware goal metric" as one of two fixes and reserved it as a DESIGN decision for the user, not an autonomous build. Nodes 14-23 pursued the other branch -- representation, benchmark, frame, action channel, forward model -- found five real defects, fixed four, and moved control by nothing. The measured reason is now on record. Next is a design decision and belongs to the user per the node-13 stop rule. One bounded check should precede it: measure Spearman(true latent cost, true position cost) directly. Node 22 showed the true latent cost yields 76%, so it should be substantial; if not, the goal metric is the defect rather than the predictor's use of it. Ledger-only. Verified: smoke 2 runs; ledger and ADR gates green. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01DJJ8y1zo7WSqEmGNW8DQap --- WorldModel/EXPERIMENT_LEDGER.md | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/WorldModel/EXPERIMENT_LEDGER.md b/WorldModel/EXPERIMENT_LEDGER.md index 4e301ac..183e465 100644 --- a/WorldModel/EXPERIMENT_LEDGER.md +++ b/WorldModel/EXPERIMENT_LEDGER.md @@ -458,3 +458,28 @@ Two reviewers verified the delivered synthetic A/Bs are sound (symlog threaded i - Decision: **REVERTED per the registered rule, without tuning K.** `synthetic_1f.py` and `eeg_jepa.py` are unchanged on main; this node is the record. Reverting is also right on the merits: the change adds a corpus mode and two parameters for no demonstrated benefit, and node 20 set the precedent. - **NODE 22's THREE-CONDITION MODEL IS NOW IN DOUBT.** It predicted that treating the last untreated condition would approach 76%. The condition was treated — measurably, 5.7x — and control did not move. Two readings remain and they are distinguishable: either (a) the residual gap between 0.00546 rollout error and the true env's exact zero is still decisive, in which case control is a near-discontinuous function of forward-model error and the 76% arm was only ever reachable with perfect dynamics; or (b) the three-condition model is incomplete and a fourth condition is unidentified. Node 22's 2x2 cannot separate these because its true-env arm used EXACT dynamics, not merely better ones. - Next question: **measure control as a function of rollout error directly**, rather than adding another candidate fix. Interpolate between the learned predictor and the true env — e.g. roll the true env but inject calibrated noise at each step, sweeping the injected magnitude across the range spanned by 0.00546 (multi-step trained), 0.0231 (incumbent) and 0 (exact). If control collapses somewhere between 0 and 0.00546, reading (a) holds and the requirement is a far more accurate forward model than any training change here will deliver — which is itself a design finding about latent-space MPC. If control stays flat until the error is large, reading (b) holds and there is a fourth condition still unfound. This is one sweep with no new model and it adjudicates a question five nodes have now circled. + +## 24 — The fourth condition is RANK PRESERVATION, and MSE does not deliver it (2026-08-02, commit ) +- Category: Benchmark +- Hypothesis: (node 23) node 22's three-condition model predicted that treating the forward model would approach 76%; it was treated (rollout error cut 5.7x) and control did not move. Two readings remained: **(a)** the residual gap between 0.00546 and exact zero is still decisive, so control is near-discontinuous in forward-model error; **(b)** a fourth condition is unidentified. Node 22's 2x2 could not separate them because its true-env arm used EXACT dynamics, not merely better ones. +- Prediction: sweep control against INJECTED latent error on an otherwise-perfect rollout, spanning the measured error levels. If control collapses below 0.00546, (a) holds. If it degrades gracefully, (b) holds. +- Implementation: true-env rollout, one-frame goal, Gaussian noise added to the encoded final latent at swept magnitude. Per-dimension sd converted to the reported MSE scale, and to a latent distance as `sqrt(d * MSE)` with d=32, against the measured goal signal. +- Evidence: + - **Latent goal signal:** median `||enc(start) - enc(goal)||` = **0.5303**. The incumbent's rollout error corresponds to a latent distance of **0.860 — 1.62x the entire goal signal**; the multi-step-trained model's to **0.418 = 0.79x**. + - **READING (a) IS REFUTED. Control degrades gracefully:** + + | noise sd | equiv MSE | error/signal | success | + |---|---|---|---| + | 0.000 | 0 | 0.00 | 80.5% | + | 0.040 | 0.00160 | 0.43 | 81.0% | + | **0.074** | **0.00548** | 0.79 | **77.0%** (multi-step level) | + | 0.100 | 0.01000 | 1.07 | 72.0% | + | **0.152** | **0.02311** | 1.62 | **65.0%** (incumbent level) | + | 0.250 | 0.06250 | 2.67 | 59.0% | + + At the INCUMBENT's own error magnitude, random error still yields **65.0%** — where the real predictor yields **35.0%**. Magnitude is not the explanation. Even error 2.67x the goal signal leaves control at 59%. + - **THE FOURTH CONDITION IS RANK PRESERVATION.** Over 120 episodes x 64 candidates: Spearman(predictor cost, true LATENT cost) = **+0.389** median, but Spearman(predictor cost, true POSITION cost) = **-0.018** median — zero. The predictor ranks candidates acceptably in latent space and its ranking is uncorrelated with the objective that decides success. Only **60.8%** of episodes reach even rho > 0.3 against the latent cost. + - **Why injected noise is benign and the predictor is not:** random noise perturbs each candidate INDEPENDENTLY, so the ordering survives in expectation and the CEM's elite set stays informative. A predictor whose error is structured — concentrated in the 31 latent directions that do not encode position, while the one direction that does is swamped — destroys the ordering at ANY MSE. The two failure modes are not comparable by magnitude, which is precisely why five nodes of accuracy-improving repairs changed nothing. +- Decision: **REJECT "improve the forward model's accuracy" as a route to control**, and with it the framing that has driven nodes 19-23. MSE over a 32-dimensional latent rewards average accuracy across dimensions dominated by the frozen factors; the planner needs correct ORDERING along the single direction that encodes position. Accuracy is neither necessary nor sufficient for that — 65% from noisy-but-rank-preserving, 35% from accurate-but-rank-destroying, at the same error magnitude. +- **This vindicates node 13's parked conclusion, three years of nodes later.** Node 13 named "a control-aware goal metric" as one of two fixes and explicitly reserved it as a DESIGN decision for the user rather than an autonomous build. Nodes 14-23 pursued the other branch — representation, benchmark, frame, action channel, forward model — found five real defects, fixed four, and moved control by nothing. The measured reason is now on record. +- Next question: this is a design decision, not a bounded build, and per the node-13 stop rule it belongs to the user. The measured requirement is a training signal that optimises RANKING with respect to the control objective rather than reconstruction error in latent space — e.g. a contrastive or ranking loss over candidate rollouts, or a cost anchored on a decoded task variable rather than on full-latent distance. One bounded thing IS available first and should precede any of it: verify the remaining link by measuring Spearman(true latent cost, true position cost) directly. Node 22 showed the true latent cost yields 76%, so that correlation should be substantial; if it is NOT, the latent cost is a weaker proxy than node 22 implied and the goal metric is the defect rather than the predictor's use of it.