Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions WorldModel/EXPERIMENT_LEDGER.md
Original file line number Diff line number Diff line change
Expand Up @@ -458,3 +458,28 @@ Two reviewers verified the delivered synthetic A/Bs are sound (symlog threaded i
- Decision: **REVERTED per the registered rule, without tuning K.** `synthetic_1f.py` and `eeg_jepa.py` are unchanged on main; this node is the record. Reverting is also right on the merits: the change adds a corpus mode and two parameters for no demonstrated benefit, and node 20 set the precedent.
- **NODE 22's THREE-CONDITION MODEL IS NOW IN DOUBT.** It predicted that treating the last untreated condition would approach 76%. The condition was treated — measurably, 5.7x — and control did not move. Two readings remain and they are distinguishable: either (a) the residual gap between 0.00546 rollout error and the true env's exact zero is still decisive, in which case control is a near-discontinuous function of forward-model error and the 76% arm was only ever reachable with perfect dynamics; or (b) the three-condition model is incomplete and a fourth condition is unidentified. Node 22's 2x2 cannot separate these because its true-env arm used EXACT dynamics, not merely better ones.
- Next question: **measure control as a function of rollout error directly**, rather than adding another candidate fix. Interpolate between the learned predictor and the true env — e.g. roll the true env but inject calibrated noise at each step, sweeping the injected magnitude across the range spanned by 0.00546 (multi-step trained), 0.0231 (incumbent) and 0 (exact). If control collapses somewhere between 0 and 0.00546, reading (a) holds and the requirement is a far more accurate forward model than any training change here will deliver — which is itself a design finding about latent-space MPC. If control stays flat until the error is large, reading (b) holds and there is a fourth condition still unfound. This is one sweep with no new model and it adjudicates a question five nodes have now circled.

## 24 — The fourth condition is RANK PRESERVATION, and MSE does not deliver it (2026-08-02, commit <pending>)
- Category: Benchmark
- Hypothesis: (node 23) node 22's three-condition model predicted that treating the forward model would approach 76%; it was treated (rollout error cut 5.7x) and control did not move. Two readings remained: **(a)** the residual gap between 0.00546 and exact zero is still decisive, so control is near-discontinuous in forward-model error; **(b)** a fourth condition is unidentified. Node 22's 2x2 could not separate them because its true-env arm used EXACT dynamics, not merely better ones.
- Prediction: sweep control against INJECTED latent error on an otherwise-perfect rollout, spanning the measured error levels. If control collapses below 0.00546, (a) holds. If it degrades gracefully, (b) holds.
- Implementation: true-env rollout, one-frame goal, Gaussian noise added to the encoded final latent at swept magnitude. Per-dimension sd converted to the reported MSE scale, and to a latent distance as `sqrt(d * MSE)` with d=32, against the measured goal signal.
- Evidence:
- **Latent goal signal:** median `||enc(start) - enc(goal)||` = **0.5303**. The incumbent's rollout error corresponds to a latent distance of **0.860 — 1.62x the entire goal signal**; the multi-step-trained model's to **0.418 = 0.79x**.
- **READING (a) IS REFUTED. Control degrades gracefully:**

| noise sd | equiv MSE | error/signal | success |
|---|---|---|---|
| 0.000 | 0 | 0.00 | 80.5% |
| 0.040 | 0.00160 | 0.43 | 81.0% |
| **0.074** | **0.00548** | 0.79 | **77.0%** (multi-step level) |
| 0.100 | 0.01000 | 1.07 | 72.0% |
| **0.152** | **0.02311** | 1.62 | **65.0%** (incumbent level) |
| 0.250 | 0.06250 | 2.67 | 59.0% |

At the INCUMBENT's own error magnitude, random error still yields **65.0%** — where the real predictor yields **35.0%**. Magnitude is not the explanation. Even error 2.67x the goal signal leaves control at 59%.
- **THE FOURTH CONDITION IS RANK PRESERVATION.** Over 120 episodes x 64 candidates: Spearman(predictor cost, true LATENT cost) = **+0.389** median, but Spearman(predictor cost, true POSITION cost) = **-0.018** median — zero. The predictor ranks candidates acceptably in latent space and its ranking is uncorrelated with the objective that decides success. Only **60.8%** of episodes reach even rho > 0.3 against the latent cost.
- **Why injected noise is benign and the predictor is not:** random noise perturbs each candidate INDEPENDENTLY, so the ordering survives in expectation and the CEM's elite set stays informative. A predictor whose error is structured — concentrated in the 31 latent directions that do not encode position, while the one direction that does is swamped — destroys the ordering at ANY MSE. The two failure modes are not comparable by magnitude, which is precisely why five nodes of accuracy-improving repairs changed nothing.
- Decision: **REJECT "improve the forward model's accuracy" as a route to control**, and with it the framing that has driven nodes 19-23. MSE over a 32-dimensional latent rewards average accuracy across dimensions dominated by the frozen factors; the planner needs correct ORDERING along the single direction that encodes position. Accuracy is neither necessary nor sufficient for that — 65% from noisy-but-rank-preserving, 35% from accurate-but-rank-destroying, at the same error magnitude.
- **This vindicates node 13's parked conclusion, three years of nodes later.** Node 13 named "a control-aware goal metric" as one of two fixes and explicitly reserved it as a DESIGN decision for the user rather than an autonomous build. Nodes 14-23 pursued the other branch — representation, benchmark, frame, action channel, forward model — found five real defects, fixed four, and moved control by nothing. The measured reason is now on record.
- Next question: this is a design decision, not a bounded build, and per the node-13 stop rule it belongs to the user. The measured requirement is a training signal that optimises RANKING with respect to the control objective rather than reconstruction error in latent space — e.g. a contrastive or ranking loss over candidate rollouts, or a cost anchored on a decoded task variable rather than on full-latent distance. One bounded thing IS available first and should precede any of it: verify the remaining link by measuring Spearman(true latent cost, true position cost) directly. Node 22 showed the true latent cost yields 76%, so that correlation should be substantial; if it is NOT, the latent cost is a weaker proxy than node 22 implied and the goal metric is the defect rather than the predictor's use of it.
Loading