Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions WorldModel/EXPERIMENT_LEDGER.md
Original file line number Diff line number Diff line change
Expand Up @@ -363,3 +363,28 @@ Two reviewers verified the delivered synthetic A/Bs are sound (symlog threaded i
- **Velocity is not destroyed.** `vel = 10 * (c1 / betaPower - 1)` exactly — max error 2e-15 over 4000 samples. It is a ratio of two observed channels: nonlinear, hence invisible to the linear probe that caps at R^2 = 0.033, but trivially within an MLP's reach. Nothing prevents the encoder learning it; nothing rewards it.
- Decision: **CLOSE the representation-objective line as specified.** No representation objective can encode a variable the generator made 0.01% of the observed variance while paying a variance penalty for ignoring the other 99.99%. This is a generator design fault, not an encoder, planner, objective or benchmark fault. Nodes 6-18 were measuring the wrong thing correctly.
- Next question: repair the generator, under node-14 blind discipline. Two candidates, NOT equivalent: **(1)** raise the velocity modulation depth — cheapest, but it is a knob, and tuning it until an arm wins is unfalsifiable; **(2)** let `chi`/`offset` vary within-trajectory — removes the free-prediction subsidy that makes the nuisance attractive without touching velocity's coefficient, and changes what the objective is REWARDED for rather than what it is SHOWN. Prefer (2). Either way: specify using only oracle/random/zero, never a candidate arm, and require the repaired generator to reproduce the existing ladder ordering (oracle > ay-oracle > ax-oracle > zero ~ random ~ MPC) before any arm is rerun. Add to the acceptance gate an assertion that a probe on the RAW observation recovers `vel` above a floor, so a future generator edit cannot silently reintroduce this.

## 20 — Generator repair attempt: within-transition nuisance jitter FAILS, structurally (2026-08-02, commit <pending>)
- Category: Benchmark
- Hypothesis: (node 19) `chi`/`peak_amp`/`offset` are held constant per transition, so predicting them at t+1 is exact and free; the JEPA loss is therefore minimised by encoding the nuisance. Jittering `chi`/`offset` between the pre- and post-window should remove that subsidy WITHOUT touching velocity's coefficient — changing what the objective is REWARDED for rather than what it is SHOWN. Node 19 preferred this over raising the velocity depth precisely because the latter is a knob that could be tuned until an arm wins.
- Prediction: acceptance criteria G1-G4 registered BEFORE implementing. Jitter depth fixed at **0.10 — the constant `_render_state` already uses for the pos/vel modulation**, chosen by symmetry with the file's own value, explicitly not tuned. Registered rule: FAIL any gate = revert, do not adjust the constant.
- Implementation: `_jitter_held` perturbing `chi` and `offset` multiplicatively for RENDERING ONLY, drawn independently for the pre- and post-windows. `_step` untouched, so dynamics, reachable set and the exact oracle are identical. `peak_amp` deliberately left fixed, matching node 19's scope.
- Evidence:
- **All four gates PASSED.** G1a `vel = 10*(c1/betaPower - 1)` exact to 2.33e-15 (jitter cancels in the ratio, as predicted). G1b raw-observation linear probe recovers `vel` at R^2 0.3605 (>= 0.30; was 0.386). G2 ladder bit-identical — ORACLE full 100.0% > ay 91.5% > ax 73.5% > zero 37.0% ~ random 34.5%. G3 `alphaPower` post-from-pre fell 1.000 -> 0.9709 (< 0.99). G4 `chi` still recoverable at 0.6653 (> 0.50).
- **And the outcome was a clear REGRESSION.** Paired, 3 seeds each, identical apart from the constant:

| metric | jitter=0 | jitter=0.10 | delta |
|---|---|---|---|
| `pos` | +0.6065 | +0.3118 | **-0.2947** |
| `vel` | +0.0227 | -0.0280 | -0.0508 |
| `chi` | +0.9566 | +0.9306 | -0.0260 |
| `peak_amp` | +0.9947 | +0.9814 | -0.0133 |
| `ax` | -0.0467 | -0.3264 | -0.2797 |
| `ay` | -0.2305 | -0.4947 | -0.2642 |

It degraded the TASK factors by 10-20x more than the NUISANCE factors — the exact inverse of its intent. `pos` halved; `chi` and `peak_amp` barely moved.
- **MECHANISM: the task factors are multiplicative modulations ON the nuisance carriers.** `c0 = alpha * (1 + 0.10*pos)` with `alpha = f(chi, peak_amp, offset)`. Carrier sd is **0.220**; the entire `pos` contribution is **0.0270** — a leverage of **8.2 : 1**. Perturbing the carrier by 10% injects 0.0259, i.e. **96% of the whole task signal**. The intervention put noise directly into the channel carrying the signal.
- **NO USABLE DEPTH EXISTS — this is structural, not mistuned.** Sweeping depth against G3 and against the injected-noise-to-task-signal ratio: 0.01 -> R^2 0.9997 (G3 fails), noise 0.10x; 0.05 -> 0.9932 (fails), 0.48x; 0.10 -> 0.9731 (passes), 0.96x; 0.20 -> 0.8953 (passes), 1.97x. **The minimum depth that removes the subsidy already injects noise the size of the entire task signal.** There is no window.
- Decision: **REVERTED, per the registered rule, without adjusting the constant.** `synthetic_1f.py` is unchanged on main; this node is the record. **REJECT node 19's preferred repair (2) — not as mistuned but as structurally unavailable**, and note that its stated advantage over repair (1) does not survive: raising the velocity depth improves SNR directly, whereas jittering the carrier degrades it, so (1) is now the better of the two even though it remains a knob.
- **THE GATE DESIGN WAS INSUFFICIENT, and that is the transferable lesson.** G1-G4 verified that the information is present in the raw observation and that the control task is unchanged. Neither is the same as "a model can still learn it". A repair can pass every raw-signal check and still halve what the encoder recovers. **Add G5 to any future generator change: factor recovery for `pos` and `vel` under the INCUMBENT objective must not degrade.** Training the incumbent is permitted under the blind rule — node 14 excludes candidate arms, not the existing baseline — and G5 would have caught this before the outcome measurement.
- Next question: the fault is the generator's STRUCTURE, not its constants. Task factors are small multiplicative modulations riding on large nuisance carriers, so any perturbation of the carrier is amplified 8.2x relative to the signal, and any attempt to suppress the nuisance suppresses the task with it. The candidate is therefore structural: give `pos` and `vel` their OWN additive observation channels, independent of `alpha`/`beta`, so the task signal is not a rounding error on a nuisance carrier and the two can be perturbed independently. That is a larger change than either node-19 candidate and needs its own pre-registration, including G5.
Loading