Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions WorldModel/EXPERIMENT_LEDGER.md
Original file line number Diff line number Diff line change
Expand Up @@ -388,3 +388,28 @@ Two reviewers verified the delivered synthetic A/Bs are sound (symlog threaded i
- Decision: **REVERTED, per the registered rule, without adjusting the constant.** `synthetic_1f.py` is unchanged on main; this node is the record. **REJECT node 19's preferred repair (2) — not as mistuned but as structurally unavailable**, and note that its stated advantage over repair (1) does not survive: raising the velocity depth improves SNR directly, whereas jittering the carrier degrades it, so (1) is now the better of the two even though it remains a knob.
- **THE GATE DESIGN WAS INSUFFICIENT, and that is the transferable lesson.** G1-G4 verified that the information is present in the raw observation and that the control task is unchanged. Neither is the same as "a model can still learn it". A repair can pass every raw-signal check and still halve what the encoder recovers. **Add G5 to any future generator change: factor recovery for `pos` and `vel` under the INCUMBENT objective must not degrade.** Training the incumbent is permitted under the blind rule — node 14 excludes candidate arms, not the existing baseline — and G5 would have caught this before the outcome measurement.
- Next question: the fault is the generator's STRUCTURE, not its constants. Task factors are small multiplicative modulations riding on large nuisance carriers, so any perturbation of the carrier is amplified 8.2x relative to the signal, and any attempt to suppress the nuisance suppresses the task with it. The candidate is therefore structural: give `pos` and `vel` their OWN additive observation channels, independent of `alpha`/`beta`, so the task signal is not a rounding error on a nuisance carrier and the two can be perturbed independently. That is a larger change than either node-19 candidate and needs its own pre-registration, including G5.

## 21 — Additive task channels: the representation is fully repaired and control does NOT follow (2026-08-02, commit <pending>)
- Category: Benchmark
- Hypothesis: (node 20) the fault is the generator's STRUCTURE. Task factors were small multiplicative modulations riding on large nuisance carriers, so the signal was a rounding error and the nuisance could not be perturbed without destroying it. Giving `pos`/`vel` their own additive channels, independent of `alpha`/`beta`, should let the encoder represent them and the planner use them.
- Prediction: G1-G5 registered BEFORE implementing, **including G5, the gate node 20 was missing**.
- Implementation: `_render_state` only — `c0 = alpha*(1 + 0.10*pos)` becomes `c0 = 1.0 + pos`, likewise `c1` for `vel`. `_step` untouched. **No tunable constant introduced**: `JEPATransitionDataset` z-scores every feature on train statistics, so any gain divides straight back out; the `1.0` is a DC offset for positivity that the same z-scoring removes. Restores `_render_state`'s own stated intent — its comment already claimed these two channels carry the task factors.
- Evidence — **every gate passed, the representation is transformed, and control is unmoved.**
- **G1** raw-observation probe: `pos` R^2 **1.0000**, `vel` **1.0000** (>0.99). **G2** oracle ladder BIT-IDENTICAL — 100.0 / 91.5 / 73.5 / 37.0 / 34.5 — confirming dynamics untouched. **G4** `chi` still recoverable at 0.6849. **G3** as declared: `alphaPower` post-from-pre remains 1.000, the free-prediction subsidy is NOT addressed by this change and must not later be called new.
- **G5 (the new gate) passes by a wide margin**, 3 seeds:

| factor | node 20 baseline | after | |
|---|---|---|---|
| `pos` | 0.6065 | **0.9824** | task |
| `vel` | 0.0227 | **0.9740** | task |
| `chi` | 0.9566 | 0.7660 | nuisance |
| `peak_amp` | 0.9947 | 0.8859 | nuisance |

The encoder inverted its priorities exactly as intended: task factors up, nuisance down. `vel` went from unencoded to near-perfect.
- **Action recovery is transformed.** `ax` **-0.1676 -> +0.8309**, `ay` **-0.1199 -> +0.3709**, with `mode_bit` still at -0.2807, so the negative control holds. Node 18's "no recoverable action information" is fully resolved. The `ay` < `ax` asymmetry is expected: `ay` moves `pos` by `0.5*ay*DT = 0.05*ay` while `ax` moves `vel` by `ax*DT = 0.1*ax`, a larger share of `vel`'s range.
- **The frame offset improved 16x without being targeted.** Node 14's A/B ratio **70.6 -> 4.25**, `centroid_gap` 1.10 -> 1.34, `procrustes_residual` 0.97 (unchanged, still non-rotational).
- **AND CONTROL DID NOT MOVE.** `mpc_success` **0.353 +/- 0.034** against a random baseline of **0.361**; log-ratio **+0.0582** against zero-action's **+0.0341**. Still at chance, still worse than doing nothing.
- **The planner remains uninformative on a representation that now contains what it needs.** correlation(needed, produced) **-0.0206** (node 17: +0.0375), sign agreement **49.0%** (45.5%), and it still produces **0.069** of the **0.222** displacement required — 31%, essentially unchanged from node 17's 33%.
- Decision: **KEEP the repair.** It passed every pre-registered gate, fixed a real and measured defect, and made the benchmark's task factors observable as its own comment always claimed. **But nodes 17-19's diagnosis is now NECESSARY BUT NOT SUFFICIENT.** The action channel was genuinely broken and is genuinely fixed; control did not follow. The bottleneck is downstream of the representation.
- **This is the third defect found, fixed, and followed by no change in control** — the frame offset (node 14, remedied, no effect), the benchmark's resolving power (node 15, repaired, still underpowered), and now the action channel (nodes 17-19, repaired, no effect). Each was real. None was binding. That pattern is itself the finding: the control failure is not a single broken component.
- Next question: the remaining untested link is the PREDICTOR'S MULTI-STEP ROLLOUT, and it now has evidence rather than suspicion. Measured after the repair, rollout error compounds **11.3x to 17.4x** from h=1 to h=8 (1-step 0.0010-0.0016 -> h=8 0.0173-0.0290) while `horizon=6`. A CEM plan feeds the predictor its own output for six steps, so the cost being minimised is computed on a state that may not correspond to the trajectory actually executed. Test it directly and cheaply: score the CEM against a rollout that uses the TRUE env instead of the predictor, on the same episodes and plans. If control jumps, the predictor's rollout is the bottleneck; if it does not, the fault is in the cost or the goal latent, not the forward model.
18 changes: 15 additions & 3 deletions WorldModel/synthetic_1f.py
Original file line number Diff line number Diff line change
Expand Up @@ -48,9 +48,21 @@ def _render_state(z: dict[str, float], t: int) -> dict[str, Any]:
theta = base * BANDS["theta"] ** (-z["chi"])
alpha = base * BANDS["alpha"] ** (-z["chi"]) + z["peak_amp"]
beta = base * BANDS["beta"] ** (-z["chi"])
# Two "electrode" channels carry the task factors, so pos/vel are also observable.
c0 = alpha * (1.0 + 0.10 * z["pos"])
c1 = beta * (1.0 + 0.10 * z["vel"])
# Two "electrode" channels carry the task factors, so pos/vel are also
# observable. ADDITIVE and independent of the 1/f carrier, since the earlier
# multiplicative form defeated that stated intent: c0 = alpha*(1 + 0.10*pos)
# left pos at 0.845% of c0's variance and vel at 0.0106% of c1's (node 19),
# with 8.2:1 carrier leverage, so the encoder learned the held factors
# (peak_amp 0.997, chi 0.950) and left vel at 0.033 (node 18). It also made
# the nuisance impossible to perturb without destroying the task signal --
# the minimum jitter depth that removed the free-prediction subsidy already
# injected noise the size of the entire pos modulation (node 20).
#
# No gain constant here, deliberately: JEPATransitionDataset z-scores every
# feature on train statistics, so any gain divides straight back out. The
# 1.0 is a DC offset for positivity and is removed by the same z-scoring.
c0 = 1.0 + z["pos"]
c1 = 1.0 + z["vel"]
return {
"timestamp": 1_700_000_000.0 + t,
"alphaPower": alpha,
Expand Down
Loading