bench(worldmodel): nuisance jitter fails structurally — reverted, recorded - #61
Merged
Merged
Conversation
…orded Ledger node 20. Node 19's preferred generator repair: jitter chi/offset between the pre- and post-window so predicting the held factors is no longer free. Acceptance criteria G1-G4 registered BEFORE implementing; depth fixed at 0.10, the constant _render_state already uses, chosen by symmetry and explicitly not tuned. Registered rule: fail any gate = revert, do not adjust the constant. ALL FOUR GATES PASSED. vel exact to 2.33e-15 (jitter cancels in the ratio); raw probe recovers vel at 0.3605; ladder bit-identical (100.0 / 91.5 / 73.5 / 37.0 / 34.5); alphaPower post-from-pre fell 1.000 -> 0.9709; chi still 0.6653. AND THE OUTCOME REGRESSED. Paired, 3 seeds, identical apart from the constant: pos +0.6065 -> +0.3118 -0.2947 vel +0.0227 -> -0.0280 -0.0508 chi +0.9566 -> +0.9306 -0.0260 peak_amp +0.9947 -> +0.9814 -0.0133 ax -0.0467 -> -0.3264 -0.2797 ay -0.2305 -> -0.4947 -0.2642 It degraded the TASK factors 10-20x more than the NUISANCE factors -- the exact inverse of its intent. MECHANISM: the task factors are multiplicative modulations ON the nuisance carriers. c0 = alpha*(1 + 0.10*pos), alpha = f(chi, peak_amp, offset). Carrier sd 0.220 against a total pos contribution of 0.0270 -- leverage 8.2:1. A 10% carrier perturbation injects 0.0259, i.e. 96% of the entire task signal. The intervention put noise straight into the channel carrying the signal. NO USABLE DEPTH EXISTS. Depth vs G3 vs injected-noise-to-signal: 0.01 -> 0.9997 (G3 fails), 0.10x; 0.05 -> 0.9932 (fails), 0.48x; 0.10 -> 0.9731 (passes), 0.96x; 0.20 -> 0.8953 (passes), 1.97x. The minimum depth that removes the subsidy already injects noise the size of the whole task signal. Structural, not mistuned. REVERTED per the registered rule, constant untouched. synthetic_1f.py is unchanged on main; this node is the record. Node 19's stated advantage of repair (2) over repair (1) does not survive: raising the velocity depth improves SNR directly, whereas jittering the carrier degrades it. THE GATE DESIGN WAS INSUFFICIENT, and that is the transferable part. G1-G4 verified the information is present in the raw observation and the control task is unchanged. Neither is the same as "a model can still learn it" -- a repair can pass every raw-signal check and still halve what the encoder recovers. Any future generator change must add G5: factor recovery for pos and vel under the INCUMBENT objective must not degrade. Training the incumbent is permitted; node 14 excludes candidate arms, not the existing baseline. Ledger-only. Verified: smoke 2 runs, ledger and ADR gates green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJJ8y1zo7WSqEmGNW8DQap
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Node 19's preferred generator repair. Acceptance criteria registered before implementing; depth fixed at 0.10 — the constant
_render_statealready uses — chosen by symmetry, explicitly not tuned. Registered rule: fail any gate = revert, do not adjust the constant.All four gates passed
vel = 10·(c1/betaPower − 1)exactvelalphaPowerpost-from-prechirecoverableAnd the outcome regressed
Paired, 3 seeds, identical apart from the constant:
posvelchipeak_ampaxayIt degraded the task factors 10–20× more than the nuisance factors — the exact inverse of its intent.
Mechanism
The task factors are multiplicative modulations on the nuisance carriers:
c0 = alpha·(1 + 0.10·pos),alpha = f(chi, peak_amp, offset).Carrier sd 0.220 against a total
poscontribution of 0.0270 — leverage 8.2 : 1. A 10% carrier perturbation injects 0.0259, i.e. 96% of the entire task signal. The intervention put noise straight into the channel carrying the signal.No usable depth exists — structural, not mistuned
The minimum depth that removes the subsidy already injects noise the size of the whole task signal.
Node 19's stated advantage of repair (2) over repair (1) does not survive: raising the velocity depth improves SNR directly, whereas jittering the carrier degrades it.
Reverted per the registered rule
synthetic_1f.pyis unchanged on main. This node is the record.The transferable part: the gate design was insufficient
G1–G4 verified the information is present in the raw observation and the control task is unchanged. Neither is the same as "a model can still learn it". A repair can pass every raw-signal check and still halve what the encoder recovers.
Any future generator change must add G5: factor recovery for
posandvelunder the incumbent objective must not degrade. Training the incumbent is permitted — node 14 excludes candidate arms, not the existing baseline — and G5 would have caught this before the outcome measurement.Next
The fault is the generator's structure, not its constants. Candidate: give
posandveltheir own additive observation channels, independent ofalpha/beta, so the task signal is not a rounding error on a nuisance carrier and the two can be perturbed independently. Larger than either node-19 candidate; needs its own pre-registration, including G5.Ledger-only. Verified: smoke 2 runs, ledger and ADR gates green.