Skip to content

bench(worldmodel): the fourth condition is rank preservation, not accuracy - #65

Merged
aurascoper merged 1 commit into
mainfrom
bench/rollout-error-sweep
Aug 3, 2026
Merged

bench(worldmodel): the fourth condition is rank preservation, not accuracy#65
aurascoper merged 1 commit into
mainfrom
bench/rollout-error-sweep

Conversation

@aurascoper

Copy link
Copy Markdown
Owner

Node 23's adjudicating sweep: inject latent error into an otherwise-perfect rollout and measure control against magnitude.

Reading (a) is refuted — control degrades gracefully

noise sd equiv MSE error / goal signal success
0.000 0 0.00 80.5%
0.040 0.00160 0.43 81.0%
0.074 0.00548 0.79 77.0% ← multi-step trained level
0.100 0.01000 1.07 72.0%
0.152 0.02311 1.62 65.0% ← incumbent level
0.250 0.06250 2.67 59.0%

Median latent goal signal is 0.5303, so the incumbent's rollout error is 1.62× the entire goal signal — and random error at that magnitude still yields 65.0%, where the real predictor yields 35.0%.

Magnitude is not the explanation. Even error 2.67× the goal signal leaves control at 59%.

The fourth condition is rank preservation

120 episodes × 64 candidates:

ranking correlation Spearman (median)
predictor cost ↔ true latent cost +0.389
predictor cost ↔ true position cost −0.018

The predictor ranks candidates acceptably in latent space, and its ranking is uncorrelated with the objective that decides success. Only 60.8% of episodes reach even ρ > 0.3 against the latent cost.

Why injected noise is benign and the predictor is not

Random noise perturbs each candidate independently, so the ordering survives in expectation and the CEM's elite set stays informative. A predictor whose error is structured — concentrated in the 31 latent directions that don't encode position, while the one that does is swamped — destroys the ordering at any MSE.

The two failure modes are not comparable by magnitude. That is exactly why five nodes of accuracy-improving repairs changed nothing.

Verdict

Reject "improve the forward model's accuracy" as a route to control, and with it the framing driving nodes 19–23. MSE over a 32-d latent rewards average accuracy across dimensions dominated by the frozen factors; the planner needs correct ordering along the single direction encoding position.

65% from noisy-but-rank-preserving versus 35% from accurate-but-rank-destroying, at the same error magnitude.

This vindicates node 13

Node 13 named "a control-aware goal metric" as one of two fixes and explicitly reserved it as a design decision for the user, not an autonomous build.

Nodes 14–23 pursued the other branch — representation, benchmark, frame, action channel, forward model. Five real defects found, four fixed, control moved by nothing. The measured reason is now on record.

Next

A design decision, and per the node-13 stop rule it belongs to you, not to an autonomous build. The measured requirement is a training signal that optimises ranking with respect to the control objective rather than reconstruction error in latent space.

One bounded check should precede it: measure Spearman(true latent cost, true position cost) directly. Node 22 showed the true latent cost yields 76%, so it should be substantial — and if it is not, the goal metric is the defect rather than the predictor's use of it.

Ledger-only. Verified: smoke 2 runs; ledger and ADR gates green.

…uracy

Ledger node 24. Node 23's adjudicating sweep: inject latent error into an
otherwise-perfect rollout and measure control against magnitude.

READING (a) IS REFUTED -- control degrades gracefully, not discontinuously.

  noise sd  equiv MSE  error/signal  success
     0.000    0.00000          0.00    80.5%
     0.074    0.00548          0.79    77.0%   <- multi-step trained level
     0.152    0.02311          1.62    65.0%   <- incumbent level
     0.250    0.06250          2.67    59.0%

Median latent goal signal is 0.5303, so the incumbent's rollout error is 1.62x
the ENTIRE goal signal -- and random error at that magnitude still yields 65.0%
where the real predictor yields 35.0%. Magnitude is not the explanation.

THE FOURTH CONDITION IS RANK PRESERVATION. Over 120 episodes x 64 candidates:

  Spearman(predictor cost, true LATENT cost)   +0.389 median
  Spearman(predictor cost, true POSITION cost) -0.018 median

The predictor ranks candidates acceptably in latent space, and its ranking is
uncorrelated with the objective that decides success.

Why injected noise is benign and the predictor is not: random noise perturbs
each candidate INDEPENDENTLY so the ordering survives and the elite set stays
informative, while a predictor whose error is structured -- concentrated in the
31 directions that do not encode position while the one that does is swamped --
destroys the ordering at ANY MSE. The two are not comparable by magnitude, which
is exactly why five nodes of accuracy-improving repairs changed nothing.

REJECT "improve the forward model's accuracy" as a route to control, and with it
the framing driving nodes 19-23. MSE over a 32-d latent rewards average accuracy
across dimensions dominated by the frozen factors; the planner needs correct
ORDERING along the one direction encoding position. 65% from
noisy-but-rank-preserving vs 35% from accurate-but-rank-destroying, same error
magnitude.

THIS VINDICATES NODE 13's PARKED CONCLUSION. Node 13 named "a control-aware goal
metric" as one of two fixes and reserved it as a DESIGN decision for the user,
not an autonomous build. Nodes 14-23 pursued the other branch -- representation,
benchmark, frame, action channel, forward model -- found five real defects, fixed
four, and moved control by nothing. The measured reason is now on record.

Next is a design decision and belongs to the user per the node-13 stop rule. One
bounded check should precede it: measure Spearman(true latent cost, true position
cost) directly. Node 22 showed the true latent cost yields 76%, so it should be
substantial; if not, the goal metric is the defect rather than the predictor's
use of it.

Ledger-only. Verified: smoke 2 runs; ledger and ADR gates green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DJJ8y1zo7WSqEmGNW8DQap
@aurascoper
aurascoper merged commit 730f2e4 into main Aug 3, 2026
2 checks passed
@aurascoper
aurascoper deleted the bench/rollout-error-sweep branch August 3, 2026 04:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant