bench(worldmodel): the fourth condition is rank preservation, not accuracy - #65
Merged
Conversation
…uracy
Ledger node 24. Node 23's adjudicating sweep: inject latent error into an
otherwise-perfect rollout and measure control against magnitude.
READING (a) IS REFUTED -- control degrades gracefully, not discontinuously.
noise sd equiv MSE error/signal success
0.000 0.00000 0.00 80.5%
0.074 0.00548 0.79 77.0% <- multi-step trained level
0.152 0.02311 1.62 65.0% <- incumbent level
0.250 0.06250 2.67 59.0%
Median latent goal signal is 0.5303, so the incumbent's rollout error is 1.62x
the ENTIRE goal signal -- and random error at that magnitude still yields 65.0%
where the real predictor yields 35.0%. Magnitude is not the explanation.
THE FOURTH CONDITION IS RANK PRESERVATION. Over 120 episodes x 64 candidates:
Spearman(predictor cost, true LATENT cost) +0.389 median
Spearman(predictor cost, true POSITION cost) -0.018 median
The predictor ranks candidates acceptably in latent space, and its ranking is
uncorrelated with the objective that decides success.
Why injected noise is benign and the predictor is not: random noise perturbs
each candidate INDEPENDENTLY so the ordering survives and the elite set stays
informative, while a predictor whose error is structured -- concentrated in the
31 directions that do not encode position while the one that does is swamped --
destroys the ordering at ANY MSE. The two are not comparable by magnitude, which
is exactly why five nodes of accuracy-improving repairs changed nothing.
REJECT "improve the forward model's accuracy" as a route to control, and with it
the framing driving nodes 19-23. MSE over a 32-d latent rewards average accuracy
across dimensions dominated by the frozen factors; the planner needs correct
ORDERING along the one direction encoding position. 65% from
noisy-but-rank-preserving vs 35% from accurate-but-rank-destroying, same error
magnitude.
THIS VINDICATES NODE 13's PARKED CONCLUSION. Node 13 named "a control-aware goal
metric" as one of two fixes and reserved it as a DESIGN decision for the user,
not an autonomous build. Nodes 14-23 pursued the other branch -- representation,
benchmark, frame, action channel, forward model -- found five real defects, fixed
four, and moved control by nothing. The measured reason is now on record.
Next is a design decision and belongs to the user per the node-13 stop rule. One
bounded check should precede it: measure Spearman(true latent cost, true position
cost) directly. Node 22 showed the true latent cost yields 76%, so it should be
substantial; if not, the goal metric is the defect rather than the predictor's
use of it.
Ledger-only. Verified: smoke 2 runs; ledger and ADR gates green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DJJ8y1zo7WSqEmGNW8DQap
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Node 23's adjudicating sweep: inject latent error into an otherwise-perfect rollout and measure control against magnitude.
Reading (a) is refuted — control degrades gracefully
Median latent goal signal is 0.5303, so the incumbent's rollout error is 1.62× the entire goal signal — and random error at that magnitude still yields 65.0%, where the real predictor yields 35.0%.
Magnitude is not the explanation. Even error 2.67× the goal signal leaves control at 59%.
The fourth condition is rank preservation
120 episodes × 64 candidates:
The predictor ranks candidates acceptably in latent space, and its ranking is uncorrelated with the objective that decides success. Only 60.8% of episodes reach even ρ > 0.3 against the latent cost.
Why injected noise is benign and the predictor is not
Random noise perturbs each candidate independently, so the ordering survives in expectation and the CEM's elite set stays informative. A predictor whose error is structured — concentrated in the 31 latent directions that don't encode position, while the one that does is swamped — destroys the ordering at any MSE.
The two failure modes are not comparable by magnitude. That is exactly why five nodes of accuracy-improving repairs changed nothing.
Verdict
Reject "improve the forward model's accuracy" as a route to control, and with it the framing driving nodes 19–23. MSE over a 32-d latent rewards average accuracy across dimensions dominated by the frozen factors; the planner needs correct ordering along the single direction encoding position.
65% from noisy-but-rank-preserving versus 35% from accurate-but-rank-destroying, at the same error magnitude.
This vindicates node 13
Node 13 named "a control-aware goal metric" as one of two fixes and explicitly reserved it as a design decision for the user, not an autonomous build.
Nodes 14–23 pursued the other branch — representation, benchmark, frame, action channel, forward model. Five real defects found, four fixed, control moved by nothing. The measured reason is now on record.
Next
A design decision, and per the node-13 stop rule it belongs to you, not to an autonomous build. The measured requirement is a training signal that optimises ranking with respect to the control objective rather than reconstruction error in latent space.
One bounded check should precede it: measure Spearman(true latent cost, true position cost) directly. Node 22 showed the true latent cost yields 76%, so it should be substantial — and if it is not, the goal metric is the defect rather than the predictor's use of it.
Ledger-only. Verified: smoke 2 runs; ledger and ADR gates green.