bench(worldmodel): the goal metric is also a defect — rho 0.25 at its best - #66
Merged
Merged
Conversation
… best Ledger node 25. Node 24's bounded check before any design decision: node 22 showed the true latent cost yields 76%, so Spearman(true latent cost, true position cost) should be substantial. It is not. 150 episodes x 64 candidates x 3 seeds, TRUE-env rollouts, no predictor: goal latent median rho mean rho<0 node 22 success one-frame (online goal) 0.2514 0.2735 17.8% 76.0% incumbent (target goal) 0.1570 0.0744 41.3% 37.5% predictor cost (node 24) -0.018 — — 35.0% THE LATENT METRIC IS WEAK EVEN AT ITS BEST. One-frame reaches only rho = 0.25. It still yields 76% because the CEM needs only to find a good candidate among 64 and rho 0.25 enriches the elite set -- but the metric is not measuring what success measures. THE TWO-ENCODER FRAME INVERTS THE RANKING RATHER THAN MERELY WEAKENING IT. With the target-encoded goal, mean rho collapses to 0.074 and 41.3% of episodes have NEGATIVE correlation: in four episodes in ten the incumbent cost ranks candidates backwards. That is the mechanism behind 76% vs 37.5%, and it is worse than a weaker signal -- it is a signal with the wrong sign nearly half the time. TWO INDEPENDENT COST DEFECTS, NOT ONE. (i) the latent distance is a weak proxy, rho ~0.25 at best; (ii) the predictor destroys even that, rho -0.018. Node 24 correctly rejected forward-model ACCURACY and named rank preservation; this refines it. A PERFECTLY rank-preserving predictor on this metric would cap near 76%, not the 93.5% ceiling. The remaining ~17.5 points are the metric's own weakness and no forward-model change recovers them. The complete measured chain, ceiling downward: true position cost 93.5% true latent cost, one frame (rho 0.25) 76.0% true latent cost, two frames (rho 0.074) 37.5% 41% sign-inverted predictor cost (rho -0.018) 35.0% Each step is a measured loss with a named cause. Next is unchanged in kind and now fully specified -- the design decision node 13 reserved for the user, with both halves quantified. A control-aware goal metric must lift rho from 0.25; a rank-preserving training signal must stop the predictor driving it to zero. They are SEPARABLE and should be attempted separately, since fixing only the predictor caps at 76% and fixing only the metric leaves the predictor destroying it. The one-frame goal is a prerequisite for either: free, removes a 41.3% sign inversion, necessary but not sufficient. Ledger-only. Verified: smoke 2 runs; ledger and ADR gates green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJJ8y1zo7WSqEmGNW8DQap
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Node 24's bounded check before any design decision. Node 22 showed the true latent cost yields 76%, so Spearman(true latent cost, true position cost) should be substantial.
It is not.
Result
150 episodes × 64 candidates × 3 seeds, true-env rollouts — no predictor involved:
The latent metric is weak even at its best
One-frame reaches only ρ = 0.25. It still yields 76% because the CEM needs only to find a good candidate among 64, and ρ=0.25 enriches the elite set enough — but the metric is not measuring what success measures.
The two-encoder frame inverts the ranking rather than weakening it
With the target-encoded goal, mean ρ collapses to 0.074 and 41.3% of episodes have negative correlation — in four episodes in ten the incumbent cost ranks candidates backwards.
That is the mechanism behind node 22's 76% vs 37.5%, and it is worse than "a weaker signal": it is a signal with the wrong sign nearly half the time.
Two independent cost defects, not one
Node 24 correctly rejected forward-model accuracy and named rank preservation. This refines it: a perfectly rank-preserving predictor on this metric would cap near 76%, not the 93.5% ceiling. The remaining ~17.5 points are the metric's own weakness and no forward-model change recovers them.
The complete measured chain
Each step is a measured loss with a named cause.
Next
Unchanged in kind and now fully specified — the design decision node 13 reserved for the user, with both halves quantified.
A control-aware goal metric must lift ρ from 0.25. A rank-preserving training signal must stop the predictor driving it to zero. They are separable and should be attempted separately, since fixing only the predictor caps at 76% and fixing only the metric leaves the predictor destroying it.
The one-frame goal is a prerequisite for either: free, removes a 41.3% sign inversion, and necessary but not sufficient per nodes 22 and 25.
Ledger-only. Verified: smoke 2 runs; ledger and ADR gates green.