Skip to content

bench(worldmodel): the goal metric is also a defect — rho 0.25 at its best - #66

Merged
aurascoper merged 1 commit into
mainfrom
bench/latent-vs-position-cost
Aug 3, 2026
Merged

bench(worldmodel): the goal metric is also a defect — rho 0.25 at its best#66
aurascoper merged 1 commit into
mainfrom
bench/latent-vs-position-cost

Conversation

@aurascoper

Copy link
Copy Markdown
Owner

Node 24's bounded check before any design decision. Node 22 showed the true latent cost yields 76%, so Spearman(true latent cost, true position cost) should be substantial.

It is not.

Result

150 episodes × 64 candidates × 3 seeds, true-env rollouts — no predictor involved:

goal latent median ρ mean ρ ρ > 0.5 ρ < 0 node 22 success
one-frame (online goal) 0.2514 0.2735 22.4% 17.8% 76.0%
incumbent (target goal) 0.1570 0.0744 26.7% 41.3% 37.5%
predictor cost (node 24) −0.018 35.0%

The latent metric is weak even at its best

One-frame reaches only ρ = 0.25. It still yields 76% because the CEM needs only to find a good candidate among 64, and ρ=0.25 enriches the elite set enough — but the metric is not measuring what success measures.

The two-encoder frame inverts the ranking rather than weakening it

With the target-encoded goal, mean ρ collapses to 0.074 and 41.3% of episodes have negative correlation — in four episodes in ten the incumbent cost ranks candidates backwards.

That is the mechanism behind node 22's 76% vs 37.5%, and it is worse than "a weaker signal": it is a signal with the wrong sign nearly half the time.

Two independent cost defects, not one

  1. the latent distance is a weak proxy — ρ ≈ 0.25 at best
  2. the predictor destroys even that — ρ = −0.018

Node 24 correctly rejected forward-model accuracy and named rank preservation. This refines it: a perfectly rank-preserving predictor on this metric would cap near 76%, not the 93.5% ceiling. The remaining ~17.5 points are the metric's own weakness and no forward-model change recovers them.

The complete measured chain

true position cost                        93.5%
true latent cost, one frame  (rho  0.25)  76.0%
true latent cost, two frames (rho  0.074) 37.5%   <- 41% sign-inverted
predictor cost               (rho -0.018) 35.0%

Each step is a measured loss with a named cause.

Next

Unchanged in kind and now fully specified — the design decision node 13 reserved for the user, with both halves quantified.

A control-aware goal metric must lift ρ from 0.25. A rank-preserving training signal must stop the predictor driving it to zero. They are separable and should be attempted separately, since fixing only the predictor caps at 76% and fixing only the metric leaves the predictor destroying it.

The one-frame goal is a prerequisite for either: free, removes a 41.3% sign inversion, and necessary but not sufficient per nodes 22 and 25.

Ledger-only. Verified: smoke 2 runs; ledger and ADR gates green.

… best

Ledger node 25. Node 24's bounded check before any design decision: node 22
showed the true latent cost yields 76%, so Spearman(true latent cost, true
position cost) should be substantial. It is not.

150 episodes x 64 candidates x 3 seeds, TRUE-env rollouts, no predictor:

  goal latent               median rho   mean    rho<0   node 22 success
  one-frame (online goal)       0.2514   0.2735   17.8%            76.0%
  incumbent (target goal)       0.1570   0.0744   41.3%            37.5%
  predictor cost (node 24)      -0.018      —        —             35.0%

THE LATENT METRIC IS WEAK EVEN AT ITS BEST. One-frame reaches only rho = 0.25.
It still yields 76% because the CEM needs only to find a good candidate among 64
and rho 0.25 enriches the elite set -- but the metric is not measuring what
success measures.

THE TWO-ENCODER FRAME INVERTS THE RANKING RATHER THAN MERELY WEAKENING IT. With
the target-encoded goal, mean rho collapses to 0.074 and 41.3% of episodes have
NEGATIVE correlation: in four episodes in ten the incumbent cost ranks candidates
backwards. That is the mechanism behind 76% vs 37.5%, and it is worse than a
weaker signal -- it is a signal with the wrong sign nearly half the time.

TWO INDEPENDENT COST DEFECTS, NOT ONE. (i) the latent distance is a weak proxy,
rho ~0.25 at best; (ii) the predictor destroys even that, rho -0.018. Node 24
correctly rejected forward-model ACCURACY and named rank preservation; this
refines it. A PERFECTLY rank-preserving predictor on this metric would cap near
76%, not the 93.5% ceiling. The remaining ~17.5 points are the metric's own
weakness and no forward-model change recovers them.

The complete measured chain, ceiling downward:
  true position cost                          93.5%
  true latent cost, one frame  (rho  0.25)    76.0%
  true latent cost, two frames (rho  0.074)   37.5%   41% sign-inverted
  predictor cost               (rho -0.018)   35.0%

Each step is a measured loss with a named cause.

Next is unchanged in kind and now fully specified -- the design decision node 13
reserved for the user, with both halves quantified. A control-aware goal metric
must lift rho from 0.25; a rank-preserving training signal must stop the
predictor driving it to zero. They are SEPARABLE and should be attempted
separately, since fixing only the predictor caps at 76% and fixing only the
metric leaves the predictor destroying it. The one-frame goal is a prerequisite
for either: free, removes a 41.3% sign inversion, necessary but not sufficient.

Ledger-only. Verified: smoke 2 runs; ledger and ADR gates green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DJJ8y1zo7WSqEmGNW8DQap
@aurascoper
aurascoper merged commit 19447e7 into main Aug 3, 2026
2 checks passed
@aurascoper
aurascoper deleted the bench/latent-vs-position-cost branch August 3, 2026 14:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant