[hand-off] experimentalist: Insight-suite holdout, Pareto integration, and two selection fixes

[aleckhoury]

Read this first: scope change

This branch is a verified prototype being handed to the Experimentalist team, not a merge candidate as it stands.

It was written by the Eval Author side while chasing an Eval Author problem, and it ended up ~2,030 lines deep in loop.py, models.py, terminator.py, and analyzer.py. That runs against the direction plugins/nemo-eval-author/tests/test_plugin_boundary.py already declares — "Eval Author is meant to end up standalone, with Experimentalist depending on it and not the other way around."

We are re-scoping to our own lane: the Eval Author will deliver a well-formed dataset of tasks, verifiers, and metrics derived from an Insight's production traces, and the Experimentalist team decides how to consume it. Concretely, the Eval Author will no longer split the suite into halves and will no longer write its metric into the optimizer's train/validation datasets. Each task carries its source_trace_ref, so a consumer can split by trace without leakage on whatever policy it prefers.

The Eval Author half of this diff is therefore superseded. What remains valuable here is the Experimentalist-side design and, especially, the two bugs below.

Take this regardless: _select_survivors discards trial detail

This one is independent of Insight suites entirely and affects plain train/validation runs today.

_select_survivors ranked on slim() copies and then returned those copies. Survivors carried into later rounds therefore lost every *_reward_details field, which starved the analyzer's trial-level diagnosis from round 2 onward and blanked the detail in metadata.json. The fix keeps ranking on slimmed copies (per-trial detail does not belong in an LLM strategy's context) but maps winners back to the caller's full objects.

Ranking ignored the held-out half

_has_converged and EvolutionTree.get_best() ranked on val_reward alone, so a run improving only on held-out evidence looked stagnant and was declared converged. Addressed by _merge_selection_rewards / node_selection_rewards, so survivor selection, convergence, and winner choice all rank on the same axes. This part is Insight-specific and only matters if you adopt a held-out dimension.

Design: held-out reward as prefixed Pareto dimensions

Validation-half metrics enter selection namespaced under insight/ so they add their own axes instead of colliding with same-named validation keys, with union-based zero-fill because _dominates treats candidates with differing key sets as incomparable — which would otherwise leave a candidate missing a score silently undominated.

Empirical evidence from the e2e run

A full 2-round run on the tau3-nooa-agent example. Replaying the shipped selection_rewards and pareto_front over the persisted candidates:

agent-0: {reward: 1.0, insight/reward: 1.0, ...}
agent-1: {reward: 1.0, insight/reward: 0.5, ...}
agent-2: {reward: 1.0, insight/reward: 1.0, ...}

front WITHOUT insight dims : [agent-0, agent-1, agent-2]
front WITH    insight dims : [agent-0, agent-2]

agent-1 ties on validation and would have survived; failing half the held-out tasks is the only thing that removed it. The run's trajectory agrees — round 2's children descend from the survivors and the winner was agent-2.

Honest caveats on that evidence:

• The Insight had 2 traces, so the held-out half was a single task run twice. The mechanism is exercised; the statistical weight behind that one demotion is thin.
• The authored LLM-judge metric never actually ran. It reads TAU2_NL_ASSERTIONS_MODEL, which defaults to an unqualified gpt-5.2 and 403'd on all 50 trials. The judge's except Exception: score = 0.0 fallback turned every failure into a valid-looking 0.0, so that axis was a dead constant and only insight/reward carried signal. Use openai/openai/gpt-5.2. The fail-closed metric pattern is being fixed on the Eval Author side.

Verification at time of writing

• plugins/nemo-eval-author: 67 passed, 2 skipped
• plugins/nemo-experimentalist: 608 passed
• ruff check / ruff format --check clean; ty check clean in nemo-eval-author

If you pick this up

• Cherry-pick the _select_survivors fix on its own — it is a live bug with no dependency on this design
• Rebase onto main (cut before refactor: make analyst/experimentalist/eval-author CLI agents-only (ASE-702) #1052; ~34 commits behind, and the earlier CLI commits here already landed as 93f0522b5f)
• Decide whether you want a held-out dimension at all now that the Eval Author ships one flat suite plus per-task trace provenance
• If yes, decide where the split belongs on your side, and re-run with a working judge and an Insight carrying more traces