feat(research): make research work reach the Brain — panels, rounds, domain studies, v2 identity - #13
Merged
Merged
Conversation
…able A UX-review panel scores 4-5 independent arms against one rubric and records runs and evaluations for every one -- and none of it was reachable by pattern mining, because nothing registered a research subject. The completion-event exporter resolves identity through `research_subject_experiments -> research_subjects` keyed on experiment_id; with no subject row, every run the panel records is identity-less and the panel's evidence is invisible no matter how many arms it scores. `research_subjects` held 0 rows in every database including months-old backups: never written, not deleted. `register_panel_subject()` registers exactly one subject per panel. Nothing is invented: the spec is the panel's real rubric prompt -- byte-identical for every arm, which is what makes one spec hash per panel correct -- and the arm set is the real evaluator list. The rubric prompt is now built once and reused for both the per-arm prompt files and the subject, instead of being rebuilt per arm. A registration failure prints to stderr and returns None rather than raising, so it cannot kill a panel that costs real agent time -- but it is never swallowed silently. A silent failure here is indistinguishable from a panel that was never meant to be mined, which is precisely how this evidence went missing. Tests: a panel registers one subject whose spec hash is the hash of the REAL rubric prompt and whose arm set retains every arm; two panels on the same app with different rubrics stay two subjects; a malformed bundle reports rather than swallows. Break/revert demonstrated twice -- collapsing the arm set to one arm forges independence and fails, and skipping the write fails the exp_id join. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…he router Route weights learn from `outcomes`, and a UX-review panel wrote none. 131 runs and 105 evaluations existed while `ux_review` appeared ZERO times in `route_weights` and had ZERO rows in `outcomes` -- so 26 panels of head-to-head agent comparison, the evidence that would say which agent reviews best, never reached the router. Meanwhile 32 of 66 routing pairs sit at n_obs=0, running on hand-set priors. An evaluation is a SCORE. Only an outcome is learnable. `panel_arm_outcome()` derives one from two signals the panel already computes, neither of which is a judgement call: * Verdict: the arm either produced parseable rubric JSON with dimension scores, or it did not. * Durability: contribution to corroborated consensus -- did any of this arm's findings survive `aggregate_accepted_findings`? An arm whose every finding was rejected as a non-finding added noise, not evidence. The clean-app case is deliberately NOT penalised. When the panel corroborated nothing at all, every parsing arm is credited `held`. Marking an arm down for reporting no findings on a sound app would train arms to invent findings -- the exact gaming this label exists to prevent, and the more dangerous failure than the missing data. A failed outcome write prints to stderr rather than raising: a learner write must never lose a completed panel that cost real agent time. Never silent. Tests: unparseable and unscored arms fail; a corroborated finding is durable; an uncorroborated one is not; a clean app does not penalise a silent arm; and every emitted durability value is one the learner actually recognises -- a label outside DURABLE_STATUSES/TERMINAL_FAILURE_DURABILITY is the same as no label. Break/revert: penalising the clean-app case fails, and emitting an unrecognised durability string fails. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`evaluations_v2` held 0 rows while `evaluations` accumulated 2,540 across 246
experiments, every one stamped `{"identity": "agent_parent_projection"}` -- the
averaged legacy projection §2 warns against. The cause was not missing code.
`exp_abcd.prepare_arms` already persists the exact v2 manifest, and
`_normalize_arm_members` already mints arm/member/profile identity, and
`record_evaluation_v2` already fires -- but only `if not impl["legacy"]`.
`experiment_members()` returns legacy members whenever meta has `agents` and no
`members`, and the legacy `exp_abcd.prepare` writes exactly that. So the guard
never opened. The v2 path was reachable only through `strategy_experiment.py`,
which is deliberately gated, while the production launcher used the legacy call.
This is a ROUTING change, not new machinery: build v2 arm descriptors from the
plan's agent list and call `prepare_arms`. One arm per agent, which is precisely
what the legacy path meant -- each agent is its own experimental condition.
`profile_id` is carried through when the plan knows it and left None when it does
not: an honest unknown, never a fabricated identity.
Selftest asserts the whole chain, not just the helper: research_v2_arms ->
_normalize_arm_members -> experiment_members must come back NON-legacy, because
that boolean is the only thing standing between this fix and another 2,540 rows
of parent projection. Duplicate agents collapse; an empty agent list yields no
arms. Break/revert: falling back to legacy `prepare` fails, and fabricating an
arm from an empty list fails.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…subjects Two capture gaps, both of which made real research unaddressable rather than mis-scored. DOMAIN RESEARCH (`domain/<slug>`). Research identity was repo-scoped end to end: `subject_identity` accepts any target string, but every producer derives its target from a repo/issue, so a topic study, a tool comparison or a technique to learn had no target shape and could not be registered at all. `runs.target` holds only `stranske/*`, `offload:<path>` and `triage:N-items` -- no domain-research target of any kind. The payoff here is RETRIEVAL before learning: a registered subject makes prior research addressable by a later session, which is what "you have access to the research project showing how to use Luminar, don't you?" was asking for and not getting. `record_domain_research` deliberately writes NO score and NO outcome. There is no un-gameable success label for "was this research any good", and inventing one would corrupt the learner far more than the missing record does. MULTI-AGENT ROUNDS (`<area>:<kind>:<date>`). Audits fan work out to several agents -- measured: 1,488 offload runs across 6 agents, up to 5 distinct agents on one target, including a 4-arm repo audit. That is comparable evidence, but 0 of those 1,488 runs carried an experiment_id, so each agent was an unrelated run against an ephemeral temp path and nothing bound them into a round. `record_research_round` registers the round as ONE subject with the real arm set, mirroring the shape UX panels already use, and `offload(research_round=...)` stamps the round on each run so the arms become attributable. An arm set is never padded. A round worked by one agent is one arm; a round with no real arm raises rather than inventing one, because a forged arm set manufactures independence the evidence does not have. Break/revert covers both: padding a solo round fails, and dropping the offload binding leaves runs unattributable. NOT DONE, on purpose: inheriting durability from the issues an audit filed. That needs the finding->issue linkage the skill knows at filing time. Until it lands, audit rounds are visible and comparable but carry no durability label. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 55 minutes Limit details: You’ve used the included review currently available. Your 67 included PR review attempts over the past 7 days set your current allowance at 1 review per hour. Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab. How can I continue?Wait for the limit to reset, then comment An organization admin can change what happens after included review limits in Billing. How do review limits work?CodeRabbit enforces per-developer PR review limits within each organization. For paid Pro and Pro+ reviews, CodeRabbit uses a developer's included PR review attempts over the past 7 days to set the current hourly allowance. At typical activity levels, the full plan allowance applies. Higher sustained activity can lower the allowance until earlier attempts leave the 7-day window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (5)
Comment |
stranske
added a commit
that referenced
this pull request
Aug 22, 2026
…5 -> 353) (#15) The floor committed in #12 was measured on a branch cut before #13 (research panels, rounds, domain studies) merged, so main's recorded floor sat 8 tests BELOW what main actually collects. A floor below reality fails in the permissive direction: those 8 tests could have stopped being collected and the check would still have passed, which is exactly the hole .verify-floor.json exists to close — a silent drop in tests-collected reads identically to tests passing. Measure the floor on the merge result, not on the branch. Ceilings unchanged (24/7/2). verify.py: 353 passed, 0 failed, 0/24 skipped, 81 selftests, 5/5 gates. Co-authored-by: Tim Stranske <tim@stranskemo.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four independent lines, each revertable on its own. All fix one class of defect: research happens, and the learner never sees it.
Measured problem
route_weightsrowsagent_parent_projection;evaluations_v2= 0Meanwhile 32 of 66
route_weightspairs sit atn_obs = 0— the router runs on hand-set priors for half its decisions while this evidence goes uncounted.The four lines
A —
ux_reviewper-arm outcomes. An evaluation is a score; only an outcome is learnable.panel_arm_outcome()uses two signals the panel already computes: did the arm produce parseable rubric JSON with dimension scores, and did any of its findings surviveaggregate_accepted_findings. The clean-app case is deliberately not penalised — marking an arm down for finding nothing on a sound app would train arms to invent findings, which is the worse failure.D — route research launches through
prepare_arms. Not new machinery.prepare_arms,_normalize_arm_membersandrecord_evaluation_v2all existed, butrecord_evaluation_v2is guarded onif not impl["legacy"], and the legacypreparewritesmeta["agents"]with nomembers, so the guard never opened. The v2 path was reachable only through the deliberately-gatedstrategy_experiment.py. The selftest asserts the whole chain returnslegacy: False, because that boolean is the only thing between this fix and another 2,540 rows of parent projection.C —
domain/<slug>namespace. Identity was repo-scoped end to end, so non-repo research could not be registered at all. Writes no score and no outcome: there is no un-gameable success label for "was this research good", and inventing one would corrupt the learner worse than the missing record does. The payoff is retrieval — making prior research addressable by a later session.B —
<area>:<kind>:<date>multi-agent rounds. Binds the fanned-out agents of one audit or study into a single subject with the real arm set, andoffload(research_round=...)stamps the round on each run so the arms become attributable.Arm sets are never padded anywhere: a solo round is one arm, and a round with no real arm raises rather than inventing one.
Verification
python3 verify.pyon the rebased tree: 351 passed, 0 failed, 351 collected (floor 330), 0/26 skipped, 81 of 81 selftests, activation audit 41 of 41, set coverage ok, admission ok, ledger valid — 5 of 5 gates green.Break→revert demonstrated for every line, including the two that matter most: penalising the clean-app case fails, and padding a solo round into false independence fails.
Rebase note
Rebased onto
5b04bb3(#10). One real conflict indispatcher.py, where #10 added theORCH_OFFLOAD_DISABLEDkill switch immediately after theoffloaddocstring this branch extends. Both were kept, and theresearch_roundparameter and itsexperiment_idbinding were re-verified after the replay.Explicit non-goals
🤖 Generated with Claude Code