test(bench-gate): sweep K=200 partitions in the first-token guard, and land corpus v0.2

[robotrocketscience]

Closes #1349.

The review on #1343 was right, and the failure it predicted was one I was about to walk into myself.

The guard could be flipped green by re-partitioning

test_directive_corpus_defeats_a_first_token_baseline asserted on a single sha1(id) % 100 < 60 split. Precision there is computed over 34 positive predictions, and the Wilson 95% interval on that is [0.7704, 0.9695] — the lower bound sits below the 0.80 gate the guard asserts the baseline clears. Four more false positives in the held-out half takes P to 0.795 and the guard passes on an unchanged corpus.

That matters because of what the guard is for. Its whole job is to prevent a false clearance of the H1 gate, and as written it could produce one: corpus v0.2 adds and edits rows, every edited id lands in a different bucket, and a partition that happens to place four more false positives held-out turns the guard green while the head-word correlation is untouched. Silent, and it reads as "corpus fixed".

Swept

The guard now sweeps K=200 salted partitions (sha1(salt + id)) and fails if the baseline clears the gate on any of them. Failing on any partition is the conservative direction — a false alarm costs a corpus inspection, a false clearance blesses an overfit detector.

Measured across K=200:

corpus	partitions where the baseline clears	precision min / median / max	pooled precision
v0.1	196 / 200 (98.0%)	0.833 / 0.941 / 1.000	0.9381, n=5,978
v0.1 + v0.2	0 / 200	0.480 / 0.613 / 0.754	0.6108, n=12,431

The populations are far enough apart that the exact K is not delicate. Runtime is ~0.2s.

This also settles the review's other point. The P=0.912 / R=0.795 I published in #1341 was one draw from that 0.833–1.000 distribution, not a property of the corpus. Both memos now carry the swept figure with its n. A memo whose thesis is that a number carried forward without its derivation does real damage should not have shipped a bare point estimate; that was my error, and it is corrected rather than defended.

The conclusion itself is unchanged and was never at risk — it rests on the partition-independent statistics (7 of 114 opening words appear in both classes, 87.9% of rows decided by their first token, no directive row opens with a one-shot task verb).

Corpus v0.2 landed

Operator funded ~220 rows / ~110 minimal pairs. v0_2.jsonl adds 225 rows lab-side across six head-word buckets — durable rules opening with task verbs (check, run, review, update, remove, write, add), absent from v0.1 entirely, and one-shot requests opening with policy verbs (always, never, avoid, use, prefer, ensure).

	v0.1	v0.1 + v0.2
rows whose first word is class-ambiguous	12.1%	71.6%
baseline clears the gate on	196/200 partitions	0/200
validity guard	red	green
real detector	P=0.706 / R=0.937	P=0.665 / R=0.636

H1 stays deferred — but now on a number that means what v2_enforcement.md § H1 intends. The detector genuinely does not clear P≥0.80 on a corpus that cannot be solved by memorising opening vocabulary.

And the shape of the problem has changed. Recall was 0.937 on v0.1 with 34 points of headroom, which is what licensed "trade recall for precision freely" — the premise every iteration path in the memo was written on. Against the union it is 0.636, barely above the 0.60 floor, because v0.2's hard positives are durable rules the 29-verb regex never fires on. Precision work now has to pay for its recall. That is exactly the constraint § Path B's risk note anticipated and could not previously measure, and the § Recommendation is rewritten around it.

On the labels

Authored per bucket, then reproduced by two independent passes with labels stripped, row ids replaced by opaque hashes, and rows reshuffled: 225/225 agreement, κ=1.000, zero unclear. I ran that twice — the first pass leaked pair structure through the row ids, so the agreement it produced was inflated and I re-ran it anonymised.

Two caveats stated plainly rather than buried: both passes are the same model family, and minimal pairs stay recognisable by topic however the ids are scrambled. This bounds label noise; it is not proof of independence. The authors also self-reported a residual shortcut — one-shot rows still carry concrete referents (a file, a PR number, a version) more often than durable rows — which is the next thing a classifier would find and what v0.3 should be sized against.

Verification

# public CI posture (corpus unset)
$ pytest -q tests/bench_gate/test_directive_detection.py
2 skipped

# lab, v0.1 only — guard correctly red
$ AELFRICE_CORPUS_ROOT=<v0.1> pytest -q ...::test_directive_corpus_defeats_a_first_token_baseline
FAILED - a first-token-only classifier clears the H1 gate on 196/200 salted
         partitions of this corpus (n=285). Worst partition: P=1.000, R=0.633

# lab, v0.1 + v0.2 — guard green, gate red on a real number
$ AELFRICE_CORPUS_ROOT=<union> pytest -q tests/bench_gate/test_directive_detection.py
1 failed, 1 passed
FAILED test_directive_detection_gate - precision 0.665 below 0.8 gate
       (TP=131, FP=66, FN=75, TN=238, n=510)

No src/ change. Detector behaviour is byte-identical.

Note on scope

Two surfaces in one PR — the guard fix (#1349) and the doc updates recording v0.2. They are coupled: the K-sweep is the thing that verifies v0.2 defeats the baseline robustly rather than on one lucky draw, and shipping the guard while the memo still said "corpus v0.2 is the blocking item" would leave a stale doc of exactly the kind this line of work exists to correct. Split into three atomic commits.

The corpus file itself is committed lab-side and does not appear here. The lab gitea remote was unreachable from this session, so that commit is local — it needs a push before anyone else can reproduce the union numbers.

Summary by Sourcery

Strengthen the directive-detection benchmark guard by sweeping multiple salted corpus partitions and update documentation and changelog to reflect corpus v0.2, the new guard behaviour, and revised detector metrics and recommendations.

Bug Fixes:

• Make the first-token validity guard robust by sweeping K=200 salted partitions and failing if the baseline clears the gate on any of them, preventing re-partitioning from silently flipping the guard green on an unchanged corpus.

Enhancements:

• Extend the first-token baseline test to report partition-level statistics across a sweep of salted splits, improving the stability and interpretability of the gate validity check.

Documentation:

• Revise the directive-detection design memo to document the partition sweep, corpus v0.2 landing, updated baseline and detector performance on the union corpus, and a new recommendation prioritising Path B with recall as the binding constraint.
• Update the v4 changelog and v2 re-entry queue to record the guard fix, corpus v0.2’s impact on the baseline and detector metrics, and that H1 remains deferred on a now-valid measurement.

[sourcery-ai]

Sorry @robotrocketscience, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

[coderabbitai]

Warning

Review limit reached

@robotrocketscience, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 2 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 107f7268-b505-479d-a70c-0b91d6ff0dce

📥 Commits

Reviewing files that changed from the base of the PR and between 42e8a31 and 7026eb5.

📒 Files selected for processing (4)

• CHANGELOG/v4.md
• docs/design/V2_REENTRY_QUEUE.md
• docs/design/v2_directive_detection.md
• tests/bench_gate/test_directive_detection.py

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[sourcery-ai]

Reviewer's Guide

This PR hardens the first-token baseline guard by sweeping multiple salted train/test partitions instead of a single split, and updates design docs and changelog to record corpus v0.2, the swept statistics, and the resulting change in directive-detector performance and iteration strategy. No src/ runtime behavior changes; all code edits are in tests and documentation.

File-Level Changes

Change	Details	Files
Guard test now sweeps K=200 salted partitions of the corpus instead of asserting on a single deterministic partition.	Introduce PARTITION_SWEEP_K constant to control the number of partitions swept. Extend the bucket helper to accept an optional salt and compute sha1(salt + id) % 100. Refactor the guard logic into a score(salt) helper that builds train/held-out splits, constructs the head-word lookup table, and returns precision/recall and confusion counts. Compute scores across K salted partitions, track which partitions clear the gate, and identify the worst partition by precision. Change the assertion to fail if any partition clears the precision/recall gate, and update the failure message to report counts, worst partition stats, and overall corpus size.	tests/bench_gate/test_directive_detection.py
Design memo for directive detection is updated to describe the swept baseline experiment, corpus v0.2, and the new recommendation/iteration plan.	Clarify status text to state that corpus v0.2 makes the gate a valid measurement and that the deferral now rests on a real number. Replace single-partition baseline description with a K=200 sweep, including partition-level and pooled precision/recall statistics and their interpretation. Document corpus v0.2 composition (minimal pairs breaking head-word/class correlation) and its quantitative impact on baseline and detector metrics. Add notes on labeling protocol, caveats about independence, and the remaining shortcut to target in corpus v0.3. Update the recommendation section to pivot from corpus construction to Path B iteration against the union corpus, stressing recall as the binding constraint and laying out next steps.	docs/design/v2_directive_detection.md
Changelog entry records the partition-sweep guard fix and the landing of corpus v0.2, with the swept figures and unchanged detector behavior.	Add a Fixed bullet explaining why a single-partition guard was fragile and how K=200 salted partitions fix the issue. Summarize swept baseline precision behavior on v0.1 and note that the earlier P=0.912 figure was a single draw, now replaced by swept statistics. Describe corpus v0.2’s size and effect on head-word ambiguity, baseline gate clearing, guard status, and detector precision/recall, noting that src/ code did not change.	CHANGELOG/v4.md
Re-entry queue doc is updated to reflect corpus v0.2’s landing, the swept baseline results, and the detector’s new metrics on the union corpus.	Modify the directive-detection row to replace the single-partition baseline result with the 196/200 vs 0/200 partition comparison. Record the detector’s P=0.665 / R=0.636 performance on the union corpus and emphasize that H1 remains deferred on a valid number with recall as the main constraint. Reference the updated directive-detection memo’s Gate validity section for details.	docs/design/V2_REENTRY_QUEUE.md

Assessment against linked issues

Issue	Objective	Addressed	Explanation
#1349	Change the first-token validity guard to evaluate the baseline over K salted partitions (sha1(salt + id)) instead of a single partition, using a conservative fail-on-any-partition rule, with K chosen from a measured run and explained in the test docstring.	✅
#1349	Update documentation (v2_directive_detection memo and CHANGELOG) so that the reported baseline figures are based on the K-partition sweep, include n and a confidence interval, and clearly describe the swept guard behavior and the effect of corpus v0.2.	✅
#1349	Ensure the guard remains bench-gated (hidden from public CI) and that against corpus v0.1 alone the guard remains red, with v0.2 affecting only the union corpus and not changing detector code.	✅

Possibly linked issues

• test(bench-gate): the first-token guard asserts on one partition, so corpus v0.2 can flip it green by re-partitioning #1349: They are the same work: replacing the single-partition guard with a K-partition sweep, measuring K, and updating docs.

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[github-actions]

PR-size soft cap

This PR is over the advisory size threshold:

• 176 changed lines (limit: 200)
• 4 changed files (limit: 3)

Bigger PRs collide with more open work, which under the parallel-session workflow tends to produce repeated attn:merge-conflict cycles (see #602). When practical, split into smaller PRs that each touch a focused surface.

This is advisory only — nothing is blocked. If the size is intentional (large refactor, module removal, generated code), apply the size:override label and this comment will be removed on the next push.

[robotrocketscience]

[claim:review:Garsecg:2026-08-05T02:31:37Z]

[robotrocketscience]

Review — this closes #1349 properly. One defect in the failure path, fixed here; one property recorded.

I filed #1349, so treat my agreement with its premise as worth nothing and the
rest on its merits. The sweep is the right shape, the K=200 numbers answer AC2,
and correcting the P=0.912 I asked about in AC3 rather than defending it is
the right call. Landing corpus v0.2 in the same PR was not something #1349 asked
for and it is what makes this a fix rather than an instrument change.

The result I did not expect and think is the most important thing here: recall
against the union is 0.636, not 0.937. Every iteration path in that memo was
written on 34 points of recall headroom, and there are now 3.6. "Trade recall
for precision freely" was a premise of the whole document and it is gone. Good
that the § Recommendation was rewritten rather than left standing.

Fixed here — the failure message could name a partition that did not clear

worst = max(results, key=lambda t: t[0]) selects on precision alone. The gate
is joint, so the global max-precision partition can have recall under the floor
— it did not clear — and it was then printed as the evidence that something did:

results = [(P=0.95, R=0.10), (P=0.82, R=0.70)]      # gate 0.80 / 0.60
clearing partitions: [1]
old reported  P=0.95 R=0.10   -> cleared? False
new reported  P=0.82 R=0.70   -> cleared? True

On v0.1 the max-precision partition happens to clear (P=1.000, R=0.633), so the
message is correct today by coincidence rather than by construction — and this
message is the artefact a future session acts on when the guard fires.

Fixed in b8548f1a: report the highest-precision partition among those that
cleared, falling back to the global max only when nothing cleared, where the
message does not render. No unit test — the logic is inline in a bench_gated
function and pulling it out to make it testable is a larger change than the
defect justifies. The demonstration above is the verification.

Recorded, not changed — the rule is monotone in K

"Clears on any of K" with salts s0..s{K-1} means raising K only adds
candidates. The guard can become redder, never greener.

So the v0.1+v0.2 pass is a K=200 statement, and its margin — worst partition
0.754 against a 0.800 gate — is the maximum of K draws, not a property of the
distribution. The comment justifies K-insensitivity from the median
separation (0.941 vs 0.613), which is the right statistic for a pooled rule and
the wrong one for a max-based rule; the median tells you nothing about where the
tail lands at K=2000.

I have not changed K or the rule. "Fail on any" is the correct conservative
direction for a guard and I would keep it. But the note now sits next to the
constant, so that a session which raises K and sees the union go red reads it as
the tail being sampled further out, reports the new count, and does not quietly
lower K back.

If you want that closed rather than documented, the K-independent form is a
quantile — "fails if ≥1% of partitions clear" is stable as K grows where "any"
is not. That is a judgement call about how conservative the guard should be, and
it is yours; I did not want to trade away strictness on my own authority.

Rebase

Behind main — #1345 landed while this was open. Rebased. CHANGELOG conflicted
against that entry; resolved insert-only (git diff github/main -- CHANGELOG/v4.md | grep -c '^-[^-]' → 0), check_changelog_dupes.py clean.

On the labels

The anonymised re-run after finding the first pass leaked pair structure through
row ids is the right instinct, and stating that both passes share a model family
— so κ=1.000 bounds label noise rather than demonstrating independence — is the
correct reading of it rather than the flattering one. The self-reported residual
shortcut (concrete referents correlating with the one-shot class) is a real
lead; that is the next head-word correlation, one level up, and sizing v0.3
against it is right.

Public CI posture verified unchanged: pytest -q tests/bench_gate/test_directive_detection.py
→ 2 skipped. Labelling once checks settle.

[robotrocketscience]

[claim:review:Setr:2026-08-05T02:37:48Z]

[robotrocketscience]

[release:review:Setr:2026-08-05T02:37:54Z]

[github-actions]

merge-train: merged 7026eb5 → main via FF push.

[robotrocketscience]

[release:review:Garsecg:2026-08-05T02:43:48Z]