Skip to content

docs(directive-detection): iteration spec for #374 H1 gate - #466

Merged
robotrocketscience merged 2 commits into
mainfrom
docs/issue-374-directive-detection-iteration
May 7, 2026
Merged

docs(directive-detection): iteration spec for #374 H1 gate#466
robotrocketscience merged 2 commits into
mainfrom
docs/issue-374-directive-detection-iteration

Conversation

@yoshi280

@yoshi280 yoshi280 commented May 7, 2026

Copy link
Copy Markdown
Collaborator

What

Adds docs/v2_directive_detection.md — an iteration spec for the deferred H1 directive-detection gate (issue #374). Cross-links from docs/V2_REENTRY_QUEUE.md to the new memo.

Does not touch src/ or tests/. Does not re-decide the deferral or the gate numbers (those stay owned by docs/v2_enforcement.md § H1). Does not claim H1 is unblocked.

Why

Issue #374 has bounced ten times across sessions because the queue surface tells autonomous workers "deferred + bench-gated" without saying which detector iteration to attempt next. Lab corpus v0.1 (285 rows) has been measured against the candidate detector at P=0.664 / R=0.937 — under the 0.80 precision floor, with 23 percentage points of recall headroom. The iteration is a precision problem, not a corpus problem; the next session that lands here needs a chosen path, not a re-derivation.

Changes

  • docs/v2_directive_detection.md (new, 103 lines) — failure-mode analysis (the FP cluster is imperative-grammar one-shot coding tasks), three iteration paths (intent-prefix filter / deontic-anchor partition / LLM classifier) with tradeoffs, recommendation (Path A), decision asks, public-tree vs lab-tree work split, unchanged re-entry trigger.
  • docs/V2_REENTRY_QUEUE.md — adds an Iteration spec line to the H1 row pointing at the new memo.

Out of scope

  • Detector source change. The memo proposes Path A but does not implement it; that lands in a separate feat(directive_detection): PR after the decision asks land.
  • Corpus changes. Lab-side per directory-of-origin rules.
  • Re-deciding the gate (P≥0.80 ∧ R≥0.60 ∧ n≥200). Owned by docs/v2_enforcement.md § H1.

Verification

  • git diff github/main...HEAD clean against the discretion grep.
  • Two atomic signed commits; no co-authorship trailers.
  • Conventional-commit prefixes: docs(directive-detection): and docs(reentry-queue):.

Closes / refs

Refs #374, #199 (umbrella). Does not close #374 — that requires a passing bench-gate run, which this PR does not produce.

Summary by Sourcery

Add an iteration spec for the deferred directive-detection gate and cross-link it from the v2 re-entry queue.

Documentation:

Summary by CodeRabbit

  • Documentation
    • Updated documentation for directive detection enforcement specifications and re-entry iteration plans
    • Clarified feature shipping status and deferral decisions for v2.0 release

Successor memo to docs/v2_enforcement.md § H1. Does not re-decide the
deferral or the gate; documents three concrete iteration paths to raise
detector precision from the measured 0.664 to the >=0.80 floor while
keeping recall above 0.60.

Recommends Path A (intent-prefix filter on imperative coding-task
verbs at the sentence head). Path B (deontic-anchor partition) and
Path C (LLM classifier) are listed with tradeoffs; Path C is flagged
non-deterministic and out of scope without an explicit ratification.

Memo also defines the public-tree vs lab-tree work split so #374 stops
bouncing in the work queue: detector source + public-CI sanity tests
ship from this repo; corpus authoring and the bench-gate run stay
lab-side per directory-of-origin rules. Closing #374 still requires a
lab-side run demonstrating P>=0.80 ^ R>=0.60 ^ n>=200.

Refs #374, #199 (umbrella), PR #377 (harness).
Adds the 'Iteration spec' line to the directive-detection (#374) row
pointing at docs/v2_directive_detection.md. The base spec link
(v2_enforcement.md § H1) stays canonical for the deferral decision
and the gate numbers; the new link is for the path-to-clearing.

Refs #374.
@sourcery-ai

sourcery-ai Bot commented May 7, 2026

Copy link
Copy Markdown

Reviewer's Guide

Adds a new directive-detection iteration spec memo and wires it into the v2 re-entry queue, defining concrete iteration paths (and a recommended one) for clearing the H1 directive-detection gate without changing any code or tests.

File-Level Changes

Change Details Files
Document the iteration plan for improving the directive detector to clear the H1 gate, including analysis, candidate paths, and decision asks.
  • Introduce a new memo describing current detector performance (P=0.664 / R=0.937) and clarifying that precision is the primary iteration target.
  • Provide failure-mode analysis showing false positives clustered on imperative one-shot coding tasks and noting minimal recall concerns.
  • Define three concrete iteration paths (intent-prefix filter, deontic-anchor partition, LLM classifier), with tradeoffs, determinism implications, and an explicit recommendation of Path A.
  • Spell out decision asks for maintainers (confirming path choice, verb anchors, connective list, and lab corpus merge) and the public-tree vs lab-tree work split.
  • Restate the H1 re-entry trigger and explicitly keep the gate deferral, clarifying what remains out of scope for this PR.
docs/v2_directive_detection.md
Connect the new directive-detection iteration spec into the v2 re-entry queue documentation.
  • Add an "Iteration spec" line under the H1 directive-detection row pointing to the new memo.
  • Annotate the iteration spec link with the candidate detector’s measured P/R and clarify that it describes paths to clear the gate.
docs/V2_REENTRY_QUEUE.md

Assessment against linked issues

Issue Objective Addressed Explanation
#374 Implement directive detection and associated TODO lifecycle (detect_directive, process_directive, detect_repetition/check_escalation) that auto-creates TODO-tagged beliefs from imperative directives in user prompts, per docs/v2_enforcement.md § H1, once the gate is met. The PR only adds an iteration-spec memo (docs/v2_directive_detection.md) and a cross-link from docs/V2_REENTRY_QUEUE.md. It explicitly does not modify src/ or tests/, does not change the detector implementation, and does not add process_directive, TODO lifecycle, repetition counters, or escalation logic.
#374 Meet and document the benchmark gate for directive detection (≥80% precision and ≥60% recall on a labeled sample of at least 200 coding prompts) and publish the evaluation sample in the repository (e.g., tests/fixtures/directive_corpus.jsonl) as specified. The PR documents existing benchmark results (P=0.664 / R=0.937 on a lab corpus) and proposes iteration paths but does not improve the detector, rerun the gate, or achieve the required precision threshold. It also does not add any corpus file under tests/fixtures/ or similar; corpus work is explicitly deferred to the lab tree.

Possibly linked issues


Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@yoshi280 yoshi280 added the attn:review Needs review (PR open, awaiting reviewer) label May 7, 2026
@github-actions github-actions Bot added the docs label May 7, 2026
@coderabbitai

coderabbitai Bot commented May 7, 2026

Copy link
Copy Markdown

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 91625aaf-5245-4e27-8977-e61b27314c47

📥 Commits

Reviewing files that changed from the base of the PR and between 7f880ca and 84499c4.

📒 Files selected for processing (2)
  • docs/V2_REENTRY_QUEUE.md
  • docs/v2_directive_detection.md

📝 Walkthrough

Walkthrough

This PR documents the directive detection enforcement H1 feature as deferred with bench-gate criteria, adding a re-entry queue row and a detailed iteration specification memo covering measured performance, identified failure modes, and three candidate implementation paths.

Changes

Directive Detection H1 Re-entry Documentation

Layer / File(s) Summary
Queue Registration
docs/V2_REENTRY_QUEUE.md
Adds a new row for directive detection (enforcement H1, issue #374) with a spec link. Clarifies H3 ships at v2.0 while H2 is dropped. Defines gate as ≥80% precision and ≥60% recall on 200 coding prompts, with deferral rationale focused on false-positive risk on imperative-grammar tasks.
Iteration Specification
docs/v2_directive_detection.md
New memo documenting current deferred status, lab-corpus v0.1 performance (P=0.664, R=0.937), and identified false-positive failure mode on coding tasks. Proposes three iteration paths (intent-prefix filter recommended), lists decision asks and lab-side actions, and defines public-tree vs lab-tree work split and unchanged re-entry trigger criteria.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

  • robotrocketscience/aelfrice#375: Updates the H1 row in docs/V2_REENTRY_QUEUE.md to point to issue #374, which this PR directly references and expands.
  • robotrocketscience/aelfrice#377: Shipped the regex-based directive-detection harness and measured bench-gate results (P=0.664, R=0.937) explicitly cited in this PR's iteration spec.

Suggested labels

docs, attn:review

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and specifically summarizes the main change: adding an iteration spec documentation for the deferred directive-detection gate (#374 H1), which is the primary objective of this PR.
Description check ✅ Passed The description comprehensively covers the PR's purpose, changes, scope boundaries, and verification steps, following the template's key sections: Summary, Linked issues, Type of change (docs), and Notes for reviewer.
Linked Issues check ✅ Passed The PR fulfills the objectives from issue #374 by providing the promised iteration spec with failure-mode analysis, multiple iteration paths, and decision asks, enabling informed work on the deferred directive-detection gate.
Out of Scope Changes check ✅ Passed All changes remain in scope: the new docs/v2_directive_detection.md and the docs/V2_REENTRY_QUEUE.md cross-link directly address issue #374's need for an iteration spec, with no detector implementation or out-of-scope modifications present.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch docs/issue-374-directive-detection-iteration

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've left some high level feedback:

  • Consider factoring the intent-prefix verbs and deontic/rule-marker lists into a single normative source (or explicitly referencing where they will live in code) so the eventual directive_detector.py implementation doesn’t silently diverge from the spec.
  • The decision asks around head-position anchors and connective lists are quite detailed; you might tighten them into a short, definitive table or checklist so future implementers can see the exact default configuration at a glance without rereading the narrative.
  • Since Path C is explicitly out of scope for now due to determinism, you could add a short note on what conditions would justify revisiting it, to avoid future ambiguity about when it becomes a realistic option.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- Consider factoring the intent-prefix verbs and deontic/rule-marker lists into a single normative source (or explicitly referencing where they will live in code) so the eventual `directive_detector.py` implementation doesn’t silently diverge from the spec.
- The decision asks around head-position anchors and connective lists are quite detailed; you might tighten them into a short, definitive table or checklist so future implementers can see the exact default configuration at a glance without rereading the narrative.
- Since Path C is explicitly out of scope for now due to determinism, you could add a short note on what conditions would justify revisiting it, to avoid future ambiguity about when it becomes a realistic option.

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

@yoshi280

yoshi280 commented May 7, 2026

Copy link
Copy Markdown
Collaborator Author

[claim:review:Gylf:2026-05-07T18:46:24Z]

@yoshi280

yoshi280 commented May 7, 2026

Copy link
Copy Markdown
Collaborator Author

[claim:review:Toug:2026-05-07T18:46:38Z]

@yoshi280

yoshi280 commented May 7, 2026

Copy link
Copy Markdown
Collaborator Author

[claim:review:Kulili:2026-05-07T18:46:41Z]

@yoshi280

yoshi280 commented May 7, 2026

Copy link
Copy Markdown
Collaborator Author

[release:review:Toug:2026-05-07T18:46:43Z]

@yoshi280

yoshi280 commented May 7, 2026

Copy link
Copy Markdown
Collaborator Author

[release:review:Kulili:2026-05-07T18:46:46Z]

@robotrocketscience
robotrocketscience merged commit 84499c4 into main May 7, 2026
22 of 29 checks passed
@robotrocketscience
robotrocketscience deleted the docs/issue-374-directive-detection-iteration branch May 7, 2026 18:47
@yoshi280

yoshi280 commented May 7, 2026

Copy link
Copy Markdown
Collaborator Author

[release:review:Gylf:2026-05-07T18:47:59Z]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

attn:review Needs review (PR open, awaiting reviewer) docs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(hook): directive detection — #199 H1 split (deferred, bench-gated)

2 participants