diff --git a/docs/V2_REENTRY_QUEUE.md b/docs/V2_REENTRY_QUEUE.md index b68903840..ad1ce9aed 100644 --- a/docs/V2_REENTRY_QUEUE.md +++ b/docs/V2_REENTRY_QUEUE.md @@ -28,6 +28,7 @@ Each row references its public spec memo and the GitHub issue that tracks re-ent ### Directive detection (enforcement H1) — issue #374 - **Spec:** [`docs/v2_enforcement.md`](v2_enforcement.md) § H1 +- **Iteration spec:** [`docs/v2_directive_detection.md`](v2_directive_detection.md) (paths to clear the gate, given the candidate detector's measured P=0.664 / R=0.937 against lab corpus v0.1). - **Disposition:** defer to v2.x. (H3 selective injection ships at v2.0; H2 compliance audit is dropped, not deferred.) - **Gate:** ≥80% precision and ≥60% recall on a labeled sample of 200 coding prompts. Below that bar, the 29-verb regex's false-positive rate (e.g., "I never push to main when I'm tired" misclassified as a directive) pollutes the belief graph faster than the captured directives help. - **Why deferred:** high false-positive risk on coding prompts; no published precision/recall numbers from the research line; ~150 LOC + TODO lifecycle is meaningful surface to ship without evidence. diff --git a/docs/v2_directive_detection.md b/docs/v2_directive_detection.md new file mode 100644 index 000000000..2283cfce0 --- /dev/null +++ b/docs/v2_directive_detection.md @@ -0,0 +1,103 @@ +# v2.x re-entry: directive detection (enforcement H1) + +Iteration spec for issue [#374](https://github.com/robotrocketscience/aelfrice/issues/374). Successor memo to [`v2_enforcement.md` § H1](v2_enforcement.md#h1-directive-detection--defer-to-v2x-with-benchmark-gate); does not re-decide the deferral or the gate, only the path to clearing it. + +Status: deferred. Harness shipped (PR [#377](https://github.com/robotrocketscience/aelfrice/pull/377)). Candidate detector measured at P=0.664 / R=0.937 against the lab corpus v0.1 (285 rows, ≥200 floor met). Below the P≥0.80 floor, so H1 stays deferred per spec; iteration target is precision, not recall. + +## What's being decided + +Which detector-iteration path to commit to before the next implementation attempt at #374. The harness is in place; what's missing is a chosen direction for raising precision from 0.664 to ≥0.80 without dropping recall below 0.60. This memo proposes three concrete paths, recommends one, and defines the public-tree vs lab-tree work split so the issue stops bouncing. + +## Substrate dependency + +None. Directive detection operates on prompt text only; no belief schema, posterior, or edge-type interaction. The detector is a pure function `str → bool`. + +## Failure-mode analysis (from the gate run on lab corpus v0.1) + +The 45 false positives that drag precision to 0.664 cluster as a single dominant pattern: **imperative-grammar one-shot coding tasks**. Examples documented publicly on the issue thread: + +- "Refactor X so it never blocks" +- "Add a test that ensures …" + +These are imperatives the user issues to the agent for the immediate session — task instructions, not durable rules to remember. The 29-verb regex correctly fires on `never` / `ensure`; the spec's three filters (wh-question, hedge, "I never X when Y" narration) do not catch this class because the surface form is a clean second-person imperative. + +The 6 false negatives are not the load-bearing problem — recall at 0.937 is comfortably above the 0.60 floor and has 23 percentage points of headroom for a more conservative detector. + +## Iteration paths + +### Path A: intent-prefix filter (recommended) + +Add a pre-filter that classifies the leading clause. Imperative coding-task prefixes (`refactor`, `add`, `implement`, `write`, `create`, `update`, `fix`, `make`, `build`, `remove`, `rename`, `extract`, `merge`, `split`, `move`, `delete`) issued as the sentence head — without any rule-marker connective ("so that", "as a rule", "from now on") elsewhere in the sentence — are treated as one-shot tasks and short-circuit to `False` regardless of downstream imperative-verb hits. + +- **Why this fits the failure mode:** the FP cluster all begin with a coding-task verb. The classification is a head-position lexical check, not semantic. +- **Precision impact:** removes the ~45 documented FPs (estimated; needs corpus re-run to confirm). +- **Recall impact:** small. The pattern "Refactor X so it never blocks as a rule" — coding task with embedded durable directive — is rare and can be opt-in via the rule-marker connective, which the filter detects. +- **Implementation surface:** ~30 LOC in `src/aelfrice/directive_detector.py`. Pure stdlib, deterministic. No corpus dependency for the public unit tests beyond what already lives in `tests/test_directive_detector.py`. +- **Determinism:** preserved. + +### Path B: deontic-anchor requirement + +Require an explicit deontic anchor (`always`, `never`, `must`, `must not`, `should`, `shall`, `forbidden`, `mandatory`, `prohibited`) — a strict subset of the current 29 verbs — and demote the remaining markers (`avoid`, `prefer`, `only`, `before`, `after`, `unless`, `whenever`, `need to`, `ensure`, `require`, …) to "supporting" status that does not fire on its own. + +- **Precision impact:** large. Imperatives without a deontic anchor (the FP cluster) all return `False`. +- **Recall impact:** unknown but plausibly significant. Many durable rules are stated without strong deontic markers ("only push from main", "before merging, run the gate"). The 0.937 → ? drop could blow the 0.60 recall floor. +- **Implementation surface:** ~10 LOC change to `_IMPERATIVE_VERBS` partition. +- **Risk:** the recall hit may be larger than the headroom allows. + +### Path C: lightweight LLM classifier + +Replace the regex with an LLM call ("does this prompt encode a durable rule?"). Cached per-prompt to amortize cost; falls back to regex on cache miss + budget exhaustion. + +- **Precision impact:** likely high; LLMs disambiguate this class easily. +- **Recall impact:** likely high. +- **Implementation surface:** large. Requires model selection, prompt design, cache schema, latency/cost budget, and a new dependency posture. +- **Determinism:** **broken.** Conflicts with `PHILOSOPHY.md` determinism property and with the `feedback_avoid_embeddings_nondeterminism` posture. Listed for completeness; not a serious option without an explicit ratification of a non-deterministic component in the rebuild path. + +## Recommendation + +**Path A (intent-prefix filter)** at v2.x re-entry. Reasons: + +1. The failure mode is monolithic — one cluster, one structural property. A targeted filter beats a coarse partition (Path B) or a model swap (Path C). +2. Determinism preserved. No new dependency. ~30 LOC. +3. Failure cost is bounded: if Path A under-performs against an updated corpus, Path B is the natural next step (already a strict subset of A's verb bank); the iteration order does not lock anything in. + +## Decision asks + +- [ ] **Confirm Path A as the iteration target.** If no, name an alternative (B, C, or a new path) before any code change. +- [ ] **Confirm head-position lexical anchors for the prefix filter.** The 16 verbs listed are reconstructed from typical session-task prefixes; the gate scores whatever lands in `directive_detector.py`, but the choice is worth ratifying so the verb bank does not drift quietly. +- [ ] **Confirm the rule-marker connective list** (`so that`, `as a rule`, `from now on`, …) that re-enables directive classification when an imperative coding-task prefix is present. Default: empty (i.e., coding-task prefix always wins). Conservative; revisit after first corpus run. +- [ ] **Lab-side action: open `tests/corpus/v2_0/directive_detection/v0_1.jsonl` PR against lab `main`** (currently on branch `exp/issue-374-directive-corpus-v0_1`). The gate harness in PR #377 cannot fire end-to-end until the corpus is on the canonical lab path (`AELFRICE_CORPUS_ROOT/directive_detection/v0_1.jsonl`). Without that merge, every public-tree session that reads `aelf-scan` and lands on #374 will continue to bounce. + +## Out of scope + +- `process_directive`, the TODO lifecycle, the repetition counter, the escalation table, hook wiring. All gated on the bench gate passing per [`v2_enforcement.md` § H1](v2_enforcement.md#h1-directive-detection--defer-to-v2x-with-benchmark-gate). Implementation work on those starts only after a detector revision lands a passing gate run. +- Corpus authoring. Per directory-of-origin rules and `tests/corpus/v2_0/README.md`, the labeled rows live under `AELFRICE_CORPUS_ROOT` (lab) and never ship to the public tree. This memo does not propose corpus changes; it assumes lab corpus v0.1 is the immediate evaluation target and that v0.2 (with hard-negative imperative-task examples added) is a natural follow-up. +- Verb-bank expansion in the existing 29-imperative regex. Path A composes with the current regex; it does not modify it. + +## Public-tree vs lab-tree work split + +To stop the 10-bounce cycle on #374, the actionable work split is: + +- **Public-tree (this repo):** detector source change in `src/aelfrice/directive_detector.py`; public-CI sanity tests in `tests/test_directive_detector.py` for the new filter behaviors; the bench-gate harness already in `tests/bench_gate/test_directive_detection.py` does not need changes. +- **Lab-tree:** corpus authoring (v0.1 → v0.2 with added hard-negatives); bench-gate run with `AELFRICE_CORPUS_ROOT` mounted to verify P/R against the gate. Lab-side P/R numbers are reported back as a comment on #374 but the rows themselves do not cross the boundary. + +A public-tree session can land Path A as a doc-only commit that preserves H1's deferred status (no claim that the gate is passing). Closing #374 still requires a lab-side run that demonstrates P≥0.80 ∧ R≥0.60 ∧ n≥200; that closing event lands as the same PR that strikes the row from `V2_REENTRY_QUEUE.md`. + +## Re-entry trigger (unchanged from § H1) + +H1 reopens for implementation when: + +- ≥0.80 precision on the lab corpus (currently `aelfrice-lab/tests/corpus/v2_0/directive_detection/`, ≥200 rows). +- ≥0.60 recall on the same corpus. +- A reproducible bench-gate run is recorded on the closing PR (lab `pytest -q tests/bench_gate/test_directive_detection.py` output; numbers cited in PR body). + +If those numbers are not met after Path A, this memo's recommendation is to revisit Path B (deontic-anchor partition) before any model-based approach. Path C stays out of scope until the determinism property is explicitly re-decided. + +## Provenance + +- Parent spec: [`docs/v2_enforcement.md` § H1](v2_enforcement.md#h1-directive-detection--defer-to-v2x-with-benchmark-gate) (PR [#257](https://github.com/robotrocketscience/aelfrice/pull/257), merged 2026-04-28). +- Re-entry queue row: [`docs/V2_REENTRY_QUEUE.md`](V2_REENTRY_QUEUE.md) § "Directive detection (enforcement H1) — issue #374". +- Harness PR: [#377](https://github.com/robotrocketscience/aelfrice/pull/377), merged 2026-05-03. +- Lab corpus v0.1 reference: issue #374 comment 2026-05-03T17:16:22Z (285 rows, P=0.664 / R=0.937 against the candidate detector). +- Failure-mode examples ("Refactor X so it never blocks", "Add a test that ensures …"): same comment. +- Umbrella: [#199](https://github.com/robotrocketscience/aelfrice/issues/199).