feat(workflows): add shared context-engineering module for ralph + deep-research

[lavaman131]

Summary

Extracts shared context-engineering utilities into a new _context module and applies them across the Ralph and deep-research-codebase builtin workflows (Claude, Copilot, and OpenCode SDKs). The changes reduce per-turn token consumption, improve cross-iteration continuity, and harden specialist stages against empty-text responses.

Key Changes

New _context module (src/sdk/workflows/builtin/_context/)

• budget.ts — Centralized token-budget thresholds (COMPACT_TRIGGER_FRACTION, CHANGESET_MASK_THRESHOLD, DIFF_STAT_TOP_N, etc.) so a single edit propagates to all consumers
• masking.ts — Pure, unit-tested utilities: changeset masking (maskChangeset, compactDiffStat, compactUncommitted), infra-discovery invalidation (isInfraPath, shouldReRunInfraDiscovery), markdown truncation (truncateMarkdownReport), history-brief derivation (deriveHistoryBrief), compact reminder formatting, prose-guard retry (queryWithProseGuard), and scratch-file compaction (compactScratchFile)
• scratchpad.ts — Per-run persistent markdown scratchpad for Ralph (.atomic/ralph/<session-id>/state.md) tracking prior RFCs, files modified, decisions, rejected approaches, and debugger reports across all iterations
• _context.test.ts — Comprehensive test suite covering all pure utility functions

Ralph workflow (ralph/)

• CAVEMAN_PREAMBLE — Token-reduction meta-prompt prepended to all stage prompts (planner, orchestrator, infra-discovery agents); exempts code blocks and structured outputs
• Per-run scratchpad — Planner output (RFC or spec-path short-circuit) and debugger reports persisted after each iteration; next-iteration planner receives prior RFC for continuity instead of re-deriving from scratch
• Prior RFC injection — priorRfc field added to PlannerContext; injected with explicit precedence note ("Debugger Report takes precedence")
• <SPEC_REMINDER> anchor — Compact session intent pinned at the bottom of planner/reviewer prompts to resist prompt drift
• Infra-discovery cache — Discovery stages now only re-run when shouldReRunInfraDiscovery detects changes to build/CI/agent-instruction files; otherwise cached result reused

Deep-research-codebase workflow (deep-research-codebase/)

• CAVEMAN_PREAMBLE — Applied to all stage prompts (scout, locator, analyzer, pattern-finder, online-researcher)
• D2: history-brief injection — deriveHistoryBrief distills the history-analyzer output (≤150 words) and injects it as <PRIOR_RESEARCH_HINT> into per-partition locator and analyzer prompts across all three SDKs
• D5: prose-guard — queryWithProseGuard wraps history-locator and history-analyzer stages; auto-retries with a follow-up prompt when the first response returns empty text
• Aggregator pre-flight — compactScratchFilesForAggregator trims oversized partition scratch files before the aggregator reads them, preventing token overruns on large codebases

[claude]

Code Review — feat/context-engineering-workflows

Ambitious and mostly well-executed. The new _context module is a nice consolidation, the test file for pure helpers is thorough (_context.test.ts, 252 lines), and the infra-discovery caching (shouldReRunInfraDiscovery) is a solid win. A few correctness and DRY issues below.

---

Bugs

🔴 [Blocking] Nested fences silently truncate stored RFCs

src/sdk/workflows/builtin/_context/scratchpad.ts:106-113 wraps the full planner output in a ```markdown ... ``` fenced block:

const block = [
  `### Iteration ${iteration}`,
  "```markdown",
  plannerOutput.trim(),
  "```",
].join("\n");

But RFCs routinely contain fenced code blocks (SQL schemas, TS signatures, diagrams, etc.). latestPriorRFC (scratchpad.ts:165) then uses:

const blocks = [...body.matchAll(/```markdown\s*\n([\s\S]*?)\n```/g)];

Because [\s\S]*? is non-greedy, it captures up to the first \n```` it encounters — which will be the opening fence of any inner typescript ` / ` sql block inside the RFC, not the outer closer. Result:priorRfc` passed back to the next-iteration planner is a silently truncated fragment. Worse, the planner block is what drives the "don't re-derive the design" guarantee the PR advertises.

Fix options: use a unique sentinel like <!--RFC_START--> / <!--RFC_END-->, a longer tilde-fence (~~~~~markdown — unlikely to collide), or base64-encode the payload.

🔴 [Blocking] extractSection terminates at any \n## in the body

scratchpad.ts:183-189:

const re = new RegExp(
  `(^|\\n)##\\s+${escapeRe(name)}\\s*\\n([\\s\\S]*?)(?=\\n##\\s|$)`,
);

RFCs, debugger reports, and reviewer output all routinely use ## Foo H2 headings. The first \n## inside the stored content will be matched as the start of the next section, so extractSection("Prior RFCs") / extractSection("Debugger Reports") return a fragment, not the full body. This is the same class of bug as #1 but affects every section helper (replaceSection, appendToSection inherit it).

Fix: switch to explicit delimiters (<!--SECTION:Prior RFCs--> … <!--/SECTION:Prior RFCs-->), or indent stored headings one level deeper so they can't collide with the outer ## schema.

Adding a round-trip test for recordPlannerOutput → latestPriorRFC with a realistic RFC containing code fences + ## Xyz headings would have caught both #1 and #2.

---

DRY / Cleanup

1. CAVEMAN_PREAMBLE duplicated verbatim

• src/sdk/workflows/builtin/ralph/helpers/prompts.ts:34 (exported)
• src/sdk/workflows/builtin/deep-research-codebase/helpers/prompts.ts:73 (module-private)

With the new _context module this is the natural home. The two copies are already drifting (Ralph's includes the destructive-SQL example; deep-research's omits it).

2. deriveSessionIntent + parseFilesFromNameStatus copy-pasted

Identical functions in all three SDK entrypoints:

• ralph/claude/index.ts:50,62
• ralph/copilot/index.ts:48,56
• ralph/opencode/index.ts:45,53

And parseFilesFromNameStatus is a near-exact duplicate of the already-present parseNameStatus in _context/masking.ts:162. Just export it.

3. Indentation regression in deep-research-codebase/claude/index.ts

Lines ~195-344 have inconsistent indentation after the queryWithProseGuard refactor — the inner async (s) => { ... } bodies are one level shallower than their declarations, and the closing ) lives at the wrong depth. Valid TS but eyesore. Run biome/prettier.

---

Minor

• compactDiffStat (masking.ts:61) uses /\|\s*(\d+)/, which silently drops binary-file entries shown as | Bin 0 -> 1234 bytes. Consider preserving them.
• recordFilesModified overwrites the trailing annotation each iteration ("_(iteration N added M)_"), so you can't trace which iteration introduced which file — only how many the latest added. If cross-iteration attribution matters, emit per-path iteration tags instead.
• compactScratchFile uses a magic 32 for the elision-marker budget. Extracting as a named constant (ELISION_MARKER_OVERHEAD) would document the intent.
• truncateMarkdownReport uses maxChars - headLen - 128 where 128 is a reserve for the elision message — same suggestion.
• initScratchpad calls readFile(filePath, "utf8") just to probe existence — access() or stat() is cheaper.
• .atomic/ralph/<uuid>/ directories accumulate forever. Consider a TTL-based cleanup in the workflow teardown (or document the sweep command).
• Heads up: Ralph's copilot/opencode queryWithProseGuard retry re-sends PROSE_GUARD_RETRY_PROMPT on the same session, so it piles extra turns onto the transcript. Acceptable but worth a comment.

---

Test coverage

Strong on the pure helpers. Not yet covered:

• queryWithProseGuard (happy path + empty-first → retry → return).
• Scratchpad I/O: recordPlannerOutput / recordDebuggerReport / latestPriorRFC round-trip on realistic RFC content (see bugs add agent instructions #1 and updates to readme and instructions #2).
• compactScratchFilesForAggregator threshold / per-file-budget behaviour.

---

Security / perf

No new attack surface. Regex helpers all use bounded lazy matching (no ReDoS risk on realistic inputs). Per-iteration fs I/O is negligible compared to the LLM calls. The infra-discovery hoist is a meaningful token-cost reduction on long runs.

---

Recommendation: fix bugs #1 and #2 before merge (with a round-trip test that exercises RFCs containing both inner fenced blocks and ## headings), then knock out the DRY items. The rest is opt-in polish.