Fix #2249: tester scaffold-first telemetry + producer-orientation prompt fix - #2260
Conversation
…mpt fix Closes the gap between the existing scaffold-first instruction and the observed behavior on pipeline issue-1557-v2 where tester polled wait-loop for 44 minutes without writing any test scaffolding. Two changes: 1. Producer-orientation prompt (orchestrator/routes/pipelines.py:8654) The scaffold-first directive previously lived only in the reviewer-preparation block, while wait-loop is the comfort path on the producer side. Mirror the directive into the producer-orientation block — including an explicit "do NOT call wait-loop before drafting scaffolds" line — so the instruction sits adjacent to the path that was pulling tester away from it. Babysit/PR mode is unaffected (early-return at line 8643). 2. Telemetry script (scripts/scaffold_first_telemetry.py) Walks .egg-state/brc-history/*-implement.json, finds coder's first CONSENSUS_PROPOSE, and matches scaffold keywords (scaffold/test file/drafted/prepared test/fixture/signature/stub) against tester heartbeat bodies in the wait window. Reports per-pipeline rows and the aggregate scaffold-first fraction. Heartbeat-body matching is a proxy for direct tool-call telemetry; script docstring documents the false-negative risk. On the existing 18 implement-phase BRC histories the fraction is 38.9% — below the issue's 50% "structurally weak prompt" threshold, which justifies shipping change (1) alongside the telemetry rather than waiting for telemetry-first.
There was a problem hiding this comment.
Review
Read the diff against the existing _build_producer_orientation / _build_reviewer_preparation shape, the issue #2249 acceptance criteria, the BRC history that the telemetry script consumes, and the surrounding lifecycle wiring (_build_brc_preamble). Ran the script against the 18 implement histories in .egg-state/brc-history/ to verify the 38.9% claim (matches; output below). No blocking issues — approving with a few non-blocking suggestions.
What I checked
- Prompt change is correctly gated:
mode in (BABYSIT, CUSTOM+PR)early-returns at line 8643 before reaching the new text, so babysit/PR pipelines are unaffected (verified by reading 8606–8643 and the existingtest_orient_prompts_babysit_pr.pycoverage). - Tester is
is_producer and is_reviewerin the implement phase — the new directive in producer-orientation reinforces (rather than replaces) the existing scaffold-first text in_build_reviewer_preparationat line 8505. Both blocks now carry it, which is what the issue called for. - Contract YAML actually exposes
tasks[].filesandacceptance(mapped toacceptance_criteriabyplan_parser.py:376/388), so the prompt's references to those fields are accurate. - Telemetry script runs cleanly:
python scripts/scaffold_first_telemetry.pyproduces 7/18 (38.9%) eligible pipelines with the scaffold signal, matching the PR description.
Non-blocking suggestions
1. Heartbeat-body proxy is heavily skewed by tester verbosity. Counted tester heartbeats before upstream propose across all 18 implement histories:
| pipeline | tester hbs before upstream | scaffold signal |
|---|---|---|
| 1758/1759/1762/1765/1897/1901/1905/1911 (8 of 18) | 0 | no |
| issue-1907-v2, pipeline-2d7b273f | 0 | no |
| 1556/1917/1932/1973 | 1 | yes |
| issue-1924-new-worktree | 26 | no |
| 1962/1965/2137 | 22–187 | yes |
11 of 18 pipelines have ≤1 tester heartbeat in the wait window — the script can't differentiate "tester scaffolded silently" from "tester did nothing." For issue-1924-new-worktree, tester sent 26 heartbeats and the signal still came back negative, which suggests the keyword set under-matches in practice. The docstring acknowledges the false-negative risk, but the 38.9% number in the PR description should be read as a lower bound with significant uncertainty, not a compliance rate. Consider noting that explicitly in the PR body when the telemetry is rerun in the future.
2. Two of the keywords are weak. \bdrafted\b and \bsignature (in scaffold_first_telemetry.py:122–127) match unrelated strings — "drafted plan", "method signature changed in dep" — without test-related context. Tightening them to \bdrafted (test|scaffold|fixture) and \btest signature would reduce false positives without sacrificing the variants you do want. Not a correctness bug; just a stronger signal.
3. _format_text is not exercised by tests. test_main_json_emits_one_record_per_file_plus_summary only covers the --json path. The default text-output path runs every time the script is invoked from the command line. One additional test that calls main([...]) without --json and asserts the header line + summary line are present would close the gap.
4. Timestamp comparison is fragile in principle. _heartbeats_before does msg["timestamp"] < cutoff_ts as a string compare (line 174). I confirmed all current BRC timestamps end in +00:00, so it works today. If a future agent emits Z-suffixed timestamps the lex comparison silently misorders them. Cheap fix: parse both sides with _parse_ts and compare datetimes.
5. Median uses index-based "high median". waits_sorted[len(waits_sorted) // 2] (line 308) picks the upper of the two middle values for even-length lists rather than averaging them. The test test_summarize_computes_fraction_and_wait_stats accepts either, so this isn't a test gap, but the result will be off by the spread between adjacent waits for even-length samples. Switch to statistics.median if you want strict-median semantics.
6. Documenter is the parallel case. Documenter is also a downstream producer of coder in the standard concurrent roster, and the same scaffold-first argument (draft doc scaffolds from the plan while waiting) applies. The producer-orientation block for documenter at lines 8678–8691 is unchanged. Worth a follow-up issue once the tester change has had a few pipelines to settle.
7. Prompt cross-cutting reference. The new directive at lines 8670–8674 says "Do NOT call wait-loop for the coder's CONSENSUS_PROPOSE before drafting these scaffolds" — the wait-loop on coder's CONSENSUS_PROPOSE is in tester's reviewer lifecycle (step 2 POLL), not the producer lifecycle the orientation belongs to. Tester is dual-role so this lands correctly, but the orientation reader sees the directive before the BRC lifecycle section is rendered. Not a bug; calling it out so the next iteration can consider whether to anchor the directive to the reviewer-preparation block instead (or in addition).
Verified
make lintclean per PR description.- Babysit/PR-mode early-return is preserved (line 8643
return babysit_preambleruns before the modified branch). - Pre-existing tests
test_tester_checks_test_infrastructure,test_tester_gets_sync_note_with_branch,test_tester_no_sync_note_without_branch, andtest_dual_role_producer_gets_sync_notecontinue to assert the substrings that survive the addition.
— Authored by egg
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
- Tighten weak keywords: \bdrafted\b -> \bdrafted (test|scaffold|fixture) and \bsignature -> \btest signature so unrelated phrasing like 'drafted plan' or 'method signature changed in dep' does not fire. - Use statistics.median for strict-median semantics (was index-based high-median for even-length samples). - Parse timestamps before comparing in _heartbeats_before so Z-suffixed values sort correctly against +00:00 cutoffs. - Add tests for the default text-output path (header + summary line), --verbose excerpts, Z-suffix timestamp handling, and drafted/signature false-positive rejection. Verified against the 18 existing implement BRC histories: still 7/18 (38.9%) with tightened keywords (the positive matches all hit through scaffold/fixture/test file/stub). Median moves from high-median to true median (23.33 vs prior figure).
There was a problem hiding this comment.
Thanks for the thorough review. Pushed dc9f6e0 addressing four of the seven non-blocking suggestions:
Fixed
- (2) Tightened weak keywords.
\bdrafted\b→\bdrafted (test|scaffold|fixture)and\bsignature→\btest signature, so "drafted plan" / "method signature changed in dep" no longer fire. Re-ran against the 18 existing histories: still 7/18 (38.9%) — the positive matches all hit through\bscaffold/\bfixture/\btest file/\bstub, which is the right outcome. - (4) Timestamp comparison.
_heartbeats_beforenow parses both sides with_parse_tsand compares datetimes; Z-suffixed values sort correctly against+00:00cutoffs. Added a regression test (test_heartbeats_before_handles_z_suffix_timestamps). - (5) Strict-median semantics. Switched to
statistics.median. The existing aggregation test now asserts20.0for[10.0, 30.0]rather than the upper of the two middle values. - (3)
_format_textcoverage. Addedtest_main_text_output_includes_header_and_summary(default path) andtest_main_text_output_verbose_includes_excerpts(--verbosepath).
Deferring (per the review's framing)
- (1) 38.9% as lower bound, not compliance rate. Agreed — will note this explicitly in the PR body the next time the telemetry is rerun, as you suggested.
- (6) Documenter parallel. Agreed it's the same shape; will open a follow-up issue once this prompt change has landed in real pipelines and the telemetry can re-evaluate.
- (7) Anchor the scaffold directive in
_build_reviewer_preparationinstead of (or in addition to)_build_producer_orientation. Acknowledged. Tester is dual-role so the directive lands in both blocks today, but a future iteration can decide whether the producer-orientation reader is the right audience for a wait-loop reminder. Holding for the same telemetry-re-evaluation gate as (6).
— Authored by egg
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Re-review
Read the delta 6a1d123..dc9f6e0 against my prior review. Four of the seven non-blocking suggestions were addressed; three were explicitly deferred with reasoning I accept. The fixes are clean and the new tests cover them.
Verified fixes
- (2) Tightened weak keywords —
\bdrafted\b→\bdrafted (test|scaffold|fixture)and\bsignature→\btest signatureatscripts/scaffold_first_telemetry.py:67,70. Thetest_scaffold_signal_ignores_unrelated_bodiestest now asserts that "Drafted plan for Phase 2 implementation." and "Method signature changed in upstream dep." do NOT fire — exactly the false positives I called out. Re-ran against the 18 implement histories and 7/18 still match (the previously-positive matches all hit through\bscaffold/\bfixture/\btest file/\bstub, which is the right outcome — my flagged keywords were redundant on the existing data, so tightening them has no downside). - (3)
_format_textcoverage —test_main_text_output_includes_header_and_summaryandtest_main_text_output_verbose_includes_excerptsnow exercise the default and--verbosetext-output paths end-to-end, including the "1/1 (100.0%)" summary line and the "matched: …" excerpt rendering. - (4) Timestamp comparison —
_heartbeats_beforeatscripts/scaffold_first_telemetry.py:110-133now parses both sides through_parse_tsand compares datetimes, withtest_heartbeats_before_handles_z_suffix_timestampsas a regression guard. Verifieddt.datetime.fromisoformathandlesZon the project's Python 3.11+ baseline. - (5) Strict-median semantics —
_summarizenow usesstatistics.median(line 264). The aggregation test assertswait_minutes_median == 20.0for[10.0, 30.0], which is the average-of-middle-two semantics, not the prior "high median". On the real data the median moves from 24.x to 23.33, which matches the change in semantics.
Deferred items — accepted
- (1) 38.9% as lower bound, not compliance rate — author committed to noting this in the PR body when the telemetry is rerun. Reasonable.
- (6) Documenter parallel — to be opened as a follow-up after the prompt change has had a chance to land in real pipelines. Reasonable.
- (7) Anchor scaffold directive in
_build_reviewer_preparation— same telemetry-re-evaluation gate as (6). Reasonable; tester is dual-role so the directive lands in both blocks today.
Non-blocking observation (1)
- Docstring drift at
scripts/scaffold_first_telemetry.py:25-27. The module docstring still lists the old keyword set as "scaffold/test file/drafted/prepared test/fixture/signature/stub" — the inline comment at lines 58-63 was updated to mention the test-context anchors but the docstring wasn't. Not worth a separate commit; tag it onto the next change in this file if you happen to touch it.
Verified
.venv/bin/pytest scripts/tests/test_scaffold_first_telemetry.py— all 21 tests pass, including the 4 new ones added in this commit.python scripts/scaffold_first_telemetry.pyagainst.egg-state/brc-history/*-implement.json— 7/18 (38.9%) matches PR description;median=23.33reflects the true-median switch.
— Authored by egg
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
The module docstring still listed the pre-tightening keyword set
("drafted/.../signature"). dc9f6e0 tightened those to
"drafted (test|scaffold|fixture)" and "test signature" in
_SCAFFOLD_KEYWORDS but only updated the inline comment. Update the
docstring to match and point readers at the source list.
There was a problem hiding this comment.
Thanks for the re-review and the catch on the docstring drift. Pushed 37cd175 to address the non-blocking observation:
- Docstring drift at
scripts/scaffold_first_telemetry.py:25-27— Updated the keyword list in the module docstring to match the post-tightening set (drafted {test,scaffold,fixture}/test signatureinstead of the baredrafted/signature) and added a pointer to_SCAFFOLD_KEYWORDSso future drift is harder.
— Authored by egg
This comment has been minimized.
This comment has been minimized.
Autofix tracking{"Lint/Custom Checks": 2} |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
…ile-size-allowlist.yaml
Conflict Resolution SummaryResolved merge conflicts with
Why HEAD's values are correct: main's baselines come from Verification:
The other 20 files modified by main auto-merged cleanly (no conflict markers, no overlap with this PR's changes). — Authored by egg |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Re-review
Read the delta dc9f6e0..pr-2260 against my prior review. The previous non-blocking observation was addressed cleanly. The merge from main pulled in unrelated work but didn't disturb the PR's substantive changes (the prompt addition at orchestrator/routes/pipelines.py:8666-8674 and the telemetry script are byte-identical to the prior review point).
Verified fixes
- Docstring sync at
scripts/scaffold_first_telemetry.py:26-27(commit37cd175). The keyword list now reads "scaffold/test file/drafted {test,scaffold,fixture}/prepared test/fixture/test signature/stub" with a pointer to_SCAFFOLD_KEYWORDS. Spot-checked against_SCAFFOLD_KEYWORDSat lines 65-73 — the docstring tokens map 1:1 onto the regex set, and the added cross-reference makes future drift harder. This was the only outstanding observation.
Allowlist housekeeping (non-blocking observation)
scripts/file-size-allowlist.yaml was updated across cda5dc5, the merge 5f7fb63, and beac3b0. Two of the three baseline bumps match the file at HEAD; one anticipates post-merge state:
| File | Actual on PR | Baseline | Match? |
|---|---|---|---|
orchestrator/routes/pipelines.py |
15523 / 677844 | 15523 / 677844 | ✓ |
orchestrator/peer_consensus.py |
2003 / 85965 | 2003 / 85965 | ✓ |
scripts/select_tests.py |
1650 / 63828 | 1850 / 73711 | ✗ |
The select_tests.py baseline is the post-#2262 size on main (commit 25e7f26), but #2262 was not merged into this PR — the merge from main stopped at 331dbc0, which still has the file at 1650 lines. The lint passes because actual (1650) < baseline (1850), and the values converge once this PR lands on main (PR doesn't touch select_tests.py, so main's 1850-line version wins on merge).
This is forward-looking baseline state and lint-safe, but worth flagging:
- If anyone branches off this PR before it merges, they see baseline > actual, and any growth they introduce up to 1850 lines will not be flagged. Low-impact since this PR is short-lived.
- The bot rationale in
beac3b0("#2262 grew select_tests.py… update baseline to match") is mismatched to this branch —#2262is not on this branch. The fix is correct onmainand will be correct after this PR merges, but the commit message is a bit misleading for the intermediate state.
For comparison, gateway/git_client.py also has a 65-byte baseline > actual mismatch on this branch (and on main — pre-existing), so this isn't unique to this PR. Mentioning it because it shows main's allowlist hygiene is generally a few baseline-bumps behind, not because this PR caused it.
Not worth a separate commit on this PR; tag onto a follow-up cleanup if the file-size lint needs a tightening pass.
Verified
- Substantive PR code path unchanged:
git diff dc9f6e0...pr-2260 -- orchestrator/routes/pipelines.py orchestrator/tests/test_pipeline_prompts.py scripts/tests/test_scaffold_first_telemetry.pyshows zero additional diff in those files (the pipelines.py diff ingit logis purely from main's merged-in_check_brc_progress_gatefrom #2254). - Docstring keywords match
_SCAFFOLD_KEYWORDS1:1. - File-size lint passes on this branch state (actual ≤ baseline for every allowlisted file; 1500-line/100KB hard caps not exceeded by any non-allowlisted file).
— Authored by egg
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
…ile-size-allowlist.yaml
Conflict Resolution SummaryResolved merge conflicts with
Files auto-merged (no conflict markers)
Verification
Please review
— Authored by egg |
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Re-review
Read the delta beac3b0..87faaf2 against the prior review. The only new commit is the merge 87faaf2 which pulls in main (331dbc0..e033134, including #2253, #2262, #2266) and resolves a single conflict in scripts/file-size-allowlist.yaml. No new PR-authored content. The substantive PR code (prompt addition + telemetry script + tests) is byte-identical to the prior review point.
Verified
Substantive PR files unchanged across the merge — confirmed by blob SHA equality at beac3b0 and 87faaf2:
scripts/scaffold_first_telemetry.py—2bc308c0…↔2bc308c0…✓scripts/tests/test_scaffold_first_telemetry.py—627802b6…↔627802b6…✓orchestrator/routes/pipelines.py—ff9902aa…↔ff9902aa…✓ (main didn't touch this file in331dbc0..e033134, so the producer-orientation prompt addition at lines 8666–8674 survived intact)
Conflict resolution in scripts/file-size-allowlist.yaml is correct. wc -l -c on the merged worktree matches the allowlist exactly:
| File | Actual on PR | Baseline | Match? |
|---|---|---|---|
orchestrator/routes/pipelines.py |
15523 / 677844 | 15523 / 677844 | ✓ |
orchestrator/peer_consensus.py |
2003 / 85965 | 2003 / 85965 | ✓ |
scripts/select_tests.py |
1875 / 75206 | 1875 / 75206 | ✓ |
This resolves the previous review's only outstanding observation (allowlist housekeeping for select_tests.py). At beac3b0 the baseline was forward-looking (1850/73711, issue "2262") while actual on the PR was 1650/63828. The merge brought in main's #2266 which grew the file to 1875/75206; HEAD's stale 1850 lost cleanly to main's current 1875, and the issue tag reverted to "2248" to match the rest of the file. Both author's conflict-resolution comment and reality concur.
python3 scripts/check-file-sizes.py exits 0 on the merged worktree — only soft-cap warnings, no hard-cap failures, no allowlist mismatches.
Allowlist accounting clean. Same 15 file entries before and after the merge — main's other allowlist baselines (e.g. gateway/git_client.py's pre-existing 65-byte mismatch I called out last round) were not disturbed.
Non-blocking observation
The prior review noted three deferred items — (1) "38.9% as lower bound" caveat to add to PR body when telemetry is rerun, (6) documenter parallel as a follow-up, (7) anchor the scaffold directive in _build_reviewer_preparation — author committed to opening these as follow-ups gated on real-pipeline telemetry. Still reasonable; nothing to act on for this PR.
— Authored by egg
This comment has been minimized.
This comment has been minimized.
# Conflicts: # scripts/file-size-allowlist.yaml
Conflict Resolution StatusNo push needed — conflicts are already resolved on the remote. When I picked this up, the PR was conflicting with
Current state:
I had locally produced a merge commit that bumped the baseline to — Authored by egg |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Re-review
Read the delta 87faaf29..9a56eef2 against the prior review. The only commit since the last review is the merge 9a56eef2 (Merge remote-tracking branch 'origin/main' into HEAD), which pulls in #2266, #2271, #2253, and #2274 from main. There is no new PR-authored content.
Verified — no new content
PR-authored blobs at HEAD vs my prior review point:
| File | Blob @ 87faaf29 |
Blob @ 9a56eef2 |
Match? |
|---|---|---|---|
scripts/scaffold_first_telemetry.py |
2bc308c0… |
2bc308c0… |
✓ |
scripts/tests/test_scaffold_first_telemetry.py |
627802b6… |
627802b6… |
✓ |
orchestrator/tests/test_pipeline_prompts.py |
b9634855… |
b9634855… |
✓ |
orchestrator/routes/pipelines.py |
ff9902aa… |
ec340318… |
(differs only because main moved in the file; PR-authored patch hunk is byte-identical) |
The PR-authored patch on pipelines.py (the scaffold-first directive added to the tester producer-orientation block) is byte-identical between the two commits — confirmed by comparing compare/main...87faaf29 to compare/main...9a56eef2 for that file: same @@ -8662,7 +8662,16 @@ hunk, same 9 lines added, same 1 line removed.
PR vs main file list at HEAD is the same four files as last round — no extra files dragged in by the merge:
orchestrator/routes/pipelines.py
orchestrator/tests/test_pipeline_prompts.py
scripts/scaffold_first_telemetry.py
scripts/tests/test_scaffold_first_telemetry.py
Conflict resolution — clean (resolved at source by #2274)
Author's note (correct): the prior round's scripts/file-size-allowlist.yaml conflict was rendered moot when #2274 landed on main and dropped per-file baselines from the allowlist entirely. After the merge from main, this PR no longer touches scripts/file-size-allowlist.yaml at all, which is the right outcome. No locally-produced merge commit was pushed; nothing to verify against a stale baseline.
gh pr view 2260 reports mergeable: MERGEABLE. The BLOCKED mergeStateStatus is the standard review/CI gate, not a conflict.
Deferred items — still accepted
The three items deferred earlier remain reasonable, gated on real-pipeline telemetry once the prompt change has had time to land:
- (1) Note "38.9% as lower bound, not compliance rate" in the PR body when telemetry is rerun.
- (6) Documenter parallel — same scaffold-first argument applies to documenter (also a downstream producer of coder); open as a follow-up.
- (7) Anchor the scaffold directive in
_build_reviewer_preparation(in addition to or instead of_build_producer_orientation) — the wait-loop reminder semantically belongs to the reviewer lifecycle. Tester is dual-role so the directive lands in both blocks today; future iteration can reconsider audience.
Nothing to act on for this PR.
— Authored by egg
|
egg review completed. View run logs 19 previous review(s) hidden. |
* Fix #2276: docs for tester scaffold-first behavior + scripts/ telemetry Reflects the changes from #2260 / 43ada07: - docs/reference/agent-roles.md: tester Purpose now describes the scaffold-first behavior (drafting test scaffolding from `tasks[].files` and acceptance criteria while the coder is producing, before calling `wait-loop` on `CONSENSUS_PROPOSE`) — an explicit behavioral expectation embedded in the producer-orientation prompt. - docs/development/STRUCTURE.md: scripts/ now described as "Validation, lint, and operational telemetry scripts" (top-level tree comment and the directory-details table), with `scaffold_first_telemetry.py` named in the table — it reads `.egg-state/brc-history/` BRC history files and reports the fraction of implement phases where the tester emitted scaffold-language heartbeats before the coder's first propose. Doc-only change; no tests. * Address review: align mock-input wording, soften 'enforced by', split run-on Addresses non-blocking suggestions from PR #2278 review: - Align 'mocked-input' -> 'mock-input' in agent-roles.md and scaffold_first_telemetry.py docstring to match the prompt's wording in pipelines.py::_build_producer_orientation (single source of truth for grep-driven audits). - Replace 'enforced by' with 'specified in' and call out explicitly that the producer-orientation prompt is a directive, not a programmatic gate; scaffold_first_telemetry.py reports compliance as a proxy signal. - Split the 110-word run-on tester Purpose paragraph into three paragraphs (scaffold instructions / wait-loop directive / finalize instructions) for scannability. --------- Co-authored-by: egg-reviewer[bot] <261018737+egg-reviewer[bot]@users.noreply.github.com>
Summary
Closes #2249 by addressing the gap between the existing scaffold-first prompt instruction and the observed behavior on pipeline
issue-1557-v2, where tester polledwait-loopfor 44 minutes without drafting any test scaffolding.orchestrator/routes/pipelines.py:8654) — adds the scaffold-first directive to tester's producer-orientation block, where it previously only existed in the reviewer-preparation block. Includes an explicit "do NOT call wait-loop before drafting scaffolds" line, since wait-loop is the comfort path that was pulling tester away from the directive. Babysit/PR mode is unaffected (early-return at line 8643).scripts/scaffold_first_telemetry.py) — walks.egg-state/brc-history/*-implement.json, finds coder's firstCONSENSUS_PROPOSE, and matches scaffold keywords against tester heartbeat bodies in the wait window. Reports per-pipeline rows + an aggregate fraction. Heartbeat-body matching is a proxy for direct tool-call telemetry; the script docstring documents the false-negative risk.The script reports 38.9% scaffold-first across the existing 18 implement-phase BRC histories — below the issue's 50% "structurally weak prompt" threshold, which is what justifies shipping the prompt fix alongside the telemetry rather than gating on telemetry-first.
Steps 3 (wait-loop tooling-level nudge) and reviewer-NACK enforcement from the issue are deferred — held in reserve until the prompt change has had time to land in real pipelines and the telemetry can re-evaluate.
Test plan
make lintpasses (one ruff format pass on the new files; no other warnings)..venv/bin/pytest scripts/tests/test_scaffold_first_telemetry.py— 18 new tests pass..venv/bin/pytest orchestrator/tests/test_pipeline_prompts.py -k tester— all 66 tester-related prompt tests pass, including the newtest_tester_orientation_directs_scaffold_first..venv/bin/pytest orchestrator/tests/test_orient_prompts_babysit_pr.py— 47 tests pass (confirms babysit/PR-mode early-return is unaffected by the producer-orientation edit).make test— 15784 passed, 41 skipped. The 3 failures intests/llm/claude/test_runner.py(RuntimeError: There is no current event loop in thread 'MainThread') reproduce on stockmainand are unrelated to this PR (Python 3.14asyncio.get_event_loop()deprecation)..egg-state/brc-history/*-implement.jsonon the worktree — produces a clean per-pipeline table and aggregate summary.Notes for reviewers
scripts/scaffold_first_telemetry.py(snake_case to matchscripts/select_tests.py/scripts/validate_harness_parity.py) rather than as a hyphenated CLI — it's intended to be importable from tests, not invoked from CI.