#179: empirical retest writeup against intuit_airflow - #182
Merged
Conversation
Phase A binary (weekly_query_cost.sql) + Phase B aggregate (15 candidates) against the substrate built for the 2026-05-30 baseline. #170 ships well- calibrated (100% match-rate, beats projection); #169 misses the projection by 18.9 pp on a 14-model sample with a named root cause (missing _ROW_COUNT_BETWEEN_SCOPE_INSTRUCTION in prompts.py per DEC-012); #171 has a drafter-calibration bug visible on every model with audit-timestamp columns. One additional ops-class finding: 170-col model blows the safety-layer audit-record size cap. Three follow-ons named for separate filing; Phase C (#171 calibration retest) deferred per epic option 3 — needs maintainer-controlled Snowflake fixture warehouse.
📝 WalkthroughWalkthroughA comprehensive research write-up documenting Issue ChangesIssue
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~3 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Comment |
This was referenced Jun 2, 2026
Closed
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/research/179-test-primitive-expansion-retest.md`:
- Around line 123-142: The fenced code blocks that begin with "5
unique_combination (+ row_count_between):", "ERROR: LLM response violated the
anchor contract...", and "PER-MODEL BREAKDOWN" are missing language identifiers
(MD040); update each opening triple-backtick to include a language tag like text
(e.g. change ``` to ```text) so markdownlint stops flagging them, and apply the
same fix to the other occurrences referenced (the similar blocks around the
other ranges) by locating the blocks by their starting lines shown above and
adding the language identifier to the opening fence for each.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 949d57dc-7081-4413-ab4b-c03e37ea5550
📒 Files selected for processing (1)
docs/research/179-test-primitive-expansion-retest.md
Comment on lines
+123
to
+142
| ``` | ||
| 5 unique_combination (+ row_count_between): | ||
| raw/taxday_auction_insights (16 cols, 9 spaced/quoted) | ||
| analytical/tvp_yelp (58 cols) | ||
| analytical/core_hourly_performance (16 cols, 8 spaced/quoted) | ||
| raw/taxday_mappings ( 7 cols) | ||
| raw/googleads_auction_insights (15 cols, 9 spaced/quoted) | ||
|
|
||
| 10 row_count_between only: | ||
| reporting/data_store_test_control (61 cols, 11 spaced/quoted) | ||
| operational/map_cid_sa360 (16 cols) | ||
| analytical/fiscal_season (12 cols) | ||
| raw/query_history (21 cols) | ||
| raw/concord_sku (12 cols) | ||
| raw/aio (14 cols) | ||
| analytical/calendar_hour ( 8 cols) | ||
| ingress/yelp_business_metrics_stg (35 cols) | ||
| reporting/cid_all_concord_sku (43 cols, 22 spaced/quoted) | ||
| raw/datashare_googlead (170 cols) | ||
| ``` |
There was a problem hiding this comment.
Add explicit language identifiers to fenced code blocks.
These three fenced blocks are missing language tags (MD040), which will keep markdown lint noisy. Please annotate them (for example, text for lists/logs).
Suggested patch
-```
+```text
5 unique_combination (+ row_count_between):
raw/taxday_auction_insights (16 cols, 9 spaced/quoted)
analytical/tvp_yelp (58 cols)
analytical/core_hourly_performance (16 cols, 8 spaced/quoted)
raw/taxday_mappings ( 7 cols)
raw/googleads_auction_insights (15 cols, 9 spaced/quoted)
@@
reporting/cid_all_concord_sku (43 cols, 22 spaced/quoted)
raw/datashare_googlead (170 cols)- +text
ERROR: LLM response violated the anchor contract (4 violation(s)).
- column test on column='creation_ts' references None
- test references nonexistent column None (available: [...])
- column test on column='update_ts' references None
- test references nonexistent column None (available: [...])
-```
+```text
PER-MODEL BREAKDOWN
taxday_auction_insights row_count_between shape-missed
taxday_auction_insights unique_combination shape-matched column-set exact match
tvp_yelp row_count_between shape-matched
@@
AGGREGATE
row_count_between matched= 8 missed= 6 failed= 1 evaluable= 14/15 coverage= 57.1%
unique_combination matched= 5 missed= 0 failed= 0 evaluable= 5/5 coverage= 100.0%
</details>
Also applies to: 150-156, 193-219
<details>
<summary>🧰 Tools</summary>
<details>
<summary>🪛 markdownlint-cli2 (0.22.1)</summary>
[warning] 123-123: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
</details>
</details>
<details>
<summary>🤖 Prompt for AI Agents</summary>
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @docs/research/179-test-primitive-expansion-retest.md around lines 123 - 142,
The fenced code blocks that begin with "5 unique_combination (+
row_count_between):", "ERROR: LLM response violated the anchor contract...", and
"PER-MODEL BREAKDOWN" are missing language identifiers (MD040); update each
opening triple-backtick to include a language tag like text (e.g. change ``` to
other occurrences referenced (the similar blocks around the other ranges) by
locating the blocks by their starting lines shown above and adding the language
identifier to the opening fence for each.
This was referenced Jun 2, 2026
This was referenced Jun 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Close-out artefact for epic #179 (Test Generation Expansion).
What this PR adds
docs/research/179-test-primitive-expansion-retest.md(404 lines) — theempirical retest of #169 / #170 / #171 against the
intuit_airflowsubstrate originally stood up for the 2026-05-30 baseline measurement.
Results
Phase A (binary, weekly_query_cost.sql):
unique_combinationPASS — structureddbt_utils.unique_combination_of_columnsreplaced the baseline's freeformGROUP BY HAVING.row_count_betweenFAIL — drafter skipped despite the model being the literal DEC-012 worked-example fixture. Root cause:_ROW_COUNT_BETWEEN_SCOPE_INSTRUCTIONnever landed inprompts.py.Phase B (aggregate, 15 candidates):
row_count_between(#169)unique_combination(#170)Findings to be filed as separate follow-on issues
_ROW_COUNT_BETWEEN_SCOPE_INSTRUCTION+ DEC-012 worked example (Phase A FAIL root cause).row_count_anomaly_by_periodto date columns on every model with audit timestamps (Add row_count_anomaly_by_period: drafter-proposed historical-trend row-count guard #171 calibration bug; reproduced 3/3 in Phase B v1/v2 before workaround).AuditRecordTooLargeErrorblocksgenerateon wide-table models (datashare_googlead, 170 cols → 4,519-byte record vs. 4,000-byte cap fromsafety-layer.mdDEC-011).Phase C (#171 calibration retest)
Deferred — requires a maintainer-controlled Snowflake fixture warehouse per the epic's option 3.
Cost + wall
~17 drafter + 4,044 grade calls; ~$3–7 LLM spend; ~95 min wall-clock.
Summary by CodeRabbit