#236: Airflow test harness + live e2e DAG + ops docs + example DAGs

[wjduenow]

Summary

Super plan for #236 — the closing test+docs child of Airflow epic #228.

Phase: detailing (awaiting approval)
Stories: 6 implementation + Quality Gate + Patterns & Memory
Decisions: 6 captured (DEC-001 … DEC-006)

Headline finding

~80% of #236's acceptance is already shipped by #229–#235 (subpackage, operators ×3, hook, drift, 877-line ops doc, 16-file gated test suite, CI airflow job, MkDocs nav). The plan is scoped to net-new gaps only (DEC-004) — no re-doing shipped work.

Net-new work

• Live e2e upgrade → real SignalForgeGenerateOperator + dag.test() (tmp AIRFLOW_HOME + airflow db migrate) on the Austin bikeshare fixture; asserts XCom tier counts + ≥1 always-passes drop (DEC-001).
• Example DAGs → rename drift_monitor → nightly_drift; add signalforge_after_dbt_build.py (DEC-002).
• Managed-runtime note (Astronomer/MWAA/Composer, "documented not certified") (DEC-005).
• SKILL.md Airflow pointer, parity-gate-safe (DEC-003).
• README v0.7 reword + CHANGELOG rollup (DEC-006 — dated Released-table move deferred to v0.7.0 release; flagged for confirmation).

Architecture review

One blocker resolved: dag.test() needs an initialized metadata DB → DEC-001 sets AIRFLOW_HOME + airflow db migrate. Rename blast-radius, packaging, docs-coherence all PASS.

Load-bearing constraints baked in

• .claude/rules/airflow-integration.md edits are orchestrator-only (workers can't write .claude/).
• Gated airflow tests are deselected in the default suite → workers validate offline; orchestrator certifies against .venv-airflow before close (the test+docs: fakesnow harness + FakeSnowflakeClient + gated live e2e (TPCH_SF1) + ops docs #124 live-debugging-pass budget).

Plan document

See plans/super/236-airflow-e2e-docs.md.

Next steps

• Review the plan in this PR
• Approve in Claude Code, then say "devolve" to create beads for Ralph

Summary by CodeRabbit

Release Notes

• New Features

  • Added an Airflow example for post-dbt build pruning.
  • Introduced a new Airflow “scheduled runs” documentation section with operator overview and credentials wiring.

• Changed

  • Renamed the nightly drift example DAG identifier to match updated guidance.

• Documentation

  • Expanded the Airflow integration guide, including managed runtime caveats and live e2e test harness requirements.

• Tests

  • Added gated Airflow end-to-end coverage using isolated DAG execution.
  • Updated DAG parsing/templating tests for the new and renamed examples; removed older live-gated test paths.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 994c149e-a4db-42c0-b575-84b571928a8b

📥 Commits

Reviewing files that changed from the base of the PR and between d368917 and f5fadc7.

📒 Files selected for processing (3)

• examples/airflow/signalforge_after_dbt_build.py
• src/signalforge/skills/signalforge/SKILL.md
• tests/airflow/test_e2e_generate_operator.py

✅ Files skipped from review due to trivial changes (1)

• src/signalforge/skills/signalforge/SKILL.md

🚧 Files skipped from review as they are similar to previous changes (2)

• tests/airflow/test_e2e_generate_operator.py
• examples/airflow/signalforge_after_dbt_build.py

---

📝 Walkthrough

Walkthrough

Closes Airflow epic #236 by renaming the drift example DAG (signalforge_drift_monitor → signalforge_nightly_drift), adding a new signalforge_after_dbt_build example DAG, replacing the inline live e2e test with a subprocess-isolated dag.test() harness in a new module, adding a managed-runtime documentation section, and propagating all updates across tests, docs, SKILL.md, README, CHANGELOG, and Claude rules.

Changes

Airflow Epic #236 Closeout

Layer / File(s)	Summary
DAG rename and new after-dbt-build example DAG examples/airflow/signalforge_nightly_drift.py, examples/airflow/signalforge_after_dbt_build.py	Renames the drift DAG id from signalforge_drift_monitor to signalforge_nightly_drift. Adds signalforge_after_dbt_build with env-var/Variable config resolution, _on_flagged validation, and a BashOperator(dbt build) >> SignalForgePruneExistingOperator task chain with templated model, schema, and as_of.
DAG parse and template-render tests tests/airflow/test_dag_parse.py	Adds DagBag parse and render_template_fields tests for signalforge_after_dbt_build. Renames the drift parse test to the nightly-drift variant and updates the loaded dag_id. Removes the previously inlined live-e2e helper and test, which moved to a dedicated module.
Subprocess-isolated dag.test() live e2e harness tests/airflow/_e2e_generate_driver.py, tests/airflow/test_e2e_generate_operator.py	Adds _e2e_generate_driver.py as a child-process script that builds a one-task DAG, runs it via dag.test(), and emits __SF_E2E_RESULT__ JSON to stdout. Adds the gated parent pytest that isolates AIRFLOW_HOME, migrates the metadata DB, rewrites profiles.yml, spawns the driver, and asserts task state == "success", XCom tier-count structure, and dropped >= 1.
Managed-runtime ops docs docs/airflow-ops.md	Updates worked-example and test-expectation references to signalforge_nightly_drift. Adds a new "Managed Airflow runtimes (Astronomer/MWAA/Composer)" section covering version constraints and secrets-backend guidance. Extends the Caveats section with a managed-runtime pointer.
Cross-cutting doc and rules updates src/signalforge/skills/signalforge/SKILL.md, README.md, CHANGELOG.md, .claude/rules/airflow-integration.md	Adds a "Scheduled runs (Airflow)" section to SKILL.md. Updates the README v0.7 roadmap entry to name all three operators and the hook. Adds CHANGELOG entries for the DAG rename and epic closure. Updates the airflow-integration rules doc with epic-closer guidance, renamed DAG references, and the new example DAG.
Super plan document plans/super/236-airflow-e2e-docs.md	Adds the complete #236 planning document: shipped-inventory, net-new gap list, architecture decisions DEC-001..DEC-006, US-001..US-008 story specs, rules-compliance gate, and the ordered Beads Manifest.

Sequence Diagram(s)

sequenceDiagram
  participant pytest as test_e2e_generate_operator
  participant driver as _e2e_generate_driver
  participant airflow as dag.test()
  participant op as SignalForgeGenerateOperator
  participant bq as BigQuery

  pytest->>pytest: skip if Airflow missing or env vars absent
  pytest->>pytest: copy fixture to tmp_path and isolate AIRFLOW_HOME
  pytest->>pytest: migrate DB and rewrite profiles.yml
  pytest->>driver: subprocess.run with project_dir arg
  driver->>airflow: DAG sf_e2e_generate .test()
  airflow->>op: execute write=False on_flagged=succeed
  op->>bq: query stg_bikeshare_trips
  op-->>airflow: xcom_push kept/kept_uncertain/dropped/flagged
  airflow-->>driver: task instance state
  driver-->>pytest: stdout SF_E2E_RESULT JSON with state and xcom
  pytest->>pytest: assert state==success and dropped>=1

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related issues

• Airflow: test harness + gated live e2e DAG + ops docs + example DAGs #236 — This PR directly implements and closes the full scope of epic #236: DAG renames, new example DAGs, gated dag.test() live e2e harness, and consolidated Airflow operations documentation.
• Epic: Airflow operator — scheduled schema-drift / signal-rot detection #228 — This PR completes the final child issue (#236) of the broader Airflow operator epic (#228), shipping the last remaining gaps (test harness, managed-runtime docs, example DAGs).
• Airflow: maintainer live-certification of the generate dag.test() e2e (#236 follow-up) #247 — Issue #247 is a direct follow-on that certifies the artifacts created here: the renamed nightly drift DAG, the new after-dbt-build DAG parse tests, and the subprocess driver and live e2e test.

Possibly related PRs

• wjduenow/SignalForge#238: This PR extends the tests/airflow gating infrastructure and DAG parse test patterns that PR #238 originally established.
• wjduenow/SignalForge#241: The new e2e harness directly exercises SignalForgeGenerateOperator and asserts its XCom tier-count contract, which was introduced in PR #241.
• wjduenow/SignalForge#245: The drift DAG rename and updated parse test assertions in this PR are directly tied to the SignalForgeDriftOperator surface covered by PR #245.

Suggested labels

airflow

🐇 The drift DAG got a brand new name,
And after_dbt_build joined the game!
A subprocess hops through dag.test() with care,
JSON hops out — XCom tiers are there!
Managed runtimes? Documented, not certified, fair! 🌟

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main changes: adding a live e2e test harness for Airflow, new example DAGs, and operations documentation updates.
Docstring Coverage	✅ Passed	Docstring coverage is 80.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[Copilot]

Pull request overview

This PR closes out Airflow epic #228’s final “test+docs” child (#236) by adding a canonical gated live E2E that runs a real SignalForgeGenerateOperator via dag.test(), extending example DAG coverage (rename drift example + add post-dbt-build prune example), and updating ops/docs surfaces to reflect the completed integration.

Changes:

• Added a gated live E2E Airflow test that runs an inline DAG through dag.test() in a subprocess with an isolated AIRFLOW_HOME + migrated metadata DB.
• Added/updated example DAGs (signalforge_nightly_drift rename + new signalforge_after_dbt_build) and expanded DAG-parse + templated-field render tests.
• Updated docs/plan/README/CHANGELOG/SKILL pointers to reflect the shipped Airflow integration and managed-runtime caveats.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tests/airflow/test_e2e_generate_operator.py	New gated live E2E test driving SignalForgeGenerateOperator via dag.test()
tests/airflow/_e2e_generate_driver.py	Subprocess driver to run dag.test() under isolated AIRFLOW_HOME
tests/airflow/test_dag_parse.py	Adds parse + render_template_fields coverage for the new/renamed example DAGs
examples/airflow/signalforge_after_dbt_build.py	New post-dbt-build prune-existing example DAG
examples/airflow/signalforge_nightly_drift.py	Renames drift monitor DAG id to signalforge_nightly_drift
docs/airflow-ops.md	Adds managed-runtime (“documented, not certified”) guidance + updates drift DAG references
src/signalforge/skills/signalforge/SKILL.md	Adds an Airflow pointer section for scheduled runs
README.md	Rewords the v0.7 roadmap row to reflect the landed Airflow integration
CHANGELOG.md	Adds #236 rollup entries (rename + epic closeout)
plans/super/236-airflow-e2e-docs.md	Adds the detailed super-plan document for #236
.claude/rules/airflow-integration.md	Updates rules to reflect the new epic-closer patterns and renamed DAG

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[wjduenow]

PR Review Summary

Addressed the Copilot review. All fixes in f5fadc7; the two E501 flags are false positives.

Fixed (3 items)

File	Line	Issue	Fix
tests/airflow/test_e2e_generate_operator.py	81	Live gate treated any non-empty value as enabled (SF_RUN_BQ=0 would run a paid test)	_live_skip_reason now requires SF_RUN_* ∈ _TRUTHY and .strip()s the secrets — mirrors test_e2e_bigquery_smoke.py
src/signalforge/skills/signalforge/SKILL.md	212	"grading existing tests" is wrong — SignalForgePruneExistingOperator is no-LLM	Reworded to "no-LLM … prunes your existing dbt tests … (no API key)"
examples/airflow/signalforge_after_dbt_build.py	146	bash_command interpolated _project_dir unquoted	shlex.quote(_project_dir)

False Positives (2 items)

File	Line	Claim	Why it's correct
tests/airflow/test_dag_parse.py	212	E501 (>100)	Line is 99 chars; ruff check + CI lint-test are green
tests/airflow/_e2e_generate_driver.py	78	E501 (>100)	Line is 75 chars; ruff check + CI lint-test are green

Validation after fixes: default suite 4119 passed, pyright 0 errors; gated airflow suite 77 passed / 1 skipped (the live e2e self-skips without creds) vs real Airflow 2.10.4.