#234: Airflow SignalForgeHook — Connection/Variable → profiles.yml + LLM key

[wjduenow]

Summary

Implements #234 (epic #228) — SignalForgeHook(BaseHook) keyed on signalforge_conn_id, mapping an Airflow Connection (+ optional Variable) → (profiles_dir, provider, LLM API key), wired through both SignalForgeGenerateOperator and SignalForgePruneExistingOperator.

Ralph-executed from plans/super/234-signalforge-hook.md (16 DECs). This PR began as the plan-revision PR and now carries the full implementation.

Stories landed

• US-001 PROVIDER_ENV_VAR_KEYS closed allowlist in signalforge.llm.providers
• US-002 logger grep-gate extended to signalforge.airflow
• US-003 airflow-free pure resolver resolve_connection + HookResolution + _ConnectionExtra (100% ungated)
• US-004 real SignalForgeHook(BaseHook) (deferred construction) + register_secret/airflow_variable_get in the one shim
• US-005 wire signalforge_conn_id → GenerateOperator + shared _merge_with_resolution / _provider_key_env helpers
• US-006 wire signalforge_conn_id → PruneExistingOperator (profiles_dir only — read-only, no key injection)
• US-007 docs/airflow-ops.md hook section + examples/airflow/signalforge_hook_dag.py (A8)
• US-008 Quality Gate (4 review passes; fixes: doc param-table rows + Generate conn_id=None no-hook test)
• US-009 Patterns & Memory (airflow-integration.md § SignalForgeHook)

Secrets hygiene (4 surfaces, each test-pinned)

Key never in: task logs (mask_secret + lazy-format JSON), XCom (counts+paths only), rendered templates (signalforge_conn_id ∉ template_fields), or __repr__.

Validation

ruff + ruff format + pyright (0 errors) + pytest (3987 passed) green. Gated airflow tests certified against the real .venv-airflow rig (airflow 2.10.4): 58 passed.

Summary by CodeRabbit

Release Notes

• New Features
  • Added signalforge_conn_id support to Airflow operators for connection-driven credentials and precedence with operator params.
  • Introduced SignalForgeHook for resolving connection-based credentials with Airflow Variable fallback and redacted output.
  • Improved secrets safety by masking resolved API keys in logs during operator runs.
• Documentation
  • Expanded the Airflow integration guide with SignalForgeHook + signalforge_conn_id setup, validation, and generate vs prune credential behavior.
  • Added an example DAG demonstrating shared connection-based configuration.

[coderabbitai]

📝 Walkthrough

Walkthrough

Implements SignalForgeHook (#234): a pure Airflow-free resolver (HookResolution + resolve_connection) with strict extra="forbid" validation, a lazy PEP 562-based hook class, a new PROVIDER_ENV_VAR_KEYS constant, register_secret/airflow_variable_get compat shim helpers, signalforge_conn_id parameter on both operators (with environment-variable injection and secrets masking for Generate, profiles-dir-only resolution for PruneExisting), an example DAG, comprehensive tests, and updated documentation and rules.

Changes

SignalForgeHook credential seam

Layer / File(s)	Summary
PROVIDER_ENV_VAR_KEYS constant src/signalforge/llm/providers.py, tests/llm/test_providers.py	Adds PROVIDER_ENV_VAR_KEYS dict mapping canonical provider names (anthropic, openai, gemini) to API-key env-var names, exported via __all__, tested for key-set completeness, UPPER_SNAKE format, and expected values.
Airflow compat shim helpers src/signalforge/airflow/_airflow_compat.py	Adds register_secret (wraps mask_secret for log redaction) and airflow_variable_get (wraps Variable.get with str|None coercion), both confining from airflow imports to function bodies, exported via __all__.
Pure HookResolution resolver src/signalforge/airflow/_resolve.py, tests/airflow/test_resolve.py	New Airflow-free module: HookResolution frozen dataclass with redacted __repr__, _ConnectionExtra Pydantic model with extra="forbid", and resolve_connection that validates extra, enforces provider allowlist, resolves api_key from password or Variable fallback, and canonicalises profiles_dir with containment check. Pure-Python tests cover all branches without importing Airflow.
Lazy-loaded SignalForgeHook src/signalforge/airflow/hooks.py, tests/airflow/test_skeleton.py, tests/airflow/test_hooks.py	Replaces the NotImplementedError stub with PEP 562 __getattr__ returning either a ModuleNotFoundError-raising placeholder (Airflow absent) or a cached real BaseHook subclass whose get_conn delegates to resolve_connection. Tests cover get_conn resolution paths, Variable fallback, error cases, and secrets hygiene (log redaction, masking, repr redaction).
signalforge_conn_id integration in both operators src/signalforge/airflow/operators.py, tests/airflow/test_operators.py, tests/airflow/test_operators_helpers.py	Adds _merge_with_resolution (precedence), _provider_key_env context manager (env injection/restore), and _resolve_hook (lazy resolution) helpers. Generate operator conditionally resolves connection, validates provider+api_key, masks key, merges profiles_dir/cache_scope, injects env var for run duration (with exception-safe restore). PruneExisting resolves profiles_dir only, no env/key injection. Tests cover injection, restore, precedence, XCom key-absence, batch iteration, and error paths.
Example DAG and DAG parse test examples/airflow/signalforge_hook_dag.py, tests/airflow/test_dag_parse.py	New example DAG wires both operators with signalforge_conn_id="signalforge_default"; _config helper resolves settings from environment, Variable, or default with graceful fallback; on_flagged validated at DAG-parse time. DAG parse test asserts two expected task IDs are present.
Docs, rules, and plan updates docs/airflow-ops.md, .claude/rules/airflow-integration.md, .claude/rules/cli-layer.md, .claude/rules/diff-renderer.md, plans/super/234-signalforge-hook.md, tests/llm/test_logger_grep_gate.py	Adds signalforge_conn_id params and full "Airflow-native credentials" section to airflow-ops.md (connection setup, extra validation, API-key precedence, provider allowlist, secrets hygiene). Updates .claude rules documenting SignalForgeHook design constraints and expanding logger grep gate to cover airflow subpackage. Updates super plan for two-operator signalforge_conn_id wiring.

Sequence Diagram(s)

sequenceDiagram
  participant DAG as Airflow DAG
  participant Op as SignalForgeGenerateOperator.execute
  participant ResolveHook as _resolve_hook
  participant Hook as SignalForgeHook.get_conn
  participant Resolver as resolve_connection
  participant Compat as _airflow_compat
  participant EnvCtx as _provider_key_env
  participant Run as run_signalforge

  DAG->>Op: execute(context)
  Op->>ResolveHook: signalforge_conn_id set?
  ResolveHook->>Hook: SignalForgeHook(conn_id).get_conn()
  Hook->>Resolver: resolve_connection(conn, variable_lookup)
  Resolver-->>Hook: HookResolution(provider, api_key, profiles_dir)
  Hook-->>ResolveHook: HookResolution
  ResolveHook-->>Op: HookResolution
  Op->>Op: validate provider + api_key present
  Op->>Compat: register_secret(api_key)
  Op->>Op: _merge_with_resolution(profiles_dir, cache_scope)
  Op->>EnvCtx: inject PROVIDER_ENV_VAR=api_key
  EnvCtx->>Run: run_signalforge(argv)
  Run-->>EnvCtx: TaskOutcome
  EnvCtx->>EnvCtx: restore prior env value
  EnvCtx-->>Op: TaskOutcome
  Op-->>DAG: XCom payload (no api_key)

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related issues

• Airflow: SignalForgeHook — Connection/Variable → profiles.yml + LLM key #234: This PR directly implements the complete scope of SignalForgeHook (#234) including the pure resolver, lazy hook class, signalforge_conn_id integration on both operators, secrets masking, and comprehensive test coverage.

Possibly related PRs

• wjduenow/SignalForge#75: Both PRs modify tests/llm/test_logger_grep_gate.py _SCAN_SUBPACKAGES; this PR adds airflow to the scanned directory set for f-string logger detection.
• wjduenow/SignalForge#241: Both PRs modify SignalForgeGenerateOperator in operators.py; this PR extends execute() with signalforge_conn_id-driven connection resolution, environment-variable injection, and HookResolution precedence merging.
• wjduenow/SignalForge#239: This PR's SignalForgeHook implementation and _airflow_compat.py extensions build directly on the lazy/placeholder shim skeleton established in that PR.

Suggested labels

airflow

🐇 A hook was born, Airflow-free at its core,
Keys masked in the logs — no secrets to deplore!
get_conn runs pure, the env is restored,
_ConnectionExtra forbids what's ignored.
With Variable fallback and frozen delight,
This rabbit hops happy — credentials done right! 🔑✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and specifically describes the main change: implementation of SignalForgeHook for Apache Airflow that maps Airflow Connections/Variables to SignalForge configuration (profiles, LLM provider, API key).
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[Copilot]

Pull request overview

Implements issue #234 by adding an Airflow-native credential seam (SignalForgeHook) that resolves an Airflow Connection (+ optional Variable fallback) into (profiles_dir, provider, api_key), and wiring signalforge_conn_id through both SignalForgeGenerateOperator (LLM + env-key injection) and SignalForgePruneExistingOperator (profiles-dir only, no key injection). This extends the existing Airflow integration (epic #228) while preserving the “no eager Airflow import” confinement pattern.

Changes:

• Added PROVIDER_ENV_VAR_KEYS as the shared, closed allowlist mapping provider → API-key env var.
• Introduced an Airflow-free pure resolver (resolve_connection) + typed HookResolution, plus the gated SignalForgeHook(BaseHook) wrapper.
• Wired signalforge_conn_id through both operators (including masking + scoped env injection for generate only), and updated docs + examples + tests accordingly.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/llm/test_providers.py	Adds unit tests pinning PROVIDER_ENV_VAR_KEYS and its export/shape.
tests/llm/test_logger_grep_gate.py	Extends logger grep gate scan to include signalforge.airflow.
tests/airflow/test_skeleton.py	Updates ungated skeleton tests to validate Airflow-free attribute access + construction-time ImportError.
tests/airflow/test_resolve.py	Adds ungated tests covering all branches of the pure Connection/Variable resolver.
tests/airflow/test_operators.py	Adds gated tests for signalforge_conn_id wiring, masking, env injection/restore, and no-leak surfaces.
tests/airflow/test_operators_helpers.py	Adds ungated tests for _merge_with_resolution and _provider_key_env.
tests/airflow/test_hooks.py	Adds gated tests for real hook behavior against Airflow Connection + Variable shim + masking.
tests/airflow/test_dag_parse.py	Ensures the new example DAG parses cleanly and asserts expected tasks.
src/signalforge/llm/providers.py	Introduces PROVIDER_ENV_VAR_KEYS and exports it via __all__.
src/signalforge/airflow/operators.py	Adds shared conn-id helpers, masks secrets, injects provider env var for generate, and wires conn-id through both operators.
src/signalforge/airflow/hooks.py	Implements deferred-construction SignalForgeHook with a gated BaseHook subclass and an airflow-missing placeholder.
src/signalforge/airflow/_resolve.py	Adds the airflow-free resolver + Pydantic-validated extra schema and redacting HookResolution.
src/signalforge/airflow/_airflow_compat.py	Adds shim-confined register_secret and airflow_variable_get.
plans/super/234-signalforge-hook.md	Updates implementation plan/DEC log reflecting the final shipped shape (incl. prune-existing wiring).
examples/airflow/signalforge_hook_dag.py	Adds an example DAG using a single Connection + Variable to configure both operators.
docs/airflow-ops.md	Documents signalforge_conn_id, Connection/Variable precedence, allowlist, hygiene guarantees, and example usage.
.claude/rules/diff-renderer.md	Updates logger-gate documentation to reflect the expanded scan set.
.claude/rules/cli-layer.md	Updates logger-gate documentation to reflect the expanded scan set and process.
.claude/rules/airflow-integration.md	Documents the hook/resolver patterns, leak-surface disciplines, and provider env-var allowlist.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[wjduenow]

PR Review Summary

Fixed (1 item)

File	Location	Issue	Commit
airflow/hooks.py + airflow/operators.py	get_conn / _resolve_hook / both execute() call sites	DEC-008 containment was dead in the wiring. _resolve_hook → get_conn called resolve_connection(project_dir=None), so a Connection extra.profiles_dir was never symlink-hardened / contained against the operator's project_dir. Now get_conn(*, project_dir=None) accepts the anchor and both operators thread self.project_dir through _resolve_hook(conn_id, self.project_dir). Added a gated regression test (test_single_model_conn_id_threads_project_dir_anchor) asserting the anchor reaches the resolver.	5c21d0a

False Positives (0 items)

None. CodeRabbit reported no actionable comments. 🎉

Validation

ruff + ruff format + pyright (0 errors) + pytest (3987 passed) green; gated airflow tests re-certified against the real .venv-airflow rig (airflow 2.10.4): 59 passed.