Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 15 additions & 13 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -254,9 +254,10 @@ jobs:
# The locked policy decision: --baseline-manifest is a required CLI
# argument; omitting it is a clap usage error (NOT a PreflightError).
# This job makes that policy a structural CI invariant — every feature
# scenario invoking the CLI must pass --baseline-manifest, except
# scenarios explicitly tagged @no-baseline-usage-error (the one
# intentional exception that exercises the usage-error path itself).
# scenario invoking the CLI must pass exactly one scope source
# (--baseline-manifest OR --scope-from-pr-diff, cute-dbt#85), except
# scenarios explicitly tagged @no-baseline-usage-error (the deliberate
# neither-source case that exercises the usage-error path itself).
name: Baseline-required scenario invariant
runs-on: ubuntu-latest
timeout-minutes: 5
Expand All @@ -276,14 +277,14 @@ jobs:
/^[[:space:]]*@no-baseline-usage-error([[:space:]]|$)/ { pending_skip=1; next }
/^[[:space:]]*Scenario(:|[[:space:]]+Outline)/ { skip=pending_skip; pending_skip=0; next }
/cute-dbt with --manifest/ && !skip {
if ($0 !~ /--baseline-manifest/) {
if ($0 !~ /--baseline-manifest/ && $0 !~ /--scope-from-pr-diff/) {
print FILENAME ":" NR ": " $0
found=1
}
}
END { exit (found ? 1 : 0) }
' features/*.feature; then
echo "::error::baseline-required-grep: scenarios invoking the CLI without --baseline-manifest must be tagged @no-baseline-usage-error"
echo "::error::baseline-required-grep: scenarios invoking the CLI without a scope source (--baseline-manifest or --scope-from-pr-diff) must be tagged @no-baseline-usage-error"
exit 1
fi

Expand All @@ -293,6 +294,7 @@ jobs:
# - 7 after unit_test_format_coverage (cute-dbt#39)
# - 8 after unit_test_yaml (cute-dbt#69)
# - 9 after consumer_report_contract (cute-dbt#71)
# - 10 after pr_diff_scoping (cute-dbt#84)
# The README is not a feature. Display-label drift (`==6`) is
# tracked by cute-dbt#68 (rename coordination with branch protection
# required).
Expand All @@ -307,8 +309,8 @@ jobs:
run: |
set -euo pipefail
count=$(find features -maxdepth 1 -name '*.feature' | wc -l | tr -d ' ')
if [ "$count" != "9" ]; then
echo "::error::feature-count: expected 9 .feature files, found $count"
if [ "$count" != "10" ]; then
echo "::error::feature-count: expected 10 .feature files, found $count"
exit 1
fi

Expand Down Expand Up @@ -639,14 +641,14 @@ jobs:

bdd:
# ATDD outer loop — the executable acceptance contract for the
# five `features/*.feature` files. Uses cucumber-rs `harness = false`
# `features/*.feature` files. Uses cucumber-rs `harness = false`
# so cargo nextest is INCOMPATIBLE; the binary is invoked via
# `cargo test --test bdd`. The `feature-count` job pins the
# `.feature` file count at 5; the `baseline-required-grep` job
# pins the `--baseline-manifest` invariant per-scenario; this job
# is the step-definition gate (every scenario has a matching step,
# all 27 scenarios pass). No untrusted GitHub event interpolation
# in any `run:` block.
# `.feature` file count; the `baseline-required-grep` job pins the
# exactly-one-scope-source invariant per-scenario; this job is the
# step-definition gate (every scenario has a matching step, all
# scenarios pass). No untrusted GitHub event interpolation in any
# `run:` block.
name: BDD outer loop (cucumber-rs, cargo test --test bdd)
runs-on: ubuntu-latest
timeout-minutes: 15
Expand Down
181 changes: 181 additions & 0 deletions features/pr_diff_scoping.feature
Original file line number Diff line number Diff line change
@@ -0,0 +1,181 @@
# Maps: NEW v0.1.x capability — `--scope-from-pr-diff` CLI flag
Comment thread
cmbays marked this conversation as resolved.
# Pipeline: cute-dbt-20260527-team-pr-review-ergonomics (Shape E, Phase 1)
# Companion to features/diff_scoping.feature (baseline-manifest path)
#
# Gherkin convention: `<changed-files>` in a When clause is a harness
# placeholder that resolves to the literal list configured by the prior
# `Given a list of changed files containing …` step. It is NOT a CLI
# argument literal. Scenario 14 uses the `@changed.txt` form instead — a
# real `@file` argument resolved to a temp file written by its Given.
#
# Manifest construction: every model / unit test a scenario needs is
# built in-memory by its own Given steps (no committed fixture files —
# the synthetic-only-fixture invariant is satisfied trivially). A bare
# model name is derived from the `.sql` file stem of its
# original_file_path (`models/marts/core/dim_payers.sql` → `dim_payers`).
Feature: Diff-scope unit tests and models via PR file diff (CI path)
As an analytics engineer setting up cute-dbt in a PR-review workflow
I want cute-dbt to derive in-scope models from the PR's file diff
So that no baseline-manifest publishing job is needed in CI

Background:
Given a compiled dbt 1.8+ manifest "current.json" with unit tests

# --- Happy paths ---

Scenario: A PR that changes one model file puts that model in scope
Given a list of changed files containing "models/marts/core/dim_payers.sql"
And the manifest contains a model with original_file_path "models/marts/core/dim_payers.sql"
And the model "dim_payers" has a unit test "test_dim_payers_injects_unknown_sentinel"
When I run cute-dbt with --manifest current.json --scope-from-pr-diff <changed-files> --project-root . --out report.html
Then the exit code is 0
And the rendered report's models-in-scope listing contains "dim_payers"
And the rendered report contains a CTE diagram for "dim_payers"
And the rendered report's test rows include "test_dim_payers_injects_unknown_sentinel"

Scenario: A PR that changes only unit-test YAML puts that test in scope
Given a list of changed files containing "models/marts/core/_core__models.yml"
And the manifest contains a model with original_file_path "models/marts/core/dim_payers.sql"
And the model "dim_payers" has a unit test "test_dim_payers_injects_unknown_sentinel" declared in "models/marts/core/_core__models.yml"
When I run cute-dbt with --manifest current.json --scope-from-pr-diff <changed-files> --project-root . --out report.html
Then the exit code is 0
And the rendered report's test rows include "test_dim_payers_injects_unknown_sentinel"

Scenario: A PR with multiple changed files puts all matching nodes in scope
Given a list of changed files containing "models/marts/core/dim_payers.sql" and "models/marts/analytics/mart_dq_summary.sql"
And the manifest contains a model with original_file_path "models/marts/core/dim_payers.sql"
And the manifest contains a model with original_file_path "models/marts/analytics/mart_dq_summary.sql"
When I run cute-dbt with --manifest current.json --scope-from-pr-diff <changed-files> --project-root . --out report.html
Then the exit code is 0
And the rendered report's models-in-scope listing contains both "dim_payers" and "mart_dq_summary"
And the rendered report contains a CTE diagram for "dim_payers"
And the rendered report contains a CTE diagram for "mart_dq_summary"

# CPO finding — proves SCOPING, not just selection.
Scenario: An unchanged sibling model is NOT in the rendered report
Given a list of changed files containing "models/marts/core/dim_payers.sql"
And the manifest contains a model with original_file_path "models/marts/core/dim_payers.sql"
And the manifest also contains an unchanged model "stg_customers" with a unit test "test_stg_customers_unique"
When I run cute-dbt with --manifest current.json --scope-from-pr-diff <changed-files> --project-root . --out report.html
Then the exit code is 0
And the rendered report's models-in-scope listing contains "dim_payers"
And the rendered report's models-in-scope listing does NOT contain "stg_customers"
And the rendered report does NOT contain a test row for "test_stg_customers_unique"

Scenario: A modified model with zero unit tests is in models_in_scope (explorer mode)
Given a list of changed files containing "models/staging/stg_payments.sql"
And the manifest contains a model with original_file_path "models/staging/stg_payments.sql"
When I run cute-dbt with --manifest current.json --scope-from-pr-diff <changed-files> --project-root . --out report.html
Then the exit code is 0
And the rendered report shows "stg_payments" with the "no unit tests wired" empty state

# --- Project-root path rewriting (load-bearing for the Action wrapper) ---

Scenario: --project-root rewrites PR-diff paths so a sub-directory dbt project is in scope
Given a list of changed files containing "dbt_project/models/marts/core/dim_payers.sql"
And the manifest (compiled with project root "dbt_project") contains a model with original_file_path "models/marts/core/dim_payers.sql"
When I run cute-dbt with --manifest current.json --scope-from-pr-diff <changed-files> --project-root dbt_project --out report.html
Then the exit code is 0
And the rendered report's models-in-scope listing contains "dim_payers"

# --- Zero-scope and non-mapping paths ---

Scenario: A PR with no dbt-relevant changes produces an empty in-scope report
Given a list of changed files containing only "README.md" and ".github/workflows/ci.yml"
When I run cute-dbt with --manifest current.json --scope-from-pr-diff <changed-files> --project-root . --out report.html
Then the exit code is 0
And the rendered report shows the "0 unit tests in scope" banner
And the rendered report contains no CTE diagrams

Scenario: A changed path that doesn't map to any manifest node is silently skipped
Given a list of changed files containing "models/deleted_model.sql"
And the manifest has no node with original_file_path "models/deleted_model.sql"
When I run cute-dbt with --manifest current.json --scope-from-pr-diff <changed-files> --project-root . --out report.html
Then the exit code is 0
And the rendered report shows the "0 unit tests in scope" banner

# Renamed-model behavior is a documented fidelity limit (PR-diff sees the
# rename as one deleted path + one added path; the deleted path maps to no
# current-manifest node, the added path maps to the new node).
# tracked: cute-dbt#80 — restoration condition: if rename detection becomes
# load-bearing, layer a git-rename signal on top of `git diff --name-only`.
# Not blocking v0.1.x.
Comment thread
cmbays marked this conversation as resolved.

# --- Mutual exclusivity at the clap usage layer ---

Scenario: Passing both --scope-from-pr-diff and --baseline-manifest is a clap usage error
Given a baseline manifest "baseline.json"
And a list of changed files containing "models/marts/core/dim_payers.sql"
When I run cute-dbt with --manifest current.json --baseline-manifest baseline.json --scope-from-pr-diff <changed-files> --project-root . --out report.html
Then the exit code is non-zero
And no file "report.html" is written
And stderr explains exactly one of --scope-from-pr-diff or --baseline-manifest must be provided

@no-baseline-usage-error
Scenario: Passing neither --scope-from-pr-diff nor --baseline-manifest is a clap usage error
When I run cute-dbt with --manifest current.json --project-root . --out report.html
Then the exit code is non-zero
And no file "report.html" is written
And stderr explains exactly one of --scope-from-pr-diff or --baseline-manifest must be provided

# --- Fail-closed contract (Stage 2 unchanged) ---

Scenario: An in-scope model with compiled_code:null fail-closes with NotCompiled
Given a list of changed files containing "models/marts/core/dim_payers.sql"
And the manifest contains a model with no compiled SQL at "models/marts/core/dim_payers.sql"
And the model "dim_payers" has a unit test "test_dim_payers_injects_unknown_sentinel"
When I run cute-dbt with --manifest current.json --scope-from-pr-diff <changed-files> --project-root . --out report.html
Then the exit code is non-zero
And no file "report.html" is written
And stderr names "dim_payers" as the offending node and recommends running "dbt compile"

# --- v0.1 fidelity limits (documented behavior, not defects) ---
Comment thread
cmbays marked this conversation as resolved.

Scenario: A change to only the YAML config block is NOT in scope via PR-diff (documented limit)
# PR-diff sees only the YAML file path; cute-dbt does not parse the YAML
# diff to distinguish a config-block change from a unit_tests-block change.
# Adopters needing config-aware scoping use --baseline-manifest instead.
Given a list of changed files containing only "dbt_project.yml"
When I run cute-dbt with --manifest current.json --scope-from-pr-diff <changed-files> --project-root . --out report.html
Then the exit code is 0
And the rendered report shows the "0 unit tests in scope" banner

Scenario: A `dbt deps` change that updates package checksums is NOT in scope via PR-diff (documented limit)
# PR-diff sees only packages.yml change; no dbt SQL files change.
# Adopters needing dependency-aware scoping use --baseline-manifest instead.
Given a list of changed files containing only "packages.yml"
When I run cute-dbt with --manifest current.json --scope-from-pr-diff <changed-files> --project-root . --out report.html
Then the exit code is 0
And the rendered report shows the "0 unit tests in scope" banner

# --- Source-of-diff: @file argument (CI path; Fix A1) ---
#
# The original scenario tested a GITHUB_EVENT_PATH JSON contract; GitHub's
# PR `event.json` does not carry the changed-file list in production, so
# the workflow computes the diff and hands cute-dbt a file (one path per
# line) via the real `@file` argument form. See observations.md (Fix A1).

Scenario: cute-dbt reads the changed file list from a file argument
Given a list of changed files containing "models/marts/core/dim_payers.sql"
And the manifest contains a model with original_file_path "models/marts/core/dim_payers.sql"
And the model "dim_payers" has a unit test "test_dim_payers_injects_unknown_sentinel"
And the changed-files list is written to a file "changed.txt" one path per line
When I run cute-dbt with --manifest current.json --scope-from-pr-diff @changed.txt --project-root . --out report.html
Then the exit code is 0
And the rendered report's test rows include "test_dim_payers_injects_unknown_sentinel"

# Scenario: `git diff --name-only origin/main...HEAD` fallback when GITHUB_EVENT_PATH is unset
# → Exercised by integration test `tests/changed_files_provider.rs`
# (cucumber-rs subprocess harness cannot cleanly stage a throwaway git
# repo with two commits per scenario; the provider's @file path is
# unit-tested in Rust against a controlled fixture instead). This
# comment preserves the contract; the BDD does not duplicate it.

# --- Path-normalization mutation kill ---
# The file-path → manifest-node mapping must handle leading "./" and
# path-separator normalization (Windows-style separators are NOT
# supported in v0.1; document and unit-test in Rust).
Comment thread
cmbays marked this conversation as resolved.
# `tests/path_matching.rs` is the Rust unit suite covering the exact-match
# / leading-"./" / project-root-strip cases (cute-dbt#81). The BDD asserts
# the user-visible behavior (model X appears in scope); the Rust suite
# kills mutants on the path-matching function.
23 changes: 13 additions & 10 deletions lefthook.yml
Original file line number Diff line number Diff line change
Expand Up @@ -50,9 +50,11 @@ pre-push:
timeout: 30s
baseline-required-grep:
# Mirror the `baseline-required-grep` CI job. Every feature line
# invoking the CLI must pass --baseline-manifest unless explicitly
# tagged @no-baseline-usage-error (the one intentional exception).
# Makes the locked gate decision a structural invariant.
# invoking the CLI must pass exactly one scope source
# (--baseline-manifest OR --scope-from-pr-diff, cute-dbt#85) unless
# explicitly tagged @no-baseline-usage-error (the deliberate
# neither-source case). Makes the locked gate decision a structural
# invariant.
run: |
set -euo pipefail
# Gherkin tag semantics: a tag line applies to the NEXT scenario.
Expand All @@ -61,27 +63,28 @@ pre-push:
/^[[:space:]]*@no-baseline-usage-error([[:space:]]|$)/ { pending_skip=1; next }
/^[[:space:]]*Scenario(:|[[:space:]]+Outline)/ { skip=pending_skip; pending_skip=0; next }
/cute-dbt with --manifest/ && !skip {
if ($0 !~ /--baseline-manifest/) {
if ($0 !~ /--baseline-manifest/ && $0 !~ /--scope-from-pr-diff/) {
print FILENAME ":" NR ": " $0
found=1
}
}
END { exit (found ? 1 : 0) }
' features/*.feature; then
echo "baseline-required-grep: scenarios invoking the CLI without --baseline-manifest must be tagged @no-baseline-usage-error"
echo "baseline-required-grep: scenarios invoking the CLI without a scope source (--baseline-manifest or --scope-from-pr-diff) must be tagged @no-baseline-usage-error"
exit 1
fi
timeout: 15s
feature-count:
# Asserts the 9-feature invariant (bumped from 8 in cute-dbt#71 when
# consumer_report_contract.feature joined the contract; previously
# bumped from 7 in cute-dbt#69 when unit_test_yaml.feature joined).
# Asserts the 10-feature invariant (bumped from 9 in cute-dbt#84 when
# pr_diff_scoping.feature joined the contract; previously bumped from
# 8 in cute-dbt#71 when consumer_report_contract.feature joined, and
# from 7 in cute-dbt#69 when unit_test_yaml.feature joined).
# Mirrors the `feature-count` CI job; the README is not a feature.
run: |
set -euo pipefail
count=$(find features -maxdepth 1 -name '*.feature' | wc -l | tr -d ' ')
if [ "$count" != "9" ]; then
echo "feature-count: expected 9 .feature files, found $count"
if [ "$count" != "10" ]; then
echo "feature-count: expected 10 .feature files, found $count"
exit 1
fi
timeout: 5s
Expand Down
2 changes: 1 addition & 1 deletion tests/bdd.rs
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
//! enumerates the kill-detection test targets so `bdd` is excluded
//! from per-mutant runs.
//!
//! The feature files under `features/` are pinned at 5 by the
//! The feature-file count under `features/` is pinned by the
//! `feature-count` CI job; every scenario in those files has a step
//! definition here, organised one module per feature file under
//! `tests/steps/`.
Expand Down
Loading
Loading