Add tests to sources, snapshots and seeds when using TestBehavior.AFTER_EACH

[tatiana]

Previously Cosmos would only create a task group when using TestBehavior.AFTER_EACH for nodes of the type DbtResourceType.MODEL. This change adds the same behavior to snapshots and seeds.

For this to work as expected with sources, we would need to create a default operator to handle DbtResourceType.SOURCE, which is outside the scope of the current ticket. Once this operator exists, sources will also lead to creating a task group.

All the test selectors were tested successfully with dbt 1.6.

This screenshot illustrates the validation of this feature, with an adapted version of Jaffle Shop:

The modifications that were done to jaffle_shop were:

Appended the following lines to dev/dags/dbt/jaffle_shop/models/schema.yml:

seeds:
  - name: raw_customers
    description: Raw data from customers
    columns:
      - name: id
        tests:
          - unique
          - not_null

snapshots:
  - name: orders_snapshot
    description: Snapshot of orders
    columns:
      - name: orders_snapshot.order_id
        tests:
          - unique
          - not_null

And created the file dev/dags/dbt/jaffle_shop/snapshots/orders_snapshot.sql with:

{% snapshot orders_snapshot %}

{{
    config(
      target_database='postgres',
      target_schema='public',
      unique_key='order_id',

      strategy='timestamp',
      updated_at='order_date',
    )
}}

select * from {{ ref('jaffle_shop', 'orders') }}

{% endsnapshot %}

Closes: #474

[sweep-ai-deprecated]

Apply Sweep Rules to your PR?

• Apply: Leftover TODOs in the code should be handled.
• Apply: All new business logic should have corresponding unit tests in the tests/ directory.
• Apply: Any clearly inefficient or repeated code should be optimized or refactored.

[netlify]

👷 Deploy Preview for amazing-pothos-a3bca0 processing.

Name	Link
🔨 Latest commit	ec4ad48
🔍 Latest deploy log	https://app.netlify.com/sites/amazing-pothos-a3bca0/deploys/652efff32f4a9f0008ae4829

[codecov]

Codecov Report

All modified lines are covered by tests ✅

Comparison is base (a433f15) 93.13% compared to head (ec4ad48) 93.15%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #599      +/-   ##
==========================================
+ Coverage   93.13%   93.15%   +0.02%     
==========================================
  Files          51       51              
  Lines        2053     2059       +6     
==========================================
+ Hits         1912     1918       +6     
  Misses        141      141              

Files	Coverage Δ
cosmos/airflow/graph.py	100.00% <100.00%> (ø)
cosmos/constants.py	100.00% <100.00%> (ø)

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[harels]

lgtm

[harels]

do we have a test case that's covering this part?

[tatiana]

@harels, we have a unit that covers the path formation for all these variations:
test_create_test_task_metadata, and I manually played with dbt 1.6 to confirm all these variations work.

ATM, we don't have an integration test for DbtResourceType.SOURCE since our only example DAG that validates this type of node overrides any default behavior. For us to have an end-to-end version of the code going through this, Cosmos would need a default behavior for SOURCE nodes, something we don't currently have.

@harels @jlaneve Should we add a ticket to define a default behavior for Source nodes? What do you think that should look like? Should it check for freshness?

[tatiana]

I'm merging since the PR was approved, and the open discussion should only be closed once we decide whether to have a follow-up ticket.