Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add documentation for unit testing incremental models #4959

Merged
merged 10 commits into from
Feb 22, 2024
67 changes: 67 additions & 0 deletions website/docs/docs/build/unit-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -254,5 +254,72 @@ unit_tests:

```

## Unit testing incremental models

When configuring your unit test, you can override the output of macros, vars, or environment variables for a given unit test. This enables you to unit test your incremental models - both in "full refresh" and "incremental" mode.
graciegoheen marked this conversation as resolved.
Show resolved Hide resolved

When testing an incremental model, the expected output is the __result of the materialization__ (i.e. what will be merged/inserted), not the resulting model itself (i.e. what the final table will look like after the merge/insert).
graciegoheen marked this conversation as resolved.
Show resolved Hide resolved

Let's say I have an incremental model in my project:
graciegoheen marked this conversation as resolved.
Show resolved Hide resolved

<file name='my_incremental_model.sql'>
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved

```sql

{{
config(
materialized='incremental'
)
}}

select * from {{ ref('events') }}
{% if is_incremental() %}
where event_time > (select max(event_time) from {{ this }})
{% endif %}

```

I can define unit tests on `my_incremental_model` to ensure my incremental logic is working as expected:
graciegoheen marked this conversation as resolved.
Show resolved Hide resolved

```yml

unit_tests:
- name: my_incremental_model_full_refresh_mode
model: my_incremental_model
overrides:
macros:
# unit test this model in "full refresh" mode
is_incremental: false
given:
- input: ref('events')
rows:
- {event_id: 1, event_time: 2020-01-01}
expect:
rows:
- {event_id: 1, event_time: 2020-01-01}

- name: my_incremental_model_incremental_mode
model: my_incremental_model
overrides:
macros:
# unit test this model in "incremental" mode
is_incremental: true
given:
- input: ref('events')
rows:
- {event_id: 1, event_time: 2020-01-01}
- {event_id: 2, event_time: 2020-01-02}
- {event_id: 3, event_time: 2020-01-03}
- input: this
# contents of current my_incremental_model
rows:
- {event_id: 1, event_time: 2020-01-01}
expect:
# what will be inserted/merged into my_incremental_model
rows:
- {event_id: 2, event_time: 2020-01-02}
- {event_id: 3, event_time: 2020-01-03}

```

There is currently no way to unit test whether the dbt framework actually inserted/merged the records into your existing model correctly or not - but [we're investigating a way to support this in the future](https://github.com/dbt-labs/dbt-core/issues/8664).
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
4 changes: 4 additions & 0 deletions website/docs/reference/resource-properties/unit-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,10 @@ When `format: csv`, can either supply:
- If you do supply an input for a seed, we will use that input instead.
- You can also have “empty” inputs, by setting rows to an empty list `rows: []`

### Overrides

When configuring your unit test, you can override the output of macros, vars, or environment variables for a given unit test.
Copy link
Collaborator Author

@graciegoheen graciegoheen Feb 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should probably have more examples / info here - but added this as a stub for now

added to follow up issue #4889


## Examples
```yml

Expand Down
Loading