Skip to content

Commit

Permalink
update source/snapshots
Browse files Browse the repository at this point in the history
  • Loading branch information
mirnawong1 committed Oct 30, 2024
1 parent 33a66a8 commit 6ebf5eb
Show file tree
Hide file tree
Showing 5 changed files with 201 additions and 9 deletions.
6 changes: 3 additions & 3 deletions website/docs/docs/deploy/advanced-ci.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,12 +32,12 @@ dbt reports the comparison differences in:
<Lightbox src="/img/docs/dbt-cloud/example-ci-compare-changes-tab.png" width="85%" title="Example of the Compare tab" />

### Considerations
You can set a [event_time](/reference/resource-configs/event-time) for a model, seed, snapshot, or source to represent the actual timestamp of the event, rather than something like loading date. This configuration is important for certain features, such as [Incremental microbatch](/docs/build/incremental-microbatch) and compare changes in CI/CD workflows, where it ensures the same time-slice of data is accurately compared between your CI and production environments.
You can set the [`event_time`](/reference/resource-configs/event-time) config for a model, seed, snapshot, or source to represent the actual timestamp of the event, rather than something like loading date. This configuration is important for certain features, such as [Incremental microbatch](/docs/build/incremental-microbatch) and compare changes in CI/CD workflows, where it ensures the same time-slice of data is accurately compared between your CI and production environments.

When configured, `event_time` enables compare changes to:

- Compare data in CI vs. production for overlapping times only, reducing false discrepancies.
- Handle scenarios where CI contains fresher data than production by using only the overlapping timeframe, which avoids incorrect row-count changes.
- Compare data in CI versus production for overlapping times only, reducing false discrepancies.
- Handle scenarios where CI has "fresher" data than production by using only the overlapping timeframe, allowing you to avoid incorrect row-count changes.
- Account for subset data builds in CI without flagging filtered-out rows as "deleted" when compared with production.
- Coming soon, you'll be able to add a flag to the `dbt compare` command to select the specific time slice to compare.

Expand Down
4 changes: 2 additions & 2 deletions website/docs/reference/resource-configs/event-time.md
Original file line number Diff line number Diff line change
Expand Up @@ -139,9 +139,9 @@ sources:
## Definition
Set the `event_time` to the name of the field that represents the timestamp of the event, as opposed to a date like data loading date. You can configure `event_time` for a [model](/docs/build/models), [seed](/docs/build/seeds), or [source](/docs/build/sources) in your `dbt_project.yml` file, property YAML file, or config block.
Set the `event_time` to the name of the field that represents the timestamp of the event, as opposed to a date like data loading date. You can configure `event_time` for a [model](/docs/build/models), [seed](/docs/build/seeds), [snapshot](/docs/build/snapshots), or [source](/docs/build/sources) in your `dbt_project.yml` file, property YAML file, or config block.

`event_time` is required for [Incremental mircrobatch](/docs/build/incremental-microbatch) and [Advanced CI compare changes](/docs/deploy/advanced-ci) in CI/CD workflows, where it ensures the same time-slice of data is correctly compared between your CI and production environments.
`event_time` is required for [Incremental mircrobatch](/docs/build/incremental-microbatch) and [Advanced CI compare changes](/docs/deploy/advanced-ci#considerations) in CI/CD workflows, where it ensures the same time-slice of data is correctly compared between your CI and production environments.

When you configure `event_time`, it enables compare changes to:

Expand Down
50 changes: 49 additions & 1 deletion website/docs/reference/seed-configs.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,8 @@ seeds:

<File name='dbt_project.yml'>

<VersionBlock lastVersion="1.8">

```yaml
seeds:
[<resource-path>](/reference/resource-configs/resource-path):
Expand All @@ -95,7 +97,28 @@ seeds:
[+](/reference/resource-configs/plus-prefix)[grants](/reference/resource-configs/grants): {<dictionary>}

```
</VersionBlock>

<VersionBlock firstVersion="1.9">

```yaml
seeds:
[<resource-path>](/reference/resource-configs/resource-path):
[+](/reference/resource-configs/plus-prefix)[enabled](/reference/resource-configs/enabled): true | false
[+](/reference/resource-configs/plus-prefix)[tags](/reference/resource-configs/tags): <string> | [<string>]
[+](/reference/resource-configs/plus-prefix)[pre-hook](/reference/resource-configs/pre-hook-post-hook): <sql-statement> | [<sql-statement>]
[+](/reference/resource-configs/plus-prefix)[post-hook](/reference/resource-configs/pre-hook-post-hook): <sql-statement> | [<sql-statement>]
[+](/reference/resource-configs/plus-prefix)[database](/reference/resource-configs/database): <string>
[+](/reference/resource-configs/plus-prefix)[schema](/reference/resource-properties/schema): <string>
[+](/reference/resource-configs/plus-prefix)[alias](/reference/resource-configs/alias): <string>
[+](/reference/resource-configs/plus-prefix)[persist_docs](/reference/resource-configs/persist_docs): <dict>
[+](/reference/resource-configs/plus-prefix)[full_refresh](/reference/resource-configs/full_refresh): <boolean>
[+](/reference/resource-configs/plus-prefix)[meta](/reference/resource-configs/meta): {<dictionary>}
[+](/reference/resource-configs/plus-prefix)[grants](/reference/resource-configs/grants): {<dictionary>}
[+](/reference/resource-configs/plus-prefix)[event_time](/reference/resource-configs/event-time): my_time_field

```
</VersionBlock>
</File>

</TabItem>
Expand All @@ -105,6 +128,8 @@ seeds:

<File name='seeds/properties.yml'>

<VersionBlock firstVersion="1.9">

```yaml
version: 2

Expand All @@ -122,13 +147,36 @@ seeds:
[full_refresh](/reference/resource-configs/full_refresh): <boolean>
[meta](/reference/resource-configs/meta): {<dictionary>}
[grants](/reference/resource-configs/grants): {<dictionary>}
[event_time](/reference/resource-configs/event-time): my_time_field

```
</VersionBlock>

<VersionBlock lastVersion="1.8">

```yaml
version: 2

seeds:
- name: [<seed-name>]
config:
[enabled](/reference/resource-configs/enabled): true | false
[tags](/reference/resource-configs/tags): <string> | [<string>]
[pre_hook](/reference/resource-configs/pre-hook-post-hook): <sql-statement> | [<sql-statement>]
[post_hook](/reference/resource-configs/pre-hook-post-hook): <sql-statement> | [<sql-statement>]
[database](/reference/resource-configs/database): <string>
[schema](/reference/resource-properties/schema): <string>
[alias](/reference/resource-configs/alias): <string>
[persist_docs](/reference/resource-configs/persist_docs): <dict>
[full_refresh](/reference/resource-configs/full_refresh): <boolean>
[meta](/reference/resource-configs/meta): {<dictionary>}
[grants](/reference/resource-configs/grants): {<dictionary>}
```
</VersionBlock>
</File>
</TabItem>

</Tabs>
## Configuring seeds
Expand Down
22 changes: 20 additions & 2 deletions website/docs/reference/snapshot-configs.md
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,24 @@ Configurations can be applied to snapshots using [YAML syntax](/docs/build/snaps

<File name='dbt_project.yml'>

<VersionBlock firstVersion="1.9">

```yaml
snapshots:
[<resource-path>](/reference/resource-configs/resource-path):
[+](/reference/resource-configs/plus-prefix)[enabled](/reference/resource-configs/enabled): true | false
[+](/reference/resource-configs/plus-prefix)[tags](/reference/resource-configs/tags): <string> | [<string>]
[+](/reference/resource-configs/plus-prefix)[alias](/reference/resource-configs/alias): <string>
[+](/reference/resource-configs/plus-prefix)[pre-hook](/reference/resource-configs/pre-hook-post-hook): <sql-statement> | [<sql-statement>]
[+](/reference/resource-configs/plus-prefix)[post-hook](/reference/resource-configs/pre-hook-post-hook): <sql-statement> | [<sql-statement>]
[+](/reference/resource-configs/plus-prefix)[persist_docs](/reference/resource-configs/persist_docs): {<dict>}
[+](/reference/resource-configs/plus-prefix)[grants](/reference/resource-configs/grants): {<dict>}
[+](/reference/resource-configs/plus-prefix)[event_time](/reference/resource-configs/event-time): my_time_field
```
</VersionBlock>
<VersionBlock lastVersion="1.8">
```yaml
snapshots:
[<resource-path>](/reference/resource-configs/resource-path):
Expand All @@ -179,6 +197,7 @@ snapshots:
[+](/reference/resource-configs/plus-prefix)[persist_docs](/reference/resource-configs/persist_docs): {<dict>}
[+](/reference/resource-configs/plus-prefix)[grants](/reference/resource-configs/grants): {<dict>}
```
</VersionBlock>
</File>
</TabItem>
Expand Down Expand Up @@ -225,6 +244,7 @@ snapshots:
[post-hook](/reference/resource-configs/pre-hook-post-hook): <sql-statement> | [<sql-statement>]
[persist_docs](/reference/resource-configs/persist_docs): {<dict>}
[grants](/reference/resource-configs/grants): {<dictionary>}
[event_time](/reference/resource-configs/event-time): my_time_field
```
</File>
Expand Down Expand Up @@ -292,7 +312,6 @@ The following examples demonstrate how to configure snapshots using the `dbt_pro
<File name='dbt_project.yml'>

```yml
snapshots:
+unique_key: id
```
Expand All @@ -307,7 +326,6 @@ The following examples demonstrate how to configure snapshots using the `dbt_pro
<File name='dbt_project.yml'>

```yml
snapshots:
jaffle_shop:
+unique_key: id
Expand Down
128 changes: 127 additions & 1 deletion website/docs/reference/source-configs.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,17 @@ import ConfigGeneral from '/snippets/_config-description-general.md';

## Available configurations

Sources only support one configuration, [`enabled`](/reference/resource-configs/enabled).
<VersionBlock lastVersion="1.8">

Sources supports [`enabled`](/reference/resource-configs/enabled) and [`meta`](/reference/resource-configs/meta).

</VersionBlock>

<VersionBlock firstVersion="1.9">

Sources configurations support [`enabled`](/reference/resource-configs/enabled), [`event_time`](/reference/resource-configs/event-time), and [`meta`](/reference/resource-configs/meta)

</VersionBlock>

### General configurations

Expand All @@ -27,12 +37,29 @@ Sources only support one configuration, [`enabled`](/reference/resource-configs/

<File name='dbt_project.yml'>

<VersionBlock firstVersion="1.9">

```yaml
sources:
[<resource-path>](/reference/resource-configs/resource-path):
[+](/reference/resource-configs/plus-prefix)[enabled](/reference/resource-configs/enabled): true | false
[+](/reference/resource-configs/plus-prefix)[event_time](/reference/resource-configs/event-time): my_time_field
[+](/reference/resource-configs/plus-prefix)[meta](/reference/resource-configs/meta):
key: value

```
</VersionBlock>

<VersionBlock lastVersion="1.8">

```yaml
sources:
[<resource-path>](/reference/resource-configs/resource-path):
[+](/reference/resource-configs/plus-prefix)[enabled](/reference/resource-configs/enabled): true | false
[+](/reference/resource-configs/plus-prefix)[meta](/reference/resource-configs/meta):
key: value
```
</VersionBlock>
</File>
Expand All @@ -43,19 +70,46 @@ sources:
<File name='models/properties.yml'>
<VersionBlock firstVersion="1.9">
```yaml
version: 2

sources:
- name: [<source-name>]
[config](/reference/resource-properties/config):
[enabled](/reference/resource-configs/enabled): true | false
[event_time](/reference/resource-configs/event-time): my_time_field
[meta](/reference/resource-configs/meta): {<dictionary>}

tables:
- name: [<source-table-name>]
[config](/reference/resource-properties/config):
[enabled](/reference/resource-configs/enabled): true | false
[event_time](/reference/resource-configs/event-time): my_time_field
[meta](/reference/resource-configs/meta): {<dictionary>}

```
</VersionBlock>

<VersionBlock lastVersion="1.8">

```yaml
version: 2

sources:
- name: [<source-name>]
[config](/reference/resource-properties/config):
[enabled](/reference/resource-configs/enabled): true | false
[meta](/reference/resource-configs/meta): {<dictionary>}
tables:
- name: [<source-table-name>]
[config](/reference/resource-properties/config):
[enabled](/reference/resource-configs/enabled): true | false
[meta](/reference/resource-configs/meta): {<dictionary>}

```
</VersionBlock>

</File>

Expand All @@ -74,18 +128,43 @@ You can disable sources imported from a package to prevent them from rendering i

<File name='dbt_project.yml'>

<VersionBlock firstVersion="1.9">

```yaml
sources:
your_project_name:
subdirectory_name:
source_name:
source_table_name:
+enabled: false
+event_time: my_time_field
```
</VersionBlock>
<VersionBlock lastVersion="1.8">
```yaml
sources:
your_project_name:

Check warning on line 148 in website/docs/reference/source-configs.md

View workflow job for this annotation

GitHub Actions / vale

[vale] website/docs/reference/source-configs.md#L148

[custom.Typos] Oops there's a typo -- did you really mean 'your_project_name'?
Raw output
{"message": "[custom.Typos] Oops there's a typo -- did you really mean 'your_project_name'? ", "location": {"path": "website/docs/reference/source-configs.md", "range": {"start": {"line": 148, "column": 5}}}, "severity": "WARNING"}
subdirectory_name:

Check warning on line 149 in website/docs/reference/source-configs.md

View workflow job for this annotation

GitHub Actions / vale

[vale] website/docs/reference/source-configs.md#L149

[custom.Typos] Oops there's a typo -- did you really mean 'subdirectory_name'?
Raw output
{"message": "[custom.Typos] Oops there's a typo -- did you really mean 'subdirectory_name'? ", "location": {"path": "website/docs/reference/source-configs.md", "range": {"start": {"line": 149, "column": 7}}}, "severity": "WARNING"}
source_name:

Check warning on line 150 in website/docs/reference/source-configs.md

View workflow job for this annotation

GitHub Actions / vale

[vale] website/docs/reference/source-configs.md#L150

[custom.Typos] Oops there's a typo -- did you really mean 'source_name'?
Raw output
{"message": "[custom.Typos] Oops there's a typo -- did you really mean 'source_name'? ", "location": {"path": "website/docs/reference/source-configs.md", "range": {"start": {"line": 150, "column": 9}}}, "severity": "WARNING"}
source_table_name:

Check warning on line 151 in website/docs/reference/source-configs.md

View workflow job for this annotation

GitHub Actions / vale

[vale] website/docs/reference/source-configs.md#L151

[custom.Typos] Oops there's a typo -- did you really mean 'source_table_name'?
Raw output
{"message": "[custom.Typos] Oops there's a typo -- did you really mean 'source_table_name'? ", "location": {"path": "website/docs/reference/source-configs.md", "range": {"start": {"line": 151, "column": 11}}}, "severity": "WARNING"}
+enabled: false
```
</VersionBlock>
</File>
### Examples
The following examples show how to configure sources in your dbt project.
&mdash; [Disable all sources imported from a package](#disable-all-sources-imported-from-a-package) <br />
&mdash; [Conditionally enable a single source](#conditionally-enable-a-single-source) <br />
&mdash; [Disable a single source from a package](#disable-a-single-source-from-a-package) <br />
&mdash; [Configure a source with an `event_time`](#configure-a-source-with-an-event_time) <br />
&mdash; [Configure meta to a source](#configure-meta-to-a-source) <br />

#### Disable all sources imported from a package
To apply a configuration to all sources included from a [package](/docs/build/packages),
state your configuration under the [project name](/reference/project-configs/name.md) in the
Expand Down Expand Up @@ -172,6 +251,53 @@ sources:
</File>


#### Configure a source with an `event_time`

<VersionBlock lastVersion="1.8">

Configuring an [`event_time`](/reference/resource-configs/event-time) for a source is only available in dbt Cloud Versionless or dbt Core versions 1.9 and later.

</VersionBlock>

<VersionBlock firstVersion="1.9">

To configure a source with an `event_time`, specify the `event_time` field in the source configuration. This field is used to represent the actual timestamp of the event, rather than something like a loading date.

For example, if you had a source table called `clickstream` in the `events` source, you can use the timestamp for each event in the `event_timestamp` column as follows:

<File name='dbt_project.yml'>

```yaml
sources:
events:
clickstream:
+event_time: event_timestamp
```
</File>

In this example, the `event_time` is set to `event_timestamp`, which has the exact time each clickstream event happened.
Not only is this required for the [incremental microbatching strategy]((/docs/build/incremental-microbatch)), but when you compare data across [CI and production](/docs/deploy/advanced-ci#considerations) environments, dbt will use `event_timestamp` to filter and match data by this event-based timeframe, ensuring that only overlapping timeframes are compared.

</VersionBlock>

#### Configure meta to a source

Use the `meta` field to assign metadata information to sources. This is useful for tracking additional context, documentation, logging, and more.

For example, you can add `meta` information to a `clickstream` source to include information about the data source system:

<File name='dbt_project.yml'>

```yaml
sources:
events:
clickstream:
+meta:
source_system: "Google analytics"
data_owner: "marketing_team"
```
</File>

## Example source configuration
The following is a valid source configuration for a project with:
* `name: jaffle_shop`
Expand Down

0 comments on commit 6ebf5eb

Please sign in to comment.