Skip to content

Latest commit

 

History

History
328 lines (231 loc) · 8.91 KB

source-configs.md

File metadata and controls

328 lines (231 loc) · 8.91 KB
Raw
title description id
Source configurations
Learn how to use source configurations in dbt.
source-configs

import ConfigGeneral from '/snippets/_config-description-general.md';

Available configurations

Sources supports enabled and meta.

Sources configurations support enabled, event_time, and meta

General configurations

<Tabs groupId="config-languages" defaultValue="project-yaml" values={[ { label: 'Project file', value: 'project-yaml', }, { label: 'Property file', value: 'property-yaml', }, ] }>

sources:
  [<resource-path>](/reference/resource-configs/resource-path):
    [+](/reference/resource-configs/plus-prefix)[enabled](/reference/resource-configs/enabled): true | false
    [+](/reference/resource-configs/plus-prefix)[event_time](/reference/resource-configs/event-time): my_time_field
    [+](/reference/resource-configs/plus-prefix)[meta](/reference/resource-configs/meta):
      key: value

sources:
  [<resource-path>](/reference/resource-configs/resource-path):
    [+](/reference/resource-configs/plus-prefix)[enabled](/reference/resource-configs/enabled): true | false
    [+](/reference/resource-configs/plus-prefix)[meta](/reference/resource-configs/meta):
      key: value
version: 2

sources:
  - name: [<source-name>]
    [config](/reference/resource-properties/config):
      [enabled](/reference/resource-configs/enabled): true | false
      [event_time](/reference/resource-configs/event-time): my_time_field
      [meta](/reference/resource-configs/meta): {<dictionary>}

    tables:
      - name: [<source-table-name>]
        [config](/reference/resource-properties/config):
          [enabled](/reference/resource-configs/enabled): true | false
          [event_time](/reference/resource-configs/event-time): my_time_field
          [meta](/reference/resource-configs/meta): {<dictionary>}

version: 2

sources:
  - name: [<source-name>]
    [config](/reference/resource-properties/config):
      [enabled](/reference/resource-configs/enabled): true | false
      [meta](/reference/resource-configs/meta): {<dictionary>}
    tables:
      - name: [<source-table-name>]
        [config](/reference/resource-properties/config):
          [enabled](/reference/resource-configs/enabled): true | false
          [meta](/reference/resource-configs/meta): {<dictionary>}

Configuring sources

Sources can be configured via a config: block within their .yml definitions, or from the dbt_project.yml file under the sources: key. This configuration is most useful for configuring sources imported from a package.

You can disable sources imported from a package to prevent them from rendering in the documentation, or to prevent source freshness checks from running on source tables imported from packages.

  • Note: To disable a source table nested in a YAML file in a subfolder, you will need to supply the subfolder(s) within the path to that YAML file, as well as the source name and the table name in the dbt_project.yml file.

    The following example shows how to disable a source table nested in a YAML file in a subfolder:

    sources:
  your_project_name:
    subdirectory_name:
      source_name:
        source_table_name:
          +enabled: false
          +event_time: my_time_field
    ```yaml sources: your_project_name: subdirectory_name: source_name: source_table_name: +enabled: false ```

Examples

The following examples show how to configure sources in your dbt project.

— Disable all sources imported from a package
— Conditionally enable a single source
— Disable a single source from a package
— Configure a source with an event_time
— Configure meta to a source

Disable all sources imported from a package

To apply a configuration to all sources included from a package, state your configuration under the project name in the sources: config as a part of the resource path.

sources:
  events:
    +enabled: false

Conditionally enable a single source

When defining a source, you can disable the entire source, or specific source tables, using the inline config property:

version: 2

sources:
  - name: my_source
    config:
      enabled: true
    tables:
      - name: my_source_table  # enabled
      - name: ignore_this_one  # not enabled
        config:
          enabled: false

You can configure specific source tables, and use variables as the input to that configuration:

version: 2

sources:
  - name: my_source
    tables:
      - name: my_source_table
        config:
          enabled: "{{ var('my_source_table_enabled', false) }}"

Disable a single source from a package

To disable a specific source from another package, qualify the resource path for your configuration with both a package name and a source name. In this case, we're disabling the clickstream source from the events package.

sources:
  events:
    clickstream:
      +enabled: false

Similarly, you can disable a specific table from a source by qualifying the resource path with a package name, source name, and table name:

sources:
  events:
    clickstream:
      pageviews:
        +enabled: false

Configure a source with an event_time

Configuring an event_time for a source is only available in dbt Cloud Versionless or dbt Core versions 1.9 and later.

To configure a source with an event_time, specify the event_time field in the source configuration. This field is used to represent the actual timestamp of the event, rather than something like a loading date.

For example, if you had a source table called clickstream in the events source, you can use the timestamp for each event in the event_timestamp column as follows:

sources:
  events:
    clickstream:
      +event_time: event_timestamp

In this example, the event_time is set to event_timestamp, which has the exact time each clickstream event happened. Not only is this required for the incremental microbatching strategy, but when you compare data across CI and production environments, dbt will use event_timestamp to filter and match data by this event-based timeframe, ensuring that only overlapping timeframes are compared.

Configure meta to a source

Use the meta field to assign metadata information to sources. This is useful for tracking additional context, documentation, logging, and more.

For example, you can add meta information to a clickstream source to include information about the data source system:

sources:
  events:
    clickstream:
      +meta:
        source_system: "Google analytics"
        data_owner: "marketing_team"

Example source configuration

The following is a valid source configuration for a project with:

  • name: jaffle_shop
  • A package called events containing multiple source tables
name: jaffle_shop
config-version: 2
...
sources:
  # project names
  jaffle_shop:
    +enabled: true

  events:
    # source names
    clickstream:
      # table names
      pageviews:
        +enabled: false
      link_clicks:
        +enabled: true