From c3bcfb6745df6995de1656a6aa78187931e94c49 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Mon, 3 Jun 2024 15:34:42 -0400 Subject: [PATCH 1/3] Deprecating v1.4 from the docs --- website/dbt-versions.js | 8 -- website/docs/docs/build/custom-aliases.md | 34 ------ website/docs/docs/build/exposures.md | 8 -- .../docs/docs/build/incremental-strategy.md | 4 - website/docs/docs/build/packages.md | 4 - website/docs/docs/build/python-models.md | 2 +- .../connect-data-platform/snowflake-setup.md | 8 +- website/docs/faqs/Git/gitignore.md | 107 ------------------ website/docs/faqs/Project/why-version-2.md | 7 -- .../docs/faqs/Troubleshooting/gitignore.md | 43 ------- .../docs/reference/artifacts/manifest-json.md | 2 +- website/docs/reference/commands/list.md | 10 -- .../reference/dbt-jinja-functions/builtins.md | 16 --- .../reference/dbt-jinja-functions/graph.md | 49 -------- website/docs/reference/dbt_project.yml.md | 6 - website/docs/reference/events-logging.md | 10 +- website/docs/reference/exposure-properties.md | 4 - .../docs/reference/global-configs/cache.md | 33 ------ website/docs/reference/global-configs/logs.md | 12 -- .../reference/global-configs/print-output.md | 6 - .../docs/reference/node-selection/defer.md | 10 -- .../docs/reference/node-selection/methods.md | 33 ------ .../docs/reference/node-selection/syntax.md | 7 -- .../node-selection/test-selection-examples.md | 72 ------------ .../node-selection/yaml-selectors.md | 52 --------- .../resource-configs/bigquery-configs.md | 11 +- .../docs/reference/resource-configs/group.md | 30 ----- .../resource-configs/singlestore-configs.md | 4 - .../resource-configs/snowflake-configs.md | 4 +- 29 files changed, 11 insertions(+), 585 deletions(-) diff --git a/website/dbt-versions.js b/website/dbt-versions.js index dc2df094918..c456bda8d8e 100644 --- a/website/dbt-versions.js +++ b/website/dbt-versions.js @@ -15,10 +15,6 @@ exports.versions = [ version: "1.5", EOLDate: "2024-04-27", }, - { - version: "1.4", - EOLDate: "2024-01-25", - }, ] exports.versionedPages = [ @@ -150,10 +146,6 @@ exports.versionedPages = [ "page": "reference/resource-properties/versions", "firstVersion": "1.5", }, - { - "page": "reference/dbt-jinja-functions/local-md5", - "firstVersion": "1.4", - }, { "page": "reference/resource-configs/on_configuration_change", "firstVersion": "1.6", diff --git a/website/docs/docs/build/custom-aliases.md b/website/docs/docs/build/custom-aliases.md index 8d54dd1a841..9e9f91f968d 100644 --- a/website/docs/docs/build/custom-aliases.md +++ b/website/docs/docs/build/custom-aliases.md @@ -73,31 +73,6 @@ To override dbt's alias name generation, create a macro named `generate_alias_na The default implementation of `generate_alias_name` simply uses the supplied `alias` config (if present) as the model alias, otherwise falling back to the model name. This implementation looks like this: - - - - -```jinja2 -{% macro generate_alias_name(custom_alias_name=none, node=none) -%} - - {%- if custom_alias_name is none -%} - - {{ node.name }} - - {%- else -%} - - {{ custom_alias_name | trim }} - - {%- endif -%} - -{%- endmacro %} - -``` - - - - - @@ -176,18 +151,9 @@ If these models should indeed have the same database identifier, you can work ar #### Model versions - - -New in v1.5 - - - - - **Related documentation:** - [Model versions](/docs/collaborate/govern/model-versions) - [`versions`](/reference/resource-properties/versions#alias) By default, dbt will create versioned models with the alias `_v`, where `` is that version's unique identifier. You can customize this behavior just like for non-versioned models by configuring a custom `alias` or re-implementing the `generate_alias_name` macro. - diff --git a/website/docs/docs/build/exposures.md b/website/docs/docs/build/exposures.md index 000d6ed571a..bcbe819d98c 100644 --- a/website/docs/docs/build/exposures.md +++ b/website/docs/docs/build/exposures.md @@ -12,8 +12,6 @@ Exposures make it possible to define and describe a downstream use of your dbt p Exposures are defined in `.yml` files nested under an `exposures:` key. - - ```yaml @@ -42,8 +40,6 @@ exposures: - - ### Available properties _Required:_ @@ -51,13 +47,9 @@ _Required:_ - **type**: one of `dashboard`, `notebook`, `analysis`, `ml`, `application` (used to organize in docs site) - **owner**: `name` or `email` required; additional properties allowed - - _Expected:_ - **depends_on**: list of refable nodes, including `metric`, `ref`, and `source`. While possible, it is highly unlikely you will ever need an `exposure` to depend on a `source` directly. - - _Optional:_ - **label**: May contain spaces, capital letters, or special characters. - **url**: Activates and populates the link to **View this exposure** in the upper right corner of the generated documentation site diff --git a/website/docs/docs/build/incremental-strategy.md b/website/docs/docs/build/incremental-strategy.md index 267ec040ca7..e1a67f93da7 100644 --- a/website/docs/docs/build/incremental-strategy.md +++ b/website/docs/docs/build/incremental-strategy.md @@ -128,8 +128,6 @@ select ... - - ### About incremental_predicates `incremental_predicates` is an advanced use of incremental models, where data volume is large enough to justify additional investments in performance. This config accepts a list of any valid SQL expression(s). dbt does not check the syntax of the SQL statements. @@ -208,8 +206,6 @@ The syntax depends on how you configure your `incremental_strategy`: - There's a decent amount of conceptual overlap with the `insert_overwrite` incremental strategy. ::: - - ### Built-in strategies Before diving into [custom strategies](#custom-strategies), it's important to understand the built-in incremental strategies in dbt and their corresponding macros: diff --git a/website/docs/docs/build/packages.md b/website/docs/docs/build/packages.md index aae8cc94361..b3705effcce 100644 --- a/website/docs/docs/build/packages.md +++ b/website/docs/docs/build/packages.md @@ -145,8 +145,6 @@ To find the latest release for a package, navigate to the `Releases` tab in the As of v0.14.0, dbt will warn you if you install a package using the `git` syntax without specifying a version (see below). - - ### Internally hosted tarball URL Some organizations have security requirements to pull resources only from internal services. To address the need to install packages from hosted environments such as Artifactory or cloud storage buckets, dbt Core enables you to install packages from internally-hosted tarball URLs. @@ -160,8 +158,6 @@ packages: Where `name: 'dbt_utils'` specifies the subfolder of `dbt_packages` that's created for the package source code to be installed within. - - ### Private packages #### SSH Key Method (Command Line only) diff --git a/website/docs/docs/build/python-models.md b/website/docs/docs/build/python-models.md index 6be1e16357b..0a089d263c7 100644 --- a/website/docs/docs/build/python-models.md +++ b/website/docs/docs/build/python-models.md @@ -256,7 +256,7 @@ def model(dbt, session): ### Materializations Python models support these materializations: -- `table` (default) +- `table` (default) - `incremental` Incremental Python models support all the same [incremental strategies](/docs/build/incremental-strategy) as their SQL counterparts. The specific strategies supported depend on your adapter. As an example, incremental models are supported on BigQuery with Dataproc for the `merge` incremental strategy; the `insert_overwrite` strategy is not yet supported. diff --git a/website/docs/docs/core/connect-data-platform/snowflake-setup.md b/website/docs/docs/core/connect-data-platform/snowflake-setup.md index 510f2f27791..24ad6f88d62 100644 --- a/website/docs/docs/core/connect-data-platform/snowflake-setup.md +++ b/website/docs/docs/core/connect-data-platform/snowflake-setup.md @@ -56,7 +56,7 @@ my-snowflake-db: connect_timeout: 10 # default: 10 retry_on_database_errors: False # default: false retry_all: False # default: false - reuse_connections: False # default: false (available v1.4+) + reuse_connections: False # default: false ``` @@ -91,7 +91,7 @@ my-snowflake-db: connect_timeout: 10 # default: 10 retry_on_database_errors: False # default: false retry_all: False # default: false - reuse_connections: False # default: false (available v1.4+) + reuse_connections: False # default: false ``` Along with adding the `authenticator` parameter, be sure to run `alter account set allow_client_mfa_caching = true;` in your Snowflake warehouse. Together, these will allow you to easily verify authentication with the DUO Mobile app (skipping this results in push notifications for every model built on every `dbt run`). @@ -263,14 +263,10 @@ The `client_session_keep_alive` feature is intended to keep Snowflake sessions a [Query tags](https://docs.snowflake.com/en/sql-reference/parameters.html#query-tag) are a Snowflake parameter that can be quite useful later on when searching in the [QUERY_HISTORY view](https://docs.snowflake.com/en/sql-reference/account-usage/query_history.html). - - ### reuse_connections During node execution (such as model and test), dbt opens connections against a Snowflake warehouse. Setting this configuration to `True` reduces execution time by verifying credentials only once for each thread. - - ### retry_on_database_errors The `retry_on_database_errors` flag along with the `connect_retries` count specification is intended to make retries configurable after the snowflake connector encounters errors of type snowflake.connector.errors.DatabaseError. These retries can be helpful for handling errors of type "JWT token is invalid" when using key pair authentication. diff --git a/website/docs/faqs/Git/gitignore.md b/website/docs/faqs/Git/gitignore.md index 6bda9611733..8d966c40e2c 100644 --- a/website/docs/faqs/Git/gitignore.md +++ b/website/docs/faqs/Git/gitignore.md @@ -123,111 +123,4 @@ dbt_modules/ - - -1. Launch the Cloud IDE into the project that is being fixed, by selecting **Develop** on the menu bar. -2. In your **File Explorer**, check to see if a `.gitignore` file exists at the root of your dbt project folder. If it doesn't exist, create a new file. -3. Open the new or existing `gitignore` file, and add the following: - -```bash -target/ -dbt_packages/ -logs/ -# legacy -- renamed to dbt_packages in dbt v1 -dbt_modules/ -``` - - * **Note** — You can place these lines anywhere in the file, as long as they're on separate lines. The lines shown are wildcards that will include all nested file and folders. Avoid adding a trailing `'*'` to the lines, such as `target/*`. - -For more info on `gitignore` syntax, refer to the [Git docs](https://git-scm.com/docs/gitignore). - -4. Save the changes but _don't commit_. -5. Restart the IDE by clicking on the three dots next to the **IDE Status button** on the lower right corner of the IDE screen and select **Restart IDE**. - - - -6. Once the IDE restarts, go to the **File Explorer** to delete the following files or folders (if they exist). No data will be lost: - * `target`, `dbt_modules`, `dbt_packages`, `logs` -7. **Save** and then **Commit and sync** the changes. -8. Restart the IDE again using the same procedure as step 5. -9. Once the IDE restarts, use the 'Create a pull request' (PR) button under the **Version Control** menu to start the process of integrating the changes. -10. When the git provider's website opens to a page with the new PR, follow the necessary steps to compelete and merge the PR into the main branch of that repository. - - * **Note** — The 'main' branch might also be called 'master', 'dev', 'qa', 'prod', or something else depending on the organizational naming conventions. The goal is to merge these changes into the root branch that all other development branches are created from. - -11. Return to the dbt Cloud IDE and use the **Change Branch** button to switch to the main branch of the project. -12. Once the branch has changed, click the **Pull from remote** button to pull in all the changes. -13. Verify the changes by making sure the files/folders in the `.gitignore `file are in italics. - - - - -### Fix in the git provider - -Sometimes it's necessary to use the git providers web interface to fix a broken `.gitignore` file. Although the specific steps may vary across providers, the general process remains the same. - -There are two options for this approach: editing the main branch directly if allowed, or creating a pull request to implement the changes if required: - - - - - -When permissions allow it, it's possible to edit the `.gitignore` directly on the main branch of your repo. Here are the following steps: - -1. Go to your repository's web interface. -2. Switch to the main branch, and the root directory of your dbt project. -3. Find the `.gitignore` file. Create a blank one if it doesn't exist. -4. Edit the file in the web interface, adding the following entries: -```bash -target/ -dbt_packages/ -logs/ -# legacy -- renamed to dbt_packages in dbt v1 -dbt_modules/ -``` -5. Commit (save) the file. -6. Delete the following folders from the dbt project root, if they exist. No data or code will be lost: - * `target`, `dbt_modules`, `dbt_packages`, `logs` -7. Commit (save) the deletions to the main branch. -8. Switch to the dbt Cloud IDE, and open the project that you're fixing. -9. Reclone your repo in the IDE by clicking on the three dots next to the **IDE Status** button on the lower right corner of the IDE screen, then select **Reclone Repo**. - * **Note** — Any saved but uncommitted changes will be lost, so make sure you copy any modified code that you want to keep in a temporary location outside of dbt Cloud. -10. Once you reclone the repo, open the `.gitignore` file in the branch you're working in. If the new changes aren't included, you'll need to merge the latest commits from the main branch into your working branch. -11. Go to the **File Explorer** to verify the `.gitignore` file contains the correct entries and make sure the untracked files/folders in the .gitignore file are in *italics*. -12. Great job 🎉! You've configured the `.gitignore` correctly and can continue with your development! - - - - -If you can't edit the `.gitignore` directly on the main branch of your repo, follow these steps: - -1. Go to your repository's web interface. -2. Switch to an existing development branch, or create a new branch just for these changes (This is often faster and cleaner). -3. Find the `.gitignore` file. Create a blank one if it doesn't exist. -4. Edit the file in the web interface, adding the following entries: -```bash -target/ -dbt_packages/ -logs/ -# legacy -- renamed to dbt_packages in dbt v1 -dbt_modules/ -``` -5. Commit (save) the file. -6. Delete the following folders from the dbt project root, if they exist. No data or code will be lost: - * `target`, `dbt_modules`, `dbt_packages`, `logs` -7. Commit (save) the deleted folders. -8. Open a merge request using the git provider web interface. The merge request should be attempting to merge the changes into the 'main' branch that all development branches are created from. -9. Follow the necessary procedures to get the branch approved and merged into the 'main' branch. You can delete the branch after the merge is complete. -10. Once the merge is complete, go back to the dbt Cloud IDE, and open the project that you're fixing. -11. Reclone your repo in the IDE by clicking on the three dots next to the **IDE Status** button on the lower right corner of the IDE screen, then select **Reclone Repo**. - * **Note** — Any saved but uncommitted changes will be lost, so make sure you copy any modified code that you want to keep in a temporary location outside of dbt Cloud. -12. Once you reclone the repo, open the `.gitignore` file in the branch you're working in. If the new changes aren't included, you'll need to merge the latest commits from the main branch into your working branch. -13. Go to the **File Explorer** to verify the `.gitignore` file contains the correct entries and make sure the untracked files/folders in the .gitignore file are in *italics*. -14. Great job 🎉! You've configured the `.gitignore` correctly and can continue with your development! - - - - - - For more info, refer to this [detailed video](https://www.loom.com/share/9b3b8e2b617f41a8bad76ec7e42dd014) for additional guidance. diff --git a/website/docs/faqs/Project/why-version-2.md b/website/docs/faqs/Project/why-version-2.md index dc7e409716e..b4e91d6a773 100644 --- a/website/docs/faqs/Project/why-version-2.md +++ b/website/docs/faqs/Project/why-version-2.md @@ -14,10 +14,3 @@ Resource yml files do not currently require this config. We only support `versio - - -Once upon a time, the structure of these `.yml` files was very different (s/o to anyone who was using dbt back then!). Adding `version: 2` allowed us to make this structure more extensible. - -Currently, Version 2 is the only supported version for these files. We kept `version:` around as a required key so that in the future, if we need to introduce a new structure for these files, we'll be able to do this more easily. - - diff --git a/website/docs/faqs/Troubleshooting/gitignore.md b/website/docs/faqs/Troubleshooting/gitignore.md index 59fd4e8c866..6ab217ebf07 100644 --- a/website/docs/faqs/Troubleshooting/gitignore.md +++ b/website/docs/faqs/Troubleshooting/gitignore.md @@ -38,49 +38,6 @@ dbt_modules/ - - - - -1. In the dbt Cloud IDE, add the following [.gitignore contents](https://github.com/dbt-labs/dbt-starter-project/blob/main/.gitignore) in your dbt project `.gitignore` file: -```bash -target/ -dbt_packages/ -logs/ -# legacy -- renamed to dbt_packages in dbt v1 -dbt_modules/ -``` -2. Go to your `dbt_project.yml` file and add `tmp/` after your `target-path:` and add `log-path: "tmp/logs"`. - * So it should look like: `target-path: "tmp/target"` and `log-path: "tmp/logs"`: - - - -3. Save your changes but _don't commit_. -4. Restart the IDE by clicking on the three dots next to the **IDE Status button** on the lower right of the IDE. - - - -5. Select **Restart IDE**. -6. Go back to your dbt project and delete the following four folders (if you have them): - * `target` - * `dbt_modules` - * `dbt_packages` - * `logs` -7. **Save** and then **Commit and sync** your changes. -8. Go back to your `dbt_project.yml` file and undo the modifications you made in **Step 2**. - - * Remove `tmp` from your `target-path` and completely remove the `log-path: "tmp/logs"` line. - - - -9. Restart the IDE again. -10. Delete the `tmp` folder in the **File Explorer**. -11. Create a pull request (PR) under the **Version Control** menu to integrate your new changes. -12. Merge the PR in your git provider page. -13. Switch to your main branch and click on **Pull from remote** to pull in all the changes you made to your main branch. You can verify the changes by making sure the files/folders in the .gitignore file are in italics. - - - For more info, refer to this [detailed video](https://www.loom.com/share/9b3b8e2b617f41a8bad76ec7e42dd014) for additional guidance. diff --git a/website/docs/reference/artifacts/manifest-json.md b/website/docs/reference/artifacts/manifest-json.md index 47a9849eda5..5a487f2f177 100644 --- a/website/docs/reference/artifacts/manifest-json.md +++ b/website/docs/reference/artifacts/manifest-json.md @@ -36,7 +36,7 @@ All resources nested within `nodes`, `sources`, `metrics`, `exposures`, `macros` - `name`: Resource name. - `unique_id`: `..`, same as dictionary key - `package_name`: Name of package that defines this resource. -- `root_path`: Absolute file path of this resource's package. (**Note:** This is removed for most node types in dbt Core v1.4 / manifest v8 to reduce duplicative information across nodes, but it is still present for seeds.) +- `root_path`: Absolute file path of this resource's package. (**Note:** This was removed for most node types in dbt Core v1.4 / manifest v8 to reduce duplicative information across nodes, but it is still present for seeds.) - `path`: Relative file path of this resource's definition within its "resource path" (`model-paths`, `seed-paths`, etc.). - `original_file_path`: Relative file path of this resource's definition, including its resource path. diff --git a/website/docs/reference/commands/list.md b/website/docs/reference/commands/list.md index 5caabdc2b2e..e73699dc78c 100644 --- a/website/docs/reference/commands/list.md +++ b/website/docs/reference/commands/list.md @@ -73,16 +73,6 @@ $ dbt ls --select snowplow.* --output json **Listing JSON output with custom keys** - - -``` -$ dbt ls --select snowplow.* --output json --output-keys "name resource_type description" -{"name": "snowplow_events", "description": "This is a pretty cool model", ...} -{"name": "snowplow_page_views", "description": "This model is even cooler", ...} -... -``` - - ``` diff --git a/website/docs/reference/dbt-jinja-functions/builtins.md b/website/docs/reference/dbt-jinja-functions/builtins.md index 9f1d139ab20..0da7c6536a8 100644 --- a/website/docs/reference/dbt-jinja-functions/builtins.md +++ b/website/docs/reference/dbt-jinja-functions/builtins.md @@ -59,22 +59,6 @@ It includes logic to extract user-provided arguments, including version - - -From dbt v1.4 and lower, use the following macro to override the `ref` method available in the model compilation context to return a [Relation](/reference/dbt-classes#relation) with the database name overriden to `dev`: - -``` - -{% macro ref(model_name) %} - - {% set rel = builtins.ref(model_name) %} - {% set newrel = rel.replace_path(database="dev") %} - {% do return(newrel) %} - -{% endmacro %} -``` - - Logic within the ref macro can also be used to control which elements of the model path are rendered when run, for example the following logic renders only the schema and object identifier, but not the database reference i.e. `my_schema.my_model` rather than `my_database.my_schema.my_model`. This is especially useful when using snowflake as a warehouse, if you intend to change the name of the database post-build and wish the references to remain accurate. ``` diff --git a/website/docs/reference/dbt-jinja-functions/graph.md b/website/docs/reference/dbt-jinja-functions/graph.md index 491b7836f45..bea09c326e8 100644 --- a/website/docs/reference/dbt-jinja-functions/graph.md +++ b/website/docs/reference/dbt-jinja-functions/graph.md @@ -23,55 +23,6 @@ to understand how to effectively use this variable. The `graph` context variable is a dictionary which maps node ids onto dictionary representations of those nodes. A simplified example might look like: - - -```json -{ - "nodes": { - "model.my_project.model_name": { - "unique_id": "model.my_project.model_name", - "config": {"materialized": "table", "sort": "id"}, - "tags": ["abc", "123"], - "path": "models/path/to/model_name.sql", - ... - }, - ... - }, - "sources": { - "source.my_project.snowplow.event": { - "unique_id": "source.my_project.snowplow.event", - "database": "analytics", - "schema": "analytics", - "tags": ["abc", "123"], - "path": "models/path/to/schema.yml", - ... - }, - ... - }, - "exposures": { - "exposure.my_project.traffic_dashboard": { - "unique_id": "source.my_project.traffic_dashboard", - "type": "dashboard", - "maturity": "high", - "path": "models/path/to/schema.yml", - ... - }, - ... - }, - "metrics": { - "metric.my_project.count_all_events": { - "unique_id": "metric.my_project.count_all_events", - "type": "count", - "path": "models/path/to/schema.yml", - ... - }, - ... - } -} -``` - - - ```json diff --git a/website/docs/reference/dbt_project.yml.md b/website/docs/reference/dbt_project.yml.md index 65f9761c9bc..6166be1df6d 100644 --- a/website/docs/reference/dbt_project.yml.md +++ b/website/docs/reference/dbt_project.yml.md @@ -3,12 +3,6 @@ Every [dbt project](/docs/build/projects) needs a `dbt_project.yml` file — thi - dbt uses [YAML](https://yaml.org/) in a few different places. If you're new to YAML, it would be worth learning how arrays, dictionaries, and strings are represented. - - -- By default, dbt looks for the `dbt_project.yml` in your current working directory and its parents, but you can set a different directory using the `--project-dir` flag. - - - - By default, dbt looks for the `dbt_project.yml` in your current working directory and its parents, but you can set a different directory using the `--project-dir` flag or the `DBT_PROJECT_DIR` environment variable. diff --git a/website/docs/reference/events-logging.md b/website/docs/reference/events-logging.md index 3eaa70c3de7..6ae0f6a95ec 100644 --- a/website/docs/reference/events-logging.md +++ b/website/docs/reference/events-logging.md @@ -32,9 +32,7 @@ As dbt runs, it generates events. The most common way to see those events is as _For more details about how the eventing system has been implemented in dbt-core, see the [`events` module README](https://github.com/dbt-labs/dbt-core/blob/HEAD/core/dbt/events/README.md)._ - - -Starting in v1.4, the structure of each event in `dbt-core` is backed by a schema defined using [protocol buffers](https://developers.google.com/protocol-buffers). All schemas are defined in the [`types.proto`](https://github.com/dbt-labs/dbt-core/blob/3bf148c443e6b1da394b62e88a08f1d7f1d8ccaa/core/dbt/events/core_types.proto) file within the `dbt-core` codebase. +The structure of each event in `dbt-core` is backed by a schema defined using [protocol buffers](https://developers.google.com/protocol-buffers). All schemas are defined in the [`types.proto`](https://github.com/dbt-labs/dbt-core/blob/3bf148c443e6b1da394b62e88a08f1d7f1d8ccaa/core/dbt/events/core_types.proto) file within the `dbt-core` codebase. Every event has the same two top-level keys: - `info`: Information common to all events. See the table below for the breakdown. @@ -67,7 +65,7 @@ Many events are fired while compiling or running a specific DAG node (model, see | `node_finished_at` | Timestamp when node processing completed | | `node_name` | Name of this model/seed/test/etc | | `node_path` | File path to where this resource is defined | -| `node_relation` | Nested object containing this node's database representation: `database`, `schema`, `alias`, and full `relation_name` with quoting & inclusion policies appliedAdded in v1.5 | +| `node_relation` | Nested object containing this node's database representation: `database`, `schema`, `alias`, and full `relation_name` with quoting & inclusion policies applied | | `node_started_at` | Timestamp when node processing started | | `node_status` | Current status of the node, either `RunningStatus` (while running) or `NodeStatus` (finished) as defined in [the result contract](https://github.com/dbt-labs/dbt-core/blob/eba90863ed4043957330ea44ca267db1a2d81fcd/core/dbt/contracts/results.py#L75-L88) | | `resource_type` | `model`, `test`, `seed`, `snapshot`, etc. | @@ -119,12 +117,10 @@ Many events are fired while compiling or running a specific DAG node (model, see } ``` - - ## Python interface :::warning -The `EVENT_HISTORY` object has been deprecated and removed in dbt Core v1.4+ +The `EVENT_HISTORY` object has been deprecated. ::: Older versions of `dbt-core` made available a full history of events fired during an invocation, in the form of an `EVENT_HISTORY` object. diff --git a/website/docs/reference/exposure-properties.md b/website/docs/reference/exposure-properties.md index ab636a3d21c..471c2a00824 100644 --- a/website/docs/reference/exposure-properties.md +++ b/website/docs/reference/exposure-properties.md @@ -18,8 +18,6 @@ You can name these files `whatever_you_want.yml`, and nest them arbitrarily deep Exposure names must contain only letters, numbers, and underscores (no spaces or special characters). For a short human-friendly name with title casing, spaces, and special characters, use the `label` property. - - ```yml @@ -51,8 +49,6 @@ exposures: ``` - - ## Example diff --git a/website/docs/reference/global-configs/cache.md b/website/docs/reference/global-configs/cache.md index a605e1e70f3..7687df30339 100644 --- a/website/docs/reference/global-configs/cache.md +++ b/website/docs/reference/global-configs/cache.md @@ -31,36 +31,3 @@ dbt --cache-selected-only run --select salesforce - - -### Cache database objects for selected resource - -:::caution Experimental config flag -This should be considered an _unstable_ config flag because it is experimental and subject to change. We reserve the right to make breaking changes to this config flag. -::: - -At the start of runs, dbt caches metadata about all the objects in all the schemas where it might materialize resources (such as models). By default, dbt caches all schemas related to the project. When this config is enabled, dbt will only cache schemas related to selected resources for the current run. This can offer significant speed improvements when running a small subset of a large project. - -For example, to improve speed and performance while focused on developing Salesforce models, which are materialized into their own dedicated schema, you would select those models and pass the `cache-selected-only` flag: - -```text - -dbt --cache-selected-only run --select salesforce - -``` - -The previous example enables you to start working with your salesforce models, and dbt will only cache those models instead of the entire project. - -To set the config in your `profile.yml`: - - -```yaml - -config: - cache_selected_only: true - -``` - - - - diff --git a/website/docs/reference/global-configs/logs.md b/website/docs/reference/global-configs/logs.md index 7ee7890e302..19ca8da6b5c 100644 --- a/website/docs/reference/global-configs/logs.md +++ b/website/docs/reference/global-configs/logs.md @@ -8,12 +8,6 @@ sidebar: "logs" dbt outputs logs to two different locations: CLI console and the log file. - - -The `LOG_FORMAT` config specifies how dbt's logs should be formatted and has three options: `json`, `text`, and `debug`. - - - The `LOG_FORMAT` and `LOG_FORMAT_FILE` configs specify how dbt's logs should be formatted, and they each have the same options: `json`, `text`, and `debug`. @@ -50,12 +44,6 @@ The `json` format outputs fully structured logs in the format {"data": {"adapter_name": "postgres", "adapter_version": "=1.8.0"}, "info": {"category": "", "code": "E034", "extra": {}, "invocation_id": "82131fa0-d2b4-4a77-9436-019834e22746", "level": "info", "msg": "Registered adapter: postgres=1.8.0", "name": "AdapterRegistered", "pid": 7875, "thread": "MainThread", "ts": "2024-05-29T23:32:56.437986Z"}} ``` - - -When the `LOG_FORMAT` is set explicitly, it will take affect in both the console and log files. - - - When the `LOG_FORMAT` is set explicitly, it will take affect in both the console and log files whereas the `LOG_FORMAT_FILE` only affects the log file. diff --git a/website/docs/reference/global-configs/print-output.md b/website/docs/reference/global-configs/print-output.md index 1d44a8bc5f0..fc129b162a7 100644 --- a/website/docs/reference/global-configs/print-output.md +++ b/website/docs/reference/global-configs/print-output.md @@ -6,12 +6,6 @@ sidebar: "Print output" ### Suppress `print()` messages in stdout - - -By default, dbt includes [`print()`](/reference/dbt-jinja-functions/print) messages in standard out (stdout). You can use the `DBT_NO_PRINT` environment variable to prevent these messages from showing up in stdout. - - - By default, dbt includes [`print()`](/reference/dbt-jinja-functions/print) messages in standard out (stdout). You can use the `DBT_PRINT` environment variable to prevent these messages from showing up in stdout. diff --git a/website/docs/reference/node-selection/defer.md b/website/docs/reference/node-selection/defer.md index 81a0f4a0328..bbcc5f7d567 100644 --- a/website/docs/reference/node-selection/defer.md +++ b/website/docs/reference/node-selection/defer.md @@ -43,26 +43,16 @@ When using defer, you may be selecting from production datasets, development dat - if you apply env-specific limits in dev but not prod, as you may end up selecting more data than you expect - when executing tests that depend on multiple parents (e.g. `relationships`), since you're testing "across" environments - - -Deferral requires both `--defer` and `--state` to be set, either by passing flags explicitly or by setting environment variables (`DBT_DEFER_TO_STATE` and `DBT_ARTIFACT_STATE_PATH`). If you use dbt Cloud, read about [how to set up CI jobs](/docs/deploy/continuous-integration). - - - Deferral requires both `--defer` and `--state` to be set, either by passing flags explicitly or by setting environment variables (`DBT_DEFER` and `DBT_STATE`). If you use dbt Cloud, read about [how to set up CI jobs](/docs/deploy/continuous-integration). - - #### Favor state You can optionally skip the second criterion by passing the `--favor-state` flag. If passed, dbt will favor using the node defined in your `--state` namespace, even if the node exists in the current target. - - ### Example In my local development environment, I create all models in my target schema, `dev_alice`. In production, the same models are created in a schema named `prod`. diff --git a/website/docs/reference/node-selection/methods.md b/website/docs/reference/node-selection/methods.md index 038cf475583..903790100fb 100644 --- a/website/docs/reference/node-selection/methods.md +++ b/website/docs/reference/node-selection/methods.md @@ -197,12 +197,6 @@ dbt test --select "test_name:range_min_max" # run all instances of a custom **N.B.** State-based selection is a powerful, complex feature. Read about [known caveats and limitations](/reference/node-selection/state-comparison-caveats) to state comparison. - - -The `state` method is used to select nodes by comparing them against a previous version of the same project, which is represented by a [manifest](/reference/artifacts/manifest-json). The file path of the comparison manifest _must_ be specified via the `--state` flag or `DBT_ARTIFACT_STATE_PATH` environment variable. - - - The `state` method is used to select nodes by comparing them against a previous version of the same project, which is represented by a [manifest](/reference/artifacts/manifest-json). The file path of the comparison manifest _must_ be specified via the `--state` flag or `DBT_STATE` environment variable. @@ -283,16 +277,6 @@ The following dbt commands produce `sources.json` artifacts whose results can be After issuing one of the above commands, you can reference the source freshness results by adding a selector to a subsequent command as follows: - - -```bash -# You can also set the DBT_ARTIFACT_STATE_PATH environment variable instead of the --state flag. -dbt source freshness # must be run again to compare current to previous state -dbt build --select "source_status:fresher+" --state path/to/prod/artifacts -``` - - - ```bash @@ -305,11 +289,6 @@ dbt build --select "source_status:fresher+" --state path/to/prod/artifacts ### The "group" method - - -Supported in v1.5 or newer. - - @@ -324,12 +303,6 @@ dbt run --select "group:finance" # run all models that belong to the finance gro ### The "access" method - - -Supported in v1.5 or newer. - - - The `access` method selects models based on their [access](/reference/resource-configs/access) property. @@ -344,12 +317,6 @@ dbt list --select "access:protected" # list all protected models ### The "version" method - - -Supported in v1.5 or newer. - - - The `version` method selects [versioned models](/docs/collaborate/govern/model-versions) based on their [version identifier](/reference/resource-properties/versions) and [latest version](/reference/resource-properties/latest_version). diff --git a/website/docs/reference/node-selection/syntax.md b/website/docs/reference/node-selection/syntax.md index da7a3af4bc3..ce7d27cb6b6 100644 --- a/website/docs/reference/node-selection/syntax.md +++ b/website/docs/reference/node-selection/syntax.md @@ -136,13 +136,6 @@ Together, the `state:` selector and deferral enable ["slim CI"](/best-practices/ State and defer can be set by environment variables as well as CLI flags: - - -- `--state` or `DBT_ARTIFACT_STATE_PATH`: file path -- `--defer` or `DBT_DEFER_TO_STATE`: boolean - - - - `--state` or `DBT_STATE`: file path diff --git a/website/docs/reference/node-selection/test-selection-examples.md b/website/docs/reference/node-selection/test-selection-examples.md index 34e5aaa73d9..11362b2364b 100644 --- a/website/docs/reference/node-selection/test-selection-examples.md +++ b/website/docs/reference/node-selection/test-selection-examples.md @@ -35,22 +35,6 @@ In both cases, `test_type` checks a property of the test itself. These are forms ### Indirect selection - - -There are three modes to configure the behavior when performing indirect selection (with `eager` as the default): - -1. `eager` (default) - include ANY test that references the selected nodes -1. `cautious` - restrict to tests that ONLY refer to selected nodes -1. `buildable` - restrict to tests that ONLY refer to selected nodes (or their ancestors) - -Note that test exclusion is always greedy: if ANY parent is explicitly excluded, the test will be excluded as well. - -:::tip Building subsets of a DAG -The `buildable` and `cautious` modes can be useful when you're only building a subset of your DAG, and you want to avoid test failures in `eager` mode caused by unbuilt resources. You can also achieve this with [deferral](/reference/node-selection/defer). -::: - - - @@ -133,62 +117,6 @@ dbt build --select "orders" --indirect-selection=empty - - - - - - - -By default, a test will run when ANY parent is selected; we call this "eager" indirect selection. In this example, that would include any test that references orders, even if it references other models as well. - -In this mode, any test that depends on unbuilt resources will raise an error. - -```shell -dbt test --select "orders" -dbt build --select "orders" -``` - - - - - -It is possible to prevent tests from running if one or more of its parents is unselected (and therefore unbuilt); we call this "cautious" indirect selection. - -It will only include tests whose references are each within the selected nodes. - -Put another way, it will prevent tests from running if one or more of its parents is unselected. - -```shell - -dbt test --select "orders" --indirect-selection=cautious -dbt build --select "orders" --indirect-selection=cautious - -``` - - - - - -This mode is similarly conservative like "cautious", but is slightly more inclusive. - -It will only include tests whose references are each within the selected nodes (or their ancestors). - -This is useful in the same scenarios as "cautious", but also includes when a test depends on a model **and** a direct ancestor of that model (like confirming an aggregation has the same totals as its input). - -```shell -dbt test --select "orders" --indirect-selection=buildable -dbt build --select "orders" --indirect-selection=buildable - -``` - - - - - - - - ### Test selection syntax examples diff --git a/website/docs/reference/node-selection/yaml-selectors.md b/website/docs/reference/node-selection/yaml-selectors.md index 24b99defea0..d911eb44baa 100644 --- a/website/docs/reference/node-selection/yaml-selectors.md +++ b/website/docs/reference/node-selection/yaml-selectors.md @@ -57,28 +57,6 @@ This is the most thorough syntax, which can include the operator-equivalent keyw Review [methods](/reference/node-selection/methods) for the available list. - - -```yml -definition: - method: tag - value: nightly - - # Optional keywords map to the `+` and `@` graph operators: - - children: true | false - parents: true | false - - children_depth: 1 # if children: true, degrees to include - parents_depth: 1 # if parents: true, degrees to include - - childrens_parents: true | false # @ operator - - indirect_selection: eager | cautious | buildable # include all tests selected indirectly? eager by default -``` - - - ```yml @@ -135,13 +113,6 @@ Note: The `exclude` argument in YAML selectors is subtly different from the `--exclude` CLI argument. Here, `exclude` _always_ returns a [set difference](https://en.wikipedia.org/wiki/Complement_(set_theory)), and it is always applied _last_ within its scope. - - -This gets us more intricate subset definitions than what's available on the CLI, -where we can only pass one "yeslist" (`--select`) and one "nolist" (`--exclude`). - - - When more than one "yeslist" (`--select`) is passed, they are treated as a [union](/reference/node-selection/set-operators#unions) rather than an [intersection](/reference/node-selection/set-operators#intersections). Same thing when there is more than one "nolist" (`--exclude`). @@ -150,29 +121,6 @@ When more than one "yeslist" (`--select`) is passed, they are treated as a [unio #### Indirect selection - - -As a general rule, dbt will indirectly select _all_ tests if they touch _any_ resource that you're selecting directly. We call this "eager" indirect selection. You can optionally switch the indirect selection mode to "cautious" or "buildable" by setting `indirect_selection` for a specific criterion: - -```yml -- union: - - method: fqn - value: model_a - indirect_selection: eager # default: will include all tests that touch model_a - - method: fqn - value: model_b - indirect_selection: cautious # will not include tests touching model_b - # if they have other unselected parents - - method: fqn - value: model_c - indirect_selection: buildable # will not include tests touching model_c - # if they have other unselected parents (unless they have an ancestor that is selected) -``` - -If provided, a YAML selector's `indirect_selection` value will take precedence over the CLI flag `--indirect-selection`. Because `indirect_selection` is defined separately for _each_ selection criterion, it's possible to mix eager/cautious/buildable modes within the same definition, to achieve the exact behavior that you need. Remember that you can always test out your critiera with `dbt ls --selector`. - - - As a general rule, dbt will indirectly select _all_ tests if they touch _any_ resource that you're selecting directly. We call this "eager" indirect selection. You can optionally switch the indirect selection mode to "cautious", "buildable", or "empty" by setting `indirect_selection` for a specific criterion: diff --git a/website/docs/reference/resource-configs/bigquery-configs.md b/website/docs/reference/resource-configs/bigquery-configs.md index 075edc8bc95..2ae314cc8f7 100644 --- a/website/docs/reference/resource-configs/bigquery-configs.md +++ b/website/docs/reference/resource-configs/bigquery-configs.md @@ -103,8 +103,6 @@ as ( - - #### Partitioning by an "ingestion" date or timestamp BigQuery supports an [older mechanism of partitioning](https://cloud.google.com/bigquery/docs/partitioned-tables#ingestion_time) based on the time when each row was ingested. While we recommend using the newer and more ergonomic approach to partitioning whenever possible, for very large datasets, there can be some performance improvements to using this older, more mechanistic approach. [Read more about the `insert_overwrite` incremental strategy below](#copying-ingestion-time-partitions). @@ -172,8 +170,6 @@ select created_date as _partitiontime, * EXCEPT(created_date) from ( - - #### Partitioning with integer buckets If the `data_type` is specified as `int64`, then a `range` key must also @@ -594,8 +590,6 @@ with events as ( ... rest of model ... ``` - - #### Copying partitions If you are replacing entire partitions in your incremental runs, you can opt to do so with the [copy table API](https://cloud.google.com/bigquery/docs/managing-tables#copy-table) and partition decorators rather than a `merge` statement. While this mechanism doesn't offer the same visibility and ease of debugging as the SQL `merge` statement, it can yield significant savings in time and cost for large datasets because the copy table API does not incur any costs for inserting the data - it's equivalent to the `bq cp` gcloud command line interface (CLI) command. @@ -639,8 +633,6 @@ from {{ ref('events') }} - - ## Controlling table expiration By default, dbt-created tables never expire. You can configure certain model(s) @@ -715,7 +707,8 @@ models: Views with this configuration will be able to select from objects in `project_1.dataset_1` and `project_2.dataset_2`, even when they are located elsewhere and queried by users who do not otherwise have access to `project_1.dataset_1` and `project_2.dataset_2`. #### Limitations -Starting in v1.4, `grant_access_to` config _is thread-safe_. In earlier versions, it wasn't safe to use multiple threads for authorizing several views at once with `grant_access_to` for the same dataset. Initially, after adding a new `grant_access_to` setting, you can execute `dbt run` in a single thread. Later runs with the same configuration won't repeat the existing access grants and can use multiple threads. + +The `grant_access_to` config _is thread-safe_. In versions prior to v1.4, it wasn't safe to use multiple threads for authorizing several views at once with `grant_access_to` for the same dataset. Initially, after adding a new `grant_access_to` setting, you can execute `dbt run` in a single thread. Later runs with the same configuration won't repeat the existing access grants and can use multiple threads. diff --git a/website/docs/reference/resource-configs/group.md b/website/docs/reference/resource-configs/group.md index 5aa8f91e4b2..7b80d70b6fc 100644 --- a/website/docs/reference/resource-configs/group.md +++ b/website/docs/reference/resource-configs/group.md @@ -18,12 +18,6 @@ id: "group" }> - - -Support for grouping models was added in dbt Core v1.5 - - - @@ -72,12 +66,6 @@ select ... - - -Support for grouping seeds was added in dbt Core v1.5 - - - @@ -107,12 +95,6 @@ seeds: - - -Support for grouping snapshots was added in dbt Core v1.5 - - - @@ -148,12 +130,6 @@ select ... - - -Support for grouping tests was added in dbt Core v1.5 - - - @@ -231,12 +207,6 @@ analyses: - - -Support for grouping metrics was added in dbt Core v1.5 - - - diff --git a/website/docs/reference/resource-configs/singlestore-configs.md b/website/docs/reference/resource-configs/singlestore-configs.md index b172a549d5f..2317c05c0e2 100644 --- a/website/docs/reference/resource-configs/singlestore-configs.md +++ b/website/docs/reference/resource-configs/singlestore-configs.md @@ -3,15 +3,11 @@ title: "SingleStore configurations" id: "singlestore-configs" --- - - ## Incremental materialization strategies The [`incremental_strategy` config](/docs/build/incremental-models#about-incremental_strategy) controls how dbt builds incremental models. Currently, SingleStoreDB supports only the `delete+insert` configuration. The `delete+insert` incremental strategy directs dbt to follow a two-step incremental approach. Initially, it identifies and removes the records flagged by the configured `is_incremental()` block. Subsequently, it re-inserts these records. - - ## Performance Optimizations [SingleStore Physical Database Schema Design documentation](https://docs.singlestore.com/managed-service/en/create-a-database/physical-database-schema-design/concepts-of-physical-database-schema-design.html) is helpful if you want to use specific options (that are described below) in your dbt project. diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md index 4e1af82d922..e2ce1ef58d5 100644 --- a/website/docs/reference/resource-configs/snowflake-configs.md +++ b/website/docs/reference/resource-configs/snowflake-configs.md @@ -301,9 +301,9 @@ models: ## Temporary tables -Beginning in dbt version 1.3, incremental table merges for Snowflake prefer to utilize a `view` rather than a `temporary table`. The reasoning was to avoid the database write step that a temporary table would initiate and save compile time. +Incremental table merges for Snowflake prefer to utilize a `view` rather than a `temporary table`. The reasoning is to avoid the database write step that a temporary table would initiate and save compile time. -However, some situations remain where a temporary table would achieve results faster or more safely. dbt v1.4 adds the `tmp_relation_type` configuration to allow you to opt in to temporary tables for incremental builds. This is defined as part of the model configuration. +However, some situations remain where a temporary table would achieve results faster or more safely. The `tmp_relation_type` configuration allowa you to opt in to temporary tables for incremental builds. This is defined as part of the model configuration. To guarantee accuracy, an incremental model using the `delete+insert` strategy with a `unique_key` defined requires a temporary table; trying to change this to a view will result in an error. From 8971b24ea104c6a6a2be2dd720360bfbe454e9ac Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Mon, 3 Jun 2024 19:40:29 -0400 Subject: [PATCH 2/3] Update website/docs/reference/resource-configs/snowflake-configs.md Co-authored-by: Ly Nguyen <107218380+nghi-ly@users.noreply.github.com> --- website/docs/reference/resource-configs/snowflake-configs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md index e2ce1ef58d5..9d70d28eae8 100644 --- a/website/docs/reference/resource-configs/snowflake-configs.md +++ b/website/docs/reference/resource-configs/snowflake-configs.md @@ -303,7 +303,7 @@ models: Incremental table merges for Snowflake prefer to utilize a `view` rather than a `temporary table`. The reasoning is to avoid the database write step that a temporary table would initiate and save compile time. -However, some situations remain where a temporary table would achieve results faster or more safely. The `tmp_relation_type` configuration allowa you to opt in to temporary tables for incremental builds. This is defined as part of the model configuration. +However, some situations remain where a temporary table would achieve results faster or more safely. The `tmp_relation_type` configuration enables you to opt in to temporary tables for incremental builds. This is defined as part of the model configuration. To guarantee accuracy, an incremental model using the `delete+insert` strategy with a `unique_key` defined requires a temporary table; trying to change this to a view will result in an error. From 58ab7a75a93a1aa5c56c75d1e88b977dfe3f2df5 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Mon, 3 Jun 2024 19:41:52 -0400 Subject: [PATCH 3/3] Update events-logging.md Removing calliout --- website/docs/reference/events-logging.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/website/docs/reference/events-logging.md b/website/docs/reference/events-logging.md index 6ae0f6a95ec..de79d8d9171 100644 --- a/website/docs/reference/events-logging.md +++ b/website/docs/reference/events-logging.md @@ -119,10 +119,6 @@ Many events are fired while compiling or running a specific DAG node (model, see ## Python interface -:::warning -The `EVENT_HISTORY` object has been deprecated. -::: - Older versions of `dbt-core` made available a full history of events fired during an invocation, in the form of an `EVENT_HISTORY` object.