From 2f1402151d3b0aea9205c263ccfaa6fd2ae6f126 Mon Sep 17 00:00:00 2001 From: L Zdanski <25642903+lzdanski@users.noreply.github.com> Date: Fri, 13 Mar 2026 19:27:35 -0400 Subject: [PATCH 1/9] Add to sidebar --- .../render-config.rst | 2 + docs/reference/glossary.rst | 62 +++++++++++++++++++ docs/reference/index.rst | 8 +++ 3 files changed, 72 insertions(+) create mode 100644 docs/reference/glossary.rst diff --git a/docs/guides/translate_dbt_to_airflow/render-config.rst b/docs/guides/translate_dbt_to_airflow/render-config.rst index 07b74feba6..8441374315 100644 --- a/docs/guides/translate_dbt_to_airflow/render-config.rst +++ b/docs/guides/translate_dbt_to_airflow/render-config.rst @@ -1,3 +1,5 @@ +.. _render-config: + Render Config ------------- diff --git a/docs/reference/glossary.rst b/docs/reference/glossary.rst new file mode 100644 index 0000000000..cec8bd51eb --- /dev/null +++ b/docs/reference/glossary.rst @@ -0,0 +1,62 @@ +.. glossary:: + +Glossary +========= + +Cosmos + An open-source Python package that allows you to write data transofrmations using dbt + and then use Apache Airflow®'s orchestration to integrate dbt projects into end-to-end workflows. + +Dag + An Airflow term derived from the mathematical structure called **Directed Acyclic Graph**. The Dag provides + a model that includes everything needed to execute an Airflow workflow. See `Dag `_. + +dbt deps + A `dbt command `_ that pulls the most recent version of dependencies contained in your packages.yml file. + +Execution mode + The execution mode describes where and how Cosmos runs dbt commands. You configure it using the ``ExecutionConfig``. See :ref:`execution-modes`. + +Load mode + The method that Cosmos uses to parse your dbt project. See :ref:`parsing-methods`. + +Manifest + A dbt artifact that This file contains a full representation of the dbt project's resources, including all node configurations and resource properties. + See `Manifest JSON file `_ in the dbt docs. + +Node + A dbt concept that encapsulates a step within a pipeline. + +Partial parsing + Partial parsing is a dbt feature that can greatly speed up dbt parsing and execution when using the ``dbt_ls`` + load mode. See :ref:`partial-parsing`. + +Profile + The authentication information used by dbt to connect to your data warehouse. See `profile `_. + +ProfileConfig + The class that determines which data warehouse Cosmos connects to when executing the dbt SQL. See :ref:`connect_database`. + +Profile mapping + A Cosmos-provided resource that translate Airflow connections into dbt profiles. See :ref:`use-profile-mapping`. + +profiles.yml + The file where dbt stores connection information for each data warehouse connection. See :ref:`connect_database`, :ref:`use-your-profiles-yml`, + or `The profiles.yml file `_ in the dbt docs. + +ProjectConfig + The Cosmos class that allows you to specify information about where your dbt project is located and project variables that should be used for rendering and execution. + +RenderConfig + The configuration in Cosmos that controls how Cosmos turns your dbt project into an Airflow dag or task group. :ref:`render-config`. + +Task + An Airflow term describing a step within a pipeline. See `Task `_. + +Test + Refers to the the `dbt test command `_. You can configure how you want + Cosmos to handle running ``dbt test`` with :ref:`testing-behavior`. + +Workflow + A dbt term describing a pipeline that contains a group of steps. dbt can run a subset + of tasks assuming upstream tasks were run. This is similar to the Airflow concept of a `Dag `_. \ No newline at end of file diff --git a/docs/reference/index.rst b/docs/reference/index.rst index 3d2980d6a9..5efc3de65f 100644 --- a/docs/reference/index.rst +++ b/docs/reference/index.rst @@ -24,6 +24,14 @@ Reference profiles/index +.. toctree:: + :maxdepth: 1 + :hidden: + :caption: Additional resources + + glossary + + Configurations ++++++++++++++ From 6dc54185c9dda9da11357981300a26566da8e1a0 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Fri, 13 Mar 2026 23:34:01 +0000 Subject: [PATCH 2/9] =?UTF-8?q?=F0=9F=8E=A8=20[pre-commit.ci]=20Auto=20for?= =?UTF-8?q?mat=20from=20pre-commit.com=20hooks?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/reference/glossary.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/glossary.rst b/docs/reference/glossary.rst index cec8bd51eb..43a0d7d29d 100644 --- a/docs/reference/glossary.rst +++ b/docs/reference/glossary.rst @@ -59,4 +59,4 @@ Test Workflow A dbt term describing a pipeline that contains a group of steps. dbt can run a subset - of tasks assuming upstream tasks were run. This is similar to the Airflow concept of a `Dag `_. \ No newline at end of file + of tasks assuming upstream tasks were run. This is similar to the Airflow concept of a `Dag `_. From 659f9dfcdc07ff0338d7fa8ef74fd704eae285bd Mon Sep 17 00:00:00 2001 From: L Zdanski <25642903+lzdanski@users.noreply.github.com> Date: Fri, 13 Mar 2026 21:40:42 -0400 Subject: [PATCH 3/9] Fix misspelling --- docs/reference/glossary.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/glossary.rst b/docs/reference/glossary.rst index 43a0d7d29d..d24596fddb 100644 --- a/docs/reference/glossary.rst +++ b/docs/reference/glossary.rst @@ -4,7 +4,7 @@ Glossary ========= Cosmos - An open-source Python package that allows you to write data transofrmations using dbt + An open-source Python package that allows you to write data transformations using dbt and then use Apache Airflow®'s orchestration to integrate dbt projects into end-to-end workflows. Dag From b599fff6b726a334a2fb936fad581996fa054e0d Mon Sep 17 00:00:00 2001 From: Pankaj Singh <98807258+pankajastro@users.noreply.github.com> Date: Wed, 29 Apr 2026 11:50:51 +0530 Subject: [PATCH 4/9] Apply suggestion from @pankajastro --- docs/reference/glossary.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/glossary.rst b/docs/reference/glossary.rst index d24596fddb..fa40ae2199 100644 --- a/docs/reference/glossary.rst +++ b/docs/reference/glossary.rst @@ -5,7 +5,7 @@ Glossary Cosmos An open-source Python package that allows you to write data transformations using dbt - and then use Apache Airflow®'s orchestration to integrate dbt projects into end-to-end workflows. + and then use Apache Airflow®'s orchestration to integrate dbt projects into end-to-end workflows. Dag An Airflow term derived from the mathematical structure called **Directed Acyclic Graph**. The Dag provides From 50d52ec9fc8cc65af5734d06bba76b7827ee98b6 Mon Sep 17 00:00:00 2001 From: Pankaj Singh Date: Wed, 29 Apr 2026 11:53:12 +0530 Subject: [PATCH 5/9] Improve glossary: fix directive, add DbtDag and DbtTaskGroup entries - Move terms inside the .. glossary:: directive so Sphinx registers them for :term: cross-references - Fix capitalisation: Dag -> DAG throughout - Add missing ExecutionConfig entry - Add DbtDag and DbtTaskGroup entries with cross-references - Fix garbled Manifest definition ("artifact that This file") - Fix double "the" in Test definition - Fix missing "See" in RenderConfig entry - Remove version-pinned dbt doc links (?version=1.12) - Fix "translate" -> "translates" in Profile mapping Co-Authored-By: Claude Sonnet 4.6 --- docs/reference/glossary.rst | 145 +++++++++++++++++++++--------------- 1 file changed, 86 insertions(+), 59 deletions(-) diff --git a/docs/reference/glossary.rst b/docs/reference/glossary.rst index fa40ae2199..6670ce4dc7 100644 --- a/docs/reference/glossary.rst +++ b/docs/reference/glossary.rst @@ -1,62 +1,89 @@ -.. glossary:: - Glossary -========= - -Cosmos - An open-source Python package that allows you to write data transformations using dbt - and then use Apache Airflow®'s orchestration to integrate dbt projects into end-to-end workflows. - -Dag - An Airflow term derived from the mathematical structure called **Directed Acyclic Graph**. The Dag provides - a model that includes everything needed to execute an Airflow workflow. See `Dag `_. - -dbt deps - A `dbt command `_ that pulls the most recent version of dependencies contained in your packages.yml file. - -Execution mode - The execution mode describes where and how Cosmos runs dbt commands. You configure it using the ``ExecutionConfig``. See :ref:`execution-modes`. - -Load mode - The method that Cosmos uses to parse your dbt project. See :ref:`parsing-methods`. - -Manifest - A dbt artifact that This file contains a full representation of the dbt project's resources, including all node configurations and resource properties. - See `Manifest JSON file `_ in the dbt docs. - -Node - A dbt concept that encapsulates a step within a pipeline. +======== -Partial parsing - Partial parsing is a dbt feature that can greatly speed up dbt parsing and execution when using the ``dbt_ls`` - load mode. See :ref:`partial-parsing`. - -Profile - The authentication information used by dbt to connect to your data warehouse. See `profile `_. - -ProfileConfig - The class that determines which data warehouse Cosmos connects to when executing the dbt SQL. See :ref:`connect_database`. - -Profile mapping - A Cosmos-provided resource that translate Airflow connections into dbt profiles. See :ref:`use-profile-mapping`. - -profiles.yml - The file where dbt stores connection information for each data warehouse connection. See :ref:`connect_database`, :ref:`use-your-profiles-yml`, - or `The profiles.yml file `_ in the dbt docs. - -ProjectConfig - The Cosmos class that allows you to specify information about where your dbt project is located and project variables that should be used for rendering and execution. - -RenderConfig - The configuration in Cosmos that controls how Cosmos turns your dbt project into an Airflow dag or task group. :ref:`render-config`. - -Task - An Airflow term describing a step within a pipeline. See `Task `_. - -Test - Refers to the the `dbt test command `_. You can configure how you want - Cosmos to handle running ``dbt test`` with :ref:`testing-behavior`. +.. glossary:: -Workflow - A dbt term describing a pipeline that contains a group of steps. dbt can run a subset - of tasks assuming upstream tasks were run. This is similar to the Airflow concept of a `Dag `_. + Cosmos + An open-source Python package that allows you to write data transformations using dbt + and then use Apache Airflow®'s orchestration to integrate dbt projects into end-to-end workflows. + + DAG + An Airflow term derived from the mathematical structure called **Directed Acyclic Graph**. A DAG + provides a model that includes everything needed to execute an Airflow workflow. + See `DAG `_. + + DbtDag + A Cosmos class that wraps a full Airflow DAG around a dbt project, rendering each dbt node as an + Airflow task. Use ``DbtDag`` when the entire Airflow DAG is dedicated to running a dbt project. + See :ref:`dbt-dag`. + + DbtTaskGroup + A Cosmos class that embeds a dbt project as an Airflow ``TaskGroup`` inside a larger DAG. Use + ``DbtTaskGroup`` when dbt is one stage of a broader pipeline (e.g. ingestion → dbt → reporting). + See :ref:`dbt-task-group`. + + dbt deps + A `dbt command `_ that pulls the most recent + version of dependencies listed in your ``packages.yml`` file. + + Execution mode + Describes where and how Cosmos runs dbt commands. You configure it using ``ExecutionConfig``. + See :ref:`execution-modes`. + + ExecutionConfig + The Cosmos class used to configure the execution mode and related runtime options such as the + dbt executable path and invocation mode. See :ref:`execution-config`. + + Load mode + The method that Cosmos uses to parse your dbt project. See :ref:`parsing-methods`. + + Manifest + A dbt artifact that contains a full representation of the dbt project's resources, including all + node configurations and resource properties. + See `Manifest JSON file `_ in the dbt docs. + + Node + A dbt concept that encapsulates a step within a pipeline, such as a model, test, seed, or source. + + Partial parsing + A dbt feature that can greatly speed up parsing and execution when using the ``dbt_ls`` load mode + by only re-parsing files that have changed. See :ref:`partial-parsing`. + + Profile + The authentication information used by dbt to connect to your data warehouse. + See `profile `_. + + ProfileConfig + The Cosmos class that determines which data warehouse Cosmos connects to when executing dbt SQL. + See :ref:`connect_database`. + + Profile mapping + A Cosmos-provided resource that translates Airflow connections into dbt profiles. + See :ref:`use-profile-mapping`. + + profiles.yml + The file where dbt stores connection information for each data warehouse connection. + See :ref:`connect_database`, :ref:`use-your-profiles-yml`, or + `The profiles.yml file `_ + in the dbt docs. + + ProjectConfig + The Cosmos class used to specify where your dbt project is located and any project variables + that should be used for rendering and execution. + + RenderConfig + The Cosmos class that controls how a dbt project is turned into an Airflow DAG or TaskGroup, + including node selection, test behaviour, and source rendering. See :ref:`render-config`. + + Task + An Airflow term describing a single unit of work within a DAG. + See `Task `_. + + Test + Refers to the `dbt test command `_. You can + configure how Cosmos runs ``dbt test`` with :ref:`testing-behavior`. + + Workflow + A dbt term describing a pipeline that contains a group of steps. dbt can run a subset of tasks + assuming upstream tasks were already run. This is similar to the Airflow concept of a + `DAG `_. From ce52c8eac552c9e88305b33abe4db03727e70e8b Mon Sep 17 00:00:00 2001 From: Pankaj Singh Date: Wed, 29 Apr 2026 12:01:00 +0530 Subject: [PATCH 6/9] Fix undefined labels: point DbtDag/DbtTaskGroup refs to core-concepts Co-Authored-By: Claude Sonnet 4.6 --- docs/reference/glossary.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/reference/glossary.rst b/docs/reference/glossary.rst index 6670ce4dc7..09b8028f3f 100644 --- a/docs/reference/glossary.rst +++ b/docs/reference/glossary.rst @@ -15,12 +15,12 @@ Glossary DbtDag A Cosmos class that wraps a full Airflow DAG around a dbt project, rendering each dbt node as an Airflow task. Use ``DbtDag`` when the entire Airflow DAG is dedicated to running a dbt project. - See :ref:`dbt-dag`. + See :ref:`core-concepts`. DbtTaskGroup A Cosmos class that embeds a dbt project as an Airflow ``TaskGroup`` inside a larger DAG. Use ``DbtTaskGroup`` when dbt is one stage of a broader pipeline (e.g. ingestion → dbt → reporting). - See :ref:`dbt-task-group`. + See :ref:`core-concepts`. dbt deps A `dbt command `_ that pulls the most recent From 1c779a2269170f8246aaac825a10932280a794b2 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Wed, 29 Apr 2026 06:43:45 +0000 Subject: [PATCH 7/9] =?UTF-8?q?=F0=9F=8E=A8=20[pre-commit.ci]=20Auto=20for?= =?UTF-8?q?mat=20from=20pre-commit.com=20hooks?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/reference/glossary.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/glossary.rst b/docs/reference/glossary.rst index f46be78ae8..51bf4037aa 100644 --- a/docs/reference/glossary.rst +++ b/docs/reference/glossary.rst @@ -97,4 +97,4 @@ Glossary Controls when and how Cosmos runs dbt tests. ``AFTER_EACH`` (default) adds a test task after each model; ``AFTER_ALL`` runs all tests in a single task at the end; ``BUILD`` runs tests as part of ``dbt build`` inside the model task; ``NONE`` skips tests entirely. - Configured via ``RenderConfig``. See :ref:`testing-behavior`. \ No newline at end of file + Configured via ``RenderConfig``. See :ref:`testing-behavior`. From 69380c4159e45b88852a910d888f28cd9ed8c57a Mon Sep 17 00:00:00 2001 From: Pankaj Singh <98807258+pankajastro@users.noreply.github.com> Date: Wed, 29 Apr 2026 12:23:50 +0530 Subject: [PATCH 8/9] Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- docs/reference/glossary.rst | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/docs/reference/glossary.rst b/docs/reference/glossary.rst index 51bf4037aa..be7907888b 100644 --- a/docs/reference/glossary.rst +++ b/docs/reference/glossary.rst @@ -27,9 +27,12 @@ Glossary version of dependencies listed in your ``packages.yml`` file. Callbacks - User-defined functions that Cosmos calls before or after a dbt task executes. Useful for - custom logging, alerting, or side effects without modifying the operator. Configured via - ``operator_args``. See :ref:`callbacks`. + User-defined functions that Cosmos invokes after the dbt command completes, during + post-execution handling and before the temporary project directory is cleaned up. + Useful for custom logging, alerting, or side effects without modifying the operator. + Configure them on Cosmos operators via ``callback`` and ``callback_args``; when using + ``DbtDag`` or ``DbtTaskGroup``, these can be passed through ``operator_args``. + See :ref:`callbacks`. ExecutionConfig The Cosmos class used to configure the execution mode and related runtime options such as the From d2f1a9b66eaa6717d30b234cd4d317ca5ba91b47 Mon Sep 17 00:00:00 2001 From: Pankaj Singh Date: Mon, 4 May 2026 14:00:04 +0530 Subject: [PATCH 9/9] Address review feedback on glossary entries - Add Interceptors entry (new in Cosmos 1.14) - Expand InvocationMode to highlight memory/CPU gains and add reference to invocation-mode docs - Replace vague "greatly" in Partial parsing with measurable description; add Cosmos v1.4 version note Co-Authored-By: Claude Sonnet 4.6 --- docs/reference/glossary.rst | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/docs/reference/glossary.rst b/docs/reference/glossary.rst index be7907888b..8a9743e515 100644 --- a/docs/reference/glossary.rst +++ b/docs/reference/glossary.rst @@ -42,10 +42,20 @@ Glossary Describes where and how Cosmos runs dbt commands (e.g. ``LOCAL``, ``WATCHER``, ``DOCKER``). Configured via ``ExecutionConfig``. See :ref:`execution-modes`. + Interceptors + An optional list of callables (new in Cosmos 1.14) that run before Cosmos builds the dbt + command for each task. Each callable receives ``(context, operator)`` and may modify + ``operator.vars`` and ``operator.env``; the modified values are then used when building and + running the dbt command. Useful for injecting runtime variables or environment values per task + run. Configured via ``operator_args={"interceptors": [...]}`` on ``DbtDag`` or ``DbtTaskGroup``. + See :ref:`operator-args`. + InvocationMode Controls how Cosmos calls dbt: ``DBT_RUNNER`` imports dbt as a Python library in the same - process (faster, no subprocess overhead), while ``SUBPROCESS`` spawns a separate process - (better isolation when dbt and Airflow share a Python environment). + process (no subprocess overhead, lower CPU and memory usage), while ``SUBPROCESS`` spawns a + separate process (better isolation when dbt and Airflow share a Python environment). + Running as a subprocess roughly doubles memory usage compared to ``DBT_RUNNER``. + See :ref:`invocation-mode`. LoadMode The method Cosmos uses to parse your dbt project (e.g. ``DBT_MANIFEST``, ``DBT_LS``). @@ -57,8 +67,10 @@ Glossary See `Manifest JSON file `_ in the dbt docs. Partial parsing - A dbt feature that can greatly speed up parsing and execution when using the ``dbt_ls`` load mode - by only re-parsing files that have changed. See :ref:`partial-parsing`. + A dbt feature, enabled in Cosmos since v1.4, that skips re-parsing files that have not + changed. Each Airflow node performs a full dbt project parse only once; subsequent task runs + reuse the cached ``partial_parse.msgpack`` artifact, reducing parse time and CPU usage. + See :ref:`partial-parsing`. Profile The authentication information used by dbt to connect to your data warehouse.