Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/guides/dbt_docs/generating-docs.rst
Original file line number Diff line number Diff line change
Expand Up @@ -121,7 +121,7 @@ Choosing a folder
All the DbtDocsOperators support specification of a custom folder (prefix) to place documentation in on the target cloud storage. This can be done by
adding a ``folder_dir`` parameter to the operator definition.

Static Flag
Static flag
'''''''''''

All of the DbtDocsOperator accept the ``--static`` flag. To learn more about the static flag, check out the `original PR on dbt-core <https://github.com/dbt-labs/dbt-docs/pull/465>`_.
Expand Down Expand Up @@ -149,7 +149,7 @@ The following code snippet shows how to provide this flag with the default jaffl
dbt_cmd_flags=["--static"],
)

Custom Callback
Custom callback
'''''''''''''''

If you want to run custom code after the docs are generated, you can use the :class:`~cosmos.operators.DbtDocsOperator`. The following code snippet shows how to do this with the default jaffle_shop project:
Expand Down
2 changes: 1 addition & 1 deletion docs/guides/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ Specify more details about how Cosmos runs both dbt commands and Airflow Dags. T
run_dbt/operators/index
run_dbt/customization/index

Multi-project Setups
Multi-project setups
++++++++++++++++++++

If you have a multi-project architecture where you have multiple dbt projects that reference each others' models, you can set up ``dbt-loom`` with Cosmos to handle cross-project references.
Expand Down
18 changes: 9 additions & 9 deletions docs/guides/multi_project/multi-project.rst
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
.. _multi-project:

Multi-Project Setups
Multi-project setups
--------------------

Cosmos supports multi-project dbt architectures where multiple dbt projects reference each other's models.
Expand All @@ -15,7 +15,7 @@ This allows you to:

Cosmos works with dbt-loom out of the box, automatically handling external node references.

How dbt-loom Works
How dbt-loom works
++++++++++++++++++

dbt-loom enables cross-project references by:
Expand All @@ -24,7 +24,7 @@ dbt-loom enables cross-project references by:
2. Injecting the upstream models' metadata into the downstream project's namespace
3. Allowing cross-project references using the dbt Mesh syntax: ``{{ ref('upstream_project', 'model_name') }}``

How Cosmos Handles dbt-loom
How Cosmos handles dbt-loom
+++++++++++++++++++++++++++

When Cosmos parses a dbt project that uses dbt-loom, it encounters two types of nodes:
Expand Down Expand Up @@ -62,10 +62,10 @@ The upstream manifest can be generated by running any dbt command that parses th
# or
dbt ls # Lists resources and generates manifest

Configuration Example
Configuration example
+++++++++++++++++++++

Project Structure
Project structure
'''''''''''''''''

A typical dbt-loom setup has an upstream project and one or more downstream projects:
Expand All @@ -92,7 +92,7 @@ A typical dbt-loom setup has an upstream project and one or more downstream proj
└── models/
└── fct_revenue.sql # References upstream models

Upstream Project Configuration
Upstream project configuration
''''''''''''''''''''''''''''''

The upstream project exposes models as ``public`` for cross-project access:
Expand All @@ -115,7 +115,7 @@ The upstream project exposes models as ``public`` for cross-project access:
+materialized: view
+access: public

Downstream Project Configuration
Downstream project configuration
''''''''''''''''''''''''''''''''

The downstream project configures dbt-loom to read from the upstream manifest.
Expand Down Expand Up @@ -335,7 +335,7 @@ To disable automatic asset emission:
# ...
)

Cross-Project Sources
Cross-project sources
+++++++++++++++++++++

dbt-loom handles **model references** but does not directly support cross-project source references
Expand Down Expand Up @@ -397,7 +397,7 @@ If you must reference the same raw table in multiple projects, define the source
This approach requires keeping source definitions in sync across projects.


Cross-Project Macros
Cross-project macros
++++++++++++++++++++

dbt-loom does **not** handle macro sharing. Macros are resolved at compile time within each project.
Expand Down
6 changes: 3 additions & 3 deletions docs/guides/run_dbt/airflow-worker/async-execution-mode.rst
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ We have `observed <https://github.com/astronomer/astronomer-cosmos/pull/1934>`_
+----------------------------------------------+--------------------------+


Getting Started with Airflow Async Mode
Getting started with Airflow async mode
+++++++++++++++++++++++++++++++++++++++

This guide walks you through setting up an Astro CLI project and running a Cosmos-based DAG with a deferrable operator, enabling asynchronous task execution in Apache Airflow.
Expand All @@ -47,7 +47,7 @@ Prerequisites
- `Astro CLI <https://www.astronomer.io/docs/astro/cli/install-cli>`_
- Airflow>=2.9

1. Create Astro-CLI Project
1. Create Astro-CLI project
'''''''''''''''''''''''''''

Run the following command in your terminal:
Expand Down Expand Up @@ -82,7 +82,7 @@ Edit your Dockerfile to ensure all necessary requirements are included.
FROM astrocrpublic.azurecr.io/runtime:3.0-2


3. Add astronomer-cosmos Dependency
3. Add astronomer-cosmos dependency
'''''''''''''''''''''''''''''''''''

In your ``requirements.txt``, add:
Expand Down
10 changes: 5 additions & 5 deletions docs/guides/run_dbt/airflow-worker/watcher-execution-mode.rst
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
.. _watcher-execution-mode:

Watcher execution mode (Experimental)
Watcher execution mode (experimental)
-------------------------------------

With the release of **Cosmos 1.11.0**, we are introducing a powerful new experimental execution mode — ``ExecutionMode.WATCHER`` — designed to drastically reduce dbt pipeline run times in `Apache Airflow® <https://airflow.apache.org/>`_.
Expand All @@ -12,7 +12,7 @@ Early benchmarks show that ``ExecutionMode.WATCHER`` can cut total DAG runtime *
- The ``poke_interval`` and ``timeout`` settings of the ``DbtConsumerWatcherSensor`` operator, which determine the frequency and duration of the sensor's polling.


Background: The Problem with the Local Execution Mode in Cosmos
Background: The problem with the local execution mode in Cosmos
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Comment thread
pankajkoti marked this conversation as resolved.

When running dbt via Cosmos using the default ``ExecutionMode.LOCAL``, each dbt model is executed as a separate Airflow task.
Expand Down Expand Up @@ -52,7 +52,7 @@ Together, these operators let you:
- Retry specific models (for resilience)


Performance Gains
Performance gains
+++++++++++++++++

We used a dbt project developed by Google, the `google/fhir-dbt-analytics <https://github.com/google/fhir-dbt-analytics>`_ project, that interfaces with BigQuery. It contains:
Expand Down Expand Up @@ -128,7 +128,7 @@ If you prefer to manage threads through Cosmos profile mappings instead of editi



Example Usage of ``ExecutionMode.WATCHER``
Example usage of ``ExecutionMode.WATCHER``
++++++++++++++++++++++++++++++++++++++++++

There are two main ways to use the new execution mode in Cosmos — directly within a ``DbtDag``, or embedded as part of a ``DbtTaskGroup`` inside a larger DAG.
Expand Down Expand Up @@ -341,7 +341,7 @@ Starting with Cosmos 1.12.0, the ``DbtConsumerWatcherSensor`` supports
which increases overall task throughput. By default, the sensor now runs in deferrable mode.


Known Limitations
Known limitations
+++++++++++++++++

Producer task implementation
Expand Down
2 changes: 1 addition & 1 deletion docs/guides/run_dbt/container/kubernetes.rst
Original file line number Diff line number Diff line change
Expand Up @@ -171,7 +171,7 @@ Enable and trigger a run of the `jaffle_shop_k8s <https://github.com/astronomer/

.. _kubernetes-known-limitations:

Known Limitations
Known limitations
+++++++++++++++++

The Kubernetes execution mode has the following limitations:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
.. _watcher-kubernetes-execution-mode:


Watcher Kubernetes execution mode (Experimental)
Watcher Kubernetes execution mode (experimental)
------------------------------------------------

.. versionadded:: 1.13.0
Expand All @@ -25,7 +25,7 @@ However, the original ``ExecutionMode.WATCHER`` requires dbt to be installed alo
For more details on the watcher concept and how it works, please refer to the :ref:`watcher-execution-mode` documentation.


How to Use
How to use
++++++++++

Comment thread
pankajkoti marked this conversation as resolved.
Users previously using ``ExecutionMode.KUBERNETES`` can simply replace the ``execution_mode`` to use ``ExecutionMode.WATCHER_KUBERNETES``.
Expand Down Expand Up @@ -61,7 +61,7 @@ The following example shows how to configure a ``DbtDag`` with ``ExecutionMode.W
For the complete setup including Kubernetes secrets, Docker image configuration, and profile setup, refer to the :ref:`kubernetes` documentation.


Performance Gains
Performance gains
+++++++++++++++++

Early benchmarks using the ``jaffle_shop_watcher_kubernetes`` DAG show significant improvements:
Expand All @@ -83,10 +83,10 @@ The performance improvement comes from:
- Eliminating repeated dbt initialization for each model


Known Limitations
Known limitations
+++++++++++++++++

Kubernetes Provider Version Compatibility
Kubernetes provider version compatibility
'''''''''''''''''''''''''''''''''''''''''

``ExecutionMode.WATCHER_KUBERNETES`` does not work with older versions of the ``apache-airflow-providers-cncf-kubernetes`` provider (<=10.7.0).
Expand Down Expand Up @@ -154,7 +154,7 @@ The ``operator_args`` must define ``get_logs`` and ``log_events_on_failure``:
)


Other Inherited Limitations
Other inherited limitations
'''''''''''''''''''''''''''

The following limitations from ``ExecutionMode.WATCHER`` also apply to ``ExecutionMode.WATCHER_KUBERNETES``:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
.. _custom-airflow-properties:

Apache Airflow® Configuration Overrides with Astronomer Cosmos
Apache Airflow® configuration overrides with Astronomer Cosmos
---------------------------------------------------------------

**Astronomer Cosmos** allows you to override Airflow configurations for each dbt task (dbt operator) via the dbt YAML file.

Sample dbt Model YAML
Sample dbt model YAML
+++++++++++++++++++++
Comment thread
pankajkoti marked this conversation as resolved.

.. code-block:: yaml
Expand Down
6 changes: 3 additions & 3 deletions docs/guides/run_dbt/customization/scheduling.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ Scheduling

Because Cosmos uses `Apache Airflow® <https://airflow.apache.org/>`_ to power scheduling, you can leverage Airflow's scheduling capabilities to schedule your dbt projects. This includes cron-based scheduling, timetables, and data-aware scheduling. For more info on Airflow's scheduling capabilities, check out the Airflow documentation or check out the `Astronomer documentation <https://docs.astronomer.io/learn/scheduling-in-airflow>`_.

Time-Based Scheduling
Time-based scheduling
+++++++++++++++++++++

To schedule a dbt project on a time-based schedule, you can use Airflow's scheduling options. For example, to run a dbt project every day starting on January 1, 2023, you can use the following DAG:
Expand All @@ -22,7 +22,7 @@ To schedule a dbt project on a time-based schedule, you can use Airflow's schedu

.. _data-aware-scheduling:

Data-Aware Scheduling
Data-aware scheduling
+++++++++++++++++++++

`Apache Airflow® <https://airflow.apache.org/>`_ 2.4 introduced the concept of `scheduling based on Datasets <https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/datasets.html>`_.
Expand Down Expand Up @@ -247,7 +247,7 @@ How Dataset Emission Differs by Execution Mode
emitted URIs by enabling the ``enable_uri_xcom`` setting (see `Emitting Dataset URIs as XCom`_).


Known Limitations
Known limitations
'''''''''''''''''

Airflow 3.0 and beyond
Expand Down
6 changes: 3 additions & 3 deletions docs/guides/translate_dbt_to_airflow/managing-sources.rst
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
.. _managing-sources:

Managing Sources
Managing sources
----------------

.. note::
Expand Down Expand Up @@ -37,7 +37,7 @@ Example:
)


Source Pruning
Source pruning
++++++++++++++
Comment thread
pankajkoti marked this conversation as resolved.

The ``source_pruning`` is a boolean parameter available in the ``RenderConfig``.
Expand All @@ -63,7 +63,7 @@ Example:
In this example, if the ``jaffle_shop`` project has multiple sources, but only some of them are upstream of the ``customers`` model, Cosmos will only render the necessary sources and prune the rest.


on_warning_callback Callback
on_warning_callback callback
++++++++++++++++++++++++++++

The ``on_warning_callback`` is a callback parameter available on the ``DbtSourceLocalOperator``. This callback is triggered when a warning occurs during the execution of the ``dbt source freshness`` command. The callback accepts the task context, which includes additional parameters: test_names and test_results
Expand Down
8 changes: 4 additions & 4 deletions docs/guides/translate_dbt_to_airflow/testing-behavior.rst
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
.. _testing-behavior:

Testing Behavior
Testing behavior
----------------

Testing Configuration
Testing configuration
+++++++++++++++++++++

By default, Cosmos will add a test after each model. This can be overridden using the ``test_behavior`` field in the ``RenderConfig`` object.
Expand Down Expand Up @@ -49,7 +49,7 @@ Finally, an example DAG and how it is rendered in the `Apache Airflow® <https:/

.. image:: ../../_static/test_behavior_build.png

Warning Behavior
Warning behavior
++++++++++++++++

.. note::
Expand Down Expand Up @@ -113,7 +113,7 @@ When at least one WARN message is present, the function passed to ``on_warning_c
``test_results`` context variables, which are specific to test-related warnings.


Tests with Multiple Parents
Tests with multiple parents
+++++++++++++++++++++++++++

It is common for dbt projects to define tests that rely on multiple upstream models, snapshots or seeds.
Expand Down
6 changes: 3 additions & 3 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -35,14 +35,14 @@ simple to manage and orchestrate your dbt workflows using `Apache Airflow® <htt
time and effort. By automatically turning dbt workflows into Airflow DAGs, Cosmos allows you to focus on building
high-quality data models without the hassle of managing complex integrations.

What Is Astronomer Cosmos?
What is Astronomer Cosmos?
--------------------------

Astronomer Cosmos is an open-source library that bridges Apache Airflow and dbt, allowing you to easily transform your
dbt projects into Airflow Dags and manage everything seamlessly. With Cosmos, you can write your data transformations
using dbt and then schedule and orchestrate them with Airflow, making the entire process smooth and straightforward.

Why Should You Use Cosmos?
Why should you use Cosmos?
--------------------------

Integrating dbt and Airflow can be complex, but Cosmos simplifies it by seamlessly connecting these
Expand All @@ -66,7 +66,7 @@ Check out the Cosmos Fundamentals, get started locally with the Quickstart Guide
You can also explore more examples in `/dev/dags <https://github.com/astronomer/astronomer-cosmos/tree/main/dev/dags>`_
or in the `cosmos-demo repo <https://github.com/astronomer/cosmos-demo>`_.

Join the Community
Join the community
------------------

Have questions, need help, or interested in contributing? We welcome all contributions and feedback!
Expand Down
2 changes: 1 addition & 1 deletion docs/optimize_performance/invocation_mode.rst
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
.. _invocation-mode:

Invocation Modes
Invocation modes
----------------
.. versionadded:: 1.4

Expand Down
Loading
Loading