Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/guides/connect_database/index.rst
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
.. _connect_database:

Connect to your database
------------------------
========================

Cosmos supports two methods of authenticating with your database:

Expand Down
6 changes: 3 additions & 3 deletions docs/guides/connect_database/profile-customise-per-node.rst
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
.. _profile-customise-per-node:

Customising the profile config per dbt node
-------------------------------------------
===========================================

.. versionadded:: 1.9.0

Expand Down Expand Up @@ -52,7 +52,7 @@ to subsets of nodes selected based on path or other criteria that dbt supports.


Dbt profile config variables
++++++++++++++++++++++++++++
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. versionadded:: 1.4.0

The parts of ``profiles.yml``, which aren't specific to a particular data platform `dbt docs <https://docs.getdbt.com/docs/core/connect-data-platform/profiles.yml>`_
Expand Down Expand Up @@ -90,7 +90,7 @@ The parts of ``profiles.yml``, which aren't specific to a particular data platfo


Disabling dbt event tracking
++++++++++++++++++++++++++++
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. note:
Deprecated in v.1.4 and will be removed in v2.0.0. Use dbt_config_vars=DbtProfileConfigVars(send_anonymous_usage_stats=False) instead.
Expand Down
2 changes: 1 addition & 1 deletion docs/guides/connect_database/use-profile-mapping.rst
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
.. _use-profile-mapping:

Use a profile mapping
---------------------
=====================

Profile mappings are utilities provided by Cosmos that translate `Apache Airflow® <https://airflow.apache.org/>`_ connections to dbt profiles. This means that
you can use the same connection objects you use in Airflow to authenticate with your database in dbt. To do so, there's
Expand Down
2 changes: 1 addition & 1 deletion docs/guides/connect_database/use-your-profiles-yml.rst
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
.. _use-your-profiles-yml:

Use your own profiles.yml file
------------------------------
==============================

If you don't want to use `Apache Airflow® <https://airflow.apache.org/>`_ connections, or if there's no readily-available profile mapping for your database,
you can use your own dbt profiles.yml file. To do so, you'll need to pass the path to your profiles.yml file to the
Expand Down
2 changes: 1 addition & 1 deletion docs/guides/cosmos_devex/compiled-sql.rst
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
.. _compiled-sql:

Compiled SQL
------------
============

When using the local execution mode, Cosmos will store the compiled SQL for each model in the ``compiled_sql`` field of the task's ``template_fields``. This allows you to view the compiled SQL in the `Apache Airflow® UI <https://airflow.apache.org/>`_.

Expand Down
10 changes: 5 additions & 5 deletions docs/guides/cosmos_devex/lineage.rst
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
.. _lineage:

Configuring Lineage
-------------------
===================

Since Cosmos 1.1, it uses internally the `openlineage-integration-common <https://github.com/OpenLineage/OpenLineage/tree/main/integration/common>`_
to parse artefacts generated with dbt commands and create lineage events.
Expand All @@ -15,7 +15,7 @@ No change to the user DAG files is required to use OpenLineage.


Known limitations
+++++++++++++++++
~~~~~~~~~~~~~~~~~

This feature is only available for the local
and virtualenv execution methods (read `execution modes <../run_dbt/execution-modes.html>`_ for more information).
Expand All @@ -36,19 +36,19 @@ Additionally, since Cosmos uses the open-source `openlineage-integration-common
Contributions are also welcome in the `OpenLineage project <https://github.com/OpenLineage/OpenLineage/blob/main/integration/common/openlineage/common/provider/dbt/processor.py#L36C1-L47C22>`_ to support more adaptors.

Installation
++++++++++++
~~~~~~~~~~~~

Install ``apache-airflow-providers-openlineage``.


Configuration
+++++++++++++
~~~~~~~~~~~~~

Follow `the instructions <https://airflow.apache.org/docs/apache-airflow-providers-openlineage/1.0.2/guides/user.html>`_ on how to configure OpenLineage.


Namespace
'''''''''
+++++++++

Cosmos will use the Airflow ``[openlineage]`` ``namespace`` property as a namespace, `if available <https://airflow.apache.org/docs/apache-airflow-providers-openlineage/1.0.2/guides/user.html>`_.

Expand Down
2 changes: 1 addition & 1 deletion docs/guides/cosmos_devex/logging.rst
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
.. _logging:

Logging
-------
=======

Cosmos allows for a custom logger implementation that adds ``(astronomer-cosmos)`` to each log message.

Expand Down
2 changes: 1 addition & 1 deletion docs/guides/cosmos_devex/task-display-name.rst
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
.. _task-display-name:

Task display name
-----------------
=================

.. note::
This feature is only available for Airflow >= 2.9.
Expand Down
18 changes: 9 additions & 9 deletions docs/guides/dbt_docs/generating-docs.rst
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
.. _generating-docs:

Generating Docs
---------------
===============

dbt allows you to generate static documentation on your models, tables, and more. You can read more about it in the `official dbt documentation <https://docs.getdbt.com/docs/building-a-dbt-project/documentation>`_. For an example of what the docs look like with the ``jaffle_shop`` project, check out `this site <http://cosmos-demo-dbt-docs.s3-website.eu-north-1.amazonaws.com/>`_.

Expand All @@ -21,10 +21,10 @@ The first four operators require you to have a connection to the target storage.


Examples
++++++++
~~~~~~~~

Upload to S3
''''''''''''
++++++++++++

S3 supports serving static files directly from a bucket. To learn more (and to set it up), check out the `official S3 documentation <https://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteHosting.html>`_.

Expand All @@ -45,7 +45,7 @@ You can use the :class:`~cosmos.operators.DbtDocsS3Operator` to generate and upl
)

Upload to S3 from Kubernetes
''''''''''''''''''''''''''''
++++++++++++++++++++++++++++

.. versionadded:: 1.15.0

Expand Down Expand Up @@ -74,7 +74,7 @@ The ``connection_id`` is resolved from Airflow and translated into AWS environme
This Kubernetes integration currently supports S3 only. If you need another storage backend, use one of the local operators or extend Cosmos with another Kubernetes docs operator.

Upload to Azure Blob Storage
''''''''''''''''''''''''''''
++++++++++++++++++++++++++++

Azure Blob Storage supports serving static files directly from a container. To learn more (and to set it up), check out the `official documentation <https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-static-website>`_.

Expand All @@ -95,7 +95,7 @@ You can use the :class:`~cosmos.operators.DbtDocsAzureStorageOperator` to genera
)

Upload to GCS
'''''''''''''
+++++++++++++

GCS supports serving static files directly from a bucket. To learn more (and to set it up), check out the `official GCS documentation <https://cloud.google.com/appengine/docs/standard/serving-static-files?tab=python>`_.

Expand All @@ -116,13 +116,13 @@ You can use the :class:`~cosmos.operators.DbtDocsGCSOperator` to generate and up
)

Choosing a folder
'''''''''''''''''
+++++++++++++++++

All the DbtDocsOperators support specification of a custom folder (prefix) to place documentation in on the target cloud storage. This can be done by
adding a ``folder_dir`` parameter to the operator definition.

Static flag
'''''''''''
+++++++++++

All of the DbtDocsOperator accept the ``--static`` flag. To learn more about the static flag, check out the `original PR on dbt-core <https://github.com/dbt-labs/dbt-docs/pull/465>`_.
The static flag is used to generate a single doc file that can be hosted directly from cloud storage.
Expand Down Expand Up @@ -150,7 +150,7 @@ The following code snippet shows how to provide this flag with the default jaffl
)

Custom callback
'''''''''''''''
+++++++++++++++

If you want to run custom code after the docs are generated, you can use the :class:`~cosmos.operators.DbtDocsOperator`. The following code snippet shows how to do this with the default jaffle_shop project:

Expand Down
28 changes: 14 additions & 14 deletions docs/guides/dbt_docs/hosting-docs.rst
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
.. hosting-docs:

Hosting Docs
------------
============

dbt docs can be served directly from the `Apache Airflow® <https://airflow.apache.org/>`_ webserver (Airflow 2) or API server (Airflow 3) with the Cosmos Airflow plugin, without requiring the user to set up anything outside of Airflow. This page describes how to host docs in the Airflow webserver/API server directly, although some users may opt to host docs externally.

Expand All @@ -13,11 +13,11 @@ Airflow 2 and Airflow 3 use different UI plugin systems.


Overview
++++++++
~~~~~~~~


Apache Airflow® 3
++++++++++++++++++
~~~~~~~~~~~~~~~~~~

The dbt docs are available in the Airflow menu under ``Browse``:

Expand Down Expand Up @@ -54,10 +54,10 @@ Endpoints per project (``<slug>``):


Local vs Remote docs (Apache Airflow® 3)
'''''''''''''''''''''''''''''''''''''''''
+++++++++++++++++++++++++++++++++++++++++

Local docs (filesystem paths)
.............................
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

- When a project's ``dir`` is a local path (e.g. ``/usr/local/airflow/dbt/jaffle_shop/target``):
- The plugin serves ``/cosmos/<slug>/dbt_docs_index.html`` by reading ``<dir>/<index>`` (default ``index.html``).
Expand All @@ -71,7 +71,7 @@ Local docs (filesystem paths)
}

Remote docs (S3, GCS, Azure, HTTP[S])
.....................................
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

- When ``dir`` is a remote URI (e.g. ``s3://...``, ``gs://...``, ``abfs://...``, ``wasb://...``, ``http(s)://...``):
- Files (index, manifest.json, catalog.json) are read via Airflow ObjectStorage (uPath/fsspec).
Expand All @@ -87,14 +87,14 @@ Remote docs (S3, GCS, Azure, HTTP[S])
}

Notes and behavior
..................
^^^^^^^^^^^^^^^^^^

- Missing artifacts return HTTP 404 with the attempted path; other IO errors return HTTP 500 and are logged.
- The docs iframe page (``/cosmos/<slug>/dbt_docs``) links to the index and shows within the Airflow UI; the index adds a small inline script to keep browser back/forward behavior intuitive.
- If you deploy behind a path prefix (e.g. Astronomer hosted Airflow deployments), menu links include the prefix automatically using ``AIRFLOW__API__BASE_URL``.

Apache Airflow® 2
++++++++++++++++++
~~~~~~~~~~~~~~~~~~

.. important::
The remainder of this page applies to Airflow 2 (Flask-AppBuilder) plugins only.
Expand Down Expand Up @@ -134,7 +134,7 @@ If your docs were generated using the ``--static`` flag, you can set the index f


Host from Cloud Storage
+++++++++++++++++++++++
~~~~~~~~~~~~~~~~~~~~~~~

For typical users, the recommended setup for hosting dbt docs would look like this:

Expand All @@ -143,7 +143,7 @@ For typical users, the recommended setup for hosting dbt docs would look like th
3. If you want to use a conn ID other than the default connection, set your ``cosmos.dbt_docs_conn_id``. Otherwise, leave this blank.

AWS S3 Example
''''''''''''''
++++++++++++++

.. code-block:: cfg

Expand All @@ -157,7 +157,7 @@ AWS S3 Example
AIRFLOW__COSMOS__DBT_DOCS_CONN_ID="aws_default"

Google Cloud Storage Example
''''''''''''''''''''''''''''
++++++++++++++++++++++++++++

.. code-block:: cfg

Expand All @@ -171,7 +171,7 @@ Google Cloud Storage Example
AIRFLOW__COSMOS__DBT_DOCS_CONN_ID="google_cloud_default"

Azure Blob Storage Example
''''''''''''''''''''''''''
++++++++++++++++++++++++++

.. code-block:: cfg

Expand All @@ -185,7 +185,7 @@ Azure Blob Storage Example
AIRFLOW__COSMOS__DBT_DOCS_CONN_ID="wasb_default"

Host from Local Storage
+++++++++++++++++++++++
~~~~~~~~~~~~~~~~~~~~~~~

By default, Cosmos will not generate docs on the fly. Local storage only works if you are pre-compiling your dbt project before deployment.

Expand Down Expand Up @@ -228,7 +228,7 @@ For example, Astronomer does not update the resources available to the webserver
cp dags/dbt/target/index.html dbt_docs_dir/index.html

Host from HTTP/HTTPS
++++++++++++++++++++
~~~~~~~~~~~~~~~~~~~~

.. code-block:: cfg

Expand Down
10 changes: 5 additions & 5 deletions docs/guides/dbt_setup/dbt-fusion.rst
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
.. _dbt_fusion:

dbt Fusion support
------------------
==================

.. note::
Available since Cosmos 1.11.0a1 when using ``ExecutionMode.LOCAL``.

Context
+++++++
~~~~~~~

dbt Labs `launched <https://www.getdbt.com/blog/dbt-launch-showcase-2025-recap>`_ `dbt Fusion <https://github.com/dbt-labs/dbt-fusion>`_ on 28 May 2025. dbt Fusion is the next-generation dbt engine that enables real-time model validation, improved SQL parsing and state-aware orchestration.
It is a unified approach that aims to merge dbt Core and dbt Cloud features via a completely new CLI,
Expand All @@ -29,14 +29,14 @@ Some reported dbt Fusion features include:
dbt Fusion is in public beta with current support for Snowflake and Databricks projects, with additional data platforms coming soon.

Support
+++++++
~~~~~~~

Cosmos 1.11.0a1 adds initial support for running dbt Fusion with Cosmos when using ``ExecutionMode.LOCAL``.

We do not have a solution for using `ExecutionMode.AIRFLOW_ASYNC <https://astronomer.github.io/astronomer-cosmos/guides/run_dbt/airflow-worker/async-execution-mode.html>`_.

How to use
++++++++++
~~~~~~~~~~

1. Install Cosmos 1.11.0a1 alpha

Expand Down Expand Up @@ -69,7 +69,7 @@ Example:


Limitations
+++++++++++
~~~~~~~~~~~

- This feature is in Cosmos 1.11 pre-release versions
- Currently (23 June 2025) dbt Fusion is still in beta
Expand Down
6 changes: 3 additions & 3 deletions docs/guides/dbt_setup/execution-modes-local-conflicts.rst
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
.. _execution-modes-local-conflicts:

Apache Airflow® and dbt dependencies conflicts
-----------------------------------------------
===============================================

When using the :ref:`local-execution` without defining a custom ``ExecutionConfig.dbt_executable_path``, you might have dependency conflicts between
`Apache Airflow® <https://airflow.apache.org/>`_ and dbt. The number of conflicts can increase depending on the Airflow providers and dbt adapters you use.
Expand Down Expand Up @@ -42,7 +42,7 @@ In the following table, ``x`` represents combinations that lead to conflicts (va
+---------------+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+------+

Examples of errors
++++++++++++++++++
~~~~~~~~~~~~~~~~~~

.. code-block:: bash

Expand Down Expand Up @@ -99,7 +99,7 @@ The conflict is caused by:


How to reproduce
++++++++++++++++
~~~~~~~~~~~~~~~~

The table was created by running `nox <https://nox.thea.codes/en/stable/>`__ with the following ``noxfile.py``:

Expand Down
Loading