[Docs] Restructure Getting Started and Guides section #2418

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

tatiana merged 29 commits into astronomer:main from lzdanski:restructure-get-started-guides

Mar 6, 2026

docs/conf.py

-Original file line number
+Diff line change
@@ Expand Up / @@ -61,7 +61,28 @@ @@
     redirects = {
         "airflow3_compatibility/index": "../policy/airflow3-compatibility.html",
         "compatibility-policy": "../policy/compatibility-policy.html",
+        "configuration/caching": "../optimize_performance/caching.html",
+        "configuration/dag-customization": "../guides/translate_dbt_to_airflow/dag-customization.html",
+        "configuration/memory_optimization": "../optimize_performance/memory_optimization.html",
+        "configuration/parsing-methods": "../guides/translate_dbt_to_airflow/parsing-methods.html",
+        "configuration/partial-parsing": "../guides/run_dbt/customization/partial-parsing.html",
+        "configuration/render-config": "../guides/translate_dbt_to_airflow/render-config.html",
+        "configuration/selecting-excluding": "../guides/translate_dbt_to_airflow/selecting-excluding.html",
+        "configuration/source-nodes-rendering": "../guides/translate_dbt_to_airflow/managing-sources.html",
+        "configuration/testing-behavior": "../guides/translate_dbt_to_airflow/testing-behavior.html",
         "contributing": "../policy/contributing.html",
         "contributors": "../policy/contributors.html",
         "contributors-roles": "../policy/contributors-roles.html",
+        "getting_started/async-execution-mode": "../guides/run_dbt/airflow-worker/async-execution-mode.html",
+        "getting_started/aws-container-run-job": "../guides/run_dbt/airflow-worker/async-execution-mode.html",
+        "getting_started/azure-container-instance": "../guides/run_dbt/container/azure-container-instance.html",
+        "getting_started/custom-airflow-properties": "../guides/run_dbt/customization/custom-airflow-properties.html",
+        "getting_started/docker": "../guides/run_dbt/container/docker.html",
+        "getting_started/execution-modes-local-conflicts": "../guides/run_dbt/airflow-worker/execution-modes-local-conflicts.html",
+        "getting_started/execution-modes": "../guides/run_dbt/execution-modes.html",
+        "getting_started/gcp-cloud-run-job": "../guides/run_dbt/container/gcp-cloud-run-job.html",
+        "getting_started/kubernetes": "../guides/run_dbt/container/kubernetes.html",
+        "getting_started/operators": "../guides/run_dbt/operators/operators.html",
+        "getting_started/watcher-execution-mode": "../guides/run_dbt/airflow-worker/watcher-execution-mode.html",
+        "getting_started/watcher-kubernetes-execution-mode": "../guides/run_dbt/container/watcher-kubernetes-execution-mode.html",
     }

docs/configuration/index.rst

This file was deleted.

docs/getting_started/astro.rst

-Original file line number
+Diff line change
@@ -1,7 +1,7 @@
     .. _astro:
-    Getting Started on Astro
-    ========================
+    Getting Started with Cosmos on Astro
+    ====================================
     While it is possible to use Cosmos on Astro with all :ref:`Execution Modes <execution-modes>`, we recommend using the ``local`` execution mode. It's the simplest to set up and use.
@@ Expand Down @@

docs/getting_started/dbt-airflow-concepts.rst

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -1,7 +1,7 @@
  
    .. _dbt-airflow-concepts:

    Similar dbt & Airflow concepts

    ==============================

    Similar dbt and Airflow concepts

    ================================

    While dbt is an open source tool for data transformations and analysis, using SQL, Airflow focuses on being a platform

    for the development, scheduling and monitoring of batch-oriented workflows, using Python. Although both tools have many

docs/getting_started/index.rst

-Original file line number
+Diff line change
@@ -1,26 +1,28 @@
     .. _getting-started:
     .. toctree::
+       :maxdepth: 1
        :hidden:
-       :caption: Contents:
+       :caption: Cosmos Fundamentals
+       Similar dbt and Airflow concepts <dbt-airflow-concepts>
+    .. toctree::
+       :maxdepth: 1
+       :hidden:
+       :caption: Quickstart
        Astro CLI quickstart <astro-cli-quickstart>
+    .. toctree::
+       :maxdepth: 1
+       :hidden:
+       :caption: Get started with Cosmos
+       Open-source Airflow <open-source>
        Astro <astro>
-       MWAA <mwaa>
-       GCC <gcc>
-       Open-Source <open-source>
-       Execution Modes <execution-modes>
-       Docker Execution Mode <docker>
-       Kubernetes Execution Mode <kubernetes>
-       Azure Container Instance Execution Mode <azure-container-instance>
-       AWS Container Run Job Execution Mode <aws-container-run-job>
-       GCP Cloud Run Job Execution Mode <gcp-cloud-run-job>
-       Airflow Async Execution Mode <async-execution-mode>
-       Watcher Execution Mode <watcher-execution-mode>
-       Watcher Kubernetes Execution Mode <watcher-kubernetes-execution-mode>
-       dbt and Airflow Similar Concepts <dbt-airflow-concepts>
-       Operators <operators>
-       Custom Airflow Properties <custom-airflow-properties>
+       Amazon Managed Workflows for Apache Airflow (MWAA) <mwaa>
+       Google Cloud Composer (GCC) <gcc>
     Getting Started
@@ Expand Down Expand Up @@
     For specific guides, see the following:
-    - `Executing dbt Dags with DockerOperators <docker.html>`__
-    - `Executing dbt Dags with KubernetesPodOperators <kubernetes.html>`__
-    - `Executing dbt Dags with Watcher Kubernetes Mode <watcher-kubernetes-execution-mode.html>`__
-    - `Executing dbt Dags with AzureContainerInstancesOperators <azure-container-instance.html>`__
-    - `Executing dbt Dags with GcpCloudRunExecuteJobOperators <gcp-cloud-run-job.html>`__
+    - `Executing dbt DAGs with DockerOperators <../../guides/run_dbt/container/docker.html>`__
+    - `Executing dbt DAGs with KubernetesPodOperators <../../guides/run_dbt/container/kubernetes.html>`__
+    - `Executing dbt DAGs with Watcher Kubernetes Mode <../../guides/run_dbt/container/watcher-kubernetes-execution-mode.html>`__
+    - `Executing dbt DAGs with AzureContainerInstancesOperators <../../guides/run_dbt/container/azure-container-instance.html>`__
+    - `Executing dbt DAGs with GcpCloudRunExecuteJobOperators <../../guides/run_dbt/container/gcp-cloud-run-job.html>`__
+    Concepts Overview
+    -----------------
+    How do dbt and Airflow concepts map to each other? Learn more `in this link <dbt-airflow-concepts.html>`__.

docs/getting_started/mwaa.rst

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -1,7 +1,7 @@
  
    .. _mwaa:

    Getting Started on MWAA

    =======================

    Getting Started with Cosmos on Amazon Managed Workflows

    =======================================================

    Users can face Python dependency issues when trying to use the Cosmos `Local Execution Mode <execution-modes.html#local>`_ in Amazon Managed Workflows for `Apache Airflow® <https://airflow.apache.org/>`_ (MWAA).

docs/getting_started/open-source.rst

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -1,7 +1,7 @@
  
    .. _open-source:

    Getting Started on Open Source Airflow

    ======================================

    Getting Started with Cosmos on Open-source Airflow

    ==================================================

    When running open-source Airflow, your setup may vary. This guide assumes you have access to edit the underlying image.

0 docs/configuration/compiled-sql.rst → docs/guides/cosmos_devex/compiled-sql.rst

File renamed without changes.

0 docs/configuration/lineage.rst → docs/guides/cosmos_devex/lineage.rst

File renamed without changes.

0 docs/configuration/logging.rst → docs/guides/cosmos_devex/logging.rst

File renamed without changes.

0 docs/configuration/task-display-name.rst → ...guides/cosmos_devex/task-display-name.rst

File renamed without changes.

0 docs/configuration/generating-docs.rst → docs/guides/dbt_docs/generating-docs.rst

File renamed without changes.

0 docs/configuration/hosting-docs.rst → docs/guides/dbt_docs/hosting-docs.rst

File renamed without changes.

0 docs/configuration/dbt-fusion.rst → docs/guides/dbt_setup/dbt-fusion.rst

File renamed without changes.

docs/guides/index.rst

-Original file line number
+Diff line change
@@ -0,0 +1,62 @@
+    .. _guides:
+    Guides
+    ======
+    Cosmos offers a number of configuration options to customize its behavior. For more info, check out the links on the left or the table of contents below.
+    .. toctree::
+       :maxdepth: 1
+       :hidden:
+       :caption: Set up dbt with Airflow
+       dbt_setup/dbt-fusion
+    .. toctree::
+       :maxdepth: 1
+       :hidden:
+       :caption: Translating dbt into Airflow
+       translate_dbt_to_airflow/parsing-methods
+       Selecting what to run <translate_dbt_to_airflow/selecting-excluding>
+       Configure tests <translate_dbt_to_airflow/testing-behavior>
+       translate_dbt_to_airflow/managing-sources
+       translate_dbt_to_airflow/render-config
+       Customize node conversion <translate_dbt_to_airflow/dag-customization>
+    .. toctree::
+       :maxdepth: 3
+       :hidden:
+       :caption: How Cosmos runs dbt
+       run_dbt/execution-modes
+       run_dbt/airflow-worker/index
+       run_dbt/container/index
+       run_dbt/callbacks/callbacks
+       run_dbt/operators/operators
+       run_dbt/customization/index
+    .. toctree::
+       :maxdepth: 1
+       :hidden:
+       :caption: Multi-project Setups
+       Handle cross-project references <multi_project/multi-project>
+    .. toctree::
+       :maxdepth: 1
+       :hidden:
+       :caption: dbt Documentation
+       dbt_docs/generating-docs
+       dbt_docs/hosting-docs
+    .. toctree::
+       :maxdepth: 1
+       :hidden:
+       :caption: Cosmos DevEx
+       cosmos_devex/lineage
+       cosmos_devex/compiled-sql
+       cosmos_devex/logging
+       cosmos_devex/task-display-name

docs/configuration/multi-project.rst → docs/guides/multi_project/multi-project.rst

-Original file line number
+Diff line change
@@ Expand Up @@
     **Option 1: Combined DAG with Task Groups using dbt ls Load Mode (Recommended)**
-    .. literalinclude:: ../../dev/dags/cross_project_dbt_ls_dag.py
+    .. literalinclude:: ../../../dev/dags/cross_project_dbt_ls_dag.py
         :language: python
         :start-after: [START cross_project_dbt_ls_dag]
         :end-before: [END cross_project_dbt_ls_dag]
@@ Expand All @@
     This option uses pre-generated ``manifest.json`` files for faster DAG parsing (no ``dbt ls`` execution required).
-    .. literalinclude:: ../../dev/dags/cross_project_manifest_dag.py
+    .. literalinclude:: ../../../dev/dags/cross_project_manifest_dag.py
         :language: python
         :start-after: [START cross_project_manifest_dag]
         :end-before: [END cross_project_manifest_dag]
@@ Expand Down @@

.../getting_started/async-execution-mode.rst → ...t/airflow-worker/async-execution-mode.rst

-Original file line number
+Diff line change
@@ -1,7 +1,5 @@
     .. _async-execution-mode:
-    .. title:: Getting Started with Deferrable Operator
     Airflow Async Execution Mode
     ============================
@@ Expand Down @@

...arted/execution-modes-local-conflicts.rst → ...orker/execution-modes-local-conflicts.rst

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -10,8 +10,8 @@ When using the `Local Execution Mode <execution-modes.html#local>`__, users may
  
    If you find errors, we recommend users isolating the installation of dbt from the Airflow installation.

    With the `Local Execution Mode <execution-modes.html#local>`__, this can be accomplished by installing dbt in a separate

    Python virtualenv and setting the `ExecutionConfig.dbt_executable_path <../configuration/execution-config.html>`_  and

    `RenderConfig.dbt_executable_path <../configuration/render-config.html>`_ parameters.

    Python virtualenv and setting the `ExecutionConfig.dbt_executable_path <../guides/execution-config.html>`_  and

    `RenderConfig.dbt_executable_path <../guides/render-config.html>`_ parameters.

    The page `execution modes <execution-modes.html>`__ describes many other methods that support isolating dbt from Airflow.

docs/guides/run_dbt/airflow-worker/index.rst

-Original file line number
+Diff line change
@@ -0,0 +1,9 @@
+    Run dbt in an Airflow worker
+    ============================
+    .. toctree::
+       :maxdepth: 1
+       :caption: Run dbt in an Airflow worker
+       async-execution-mode
+       watcher-execution-mode

...etting_started/watcher-execution-mode.rst → ...airflow-worker/watcher-execution-mode.rst

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -144,7 +144,7 @@ Example 1 — Using ``DbtDag`` with ``ExecutionMode.WATCHER``
  
    You can enable WATCHER mode directly in your ``DbtDag`` configuration.

    This approach is best when your Airflow DAG is fully dedicated to a dbt project.

    .. literalinclude:: ../../dev/dags/example_watcher.py

    .. literalinclude:: ../../../../dev/dags/example_watcher.py

        :language: python

        :start-after: [START example_watcher]

        :end-before: [END example_watcher]

    @@ -370,7 +370,7 @@ Source freshness nodes
  
    Since Cosmos 1.6, it `supports the rendering of source nodes <https://www.astronomer.io/blog/native-support-for-source-node-rendering-in-cosmos/>`_.

    We noticed some Cosmos users use this feature alongside `overriding Cosmos source nodes <https://astronomer.github.io/astronomer-cosmos/configuration/render-config.html#customizing-how-nodes-are-rendered-experimental>`_ as sensors or another operator that allows them to skip the following branch of the DAG if the source is not fresh.

    We noticed some Cosmos users use this feature alongside `overriding Cosmos source nodes <https://astronomer.github.io/astronomer-cosmos/guides/render-config.html#customizing-how-nodes-are-rendered-experimental>`_ as sensors or another operator that allows them to skip the following branch of the DAG if the source is not fresh.

    This use case is not currently supported by the ``ExecutionMode.WATCHER``, since the ``dbt build`` command does not run `source freshness checks <https://docs.getdbt.com/reference/commands/build#source-freshness-checks>`_.

    @@ -451,7 +451,7 @@ Asynchronous sensor execution
  
    To disable asynchronous execution, set the ``deferrable`` flag to ``False`` in the ``operator_args``.

    .. literalinclude:: ../../dev/dags/example_watcher.py

    .. literalinclude:: ../../../../dev/dags/example_watcher.py

       :language: python

       :start-after: [START example_watcher_synchronous]

       :end-before: [END example_watcher_synchronous]

docs/configuration/callbacks.rst → docs/guides/run_dbt/callbacks/callbacks.rst

-Original file line number
+Diff line change
@@ Expand Up / @@ -34,7 +34,7 @@ Example: Using Callbacks with a Single Operator @@
     To demonstrate how to specify a callback function for uploading files from the target directory, here’s an example
     using a single operator in an Airflow DAG:
-    .. literalinclude:: ../../dev/dags/example_operators.py
+    .. literalinclude:: ../../../../dev/dags/example_operators.py
         :language: python
         :start-after: [START single_operator_callback]
         :end-before: [END single_operator_callback]
@@ Expand All @@
     from the target directory to a remote storage. Below is an example of how to define a callback helper function in your
     ``DbtDag`` that utilizes this configuration:
-    .. literalinclude:: ../../dev/dags/cosmos_callback_dag.py
+    .. literalinclude:: ../../../../dev/dags/cosmos_callback_dag.py
         :language: python
         :start-after: [START cosmos_callback_example]
         :end-before: [END cosmos_callback_example]
@@ Expand Down @@

...getting_started/aws-container-run-job.rst → ...n_dbt/container/aws-container-run-job.rst

-Original file line number
+Diff line change
@@ -1,7 +1,5 @@
     .. _aws-container-run-job:
-    .. title:: Getting Started with Astronomer Cosmos on AWS ECS
     Getting Started with Astronomer Cosmos on AWS ECS
     ==================================================
@@ Expand Down @@

0 ...ting_started/azure-container-instance.rst → ...bt/container/azure-container-instance.rst

File renamed without changes.

0 docs/getting_started/docker.rst → docs/guides/run_dbt/container/docker.rst

File renamed without changes.

0 docs/getting_started/gcp-cloud-run-job.rst → ...s/run_dbt/container/gcp-cloud-run-job.rst

File renamed without changes.

docs/guides/run_dbt/container/index.rst

-Original file line number
+Diff line change
@@ -0,0 +1,13 @@
+    Run dbt in a container
+    ======================
+    .. toctree::
+       :maxdepth: 1
+       :caption: Run dbt in a container
+       docker
+       kubernetes
+       watcher-kubernetes-execution-mode
+       aws-container-run-job
+       azure-container-instance
+       gcp-cloud-run-job

docs/getting_started/kubernetes.rst → docs/guides/run_dbt/container/kubernetes.rst

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -28,7 +28,7 @@ Additional KubernetesPodOperator parameters can be added to the ``operator_args`
  
    For instance,

    .. literalinclude:: ../../dev/dags/jaffle_shop_kubernetes.py

    .. literalinclude:: ../../../../dev/dags/jaffle_shop_kubernetes.py

       :language: python

       :start-after: [START kubernetes_tg_example]

       :end-before: [END kubernetes_tg_example]

    @@ -161,7 +161,7 @@ The Kubernetes execution mode has the following limitations:
  
    - Does not emit Airflow datasets, assets, and dataset aliases (there is an `open ticket #2329 <https://github.com/astronomer/astronomer-cosmos/issues/2329>`__ to address this)

    - Does not handle installing dbt deps for users (there is an `open ticket #679 <https://github.com/astronomer/astronomer-cosmos/issues/679>`__ to address this)

    - Does not support `ProfileMapping <https://astronomer.github.io/astronomer-cosmos/profiles/index.html#using-a-profile-mapping>`_ (there is an `open ticket #749 <https://github.com/astronomer/astronomer-cosmos/issues/749>`__ to address this)

    - Does not support `Callbacks <https://astronomer.github.io/astronomer-cosmos/configuration/callbacks.html>`_ (there is an `open ticket #1575 <https://github.com/astronomer/astronomer-cosmos/issues/1575>`__ to address this)

    - Does not expose Compiled SQL as a `templated field <https://astronomer.github.io/astronomer-cosmos/configuration/compiled-sql.html>`_

    - Does not benefit from `Cosmos caching mechanisms <https://astronomer.github.io/astronomer-cosmos/configuration/caching.html>`_

    - Does not support `generating dbt docs & uploading to an object store <https://astronomer.github.io/astronomer-cosmos/configuration/generating-docs.html>`_ (there is a `PR <https://github.com/astronomer/astronomer-cosmos/pull/2058>`_ to solve this for S3)

    - Does not support `Callbacks <https://astronomer.github.io/astronomer-cosmos/guides/callbacks.html>`_ (there is an `open ticket #1575 <https://github.com/astronomer/astronomer-cosmos/issues/1575>`__ to address this)

    - Does not expose Compiled SQL as a `templated field <https://astronomer.github.io/astronomer-cosmos/guides/compiled-sql.html>`_

    - Does not benefit from `Cosmos caching mechanisms <https://astronomer.github.io/astronomer-cosmos/guides/caching.html>`_

    - Does not support `generating dbt docs & uploading to an object store <https://astronomer.github.io/astronomer-cosmos/guides/generating-docs.html>`_ (there is a `PR <https://github.com/astronomer/astronomer-cosmos/pull/2058>`_ to solve this for S3)

...ted/watcher-kubernetes-execution-mode.rst → ...ner/watcher-kubernetes-execution-mode.rst

-Original file line number
+Diff line change
@@ Expand Up / @@ -183,7 +183,7 @@ Example DAG @@
     Below is a complete example of a DAG using ``ExecutionMode.WATCHER_KUBERNETES``:
-    .. literalinclude:: ../../dev/dags/jaffle_shop_watcher_kubernetes.py
+    .. literalinclude:: ../../../../dev/dags/jaffle_shop_watcher_kubernetes.py
         :language: python
     -------------------------------------------------------------------------------
@@ Expand Down @@

...ing_started/custom-airflow-properties.rst → ...stomization/custom-airflow-properties.rst

-Original file line number
+Diff line change
@@ Expand Up @@
     For example, in the YAML above, the **pool** setting is applied to the specific dbt task.
     This approach allows for more granular control over Airflow settings per task within your dbt model definitions.
-    .. image:: ../_static/custom_airflow_pool.png
+    .. image:: ../../../_static/custom_airflow_pool.png
        :alt: Result of applying Custom Airflow Pool

docs/guides/run_dbt/customization/index.rst

-Original file line number
+Diff line change
@@ -0,0 +1,11 @@
+    Additional Customization
+    ========================
+    .. toctree::
+       :maxdepth: 1
+       :caption: Additional Customization
+       scheduling
+       operator-args
+       partial-parsing
+       custom-airflow-properties

0 docs/configuration/operator-args.rst → ...s/run_dbt/customization/operator-args.rst

File renamed without changes.

0 docs/configuration/partial-parsing.rst → ...run_dbt/customization/partial-parsing.rst

File renamed without changes.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Docs] Restructure Getting Started and Guides section #2418

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!