Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions docs/getting_started/astro-cli-quickstart.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
.. title:: Run Cosmos with the Astro CLI

Run Cosmos with the Astro CLI
-----------------------------
=============================

Work locally with `Apache Airflow® <https://airflow.apache.org/>`_, dbt, and Astronomer Cosmos using the `Astro CLI <https://github.com/astronomer/astro-cli>`__. While Cosmos fully works with standard Airflow and independently of the Astro CLI, the Astro CLI can simplify creating and running Airflow projects. If you want to get started with Cosmos using only Airflow, see `Getting Started on Open Source Airflow <open-source.html>`_.

Expand All @@ -16,7 +16,7 @@ By the end of this quickstart you will:
- Run a simple Dag that uses dbt to load, run, and test sample e-commerce data

Prerequisites
+++++++++++++
~~~~~~~~~~~~~

- `Python <https://www.python.org/downloads/>`_
- Install `git <https://git-scm.com/install/>`_
Expand All @@ -27,7 +27,7 @@ Prerequisites
Depending on your operating system, you might also need to install a separate installation manager like `Homebrew <https://brew.sh>`_ or `WinGet <https://learn.microsoft.com/en-us/windows/package-manager/winget/>`_.

Clone the demo repo
+++++++++++++++++++
~~~~~~~~~~~~~~~~~~~

1. Open a terminal in the directory where you want to clone your sample repo.
2. Clone the ``cosmos-demo`` repo.
Expand All @@ -37,7 +37,7 @@ Clone the demo repo
git clone https://github.com/astronomer/cosmos-demo.git

Start Apache Airflow® locally
++++++++++++++++++++++++++++++
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

1. Open a terminal at the root of the ``cosmos-demo`` repo.
2. Run ``astro dev start --verbosity debug`` to start your Aiflow instance. Including the ``--verbosity debug`` flag allows the command to display the steps the Astro CLI takes to set up your Airflow project.
Expand All @@ -47,7 +47,7 @@ Start Apache Airflow® locally
The `Airflow UI <https://www.astronomer.io/docs/learn/airflow-ui>`_ enables you to start, stop, troubleshoot, or manage your Dags.

Run a simple Cosmos Dag
+++++++++++++++++++++++
~~~~~~~~~~~~~~~~~~~~~~~

1. From the Airflow Dashboard, click **Dags**. This opens a view where you can see all available Dags. Or, you can see if there were problems loading Dags to your Airflow project.
2. Select **Simple Dag** from the list to access the `Dag view <https://www.astronomer.io/docs/learn/airflow-ui#individual-dag>`_ in the Airflow UI. Click **Code** to see the Dag code.
Expand Down Expand Up @@ -81,7 +81,7 @@ For example in the ``stg_customers``, task group, in the ``run`` task, the logs
This log indicates that the Dag triggers Cosmos to initiate the ``dbt run`` command following the sql actions defined in the ``stg_customers.sql``.

View results with a database viewer
+++++++++++++++++++++++++++++++++++
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To view the transformations completed by the Dag, you must use a database viewer to access the views and materializations completed by your dbt project.

Expand Down Expand Up @@ -116,7 +116,7 @@ You have several options to locate this kind of information in dbt projects. For
:alt: dBeaver user interface displaying the Customers table view produced by the dbt code. This table includes data that has been joined together from three separate raw database sources.

Key Concepts
++++++++++++
~~~~~~~~~~~~

Congratulations! You ran a dbt project successfully on Airflow! This quickstart includes the minimal steps required to get started working with Cosmos. Specifically it includes:

Expand All @@ -133,7 +133,7 @@ This demo repo also includes a dbt project with configurations that allow you to
Cosmos does not require you to use the specific project architecture shown in the ``cosmos-demo`` to run successfully. However, it can serve as a template or example for you to adapt your dbt or Airflow projects to work cohesively.

Next steps
++++++++++
~~~~~~~~~~

- Follow one of the Getting Started Guides where you can bring your own dbt projects and/or Dag code:
- `Getting Started on Open-Source <open-source.html>`__
Expand Down
14 changes: 7 additions & 7 deletions docs/getting_started/astro.rst
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
.. _astro:

Getting Started with Cosmos on Astro
------------------------------------
====================================

While it is possible to use Cosmos on Astro with all :ref:`Execution Modes <execution-modes>`, we recommend using the ``local`` execution mode. It's the simplest to set up and use.

Expand All @@ -10,7 +10,7 @@ If you'd like to see a fully functional project to run in Astro (CLI or Cloud),
Below you can find a step-by-step guide to run your own dbt project within Astro.

Pre-requisites
++++++++++++++
~~~~~~~~~~~~~~

To get started, you should have:

Expand All @@ -19,7 +19,7 @@ To get started, you should have:
- A dbt project. The `jaffle shop example <https://github.com/dbt-labs/jaffle-shop-classic>`_ is a good example.

Create a virtual environment
++++++++++++++++++++++++++++
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Create a virtual environment in your ``Dockerfile`` using the sample below. Be sure to replace ``<your-dbt-adapter>`` with the actual adapter you need (i.e. ``dbt-redshift``, ``dbt-snowflake``). It's recommended to use a virtual environment because dbt and `Apache Airflow® <https://airflow.apache.org/>`_ can have conflicting dependencies.

Expand All @@ -34,7 +34,7 @@ Create a virtual environment in your ``Dockerfile`` using the sample below. Be s
An example of dbt adapter is ``dbt-postgres``.

Install Cosmos
++++++++++++++
~~~~~~~~~~~~~~

Add Cosmos to your project's ``requirements.txt``.

Expand All @@ -44,7 +44,7 @@ Add Cosmos to your project's ``requirements.txt``.


Move your dbt project into the DAGs directory
+++++++++++++++++++++++++++++++++++++++++++++
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Make a new folder, ``dbt``, inside your local project's ``dags`` folder. Then, copy/paste your dbt project into the directory and create a file called ``my_cosmos_dag.py`` in the root of your DAGs directory. Your project structure should look like this:

Expand Down Expand Up @@ -84,7 +84,7 @@ For example, if you wanted to put your dbt project in a directory relative to yo
)

Create a dagfile
++++++++++++++++
~~~~~~~~~~~~~~~~

In your ``my_cosmos_dag.py`` file, import the ``DbtDag`` class from Cosmos and create a new DAG instance. Make sure to use the ``dbt_executable_path`` argument to point to the virtual environment you created in step 1.

Expand Down Expand Up @@ -128,7 +128,7 @@ In your ``my_cosmos_dag.py`` file, import the ``DbtDag`` class from Cosmos and c
This error can be resolved by increasing the value of the Airflow configuration `core.dagbag_import_timeout <https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#dagbag-import-timeout>`_.

Start your project
++++++++++++++++++
~~~~~~~~~~~~~~~~~~

Start your project with ``astro dev start``. You should see your Airflow DAG in the Airflow UI (``localhost:8080`` by default), where you can trigger it.

Expand Down
4 changes: 2 additions & 2 deletions docs/getting_started/bring-your-own.rst
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
.. _bring-your-own:

Get started with your dbt project
---------------------------------
=================================

If you have a sample project that you want to use to test out Cosmos functionality, or, you want to see how to set up a more custom project, check out the Bring your Own project guides.
These guides provide the general structure for how to set up a dbt project in Cosmos.
Expand All @@ -14,7 +14,7 @@ These guides provide the general structure for how to set up a dbt project in Co
When you're ready to set up and customize your Cosmos project, see the :ref:`guides`.

Example Demo: Jaffle Shop Project
+++++++++++++++++++++++++++++++++
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If you don't have your own project that you want to set up as a demo, you can explore a practical example in the **Bring your own project** guides to see how Cosmos can convert the dbt workflow into an `Apache Airflow® <https://airflow.apache.org/>`_ Dag.

Expand Down
26 changes: 13 additions & 13 deletions docs/getting_started/core-concepts.rst
Original file line number Diff line number Diff line change
@@ -1,18 +1,18 @@
.. _core-concepts:

Cosmos core concepts
--------------------
====================

**Cosmos** is an open-source library that helps you to run dbt code in Apache Airflow. Because it operates at the interface between Airflow and dbt, becoming familiar with the commonly used terminology and foundational concepts can help you get started.


Similar dbt and Apache Airflow® concepts
+++++++++++++++++++++++++++++++++++++++++
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

dbt and Airflow each solve different kinds of data engineering problems. However, they do share some useful concepts. See :ref:`dbt-airflow-concepts`.

How Cosmos works
++++++++++++++++
~~~~~~~~~~~~~~~~

Cosmos provides an exceptional amount of control and ability to customize how Cosmos runs your dbt project in Airflow. When starting with Cosmos, you can think of it as performing two core functions:

Expand All @@ -22,7 +22,7 @@ Cosmos provides an exceptional amount of control and ability to customize how Co


DbtDag and DbtTaskGroup
+++++++++++++++++++++++
~~~~~~~~~~~~~~~~~~~~~~~

During the dbt code parsing phase, you can choose how you want your dbt workflow to translate into an Airflow Dag:

Expand All @@ -32,17 +32,17 @@ During the dbt code parsing phase, you can choose how you want your dbt workflow
You can see example Dags for both a simple `DbtDag <https://github.com/astronomer/astronomer-cosmos/blob/main/dev/dags/basic_cosmos_dag.py>`_ and `DbtTaskGroup <https://github.com/astronomer/astronomer-cosmos/blob/main/dev/dags/basic_cosmos_task_group.py>`_ in the Cosmos `dev/dags directory <https://github.com/astronomer/astronomer-cosmos/tree/main/dev/dags>`_.

ProjectConfig
+++++++++++++
~~~~~~~~~~~~~

The ``ProjectConfig`` contains information about which dbt project a Cosmos Dag or task group executes, as well as configurations that apply to both, rendering and execution. See :ref:`project-config`.

ProfileConfig
+++++++++++++
~~~~~~~~~~~~~

The ``ProfileConfig`` class determines which data warehouse Cosmos connects to when executing the dbt SQL.

Connecting Cosmos to your data warehouse
''''''''''''''''''''''''''''''''''''''''
++++++++++++++++++++++++++++++++++++++++

There are two ways to connect Cosmos projects to the data warehouse used by your dbt project:

Expand All @@ -52,7 +52,7 @@ There are two ways to connect Cosmos projects to the data warehouse used by your
See :ref:`profile-config` for more information about the ``ProfileConfig``, ``profiles.yml``, and ``profile_mapping``.

Execution Modes
+++++++++++++++
~~~~~~~~~~~~~~~

You define the execution mode for your project by using the ``ExecutionConfig`` class, which determines where and how dbt commands are run within Cosmos.

Expand All @@ -70,28 +70,28 @@ There are two main categories of execution modes:
See :ref:`execution-modes` and :ref:`execution-config`.

Parsing
+++++++
~~~~~~~

Parsing generally refers to the processes and configurations that Cosmos uses to parse your dbt project. See :ref:`parsing-methods`.

Invocation Mode
+++++++++++++++
~~~~~~~~~~~~~~~

The method that Cosmos uses to parse the dbt object, whether its by using the Python ``subprocess`` module or with a ``dbt_runner``. See `Invocation modes <https://astronomer.github.io/astronomer-cosmos/guides/run_dbt/execution-modes.html#invocation-modes>`_.

Rendering
+++++++++
~~~~~~~~~

After parsing your dbt project, Cosmos *renders* the project as an Airflow Dag or Task Group. Depending on the specifics of your dbt project, you can choose customizations that speed up overall performance. You can find more information about rendering options in the `RenderConfig <../guides/translate_dbt_to_airflow/render-config.html>`_.


Testing Strategy
''''''''''''''''
++++++++++++++++

By default, Cosmos adds a dbt test after it completes a model. However, you can change this behavior by configuring the test behavior in the ``RenderConfig``. Learn more in :ref:`testing-behavior`.


Select and exclude
''''''''''''''''''
++++++++++++++++++

You can filter your dbt project to only parse a subset of your dbt project by using the ``RenderConfig``, and define the ``select`` and ``exclude`` parameters. See :ref:`selecting-excluding`.
2 changes: 1 addition & 1 deletion docs/getting_started/dbt-airflow-concepts.rst
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
.. _dbt-airflow-concepts:

Similar dbt and Apache Airflow® concepts
-----------------------------------------
=========================================

While dbt is an open source tool for data transformations and analysis, using SQL, `Apache Airflow® <https://airflow.apache.org/>`_ focuses on being a platform
for the development, scheduling and monitoring of batch-oriented workflows, using Python. Although both tools have many
Expand Down
8 changes: 4 additions & 4 deletions docs/getting_started/gcc.rst
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
.. _gcc:

Getting Started on Google Cloud Composer (GCC)
----------------------------------------------
==============================================

Because there's no straightforward way of creating a Python virtual environment in Google Cloud Composer (GCC) , we recommend using Cosmos' built-in virtual environment functionality to run dbt.

Install Cosmos
++++++++++++++
~~~~~~~~~~~~~~

Add the following to your base project ``requirements.txt``:

Expand All @@ -16,7 +16,7 @@ Add the following to your base project ``requirements.txt``:


Move your dbt project into the DAGs directory
+++++++++++++++++++++++++++++++++++++++++++++
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Make a new folder, ``dbt``, inside your local ``dags`` folder. Then, copy/paste your dbt project into the directory and create a file called ``my_cosmos_dag.py`` in the root of your DAGs directory.

Expand All @@ -43,7 +43,7 @@ For example, if you wanted to put your dbt project in the ``/usr/local/airflow/d
You can also exclude the ``manifest_path=...`` from the ``ProjectConfig``. Excluding a ``manifest_path`` file will by default use Cosmos's ``custom`` parsing method, which may be less accurate at parsing a dbt project compared to providing a ``manifest.json``.

Create your DAG
+++++++++++++++
~~~~~~~~~~~~~~~

In your ``my_cosmos_dag.py`` file, import the ``DbtDag`` class from Cosmos and create a new DAG instance. You need to supply additional arguments in the ``operator_args`` dictionary to tell Cosmos which packages are required.

Expand Down
8 changes: 4 additions & 4 deletions docs/getting_started/how-cosmos-works.rst
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
.. _how-cosmos-works:

How Cosmos works
----------------
================

Cosmos is an open-source Python package developed by Astronomer under the Apache License 2.0. The
package includes classes that automatically turn dbt Core and dbt Fusion projects into `Apache Airflow® <https://airflow.apache.org/>`_
Expand All @@ -12,7 +12,7 @@ advanced orchestration capabilities to integrate your dbt projects into your end
run automatically.

Overview
++++++++
~~~~~~~~

Cosmos creates an interface between a dbt project and Airflow, allowing you to translate your dbt project into a Dag. Then, your Cosmos configuration provides the necessary configurations to Airflow so that it can schedule and initiate running your dbt code when you want.

Expand All @@ -26,12 +26,12 @@ You have a number of configuration options, but fundamentally, Cosmos provides t
:alt: Diagram showing that Cosmos parses dbt projects, with options defined by the ProfileConfig and RenderConfig. Then, Cosmos executes the resulting Dag, with options defined by ExecutionConfig and ProjectConfig.

Quickstart
++++++++++
~~~~~~~~~~

Even though Cosmos is highly extensible, and you have many advanced customization options, you can run a demo with the `Astro CLI <astro-cli-quickstart.html>`_ in just a few minutes. This demo introduces you to the key elements required for Cosmos to parse dbt projects and run Dags.

Bring your own project
++++++++++++++++++++++
~~~~~~~~~~~~~~~~~~~~~~

If you have existing resources for Airflow or dbt, and want to start exploring how to get started with a project more similar to your use case, check out the **Get started with Cosmos** guides. These resources provide more general recommendations for how to create a new project.

Expand Down
8 changes: 4 additions & 4 deletions docs/getting_started/index.rst
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
.. _getting_started:

Getting started with Cosmos
---------------------------
===========================

.. toctree::
:maxdepth: 0
Expand All @@ -10,7 +10,7 @@ Getting started with Cosmos
self

Cosmos fundamentals
+++++++++++++++++++
~~~~~~~~~~~~~~~~~~~

Learn the important concepts and key ideas for how Cosmos works with Airflow and dbt.

Expand All @@ -23,7 +23,7 @@ Learn the important concepts and key ideas for how Cosmos works with Airflow and
Cosmos core concepts <core-concepts>

Quickstart
++++++++++
~~~~~~~~~~

Run Cosmos in just a few minutes using a pre-configured demo, such as the `cosmos-demo <https://github.com/astronomer/cosmos-demo>`_.
These guides are an ideal place to start if you don't have a particular project that you want to try out, and just want walk through the process of setting up Cosmos, Airflow, and a dbt project to work together.
Expand All @@ -36,7 +36,7 @@ These guides are an ideal place to start if you don't have a particular project
Astro CLI quickstart <astro-cli-quickstart>

Bring your own project
++++++++++++++++++++++
~~~~~~~~~~~~~~~~~~~~~~

An example workflow for setting up Cosmos, depending on how you run Airflow. For guides that describe specific scenarios or platforms that you want to use, see the following:

Expand Down
Loading
Loading