Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 53 additions & 0 deletions docs/getting_started/bring-your-own.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
.. _bring-your-own:

Get started with your dbt project
=================================

If you have a sample project that you want to use to test out Cosmos functionality, or, you want to see how to set up a more custom project, check out the Bring your Own project guides.
These guides provide the general structure for how to set up a dbt project in Cosmos.

- :ref:`Open-source Airflow <open-source>`
- :ref:`Astro <astro>`
- :ref:`AWS MWAA <mwaa>`
- :ref:`Google Cloud Composer <gcc>`

When you're ready to set up and customize your Cosmos project, see the :ref:`guides`.

Example Demo: Jaffle Shop Project
__________________________________

If you don't have your own project that you want to set up as a demo, you can explore a practical example in the **Bring your own project** guides to see how Cosmos can convert the dbt workflow into an Airflow Dag.

The `jaffle_shop project <https://github.com/dbt-labs/jaffle_shop>`_ is a sample dbt project that simulates an e-commerce store's data.
The project includes a series of dbt models that transform raw data into structured tables, such as sales, customers, and products.

The following diagram shows the original dbt workflow in a lineage graph. This graph illustrates the relationships between different models:

.. image:: /_static/jaffle_shop_dbt_graph.png

Cosmos can take this dbt workflow and convert it into an Airflow Dag, allowing you to leverage Airflow's scheduling and
orchestration capabilities.

To convert this dbt workflow into an Airflow Dag, create a new Dag definition file, import ``DbtDag`` from the Cosmos library,
and fill in a few parameters, such as the dbt project directory path and the profile name:

..
The following renders in Sphinx but not Github:

.. literalinclude:: ./../../dev/dags/basic_cosmos_dag.py
:language: python
:start-after: [START local_example]
:end-before: [END local_example]


This code snippet then generates an Airflow Dag like this:

.. image:: https://raw.githubusercontent.com/astronomer/astronomer-cosmos/main/docs/_static/jaffle_shop_dag.png

``DbtDag`` is a custom Dag generator that converts dbt projects into Airflow Dags. It also accepts Cosmos-specific arguments like
``fail_fast``, to immediately fail a dag if dbt fails to process a resource, or ``cancel_query_on_kill``, to cancel any running
queries if the task is externally killed or manually set to failed in Airflow. ``DbtDag`` also accepts standard Dag arguments such
as ``max_active_tasks``, ``max_active_runs``, and ``default_args``.

With Cosmos, transitioning from a dbt workflow to an Airflow Dag is seamless, giving you the best of both tools
for managing and scaling your data workflows.
83 changes: 27 additions & 56 deletions docs/getting_started/index.rst
Original file line number Diff line number Diff line change
@@ -1,78 +1,49 @@
.. _getting-started:
.. _getting_started:

Getting started with Cosmos
===========================

.. toctree::
:maxdepth: 1
:maxdepth: 0
:hidden:

self

Cosmos fundamentals
~~~~~~~~~~~~~~~~~~~~

Learn the important concepts and key ideas for how Cosmos works with Airflow and dbt.
Comment thread
pankajkoti marked this conversation as resolved.

.. toctree::
:maxdepth: 1
:caption: Cosmos Fundamentals

Similar dbt and Airflow concepts <dbt-airflow-concepts>

Quickstart
~~~~~~~~~~~

Run Cosmos in just a few minutes using a pre-configured demo, such as the `cosmos-demo <https://github.com/astronomer/cosmos-demo>`_.
These guides are an ideal place to start if you don't have a particular project that you want to try out, and just want walk through the process of setting up Cosmos, Airflow, and a dbt project to work together.

.. toctree::
:maxdepth: 1
:hidden:
:caption: Quickstart

Open-source quickstart <oss-quickstart>
Astro CLI quickstart <astro-cli-quickstart>

Bring your own project
~~~~~~~~~~~~~~~~~~~~~~

An example workflow for setting up Cosmos, depending on how you run Airflow. For guides that describe specific scenarios or platforms that you want to use, see the following:

.. toctree::
:maxdepth: 1
:hidden:
:caption: Bring your own project

Get started with your dbt project <bring-your-own>
Open-source Airflow <open-source>
Astro <astro>
Amazon Managed Workflows for Apache Airflow (MWAA) <mwaa>
Google Cloud Composer (GCC) <gcc>


Getting Started
===============

The recommended way to install and run Cosmos depends on how you run Airflow. For specific guides, see the following:

- `Getting Started on Open-Source <open-source.html>`__
- `Getting Started on Astro <astro.html>`__
- `Getting Started on MWAA <mwaa.html>`__
- `Getting Started on GCC <gcc.html>`__

You might require a different setup depending on your particular configuration. See :ref:`execution-modes`.

Example Demo: Jaffle Shop Project
__________________________________

You can explore a practical example in the **Getting Started** guides to see how Cosmos can convert the dbt workflow into an Airflow Dag.

The `jaffle_shop project <https://github.com/dbt-labs/jaffle_shop>`_ is a sample dbt project that simulates an e-commerce store's data.
The project includes a series of dbt models that transform raw data into structured tables, such as sales, customers, and products.

The following diagram shows the original dbt workflow in a lineage graph. This graph illustrates the relationships between different models:

.. image:: /_static/jaffle_shop_dbt_graph.png

Cosmos can take this dbt workflow and convert it into an Airflow Dag, allowing you to leverage Airflow's scheduling and
orchestration capabilities.

To convert this dbt workflow into an Airflow Dag, create a new Dag definition file, import ``DbtDag`` from the Cosmos library,
and fill in a few parameters, such as the dbt project directory path and the profile name:

..
The following renders in Sphinx but not Github:

.. literalinclude:: ./../../dev/dags/basic_cosmos_dag.py
:language: python
:start-after: [START local_example]
:end-before: [END local_example]


This code snippet then generates an Airflow Dag like this:

.. image:: https://raw.githubusercontent.com/astronomer/astronomer-cosmos/main/docs/_static/jaffle_shop_dag.png

``DbtDag`` is a custom Dag generator that converts dbt projects into Airflow Dags. It also accepts Cosmos-specific arguments like
``fail_fast``, to immediately fail a dag if dbt fails to process a resource, or ``cancel_query_on_kill``, to cancel any running
queries if the task is externally killed or manually set to failed in Airflow. ``DbtDag`` also accepts standard Dag arguments such
as ``max_active_tasks``, ``max_active_runs``, and ``default_args``.

With Cosmos, transitioning from a dbt workflow to an Airflow Dag is seamless, giving you the best of both tools
for managing and scaling your data workflows.
24 changes: 14 additions & 10 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -35,35 +35,39 @@ simple to manage and orchestrate your dbt workflows using `Apache Airflow® <htt
time and effort. By automatically turning dbt workflows into Airflow DAGs, Cosmos allows you to focus on building
high-quality data models without the hassle of managing complex integrations.

To get started right away, check out the `Quickstart Guides <https://astronomer.github.io/astronomer-cosmos/getting_started/index.html>`_.
You can also explore more examples in `/dev/dags <https://github.com/astronomer/astronomer-cosmos/tree/main/dev/dags>`_
or in the `cosmos-demo repo <https://github.com/astronomer/cosmos-demo>`_.

What Is Astronomer Cosmos?
___________________________
~~~~~~~~~~~~~~~~~~~~~~~~~~

Astronomer Cosmos is an open-source library that bridges Apache Airflow and dbt, allowing you to easily transform your
dbt projects into Airflow DAGs and manage everything seamlessly. With Cosmos, you can write your data transformations
dbt projects into Airflow Dags and manage everything seamlessly. With Cosmos, you can write your data transformations
using dbt and then schedule and orchestrate them with Airflow, making the entire process smooth and straightforward.

Why Should You Use Cosmos?
___________________________
~~~~~~~~~~~~~~~~~~~~~~~~~~

Integrating dbt and Airflow can be complex, but Cosmos simplifies it by seamlessly connecting these
powerful tools—letting you focus on what matters most: delivering impactful data models and results without getting
bogged down by technical challenges.

Cosmos makes orchestrating dbt workflows:

- **Effortless**: Transform your dbt projects into Airflow DAGs without writing extra code—Cosmos handles the heavy lifting.
- **Effortless**: Transform your dbt projects into Airflow Dags without writing extra code—Cosmos handles the heavy lifting.
- **Reliable**: Rely on Airflow's robust scheduling and monitoring features to ensure your dbt workflows run smoothly and efficiently.
- **Scalable**: Easily scale your workflows to match growing data demands, thanks to Airflow's distributed capabilities.

Whether you're handling intricate data tasks or looking to streamline your processes, Cosmos helps you orchestrate dbt
with Airflow effortlessly, saving you time and letting you focus on what truly matters—creating impactful insights.

Ready to get started?
~~~~~~~~~~~~~~~~~~~~~

Check out the Cosmos Fundamentals, get started locally with the Quickstart Guides, or find instructions for how to bring your own dbt project to Cosmos in `Get started with Cosmos <https://astronomer.github.io/astronomer-cosmos/getting_started/index.html>`_.

You can also explore more examples in `/dev/dags <https://github.com/astronomer/astronomer-cosmos/tree/main/dev/dags>`_
or in the `cosmos-demo repo <https://github.com/astronomer/cosmos-demo>`_.

Join the Community
__________________
~~~~~~~~~~~~~~~~~~

Have questions, need help, or interested in contributing? We welcome all contributions and feedback!

Expand All @@ -76,7 +80,7 @@ Note that contributors and maintainers are expected to abide by the
`Contributor Code of Conduct <https://github.com/astronomer/astronomer-cosmos/blob/main/CODE_OF_CONDUCT.md>`_.

License
_______
~~~~~~~

`Apache License 2.0 <https://github.com/astronomer/astronomer-cosmos/blob/main/LICENSE>`_

Expand Down