diff --git a/docs/getting_started/bring-your-own.rst b/docs/getting_started/bring-your-own.rst new file mode 100644 index 0000000000..b496d18781 --- /dev/null +++ b/docs/getting_started/bring-your-own.rst @@ -0,0 +1,53 @@ +.. _bring-your-own: + +Get started with your dbt project +================================= + +If you have a sample project that you want to use to test out Cosmos functionality, or, you want to see how to set up a more custom project, check out the Bring your Own project guides. +These guides provide the general structure for how to set up a dbt project in Cosmos. + +- :ref:`Open-source Airflow ` +- :ref:`Astro ` +- :ref:`AWS MWAA ` +- :ref:`Google Cloud Composer ` + +When you're ready to set up and customize your Cosmos project, see the :ref:`guides`. + +Example Demo: Jaffle Shop Project +__________________________________ + +If you don't have your own project that you want to set up as a demo, you can explore a practical example in the **Bring your own project** guides to see how Cosmos can convert the dbt workflow into an Airflow Dag. + +The `jaffle_shop project `_ is a sample dbt project that simulates an e-commerce store's data. +The project includes a series of dbt models that transform raw data into structured tables, such as sales, customers, and products. + +The following diagram shows the original dbt workflow in a lineage graph. This graph illustrates the relationships between different models: + +.. image:: /_static/jaffle_shop_dbt_graph.png + +Cosmos can take this dbt workflow and convert it into an Airflow Dag, allowing you to leverage Airflow's scheduling and +orchestration capabilities. + +To convert this dbt workflow into an Airflow Dag, create a new Dag definition file, import ``DbtDag`` from the Cosmos library, +and fill in a few parameters, such as the dbt project directory path and the profile name: + +.. + The following renders in Sphinx but not Github: + +.. literalinclude:: ./../../dev/dags/basic_cosmos_dag.py + :language: python + :start-after: [START local_example] + :end-before: [END local_example] + + +This code snippet then generates an Airflow Dag like this: + +.. image:: https://raw.githubusercontent.com/astronomer/astronomer-cosmos/main/docs/_static/jaffle_shop_dag.png + +``DbtDag`` is a custom Dag generator that converts dbt projects into Airflow Dags. It also accepts Cosmos-specific arguments like +``fail_fast``, to immediately fail a dag if dbt fails to process a resource, or ``cancel_query_on_kill``, to cancel any running +queries if the task is externally killed or manually set to failed in Airflow. ``DbtDag`` also accepts standard Dag arguments such +as ``max_active_tasks``, ``max_active_runs``, and ``default_args``. + +With Cosmos, transitioning from a dbt workflow to an Airflow Dag is seamless, giving you the best of both tools +for managing and scaling your data workflows. diff --git a/docs/getting_started/index.rst b/docs/getting_started/index.rst index ccad2a68f2..dcc7138b05 100644 --- a/docs/getting_started/index.rst +++ b/docs/getting_started/index.rst @@ -1,78 +1,49 @@ -.. _getting-started: +.. _getting_started: + +Getting started with Cosmos +=========================== .. toctree:: - :maxdepth: 1 + :maxdepth: 0 :hidden: + + self + +Cosmos fundamentals +~~~~~~~~~~~~~~~~~~~~ + +Learn the important concepts and key ideas for how Cosmos works with Airflow and dbt. + +.. toctree:: + :maxdepth: 1 :caption: Cosmos Fundamentals Similar dbt and Airflow concepts +Quickstart +~~~~~~~~~~~ + +Run Cosmos in just a few minutes using a pre-configured demo, such as the `cosmos-demo `_. +These guides are an ideal place to start if you don't have a particular project that you want to try out, and just want walk through the process of setting up Cosmos, Airflow, and a dbt project to work together. + .. toctree:: :maxdepth: 1 - :hidden: :caption: Quickstart Open-source quickstart Astro CLI quickstart +Bring your own project +~~~~~~~~~~~~~~~~~~~~~~ + +An example workflow for setting up Cosmos, depending on how you run Airflow. For guides that describe specific scenarios or platforms that you want to use, see the following: + .. toctree:: :maxdepth: 1 - :hidden: :caption: Bring your own project + Get started with your dbt project Open-source Airflow Astro Amazon Managed Workflows for Apache Airflow (MWAA) Google Cloud Composer (GCC) - - -Getting Started -=============== - -The recommended way to install and run Cosmos depends on how you run Airflow. For specific guides, see the following: - -- `Getting Started on Open-Source `__ -- `Getting Started on Astro `__ -- `Getting Started on MWAA `__ -- `Getting Started on GCC `__ - -You might require a different setup depending on your particular configuration. See :ref:`execution-modes`. - -Example Demo: Jaffle Shop Project -__________________________________ - -You can explore a practical example in the **Getting Started** guides to see how Cosmos can convert the dbt workflow into an Airflow Dag. - -The `jaffle_shop project `_ is a sample dbt project that simulates an e-commerce store's data. -The project includes a series of dbt models that transform raw data into structured tables, such as sales, customers, and products. - -The following diagram shows the original dbt workflow in a lineage graph. This graph illustrates the relationships between different models: - -.. image:: /_static/jaffle_shop_dbt_graph.png - -Cosmos can take this dbt workflow and convert it into an Airflow Dag, allowing you to leverage Airflow's scheduling and -orchestration capabilities. - -To convert this dbt workflow into an Airflow Dag, create a new Dag definition file, import ``DbtDag`` from the Cosmos library, -and fill in a few parameters, such as the dbt project directory path and the profile name: - -.. - The following renders in Sphinx but not Github: - -.. literalinclude:: ./../../dev/dags/basic_cosmos_dag.py - :language: python - :start-after: [START local_example] - :end-before: [END local_example] - - -This code snippet then generates an Airflow Dag like this: - -.. image:: https://raw.githubusercontent.com/astronomer/astronomer-cosmos/main/docs/_static/jaffle_shop_dag.png - -``DbtDag`` is a custom Dag generator that converts dbt projects into Airflow Dags. It also accepts Cosmos-specific arguments like -``fail_fast``, to immediately fail a dag if dbt fails to process a resource, or ``cancel_query_on_kill``, to cancel any running -queries if the task is externally killed or manually set to failed in Airflow. ``DbtDag`` also accepts standard Dag arguments such -as ``max_active_tasks``, ``max_active_runs``, and ``default_args``. - -With Cosmos, transitioning from a dbt workflow to an Airflow Dag is seamless, giving you the best of both tools -for managing and scaling your data workflows. diff --git a/docs/index.rst b/docs/index.rst index cd79000a16..c403856e22 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -35,19 +35,15 @@ simple to manage and orchestrate your dbt workflows using `Apache Airflow® `_. -You can also explore more examples in `/dev/dags `_ -or in the `cosmos-demo repo `_. - What Is Astronomer Cosmos? -___________________________ +~~~~~~~~~~~~~~~~~~~~~~~~~~ Astronomer Cosmos is an open-source library that bridges Apache Airflow and dbt, allowing you to easily transform your -dbt projects into Airflow DAGs and manage everything seamlessly. With Cosmos, you can write your data transformations +dbt projects into Airflow Dags and manage everything seamlessly. With Cosmos, you can write your data transformations using dbt and then schedule and orchestrate them with Airflow, making the entire process smooth and straightforward. Why Should You Use Cosmos? -___________________________ +~~~~~~~~~~~~~~~~~~~~~~~~~~ Integrating dbt and Airflow can be complex, but Cosmos simplifies it by seamlessly connecting these powerful tools—letting you focus on what matters most: delivering impactful data models and results without getting @@ -55,15 +51,23 @@ bogged down by technical challenges. Cosmos makes orchestrating dbt workflows: -- **Effortless**: Transform your dbt projects into Airflow DAGs without writing extra code—Cosmos handles the heavy lifting. +- **Effortless**: Transform your dbt projects into Airflow Dags without writing extra code—Cosmos handles the heavy lifting. - **Reliable**: Rely on Airflow's robust scheduling and monitoring features to ensure your dbt workflows run smoothly and efficiently. - **Scalable**: Easily scale your workflows to match growing data demands, thanks to Airflow's distributed capabilities. Whether you're handling intricate data tasks or looking to streamline your processes, Cosmos helps you orchestrate dbt with Airflow effortlessly, saving you time and letting you focus on what truly matters—creating impactful insights. +Ready to get started? +~~~~~~~~~~~~~~~~~~~~~ + +Check out the Cosmos Fundamentals, get started locally with the Quickstart Guides, or find instructions for how to bring your own dbt project to Cosmos in `Get started with Cosmos `_. + +You can also explore more examples in `/dev/dags `_ +or in the `cosmos-demo repo `_. + Join the Community -__________________ +~~~~~~~~~~~~~~~~~~ Have questions, need help, or interested in contributing? We welcome all contributions and feedback! @@ -76,7 +80,7 @@ Note that contributors and maintainers are expected to abide by the `Contributor Code of Conduct `_. License -_______ +~~~~~~~ `Apache License 2.0 `_