-
Notifications
You must be signed in to change notification settings - Fork 294
Getting Started: Update overview and index pages. #2452
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
+94
−66
Merged
Changes from all commits
Commits
Show all changes
14 commits
Select commit
Hold shift + click to select a range
1fa396b
Add bring your own guide
lzdanski 7c25374
🎨 [pre-commit.ci] Auto format from pre-commit.com hooks
pre-commit-ci[bot] 83912c3
Merge branch 'main' into get-started-overview
lzdanski 79df9a3
🎨 [pre-commit.ci] Auto format from pre-commit.com hooks
pre-commit-ci[bot] 67950e4
move section back to main landing page
lzdanski 0af06b0
Address review feedback
lzdanski 1159410
Fix header levels on index
lzdanski 38c46fa
Reorder sections
lzdanski fba1566
Merge branch 'main' into get-started-overview
lzdanski 756142d
Address getting started
lzdanski 59ce3ec
Apply suggestion from @lzdanski
lzdanski a6bccff
Fix header levels
lzdanski c39729a
Fix page title alignment to match sidebar
lzdanski 748b7d8
Merge branch 'main' into get-started-overview
pankajkoti File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| .. _bring-your-own: | ||
|
|
||
| Get started with your dbt project | ||
| ================================= | ||
|
|
||
| If you have a sample project that you want to use to test out Cosmos functionality, or, you want to see how to set up a more custom project, check out the Bring your Own project guides. | ||
| These guides provide the general structure for how to set up a dbt project in Cosmos. | ||
|
|
||
| - :ref:`Open-source Airflow <open-source>` | ||
| - :ref:`Astro <astro>` | ||
| - :ref:`AWS MWAA <mwaa>` | ||
| - :ref:`Google Cloud Composer <gcc>` | ||
|
|
||
| When you're ready to set up and customize your Cosmos project, see the :ref:`guides`. | ||
|
|
||
| Example Demo: Jaffle Shop Project | ||
| __________________________________ | ||
|
|
||
| If you don't have your own project that you want to set up as a demo, you can explore a practical example in the **Bring your own project** guides to see how Cosmos can convert the dbt workflow into an Airflow Dag. | ||
|
|
||
| The `jaffle_shop project <https://github.com/dbt-labs/jaffle_shop>`_ is a sample dbt project that simulates an e-commerce store's data. | ||
| The project includes a series of dbt models that transform raw data into structured tables, such as sales, customers, and products. | ||
|
|
||
| The following diagram shows the original dbt workflow in a lineage graph. This graph illustrates the relationships between different models: | ||
|
|
||
| .. image:: /_static/jaffle_shop_dbt_graph.png | ||
|
|
||
| Cosmos can take this dbt workflow and convert it into an Airflow Dag, allowing you to leverage Airflow's scheduling and | ||
| orchestration capabilities. | ||
|
|
||
| To convert this dbt workflow into an Airflow Dag, create a new Dag definition file, import ``DbtDag`` from the Cosmos library, | ||
| and fill in a few parameters, such as the dbt project directory path and the profile name: | ||
|
|
||
| .. | ||
| The following renders in Sphinx but not Github: | ||
|
|
||
| .. literalinclude:: ./../../dev/dags/basic_cosmos_dag.py | ||
| :language: python | ||
| :start-after: [START local_example] | ||
| :end-before: [END local_example] | ||
|
|
||
|
|
||
| This code snippet then generates an Airflow Dag like this: | ||
|
|
||
| .. image:: https://raw.githubusercontent.com/astronomer/astronomer-cosmos/main/docs/_static/jaffle_shop_dag.png | ||
|
|
||
| ``DbtDag`` is a custom Dag generator that converts dbt projects into Airflow Dags. It also accepts Cosmos-specific arguments like | ||
| ``fail_fast``, to immediately fail a dag if dbt fails to process a resource, or ``cancel_query_on_kill``, to cancel any running | ||
| queries if the task is externally killed or manually set to failed in Airflow. ``DbtDag`` also accepts standard Dag arguments such | ||
| as ``max_active_tasks``, ``max_active_runs``, and ``default_args``. | ||
|
|
||
| With Cosmos, transitioning from a dbt workflow to an Airflow Dag is seamless, giving you the best of both tools | ||
| for managing and scaling your data workflows. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,78 +1,49 @@ | ||
| .. _getting-started: | ||
| .. _getting_started: | ||
|
|
||
| Getting started with Cosmos | ||
| =========================== | ||
|
|
||
| .. toctree:: | ||
| :maxdepth: 1 | ||
| :maxdepth: 0 | ||
| :hidden: | ||
|
|
||
| self | ||
|
|
||
| Cosmos fundamentals | ||
| ~~~~~~~~~~~~~~~~~~~~ | ||
|
|
||
| Learn the important concepts and key ideas for how Cosmos works with Airflow and dbt. | ||
|
|
||
| .. toctree:: | ||
| :maxdepth: 1 | ||
| :caption: Cosmos Fundamentals | ||
|
|
||
| Similar dbt and Airflow concepts <dbt-airflow-concepts> | ||
|
|
||
| Quickstart | ||
| ~~~~~~~~~~~ | ||
|
|
||
| Run Cosmos in just a few minutes using a pre-configured demo, such as the `cosmos-demo <https://github.com/astronomer/cosmos-demo>`_. | ||
| These guides are an ideal place to start if you don't have a particular project that you want to try out, and just want walk through the process of setting up Cosmos, Airflow, and a dbt project to work together. | ||
|
|
||
| .. toctree:: | ||
| :maxdepth: 1 | ||
| :hidden: | ||
| :caption: Quickstart | ||
|
|
||
| Open-source quickstart <oss-quickstart> | ||
| Astro CLI quickstart <astro-cli-quickstart> | ||
|
|
||
| Bring your own project | ||
| ~~~~~~~~~~~~~~~~~~~~~~ | ||
|
|
||
| An example workflow for setting up Cosmos, depending on how you run Airflow. For guides that describe specific scenarios or platforms that you want to use, see the following: | ||
|
|
||
| .. toctree:: | ||
| :maxdepth: 1 | ||
| :hidden: | ||
| :caption: Bring your own project | ||
|
|
||
| Get started with your dbt project <bring-your-own> | ||
| Open-source Airflow <open-source> | ||
| Astro <astro> | ||
| Amazon Managed Workflows for Apache Airflow (MWAA) <mwaa> | ||
| Google Cloud Composer (GCC) <gcc> | ||
|
|
||
|
|
||
| Getting Started | ||
| =============== | ||
|
|
||
| The recommended way to install and run Cosmos depends on how you run Airflow. For specific guides, see the following: | ||
|
|
||
| - `Getting Started on Open-Source <open-source.html>`__ | ||
| - `Getting Started on Astro <astro.html>`__ | ||
| - `Getting Started on MWAA <mwaa.html>`__ | ||
| - `Getting Started on GCC <gcc.html>`__ | ||
|
|
||
| You might require a different setup depending on your particular configuration. See :ref:`execution-modes`. | ||
|
|
||
| Example Demo: Jaffle Shop Project | ||
| __________________________________ | ||
|
|
||
| You can explore a practical example in the **Getting Started** guides to see how Cosmos can convert the dbt workflow into an Airflow Dag. | ||
|
|
||
| The `jaffle_shop project <https://github.com/dbt-labs/jaffle_shop>`_ is a sample dbt project that simulates an e-commerce store's data. | ||
| The project includes a series of dbt models that transform raw data into structured tables, such as sales, customers, and products. | ||
|
|
||
| The following diagram shows the original dbt workflow in a lineage graph. This graph illustrates the relationships between different models: | ||
|
|
||
| .. image:: /_static/jaffle_shop_dbt_graph.png | ||
|
|
||
| Cosmos can take this dbt workflow and convert it into an Airflow Dag, allowing you to leverage Airflow's scheduling and | ||
| orchestration capabilities. | ||
|
|
||
| To convert this dbt workflow into an Airflow Dag, create a new Dag definition file, import ``DbtDag`` from the Cosmos library, | ||
| and fill in a few parameters, such as the dbt project directory path and the profile name: | ||
|
|
||
| .. | ||
| The following renders in Sphinx but not Github: | ||
|
|
||
| .. literalinclude:: ./../../dev/dags/basic_cosmos_dag.py | ||
| :language: python | ||
| :start-after: [START local_example] | ||
| :end-before: [END local_example] | ||
|
|
||
|
|
||
| This code snippet then generates an Airflow Dag like this: | ||
|
|
||
| .. image:: https://raw.githubusercontent.com/astronomer/astronomer-cosmos/main/docs/_static/jaffle_shop_dag.png | ||
|
|
||
| ``DbtDag`` is a custom Dag generator that converts dbt projects into Airflow Dags. It also accepts Cosmos-specific arguments like | ||
| ``fail_fast``, to immediately fail a dag if dbt fails to process a resource, or ``cancel_query_on_kill``, to cancel any running | ||
| queries if the task is externally killed or manually set to failed in Airflow. ``DbtDag`` also accepts standard Dag arguments such | ||
| as ``max_active_tasks``, ``max_active_runs``, and ``default_args``. | ||
|
|
||
| With Cosmos, transitioning from a dbt workflow to an Airflow Dag is seamless, giving you the best of both tools | ||
| for managing and scaling your data workflows. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.