-
Notifications
You must be signed in to change notification settings - Fork 296
Getting Started: How cosmos works #2449
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
10 commits
Select commit
Hold shift + click to select a range
f46437e
Add skeleton
lzdanski e5918ec
Add how cosmos works landing page
lzdanski 3123afa
Add Get started disambiguation
lzdanski 3be4983
first pass - how cosmos works
lzdanski 16ca5df
Merge branch 'astronomer:main' into how-cosmos-works
lzdanski 1dfb4c6
Follow Airflow copyright conventios
lzdanski c3df3b8
Merge branch 'main' into how-cosmos-works
lzdanski 9f4f285
Remove image for now
lzdanski 1612681
Fix header for consistency
lzdanski af078c5
Add image
lzdanski File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| .. _how-cosmos-works: | ||
|
|
||
| How Cosmos works | ||
| ================= | ||
|
|
||
| Cosmos is an open-source Python package developed by Astronomer under the Apache License 2.0. The | ||
| package includes classes that automatically turn dbt Core and dbt Fusion projects into `Apache Airflow® <https://airflow.apache.org/>`_ | ||
| workflows. | ||
|
|
||
| When using Cosmos you can write your data transformations using dbt and then use Airflow’s | ||
| advanced orchestration capabilities to integrate your dbt projects into your end to end workflows, scheduled to | ||
| run automatically. | ||
|
|
||
| Overview | ||
| ~~~~~~~~~ | ||
|
|
||
| Cosmos creates an interface between a dbt project and Airflow, allowing you to translate your dbt project into a Dag. Then, your Cosmos configuration provides the necessary configurations to Airflow so that it can schedule and initiate running your dbt code when you want. | ||
|
|
||
| You have a number of configuration options, but fundamentally, Cosmos provides the following two functions: | ||
|
|
||
| - **Parse your dbt project**: Cosmos parses your dbt project, and translates it into an Airflow Dag. This process uses the `ProjectConfig <../reference/configs/project-config.html>`_ and `RenderConfig <../guides/translate_dbt/render-config.html>`_ to customize specific behavior, allowing you to optimize how your dbt project is represented as a Dag. | ||
|
|
||
| - **Execute the dbt commands**: Cosmos then executes the Dag, using the execution options in your `ExecutionConfig <../reference/configs/execution-config.html>`_ and `ProjectConfig <../reference/configs/project-config.html>`_ to run dbt commands with the appropriate dbt adapter, finally resulting in your dbt SQL running in your data warehouse. Cosmos uses a connection defined in the `ProfileConfig <../profiles/index.html>`_ to execute your SQL in your data warehouse. | ||
|
|
||
| .. image:: ../_static/how_cosmos_works.png | ||
| :alt: Diagram showing that Cosmos parses dbt projects, with options defined by the ProfileConfig and RenderConfig. Then, Cosmos executes the resulting Dag, with options defined by ExecutionConfig and ProjectConfig. | ||
|
|
||
| Quickstart | ||
| ~~~~~~~~~~~ | ||
|
|
||
| Even though Cosmos is highly extensible, and you have many advanced customization options, you can run a demo with the `Astro CLI <astro-cli-quickstart.html>`_ in just a few minutes. This demo introduces you to the key elements required for Cosmos to parse dbt projects and run Dags. | ||
|
|
||
| Bring your own project | ||
| ~~~~~~~~~~~~~~~~~~~~~~~~ | ||
|
|
||
| If you have existing resources for Airflow or dbt, and want to start exploring how to get started with a project more similar to your use case, check out the **Get started with Cosmos** guides. These resources provide more general recommendations for how to create a new project. | ||
|
|
||
| - :ref:`open-source` | ||
| - :ref:`astro` | ||
| - :ref:`mwaa` | ||
| - :ref:`gcc` | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.