diff --git a/docs/_static/jaffle_shop_dbt_graph.png b/docs/_static/jaffle_shop_dbt_graph.png new file mode 100644 index 0000000000..2b5a79426d Binary files /dev/null and b/docs/_static/jaffle_shop_dbt_graph.png differ diff --git a/docs/index.rst b/docs/index.rst index 0bbbe0e9e1..aea7dc706a 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -28,19 +28,62 @@ |fury| |ossrank| |downloads| |pre-commit| -Run your dbt Core projects as `Apache Airflow® `_ DAGs and Task Groups with a few lines of code. Benefits include: -- Run dbt projects against Airflow connections instead of dbt profiles -- Native support for installing and running dbt in a virtual environment to avoid dependency conflicts with Airflow -- Run tests immediately after a model is done to catch issues early -- Utilize Airflow's data-aware scheduling to run models immediately after upstream ingestion -- Turn each dbt model into a task/task group complete with retries, alerting, etc. +Welcome to Astronomer Cosmos! Whether you're an experienced data practitioner or just getting started, Cosmos makes it +simple to manage and orchestrate your dbt workflows using `Apache Airflow® `_, saving you +time and effort. By automatically turning dbt workflows into Airflow DAGs, Cosmos allows you to focus on building +high-quality data models without the hassle of managing complex integrations. +To get started right away, please check out our `Quickstart Guides `_. +You can also explore more examples in `/dev/dags `_ +or in the `cosmos-demo repo `_. -Example Usage -___________________ +To learn more about about Cosmos, please read on. -You can render a Cosmos Airflow DAG using the ``DbtDag`` class. Here's an example with the `jaffle_shop project `_: + +What Is Astronomer Cosmos? +___________________________ + +Astronomer Cosmos is an open-source library that bridges Apache Airflow and dbt, allowing you to easily transform your +dbt projects into Airflow DAGs and manage everything seamlessly. With Cosmos, you can write your data transformations +using dbt and then schedule and orchestrate them with Airflow, making the entire process smooth and straightforward. + +**Why Cosmos?** Integrating dbt and Airflow can be complex, but Cosmos simplifies it by seamlessly connecting these +powerful tools—letting you focus on what matters most: delivering impactful data models and results without getting +bogged down by technical challenges. + + +Why Should You Use Cosmos? +___________________________ + +Cosmos makes orchestrating dbt workflows: + +- **Effortless**: Transform your dbt projects into Airflow DAGs without writing extra code—Cosmos handles the heavy lifting. +- **Reliable**: Rely on Airflow's robust scheduling and monitoring features to ensure your dbt workflows run smoothly and efficiently. +- **Scalable**: Easily scale your workflows to match growing data demands, thanks to Airflow's distributed capabilities. + +Whether you're handling intricate data tasks or looking to streamline your processes, Cosmos helps you orchestrate dbt +with Airflow effortlessly, saving you time and letting you focus on what truly matters—creating impactful insights. + + +Example Usage: Jaffle Shop Project +__________________________________ + +Let's explore a practical example to see how Cosmos can convert the dbt workflow into an Airflow DAG. + +The `jaffle_shop project `_ is a sample dbt project that simulates an e-commerce store's data. +The project includes a series of dbt models that transform raw data into structured tables, such as sales, customers, and products. + +Below, you can see what the original dbt workflow looks like in a lineage graph. This graph helps illustrate the +relationships between different models: + +.. image:: /_static/jaffle_shop_dbt_graph.png + +Cosmos can take this dbt workflow and convert it into an Airflow DAG, allowing you to leverage Airflow's scheduling and +orchestration capabilities. + +To convert this dbt workflow into an Airflow DAG, create a new DAG definition file, import ``DbtDag`` from the Cosmos library, +and fill in a few parameters, such as the dbt project directory path and the profile name: .. The following renders in Sphinx but not Github: @@ -51,32 +94,41 @@ You can render a Cosmos Airflow DAG using the ``DbtDag`` class. Here's an exampl :end-before: [END local_example] -This will generate an Airflow DAG that looks like this: +This code snippet will generate an Airflow DAG that looks like this: .. image:: https://raw.githubusercontent.com/astronomer/astronomer-cosmos/main/docs/_static/jaffle_shop_dag.png -Getting Started -_______________ +``DbtDag`` is a custom DAG generator that converts dbt projects into Airflow DAGs and accepts Cosmos-specific args like +``fail_fast`` to immediately fail a dag if dbt fails to process a resource, or ``cancel_query_on_kill`` to cancel any running +queries if the task is externally killed or manually set to failed in Airflow. ``DbtDag`` also accepts standard DAG arguments such +as ``max_active_tasks``, ``max_active_runs`` and ``default_args``. -Check out the Quickstart guide on our `docs `_. See more examples at `/dev/dags `_ and at the `cosmos-demo repo `_. +With Cosmos, transitioning from a dbt workflow to a proper Airflow DAG is seamless, giving you the best of both tools +for managing and scaling your data workflows. Changelog _________ We follow `Semantic Versioning `_ for releases. -Check `CHANGELOG.rst `_ +Refer to `CHANGELOG.rst `_ for the latest changes. -Contributing Guide + +Join the Community __________________ -All contributions, bug reports, bug fixes, documentation improvements, enhancements are welcome. +Have questions, need help, or interested in contributing? We welcome all contributions and feedback! + +- Join the community on Slack! You can find us in the Airflow Slack workspace `#airflow-dbt `_ channel. If you don't have an account, click `here `_ to sign up. + +- Report bugs, request features, or ask questions by creating an issue in the `GitHub repository `_. + +- Want to contribute new features, bug fixes or documentation enhancements? Please refer to our `Contributing Guide `_. -A detailed overview on how to contribute can be found in the `Contributing Guide `_. -Find out more about `our contributors `_. +- Check out this `link `_. to learn more about our current contributors -As contributors and maintainers to this project, you are expected to abide by the +Note that contributors and maintainers are expected to abide by the `Contributor Code of Conduct `_.