Awesome Dagster

Please visit the offical awesome-dagster repo by dagster. I moved the relevant content over there.

A curated list of dagster code snippets for data engineers

List of content

1. Resources
2. Solids
3. Hooks
4. Types
5. Docker Images

Resources

• Data Warehouse
  • Apache Druid Connector resource to send GET/POST REST-Requests
• Relational
  • MySQL Instantiate a connection to a MySQL database and run queries
• Monitoring
  • HealthcheckIO Send updates to Cron job monitoring HealthcheckIO

Solids

• Spark & Delta-Lake
  • Spark and Delta Lake Solids to create Delta-Table create_delta_table, load data from a Delta-Table into a Spark-df create_delta_table
• Scrapting
  • Web-Scraping and Caching Web-scraping with BeautifulSoup

Hooks

• Success and Failure Hook and Pipeline Example How to use Hooks on failure or on error

Types

• JsonType A valid representation of a JSON, validated with json.loads()

Docker Images

• Everything but the Kitchen Sink Image Image to run Spark and a whole lot more
• Spark Image to run Spark and Delta-Lake

Community

Forums

• GitHub Discussions News, ideas, Q&A and more about Dagster
• Slack Channel Questions and Discussions

Podcasts

• Data Engineering Podcast - Build Maintainable And Testable Data Applications With Dagster - Episode 104 October 28, 2019: Fist podcast about Dagster where Nick Schrock the founder explains the motivation and the story behind Dagster.
• Software Engineering Daily - Dagster with Nick Schrock November 15, 2019: Second podcast with different insight from the software engineering perspective
• Python Podcast - Digging Into Dagster: An Opinionated Open Source Framework For Data Orchestration - Episode 279 September 7, 2020: Update on Dagster after a year.

Slides

• Running Pipelines In Production Using Dagster to replace a homegrown Cron job
• Bestplaces and their Journey A journey from various heterogeneous Tools to integrated Dagster Environment
• Dagster Migration At Prezi From JSON based homegrown orchestrator to Dagster

Inspired by the Awesome list of Data Engineering by Igor Barinov.

License

To the extent possible under law, Simon Späti has waived all copyright and related or neighboring rights to this work.