Below is a visual representation of how data flows through our integrated tools in this Quickstart. This comes from Dagster's global asset lineage view:
Before you embark on this integration, ensure you have the following set up and ready:
-
Python 3.10 or later: If not installed, download and install it from Python's official website.
-
Docker and Docker Compose (Docker Desktop): Install Docker following the official documentation for your specific OS.
-
Airbyte OSS version: Deploy the open-source version of Airbyte locally. Follow the installation instructions from the Airbyte Documentation.
-
Terraform (Optional): Terraform will help you provision and manage the Airbyte resources. If you haven't installed it, follow the official Terraform installation guide. This is an optional step because you can also create and manage Airbyte resources via the UI.
-
Google Cloud account with BigQuery: You will also need to add the necessary permissions to allow Airbyte and dbt to access the data in BigQuery. A step-by-step guide is provided below.