-
Notifications
You must be signed in to change notification settings - Fork 297
Release 1.13.1 #2388
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Release 1.13.1 #2388
Changes from all commits
Commits
Show all changes
24 commits
Select commit
Hold shift + click to select a range
a9cba2b
Refactor to avoid potential future `UnboundLocalError` for `producer_…
RinZ27 35b7b15
Document `ExecutionMode.KUBERNETES` limitations (#2326)
tatiana a8e2c1e
Improve test coverage for PR #2307 (#2308)
tatiana 24868a6
Fix leaked semaphore warnings in Airflow 3 by resetting dbt adapters …
pankajkoti 6f3ef0a
Fix cache swap issue (#2332)
YourRoyalLinus bf31d5c
Handle Param Validation errors (#2358)
tatiana a5b5df3
Fix selecting model when it has the same name as folder (#2328)
pankajastro c6121db
Populate compiled_sql for InvocationMode.SUBPROCESS in ExecutionMode.…
pankajkoti 9b5e0d9
Error handle invalid YAML with `LoadMode.DBT_MANIFEST` and `RenderCon…
YourRoyalLinus ba6d2d1
Change Snowflake profile mapppings to default to four threads (#2374)
tatiana c84f244
Preserve extra_context for watcher consumer task instances (#2381)
pankajkoti 692366e
fix(watcher): Respect deferrable=False from operator_args on consumer…
pankajkoti 9868d29
Release 1.13.1
tatiana ef5c447
Fix codespell rin issue
tatiana c396c0e
Improve changelog
tatiana e06654a
Fix KeyError in graph selector when using + operator with dbt-loom ex…
award1230 d716c05
Address feedback from code review #2389 (#2394)
evanvolgas 8558067
Improve 1.13.1 changelog further
tatiana 52745bb
Release 1.13.1a1
tatiana 8d028e3
🎨 [pre-commit.ci] Auto format from pre-commit.com hooks
pre-commit-ci[bot] ac5d130
Apply suggestion from @tatiana
tatiana c8cc599
Add .airflow-registry.yaml for Airflow Provider Registry (#2387)
kaxil 7dd5209
Release 1.13.1
tatiana 38df404
Remove empty line from changelog
tatiana File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,219 @@ | ||
| # .airflow-registry.yaml — metadata for the Apache Airflow Provider Registry | ||
| # Place this file in the root of your provider repository. | ||
| # Docs: https://github.com/apache/airflow/blob/main/dev/registry/README.md | ||
|
|
||
| package-name: astronomer-cosmos | ||
| display-name: Cosmos (dbt) | ||
| description: Run dbt projects as Airflow DAGs and Task Groups. Supports both dbt Core and dbt Fusion. | ||
| logo: https://raw.githubusercontent.com/astronomer/astronomer-cosmos/main/docs/_static/cosmos-logo.svg | ||
| docs-url: https://astronomer.github.io/astronomer-cosmos | ||
| source-url: https://github.com/astronomer/astronomer-cosmos | ||
|
|
||
| operators: | ||
| # Execute dbt commands using a local dbt installation, either within the Airflow Python virtual environment or as a standalone dbt binary. | ||
| - module: cosmos.operators.local.DbtRunLocalOperator | ||
| description: Run `dbt run` using a local dbt installation | ||
| - module: cosmos.operators.local.DbtTestLocalOperator | ||
| description: Run `dbt test` using a local dbt installation | ||
| - module: cosmos.operators.local.DbtSeedLocalOperator | ||
| description: Run `dbt seed` using a local dbt installation | ||
| - module: cosmos.operators.local.DbtSnapshotLocalOperator | ||
| description: Run `dbt snapshot` using a local dbt installation | ||
| - module: cosmos.operators.local.DbtCompileLocalOperator | ||
| description: Run `dbt compile` using a local dbt installation | ||
| - module: cosmos.operators.local.DbtLSLocalOperator | ||
| description: Run `dbt ls` using a local dbt installation | ||
| - module: cosmos.operators.local.DbtRunOperationLocalOperator | ||
| description: Run `dbt run-operation` using a local dbt installation | ||
| - module: cosmos.operators.local.DbtBuildLocalOperator | ||
| description: Run `dbt build` using a local dbt installation | ||
| - module: cosmos.operators.local.DbtCloneLocalOperator | ||
| description: Run `dbt clone` using a local dbt installation | ||
| - module: cosmos.operators.local.DbtSourceLocalOperator | ||
| description: Run `dbt source freshness` using a local dbt installation | ||
| - module: cosmos.operators.local.DbtDocsLocalOperator | ||
| description: Run `dbt docs generate` using a local dbt installation | ||
| - module: cosmos.operators.local.DbtDocsS3LocalOperator | ||
| description: Run `dbt docs generate` and upload the generated docs to AWS S3 | ||
| - module: cosmos.operators.local.DbtDocsAzureStorageLocalOperator | ||
| description: Run `dbt docs generate` and upload the generated docs to Azure Blob Storage | ||
| - module: cosmos.operators.local.DbtDocsGCSLocalOperator | ||
| description: Run `dbt docs generate` and upload the generated docs to Google Cloud Storage (GCS) | ||
|
|
||
| # Execute dbt commands by running each command in its own (local) Docker container. | ||
| - module: cosmos.operators.docker.DbtBuildDockerOperator | ||
| description: Run `dbt build` inside a Docker container. | ||
| - module: cosmos.operators.docker.DbtCloneDockerOperator | ||
| description: Run `dbt clone` inside a Docker container. | ||
| - module: cosmos.operators.docker.DbtLSDockerOperator | ||
| description: Run `dbt ls` inside a Docker container. | ||
| - module: cosmos.operators.docker.DbtRunDockerOperator | ||
| description: Run `dbt run` inside a Docker container. | ||
| - module: cosmos.operators.docker.DbtRunOperationDockerOperator | ||
| description: Run `dbt run-operation` inside a Docker container. | ||
| - module: cosmos.operators.docker.DbtSeedDockerOperator | ||
| description: Run `dbt seed` inside a Docker container. | ||
| - module: cosmos.operators.docker.DbtSnapshotDockerOperator | ||
| description: Run `dbt snapshot` inside a Docker container. | ||
| - module: cosmos.operators.docker.DbtSourceDockerOperator | ||
| description: Run `dbt source freshness` inside a Docker container. | ||
| - module: cosmos.operators.docker.DbtTestDockerOperator | ||
| description: Run `dbt test` inside a Docker container. | ||
|
|
||
| # Execute dbt commands via dedicated Kubernetes pods | ||
| - module: cosmos.operators.kubernetes.DbtBuildKubernetesOperator | ||
| description: Run `dbt build` in a Kubernetes pod. | ||
| - module: cosmos.operators.kubernetes.DbtCloneKubernetesOperator | ||
| description: Run `dbt clone` in a Kubernetes pod. | ||
| - module: cosmos.operators.kubernetes.DbtLSKubernetesOperator | ||
| description: Run `dbt ls` in a Kubernetes pod. | ||
| - module: cosmos.operators.kubernetes.DbtRunKubernetesOperator | ||
| description: Run `dbt run` in a Kubernetes pod. | ||
| - module: cosmos.operators.kubernetes.DbtRunOperationKubernetesOperator | ||
| description: Run `dbt run-operation` in a Kubernetes pod. | ||
| - module: cosmos.operators.kubernetes.DbtSeedKubernetesOperator | ||
| description: Run `dbt seed` in a Kubernetes pod. | ||
| - module: cosmos.operators.kubernetes.DbtSnapshotKubernetesOperator | ||
| description: Run `dbt snapshot` in a Kubernetes pod. | ||
| - module: cosmos.operators.kubernetes.DbtSourceKubernetesOperator | ||
| description: Run `dbt source freshness` in a Kubernetes pod. | ||
| - module: cosmos.operators.kubernetes.DbtTestKubernetesOperator | ||
| description: Run `dbt test` in a Kubernetes pod. | ||
|
|
||
| # Execute dbt commands via isolated Python virtual environments created and managed by Cosmos | ||
| - module: cosmos.operators.virtualenv.DbtBuildVirtualenvOperator | ||
| description: Run `dbt build` inside a Cosmos-managed Python virtual environment. | ||
| - module: cosmos.operators.virtualenv.DbtCloneVirtualenvOperator | ||
| description: Run `dbt clone` inside a Cosmos-managed Python virtual environment. | ||
| - module: cosmos.operators.virtualenv.DbtDocsVirtualenvOperator | ||
| description: Run `dbt docs generate` inside a Cosmos-managed Python virtual environment. | ||
| - module: cosmos.operators.virtualenv.DbtLSVirtualenvOperator | ||
| description: Run `dbt ls` inside a Cosmos-managed Python virtual environment. | ||
| - module: cosmos.operators.virtualenv.DbtRunVirtualenvOperator | ||
| description: Run `dbt run` inside a Cosmos-managed Python virtual environment. | ||
| - module: cosmos.operators.virtualenv.DbtRunOperationVirtualenvOperator | ||
| description: Run `dbt run-operation` inside a Cosmos-managed Python virtual environment. | ||
| - module: cosmos.operators.virtualenv.DbtSeedVirtualenvOperator | ||
| description: Run `dbt seed` inside a Cosmos-managed Python virtual environment. | ||
| - module: cosmos.operators.virtualenv.DbtSnapshotVirtualenvOperator | ||
| description: Run `dbt snapshot` inside a Cosmos-managed Python virtual environment. | ||
| - module: cosmos.operators.virtualenv.DbtSourceVirtualenvOperator | ||
| description: Run `dbt source freshness` inside a Cosmos-managed Python virtual environment. | ||
| - module: cosmos.operators.virtualenv.DbtTestVirtualenvOperator | ||
| description: Run `dbt test` inside a Cosmos-managed Python virtual environment. | ||
|
|
||
| # Pre-compile the project using dbt Core and use Airflow `BigQueryInsertJobOperator` to asynchronously run the transformations. | ||
| - module: cosmos.operators._asynchronous.SetupAsyncOperator | ||
| description: Monkey-patch `dbt run` command to pre-generate the dbt models SQL and export them to XCom | ||
| - module: cosmos.operators.airflow_async.DbtRunAirflowAsyncOperator | ||
| description: Retrieve the SQL from the setup task and execute each model natively to Airflow via a subclass of `BigQueryInsertJobOperator` | ||
| - module: cosmos.operators._asynchronous.TeardownAsyncOperator | ||
| description: Clear all the SQL files related to this DAG run from corresponding XComs | ||
|
|
||
| # Run dbt commands as containers via the Amazon Elastic Container Service (AWS ECS) | ||
| - module: cosmos.operators.aws_ecs.DbtBuildAwsEcsOperator | ||
| description: Run `dbt build` as a task in AWS ECS. | ||
| - module: cosmos.operators.aws_ecs.DbtLSAwsEcsOperator | ||
| description: Run `dbt ls` as a task in AWS ECS. | ||
| - module: cosmos.operators.aws_ecs.DbtRunAwsEcsOperator | ||
| description: Run `dbt run` as a task in AWS ECS. | ||
| - module: cosmos.operators.aws_ecs.DbtRunOperationAwsEcsOperator | ||
| description: Run `dbt run-operation` as a task in AWS ECS. | ||
| - module: cosmos.operators.aws_ecs.DbtSeedAwsEcsOperator | ||
| description: Run `dbt seed` as a task in AWS ECS. | ||
| - module: cosmos.operators.aws_ecs.DbtSnapshotAwsEcsOperator | ||
| description: Run `dbt snapshot` as a task in AWS ECS. | ||
| - module: cosmos.operators.aws_ecs.DbtSourceAwsEcsOperator | ||
| description: Run `dbt source freshness` as a task in AWS ECS. | ||
| - module: cosmos.operators.aws_ecs.DbtTestAwsEcsOperator | ||
| description: Run `dbt test` as a task in AWS ECS. | ||
|
|
||
| # Run dbt commands as Kubernetes pods via the Amazon Elastic Kubernetes Service (AWS EKS). | ||
| - module: cosmos.operators.aws_eks.DbtBuildAwsEksOperator | ||
| description: Run `dbt build` in a pod on AWS EKS. | ||
| - module: cosmos.operators.aws_eks.DbtCloneAwsEksOperator | ||
| description: Run `dbt clone` in a pod on AWS EKS. | ||
| - module: cosmos.operators.aws_eks.DbtLSAwsEksOperator | ||
| description: Run `dbt ls` in a pod on AWS EKS. | ||
| - module: cosmos.operators.aws_eks.DbtRunAwsEksOperator | ||
| description: Run `dbt run` in a pod on AWS EKS. | ||
| - module: cosmos.operators.aws_eks.DbtRunOperationAwsEksOperator | ||
| description: Run `dbt run-operation` in a pod on AWS EKS. | ||
| - module: cosmos.operators.aws_eks.DbtSeedAwsEksOperator | ||
| description: Run `dbt seed` in a pod on AWS EKS. | ||
| - module: cosmos.operators.aws_eks.DbtSnapshotAwsEksOperator | ||
| description: Run `dbt snapshot` in a pod on AWS EKS. | ||
| - module: cosmos.operators.aws_eks.DbtTestAwsEksOperator | ||
| description: Run `dbt test` in a pod on AWS EKS. | ||
|
|
||
| # Run dbt commands as remote containers via the Azure Container Instance service. | ||
| - module: cosmos.operators.azure_container_instance.DbtBuildAzureContainerInstanceOperator | ||
| description: Run `dbt build` in an Azure Container Instance. | ||
| - module: cosmos.operators.azure_container_instance.DbtCloneAzureContainerInstanceOperator | ||
| description: Run `dbt clone` in an Azure Container Instance. | ||
| - module: cosmos.operators.azure_container_instance.DbtLSAzureContainerInstanceOperator | ||
| description: Run `dbt ls` in an Azure Container Instance. | ||
| - module: cosmos.operators.azure_container_instance.DbtRunAzureContainerInstanceOperator | ||
| description: Run `dbt run` in an Azure Container Instance. | ||
| - module: cosmos.operators.azure_container_instance.DbtRunOperationAzureContainerInstanceOperator | ||
| description: Run `dbt run-operation` in an Azure Container Instance. | ||
| - module: cosmos.operators.azure_container_instance.DbtSeedAzureContainerInstanceOperator | ||
| description: Run `dbt seed` in an Azure Container Instance. | ||
| - module: cosmos.operators.azure_container_instance.DbtSnapshotAzureContainerInstanceOperator | ||
| description: Run `dbt snapshot` in an Azure Container Instance. | ||
| - module: cosmos.operators.azure_container_instance.DbtSourceAzureContainerInstanceOperator | ||
| description: Run `dbt source freshness` in an Azure Container Instance. | ||
| - module: cosmos.operators.azure_container_instance.DbtTestAzureContainerInstanceOperator | ||
| description: Run `dbt test` in an Azure Container Instance. | ||
|
|
||
| # Run dbt commands as Google Cloud (GCP) Platform Cloud Run Jobs | ||
| - module: cosmos.operators.gcp_cloud_run_job.DbtBuildGcpCloudRunJobOperator | ||
| description: Run `dbt build` as a Google Cloud Run Job. | ||
| - module: cosmos.operators.gcp_cloud_run_job.DbtCloneGcpCloudRunJobOperator | ||
| description: Run `dbt clone` as a Google Cloud Run Job. | ||
| - module: cosmos.operators.gcp_cloud_run_job.DbtLSGcpCloudRunJobOperator | ||
| description: Run `dbt ls` as a Google Cloud Run Job. | ||
| - module: cosmos.operators.gcp_cloud_run_job.DbtRunGcpCloudRunJobOperator | ||
| description: Run `dbt run` as a Google Cloud Run Job. | ||
| - module: cosmos.operators.gcp_cloud_run_job.DbtRunOperationGcpCloudRunJobOperator | ||
| description: Run `dbt run-operation` as a Google Cloud Run Job. | ||
| - module: cosmos.operators.gcp_cloud_run_job.DbtSeedGcpCloudRunJobOperator | ||
| description: Run `dbt seed` as a Google Cloud Run Job. | ||
| - module: cosmos.operators.gcp_cloud_run_job.DbtSnapshotGcpCloudRunJobOperator | ||
| description: Run `dbt snapshot` as a Google Cloud Run Job. | ||
| - module: cosmos.operators.gcp_cloud_run_job.DbtSourceGcpCloudRunJobOperator | ||
| description: Run `dbt source freshness` as a Google Cloud Run Job. | ||
| - module: cosmos.operators.gcp_cloud_run_job.DbtTestGcpCloudRunJobOperator | ||
| description: Run `dbt test` as a Google Cloud Run Job. | ||
|
|
||
| # Watcher pattern (local) | ||
| # Run a single local `dbt build` or `dbt run` command per pipeline, via a producer task, and have Airflow deferrable sensors (consumers) tracking progress for individual dbt nodes. | ||
| - module: cosmos.operators.watcher.DbtProducerWatcherOperator | ||
| description: Run a single dbt command (build or run) for the entire dbt project and produce one XCom containing the status of each dbt node (model, seed, snapshot). | ||
| - module: cosmos.operators.watcher.DbtRunWatcherOperator | ||
| description: Monitor a dbt model using Airflow deferrable XCom sensors. On retry, executes `dbt run` for the specific model via `DbtRunLocalOperator`. | ||
| - module: cosmos.operators.watcher.DbtSeedWatcherOperator | ||
| description: Monitor a dbt seed using Airflow deferrable XCom sensors. On retry, executes `dbt seed` for the specific seed via `DbtSeedLocalOperator`. | ||
| - module: cosmos.operators.watcher.DbtSnapshotWatcherOperator | ||
| description: Monitor a dbt snapshot using Airflow deferrable XCom sensors. On retry, executes `dbt snapshot` for the specific snapshot via `DbtSnapshotLocalOperator`. | ||
| - module: cosmos.operators.watcher.DbtTestWatcherOperator | ||
| description: Runs `dbt test` using the local execution pattern, or acts as an `EmptyOperator` for `TestBehavior.AFTER_EACH`. | ||
|
|
||
| # Watcher pattern (Kubernetes) | ||
| # Run a single `dbt build` or `dbt run` command per pipeline (producer), via a Kubernetes pod, and have Airflow deferrable sensors (consumers) tracking progress for individual dbt nodes. | ||
| - module: cosmos.operators.watcher_kubernetes.DbtProducerWatcherKubernetesOperator | ||
| description: Run a single dbt command (build or run) via a Kubernetes Pod Operator for the entire dbt project and produce one XCom containing the status of each dbt node (model, seed, snapshot). | ||
| - module: cosmos.operators.watcher_kubernetes.DbtRunWatcherKubernetesOperator | ||
| description: Monitor a dbt model using Airflow deferrable XCom sensors. On retry, executes `dbt run` for the specific model via `DbtRunKubernetesOperator`. | ||
| - module: cosmos.operators.watcher_kubernetes.DbtSeedWatcherKubernetesOperator | ||
| description: Monitor a dbt seed using Airflow deferrable XCom sensors. On retry, executes `dbt seed` for the specific seed via `DbtSeedKubernetesOperator`. | ||
| - module: cosmos.operators.watcher_kubernetes.DbtSnapshotWatcherKubernetesOperator | ||
| description: Monitor a dbt snapshot using Airflow deferrable XCom sensors. On retry, executes `dbt snapshot` for the specific snapshot via `DbtSnapshotKubernetesOperator`. | ||
| - module: cosmos.operators.watcher_kubernetes.DbtTestWatcherKubernetesOperator | ||
| description: Runs `dbt test` using the Kubernetes execution pattern, or acts as an `EmptyOperator` for `TestBehavior.AFTER_EACH`. | ||
|
|
||
| other: | ||
| - module: cosmos.airflow.dag.DbtDag | ||
| description: Render a complete dbt project as an Airflow DAG | ||
| - module: cosmos.airflow.task_group.DbtTaskGroup | ||
| description: Render a dbt project as an Airflow Task Group |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| rin |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.