Release 1.2.4#676
Merged
Merged
Conversation
…AL` (#659) Extends the local operator when running `dbt deps` with the provides profile flags. This makes the logic consistent between DAG parsing and task running as referenced below https://github.com/astronomer/astronomer-cosmos/blob/8e2d5908ce89aa98813af6dfd112239e124bd69a/cosmos/dbt/graph.py#L247-L266 Closes: #658 (cherry picked from commit 6e41471)
Since Cosmos 1.2.2 users who used `ExecutionMode.DBT_LS` (directly or
via `ExecutionMode.AUTOMATIC`) and set
`ExecutionConfig.dbt_executable_path` (most, if not all, Astro CLI
users), like:
```
execution_config = ExecutionConfig(
dbt_executable_path = f"{os.environ['AIRFLOW_HOME']}/dbt_venv/bin/dbt",
)
```
Started facing the issue:
```
Broken DAG: [/usr/local/airflow/dags/my_example.py] Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/cosmos/dbt/graph.py", line 178, in load
self.load_via_dbt_ls()
File "/usr/local/lib/python3.11/site-packages/cosmos/dbt/graph.py", line 233, in load_via_dbt_ls
raise CosmosLoadDbtException(f"Unable to find the dbt executable: {self.dbt_cmd}")
cosmos.dbt.graph.CosmosLoadDbtException: Unable to find the dbt executable: dbt
```
This issue was initially reported in the Airflow #airflow-astronomer
Slack channel:
https://apache-airflow.slack.com/archives/C03T0AVNA6A/p1699584315506629
The workaround to avoid this error in Cosmos 1.2.2 and 1.2.3 is to set
the `dbt_executable_path` in the `RenderConfig`:
```
render_config=RenderConfig(dbt_executable_path = f"{os.environ['AIRFLOW_HOME']}/dbt_venv/bin/dbt",),
```
This PR solves the bug from Cosmos 1.2.4 onwards.
(cherry picked from commit 2b61364)
Update `DbtLocalBaseOperator` code to store `compiled_sql` prior to exception handling so that when a task fails, the `compiled_sql` can still be reviewed. In the process found and fixed a related bug where `compiled_sql` was being dropped on some operations due to the way that the `full_refresh` field was being added to the `template_fields`. Closes #369 Fixes bug introduced in #623 where compiled_sql was being lost in `DbtSeedLocalOperator` and `DbtRunLocalOperator` Co-authored-by: Andrew Greenburg <agreenburg@vergeventures.net> (cherry picked from commit ee91ece)
Bug fixes * Store `compiled_sql` even when task fails by @agreenburg in #671 * Refactor `LoadMethod.LOCAL` to use symlinks instead of copying directory by @jbandoro in #660 * Fix 'Unable to find the dbt executable: dbt' error by @tatiana in #666 * Fix installing deps when using `profile_mapping` & `ExecutionMode.LOCAL` by @joppevos in #659 Others * Docs fix: add execution config to MWAA code example by @ugmuka in #674
dosubot
Bot
added
area:execution
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bug fixes
compiled_sqleven when task fails by @agreenburg in Store compiled_sql even when task fails (fixes issue #369) #671LoadMethod.LOCALto use symlinks instead of copying directory by @jbandoro in Refactor LoadMethod.LOCAL to use symlinks instead of copying directory #660profile_mapping&ExecutionMode.LOCALby @joppevos in Fix installing deps when usingprofile_mapping&ExecutionMode.LOCAL#659Others