Skip to content

Fix reusing config accross TaskGroups/DAGs#664

Merged
tatiana merged 1 commit into
mainfrom
fix-config-reuse
Nov 9, 2023
Merged

Fix reusing config accross TaskGroups/DAGs#664
tatiana merged 1 commit into
mainfrom
fix-config-reuse

Conversation

@tatiana
Copy link
Copy Markdown
Collaborator

@tatiana tatiana commented Nov 9, 2023

If execution_config was reused, Cosmos 1.2.2 would raise:

astronomer-cosmos/dags/basic_cosmos_task_group.py
Traceback (most recent call last):
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/venv-38/lib/python3.8/site-packages/airflow/models/dagbag.py", line 343, in parse
    loader.exec_module(new_module)
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/dags/basic_cosmos_task_group.py", line 74, in <module>
    basic_cosmos_task_group()
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/venv-38/lib/python3.8/site-packages/airflow/models/dag.py", line 3817, in factory
    f(**f_kwargs)
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/dags/basic_cosmos_task_group.py", line 54, in basic_cosmos_task_group
    orders = DbtTaskGroup(
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/cosmos/airflow/task_group.py", line 26, in __init__
    DbtToAirflowConverter.__init__(self, *args, **specific_kwargs(**kwargs))
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/cosmos/converter.py", line 113, in __init__
    raise CosmosValueError(
cosmos.exceptions.CosmosValueError: ProjectConfig.dbt_project_path is mutually exclusive with RenderConfig.dbt_project_path and ExecutionConfig.dbt_project_path.If using RenderConfig.dbt_project_path or ExecutionConfig.dbt_project_path, ProjectConfig.dbt_project_path should be None

This has been raised by an Astro customer and our field engineer, who tried to run:
https://github.com/astronomer/cosmos-demo

If execution_config was reused, Cosmos 1.2.2 would raise:

```
astronomer-cosmos/dags/basic_cosmos_task_group.py
Traceback (most recent call last):
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/venv-38/lib/python3.8/site-packages/airflow/models/dagbag.py", line 343, in parse
    loader.exec_module(new_module)
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/dags/basic_cosmos_task_group.py", line 74, in <module>
    basic_cosmos_task_group()
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/venv-38/lib/python3.8/site-packages/airflow/models/dag.py", line 3817, in factory
    f(**f_kwargs)
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/dags/basic_cosmos_task_group.py", line 54, in basic_cosmos_task_group
    orders = DbtTaskGroup(
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/cosmos/airflow/task_group.py", line 26, in __init__
    DbtToAirflowConverter.__init__(self, *args, **specific_kwargs(**kwargs))
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/cosmos/converter.py", line 113, in __init__
    raise CosmosValueError(
cosmos.exceptions.CosmosValueError: ProjectConfig.dbt_project_path is mutually exclusive with RenderConfig.dbt_project_path and ExecutionConfig.dbt_project_path.If using RenderConfig.dbt_project_path or ExecutionConfig.dbt_project_path, ProjectConfig.dbt_project_path should be None
```
@tatiana tatiana requested a review from a team as a code owner November 9, 2023 11:52
@tatiana tatiana requested a review from a team November 9, 2023 11:52
@netlify
Copy link
Copy Markdown

netlify Bot commented Nov 9, 2023

👷 Deploy Preview for amazing-pothos-a3bca0 processing.

Name Link
🔨 Latest commit 6091a3b
🔍 Latest deploy log https://app.netlify.com/sites/amazing-pothos-a3bca0/deploys/654cc7e2353036000841340a

@dosubot dosubot Bot added area:config Related to configuration, like YAML files, environment variables, or executer configuration parsing:custom Related to custom parsing, like custom DAG parsing, custom DBT parsing, etc priority:high High priority issues are blocking or critical issues without a workaround and large impact labels Nov 9, 2023
@tatiana tatiana added area:parsing Related to parsing DAG/DBT improvement, issues, or fixes and removed parsing:custom Related to custom parsing, like custom DAG parsing, custom DBT parsing, etc labels Nov 9, 2023
@codecov
Copy link
Copy Markdown

codecov Bot commented Nov 9, 2023

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (9cd46d2) 92.67% compared to head (6091a3b) 92.68%.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #664   +/-   ##
=======================================
  Coverage   92.67%   92.68%           
=======================================
  Files          54       54           
  Lines        2212     2215    +3     
=======================================
+ Hits         2050     2053    +3     
  Misses        162      162           
Files Coverage Δ
cosmos/converter.py 97.01% <100.00%> (+0.13%) ⬆️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@tatiana tatiana added this to the 1.2.3 milestone Nov 9, 2023
@tatiana tatiana merged commit f27cb25 into main Nov 9, 2023
@tatiana tatiana deleted the fix-config-reuse branch November 9, 2023 16:09
tatiana added a commit that referenced this pull request Nov 9, 2023
If execution_config was reused, Cosmos 1.2.2 would raise:

```
astronomer-cosmos/dags/basic_cosmos_task_group.py
Traceback (most recent call last):
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/venv-38/lib/python3.8/site-packages/airflow/models/dagbag.py", line 343, in parse
    loader.exec_module(new_module)
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/dags/basic_cosmos_task_group.py", line 74, in <module>
    basic_cosmos_task_group()
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/venv-38/lib/python3.8/site-packages/airflow/models/dag.py", line 3817, in factory
    f(**f_kwargs)
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/dags/basic_cosmos_task_group.py", line 54, in basic_cosmos_task_group
    orders = DbtTaskGroup(
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/cosmos/airflow/task_group.py", line 26, in __init__
    DbtToAirflowConverter.__init__(self, *args, **specific_kwargs(**kwargs))
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/cosmos/converter.py", line 113, in __init__
    raise CosmosValueError(
cosmos.exceptions.CosmosValueError: ProjectConfig.dbt_project_path is mutually exclusive with RenderConfig.dbt_project_path and ExecutionConfig.dbt_project_path.If using RenderConfig.dbt_project_path or ExecutionConfig.dbt_project_path, ProjectConfig.dbt_project_path should be None
```

This has been raised by an Astro customer and our field engineer, who
tried to run: https://github.com/astronomer/cosmos-demo
tatiana added a commit that referenced this pull request Nov 9, 2023
If execution_config was reused, Cosmos 1.2.2 would raise:

```
astronomer-cosmos/dags/basic_cosmos_task_group.py
Traceback (most recent call last):
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/venv-38/lib/python3.8/site-packages/airflow/models/dagbag.py", line 343, in parse
    loader.exec_module(new_module)
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/dags/basic_cosmos_task_group.py", line 74, in <module>
    basic_cosmos_task_group()
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/venv-38/lib/python3.8/site-packages/airflow/models/dag.py", line 3817, in factory
    f(**f_kwargs)
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/dags/basic_cosmos_task_group.py", line 54, in basic_cosmos_task_group
    orders = DbtTaskGroup(
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/cosmos/airflow/task_group.py", line 26, in __init__
    DbtToAirflowConverter.__init__(self, *args, **specific_kwargs(**kwargs))
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/cosmos/converter.py", line 113, in __init__
    raise CosmosValueError(
cosmos.exceptions.CosmosValueError: ProjectConfig.dbt_project_path is mutually exclusive with RenderConfig.dbt_project_path and ExecutionConfig.dbt_project_path.If using RenderConfig.dbt_project_path or ExecutionConfig.dbt_project_path, ProjectConfig.dbt_project_path should be None
```

This has been raised by an Astro customer and our field engineer, who
tried to run: https://github.com/astronomer/cosmos-demo

(cherry picked from commit 9001c98)
tatiana added a commit that referenced this pull request Nov 9, 2023
Bug fixes

* Fix reusing config accross TaskGroups/DAGs by @tatiana in #664
@tatiana tatiana mentioned this pull request Nov 9, 2023
tatiana added a commit that referenced this pull request Nov 9, 2023
Bug fix

* Fix reusing config accross TaskGroups/DAGs by @tatiana in #664
LennartKloppenburg pushed a commit to LennartKloppenburg/astronomer-cosmos that referenced this pull request Dec 17, 2023
If execution_config was reused, Cosmos 1.2.2 would raise:

```
astronomer-cosmos/dags/basic_cosmos_task_group.py
Traceback (most recent call last):
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/venv-38/lib/python3.8/site-packages/airflow/models/dagbag.py", line 343, in parse
    loader.exec_module(new_module)
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/dags/basic_cosmos_task_group.py", line 74, in <module>
    basic_cosmos_task_group()
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/venv-38/lib/python3.8/site-packages/airflow/models/dag.py", line 3817, in factory
    f(**f_kwargs)
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/dags/basic_cosmos_task_group.py", line 54, in basic_cosmos_task_group
    orders = DbtTaskGroup(
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/cosmos/airflow/task_group.py", line 26, in __init__
    DbtToAirflowConverter.__init__(self, *args, **specific_kwargs(**kwargs))
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/cosmos/converter.py", line 113, in __init__
    raise CosmosValueError(
cosmos.exceptions.CosmosValueError: ProjectConfig.dbt_project_path is mutually exclusive with RenderConfig.dbt_project_path and ExecutionConfig.dbt_project_path.If using RenderConfig.dbt_project_path or ExecutionConfig.dbt_project_path, ProjectConfig.dbt_project_path should be None
```

This has been raised by an Astro customer and our field engineer, who
tried to run: https://github.com/astronomer/cosmos-demo
tatiana added a commit that referenced this pull request Jul 4, 2024
If execution_config was reused, Cosmos 1.2.2 would raise:

```
astronomer-cosmos/dags/basic_cosmos_task_group.py
Traceback (most recent call last):
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/venv-38/lib/python3.8/site-packages/airflow/models/dagbag.py", line 343, in parse
    loader.exec_module(new_module)
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/dags/basic_cosmos_task_group.py", line 74, in <module>
    basic_cosmos_task_group()
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/venv-38/lib/python3.8/site-packages/airflow/models/dag.py", line 3817, in factory
    f(**f_kwargs)
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/dags/basic_cosmos_task_group.py", line 54, in basic_cosmos_task_group
    orders = DbtTaskGroup(
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/cosmos/airflow/task_group.py", line 26, in __init__
    DbtToAirflowConverter.__init__(self, *args, **specific_kwargs(**kwargs))
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/cosmos/converter.py", line 113, in __init__
    raise CosmosValueError(
cosmos.exceptions.CosmosValueError: ProjectConfig.dbt_project_path is mutually exclusive with RenderConfig.dbt_project_path and ExecutionConfig.dbt_project_path.If using RenderConfig.dbt_project_path or ExecutionConfig.dbt_project_path, ProjectConfig.dbt_project_path should be None
```

This has been raised by an Astro customer and our field engineer, who
tried to run: https://github.com/astronomer/cosmos-demo
arojasb3 pushed a commit to arojasb3/astronomer-cosmos that referenced this pull request Jul 14, 2024
If execution_config was reused, Cosmos 1.2.2 would raise:

```
astronomer-cosmos/dags/basic_cosmos_task_group.py
Traceback (most recent call last):
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/venv-38/lib/python3.8/site-packages/airflow/models/dagbag.py", line 343, in parse
    loader.exec_module(new_module)
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/dags/basic_cosmos_task_group.py", line 74, in <module>
    basic_cosmos_task_group()
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/venv-38/lib/python3.8/site-packages/airflow/models/dag.py", line 3817, in factory
    f(**f_kwargs)
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/dags/basic_cosmos_task_group.py", line 54, in basic_cosmos_task_group
    orders = DbtTaskGroup(
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/cosmos/airflow/task_group.py", line 26, in __init__
    DbtToAirflowConverter.__init__(self, *args, **specific_kwargs(**kwargs))
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/cosmos/converter.py", line 113, in __init__
    raise CosmosValueError(
cosmos.exceptions.CosmosValueError: ProjectConfig.dbt_project_path is mutually exclusive with RenderConfig.dbt_project_path and ExecutionConfig.dbt_project_path.If using RenderConfig.dbt_project_path or ExecutionConfig.dbt_project_path, ProjectConfig.dbt_project_path should be None
```

This has been raised by an Astro customer and our field engineer, who
tried to run: https://github.com/astronomer/cosmos-demo
tatiana added a commit that referenced this pull request Jul 29, 2024
If execution_config was reused, Cosmos 1.2.2 would raise:

```
astronomer-cosmos/dags/basic_cosmos_task_group.py
Traceback (most recent call last):
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/venv-38/lib/python3.8/site-packages/airflow/models/dagbag.py", line 343, in parse
    loader.exec_module(new_module)
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/dags/basic_cosmos_task_group.py", line 74, in <module>
    basic_cosmos_task_group()
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/venv-38/lib/python3.8/site-packages/airflow/models/dag.py", line 3817, in factory
    f(**f_kwargs)
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/dags/basic_cosmos_task_group.py", line 54, in basic_cosmos_task_group
    orders = DbtTaskGroup(
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/cosmos/airflow/task_group.py", line 26, in __init__
    DbtToAirflowConverter.__init__(self, *args, **specific_kwargs(**kwargs))
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/cosmos/converter.py", line 113, in __init__
    raise CosmosValueError(
cosmos.exceptions.CosmosValueError: ProjectConfig.dbt_project_path is mutually exclusive with RenderConfig.dbt_project_path and ExecutionConfig.dbt_project_path.If using RenderConfig.dbt_project_path or ExecutionConfig.dbt_project_path, ProjectConfig.dbt_project_path should be None
```

This has been raised by an Astro customer and our field engineer, who
tried to run: https://github.com/astronomer/cosmos-demo
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:config Related to configuration, like YAML files, environment variables, or executer configuration area:parsing Related to parsing DAG/DBT improvement, issues, or fixes priority:high High priority issues are blocking or critical issues without a workaround and large impact

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants