Skip to content

Fix dag randering for taskflow + DbtTaskGroup combo#1360

Merged
pankajastro merged 6 commits into
mainfrom
fix_dag_rendering
Dec 19, 2024
Merged

Fix dag randering for taskflow + DbtTaskGroup combo#1360
pankajastro merged 6 commits into
mainfrom
fix_dag_rendering

Conversation

@pankajastro
Copy link
Copy Markdown
Contributor

@pankajastro pankajastro commented Dec 2, 2024

Description

I believe the deepcopy results in two separate DAG objects—the original object and its deepcopy and the task is referencing to both objects causing failure

airflow.exceptions.AirflowException: Tried to set relationships between tasks in more than one DAG: {<DAG: DAG_NAME>, <DAG: DAG_NAME>}
Screenshot 2024-12-03 at 2 30 23 AM

DAG Code

from datetime import datetime

from airflow.decorators import task, dag
from cosmos import DbtTaskGroup, ProjectConfig
from include.constants import jaffle_shop_path, venv_execution_config, manifest_path
from include.profiles import airflow_db

@task(task_id="build_partial_dbt_env_vars_operator")
def build_partial_dbt_env():
    # some code
    # This return is for demonstration purposes only
    return {
        "ENV_VAR_NAME": "value",
        "ENV_VAR_NAME_2": False
    }


# partial_dbt_env = build_partial_dbt_env()



@dag(
    schedule_interval="@daily",
    start_date=datetime(2023, 1, 1),
    catchup=False,
    tags=["simple"],
)
def simple_task_group1() -> None:
    DbtTaskGroup(
        group_id="transform_task_group",
        project_config=ProjectConfig(
        dbt_project_path=jaffle_shop_path,
        manifest_path=manifest_path,
        env_vars=build_partial_dbt_env()
    ),
        profile_config=airflow_db,
        execution_config=venv_execution_config,
    )

simple_task_group1()
#partial_dbt_env >> transform_task_group

Related Issue(s)

closes: #1218

Breaking Change?

Checklist

  • I have made corresponding changes to the documentation (if required)
  • I have added tests that prove my fix is effective or that my feature works

@cloudflare-workers-and-pages
Copy link
Copy Markdown

cloudflare-workers-and-pages Bot commented Dec 2, 2024

Deploying astronomer-cosmos with  Cloudflare Pages  Cloudflare Pages

Latest commit: 41e892e
Status: ✅  Deploy successful!
Preview URL: https://6d76dd4b.astronomer-cosmos.pages.dev
Branch Preview URL: https://fix-dag-rendering.astronomer-cosmos.pages.dev

View logs

@codecov
Copy link
Copy Markdown

codecov Bot commented Dec 2, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.52%. Comparing base (ea67123) to head (41e892e).
Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1360   +/-   ##
=======================================
  Coverage   96.52%   96.52%           
=======================================
  Files          71       71           
  Lines        4228     4228           
=======================================
  Hits         4081     4081           
  Misses        147      147           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@pankajastro pankajastro changed the title Fix dag randering for config using from upstream task Fix dag randering for taskflow + DbtTaskGroup combo Dec 10, 2024
@pankajastro pankajastro marked this pull request as ready for review December 18, 2024 20:23
@dosubot dosubot Bot added size:XS This PR changes 0-9 lines, ignoring generated files. area:rendering Related to rendering, like Jinja, Airflow tasks, etc labels Dec 18, 2024
@pankajastro
Copy link
Copy Markdown
Contributor Author

Comment thread cosmos/converter.py
@tatiana tatiana added this to the Cosmos 1.8.0 milestone Dec 19, 2024
@dosubot dosubot Bot added size:M This PR changes 30-99 lines, ignoring generated files. and removed size:XS This PR changes 0-9 lines, ignoring generated files. labels Dec 19, 2024
@pankajastro pankajastro merged commit c21f8c9 into main Dec 19, 2024
@pankajastro pankajastro deleted the fix_dag_rendering branch December 19, 2024 17:27
@tatiana tatiana mentioned this pull request Dec 20, 2024
tatiana added a commit that referenced this pull request Dec 20, 2024
**New Features**

* Support customizing Airflow operator arguments per dbt node by @wornjs
in #1339. [More
information](https://astronomer.github.io/astronomer-cosmos/getting_started/custom-airflow-properties.html).
* Support uploading dbt artifacts to remote cloud storages via callback
by @pankajkoti in #1389. [Read
more](https://astronomer.github.io/astronomer-cosmos/configuration/callbacks.html).
* Add support to ``TestBehavior.BUILD`` by @tatiana in #1377.
[Documentation](https://astronomer.github.io/astronomer-cosmos/configuration/testing-behavior.html).
* Add support for the "at" operator when using ``LoadMode.DBT_MANIFEST``
or ``CUSTOM`` by @benjy44 in #1372
* Add dbt clone operator by @pankajastro in #1326, as documented in
[here](https://astronomer.github.io/astronomer-cosmos/getting_started/operators.html).
* Support rendering tasks with non-ASCII characters by @t0momi219 in
#1278 [Read
more](https://astronomer.github.io/astronomer-cosmos/configuration/task-display-name.html)
* Add warning callback on source freshness by @pankajastro in #1400
[Read
more](https://astronomer.github.io/astronomer-cosmos/configuration/source-nodes-rendering.html#on-warning-callback-callback)
* Add Oracle Profile mapping by @slords and @pankajkoti in #1190 and
#1404
* Emit telemetry to Scarf during DAG run by @tatiana in #1397
* Save tasks map as ``DbtToAirflowConverter`` property by
@internetcoffeephone and @hheemskerk in #1362

**Bug Fixes**

* Fix the mock value of port in ``TrinoBaseProfileMapping`` to be an
integer by @dwolfeu #1322
* Fix access to the ``dbt docs`` menu item outside of Astro cloud by
@tatiana in #1312
* Add missing ``DbtSourceGcpCloudRunJobOperator`` in module
``cosmos.operators.gcp_cloud_run_job`` by @anai-s in #1290
* Support building ``DbtDag`` without setting paths in ``ProjectConfig``
by @tatiana in #1307
* Fix parsing dbt ls outputs that contain JSONs that are not dbt nodes
by @tatiana in #1296
* Fix Snowflake Profile mapping when using AWS default region by
@tatiana in #1406
* Fix dag rendering for taskflow + DbtTaskGroup combo by @pankajastro in
#1360

**Enhancements**

* Improve dbt command execution logs to troubleshoot ``None`` values by
@tatiana in #1392
* Add logging of stdout to dbt graph run_command by @KarolGongola in
#1390
* Save tasks map as DbtToAirflowConverter property by
@internetcoffeephone and @hheemskerk in #1362
* Support rendering build operator task-id with non-ASCII characters by
@pankajastro in #1415

**Docs**

* Remove extra ` char from docs by @pankajastro in #1345
* Add limitation about copying target dir files to remote by @pankajkoti
in #1305
* Generalise example from README by @ReadytoRocc in #1311
* Add security policy by @tatiana, @chaosmaw and @lzdanski in # 1385
* Mention in documentation that the callback functionality is supported
in ``ExecutionMode.VIRTUALENV`` by @pankajkoti in #1401

**Others**

* Restore Jaffle Shop so that ``basic_cosmos_dag`` works as documented
by @tatiana in #1374
* Remove Pytest durations from tests scripts by @tatiana in #1383
* Remove typing-extensions as dependency by @pankajastro in #1381
* Pin dbt-databricks version to < 1.9 by @pankajastro in #1376
* Refactor ``dbt-sqlite`` tests to use ``dbt-postgres`` by @pankajastro
in #1366
* Remove 'dbt-core<1.8.9' pin by @tatiana in #1371
* Remove dependency ``eval_type_backport`` by @tatiana in #1370
* Enable kubernetes tests for dbt>=1.8 by @pankajastro #1364
* CI Workaround: Pin dbt-core, Disable SQLite Tests, and Correctly
Ignore Clone Test to Pass CI by @pankajastro in #1337
* Enable Azure task in the remote store manifest example DAG by
@pankajkoti in #1333
* Enable GCP remote manifest task by @pankajastro in #1332
* Add exempt label option in GH action stale job by @pankajastro in
#1328
* Add integration test for source node rendering by @pankajastro in
#1327
* Fix vulnerability issue on docs dependency by @tatiana in #1313
* Add postgres pod status check for k8s tests in CI by @pankajkoti in
#1320
* [CI] Reduce the amount taking to run tests in the CI from 5h to 11min
by @tatiana in #1297
* Enable secret detection precommit check by @pankajastro in #1302
* Fix security vulnerability, by not pinning Airflow 2.10.0 by @tatiana
in #1298
* Fix Netlify build timeouts by @tatiana in #1294
* Add stalebot to label/close stale PRs and issues by @tatiana in #1288
* Unpin dbt-databricks version by @pankajastro in #1409
* Fix source resource type tests by @pankajastro in #1405
* Increase performance tests models by @tatiana in #1403
* Drop running 1000 models in the CI by @pankajkoti in #1411
* Fix releasing package to PyPI by @tatiana in #1396
* Pre-commit hook updates in #1394, #1373, #1358, #1340, #1331, #1314,
#1301

Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>
Co-authored-by: Pankaj Singh <pankaj.singh@astronomer.io>

Closes: #1193

---------

Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>
Co-authored-by: Pankaj Singh <98807258+pankajastro@users.noreply.github.com>
@anyapriya anyapriya mentioned this pull request Oct 9, 2025
2 tasks
tatiana pushed a commit that referenced this pull request Oct 13, 2025
Similar to the issue
[here](#1360). With
task flow for operator_args, copy.deepcopy(operator_args) was raising an
error due to a completely new copy of the taskflow output being passed
in. Using copy.copy(operator_args) instead means we're still referencing
the original taskflow output, but still have a new copy of the
operator_args itself.

Note: I was struggling to get the cosmos tests to run locally so
apologies that I haven't actually run the new test! But I did recreate
the issue in my local airflow environment, install this version of the
package, and confirm that this seems to fix it.

Closes: #1984
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:rendering Related to rendering, like Jinja, Airflow tasks, etc size:M This PR changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Tried to set relationships between tasks in more than one DAG

3 participants