Skip to content

Fix DbtRunLocalOperator.partial() support#1609

Merged
pankajkoti merged 5 commits into
mainfrom
issue-1546
Mar 13, 2025
Merged

Fix DbtRunLocalOperator.partial() support#1609
pankajkoti merged 5 commits into
mainfrom
issue-1546

Conversation

@tatiana
Copy link
Copy Markdown
Collaborator

@tatiana tatiana commented Mar 13, 2025

Since Cosmos 1.9.0, users who attempted to use:

DbtRunLocalOperator.partial(task_id="foo", project_dir="foo")

Started facing the issue:

File /usr/local/lib/python3.11/site-packages/airflow/models/baseoperator.py:284, in partial(operator_class, task_id, dag, task_group, start_date, end_date, owner, email, params, resources, trigger_rule, depends_on_past, ignore_first_depends_on_past, wait_for_past_depends_before_skipping, wait_for_downstream, retries, queue, pool, pool_slots, execution_timeout, max_retry_delay, retry_delay, retry_exponential_backoff, priority_weight, weight_rule, sla, map_index_template, max_active_tis_per_dag, max_active_tis_per_dagrun, on_execute_callback, on_failure_callback, on_success_callback, on_retry_callback, on_skipped_callback, run_as_user, executor, executor_config, inlets, outlets, doc, doc_md, doc_json, doc_yaml, doc_rst, task_display_name, logger_name, allow_nested_operators, **kwargs)
281 from airflow.models.dag import DagContext
282 from airflow.utils.task_group import TaskGroupContext
--> 284 validate_mapping_kwargs(operator_class, "partial", kwargs)
286 dag = dag or DagContext.get_current_dag()
287 if dag:

File /usr/local/lib/python3.11/site-packages/airflow/models/mappedoperator.py:123, in validate_mapping_kwargs(op, func, value)
121 names = ", ".join(repr(n) for n in unknown_args)
122 error = f"unexpected keyword arguments {names}"
--> 123 raise TypeError(f"{op.name}.{func}() got {error}")

TypeError: DbtRunLocalOperator.partial() got an unexpected keyword argument 'project_dir'`

This was introduced given the changes in how Cosmos operators subclass, that was introduced to allow to dynamically chose which Airflow operator is run during DAG rendering time.

Closes: #1546

To validate it, we introduced a new small dbt project and an example DAG, and it can be tested by running:

airflow dags test example_task_mapping

Co-authored-by: Ash Berlin-Taylor ash@astronomer.io

@dosubot dosubot Bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Mar 13, 2025
@netlify
Copy link
Copy Markdown

netlify Bot commented Mar 13, 2025

Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name Link
🔨 Latest commit c2e31ad
🔍 Latest deploy log https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/67d304708975ad0008438913

@cloudflare-workers-and-pages
Copy link
Copy Markdown

cloudflare-workers-and-pages Bot commented Mar 13, 2025

Deploying astronomer-cosmos with  Cloudflare Pages  Cloudflare Pages

Latest commit: c2e31ad
Status: ✅  Deploy successful!
Preview URL: https://cb8d04fa.astronomer-cosmos.pages.dev
Branch Preview URL: https://issue-1546.astronomer-cosmos.pages.dev

View logs

@dosubot dosubot Bot added size:M This PR changes 30-99 lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels Mar 13, 2025
@tatiana tatiana added this to the Cosmos 1.9.1 milestone Mar 13, 2025
@dosubot dosubot Bot added dbt:run Primarily related to dbt run command or functionality execution:local Related to Local execution environment labels Mar 13, 2025
tatiana added 5 commits March 13, 2025 16:13
There were a few tests taking a long time to run when using `hatch run tests.py3.9-2.9:test-cov`.
This PR marks them as integration tests, since they try to connect to Object Storages
@tatiana tatiana added the customer request An Astronomer customer made requested this label Mar 13, 2025
@tatiana tatiana mentioned this pull request Mar 13, 2025
@codecov
Copy link
Copy Markdown

codecov Bot commented Mar 13, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 97.43%. Comparing base (d494dcd) to head (c2e31ad).
Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1609   +/-   ##
=======================================
  Coverage   97.43%   97.43%           
=======================================
  Files          80       80           
  Lines        4943     4948    +5     
=======================================
+ Hits         4816     4821    +5     
  Misses        127      127           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@pankajkoti
Copy link
Copy Markdown
Contributor

I am merging the PR so that we can cherry-pick it in #1607

@pankajkoti pankajkoti merged commit c8c148b into main Mar 13, 2025
@pankajkoti pankajkoti deleted the issue-1546 branch March 13, 2025 16:45
pankajkoti pushed a commit that referenced this pull request Mar 13, 2025
Since Cosmos 1.9.0, users who attempted to use:
```
DbtRunLocalOperator.partial(task_id="foo", project_dir="foo")
```

Started facing the issue:
```
File /usr/local/lib/python3.11/site-packages/airflow/models/baseoperator.py:284, in partial(operator_class, task_id, dag, task_group, start_date, end_date, owner, email, params, resources, trigger_rule, depends_on_past, ignore_first_depends_on_past, wait_for_past_depends_before_skipping, wait_for_downstream, retries, queue, pool, pool_slots, execution_timeout, max_retry_delay, retry_delay, retry_exponential_backoff, priority_weight, weight_rule, sla, map_index_template, max_active_tis_per_dag, max_active_tis_per_dagrun, on_execute_callback, on_failure_callback, on_success_callback, on_retry_callback, on_skipped_callback, run_as_user, executor, executor_config, inlets, outlets, doc, doc_md, doc_json, doc_yaml, doc_rst, task_display_name, logger_name, allow_nested_operators, **kwargs)
281 from airflow.models.dag import DagContext
282 from airflow.utils.task_group import TaskGroupContext
--> 284 validate_mapping_kwargs(operator_class, "partial", kwargs)
286 dag = dag or DagContext.get_current_dag()
287 if dag:

File /usr/local/lib/python3.11/site-packages/airflow/models/mappedoperator.py:123, in validate_mapping_kwargs(op, func, value)
121 names = ", ".join(repr(n) for n in unknown_args)
122 error = f"unexpected keyword arguments {names}"
--> 123 raise TypeError(f"{op.name}.{func}() got {error}")

TypeError: DbtRunLocalOperator.partial() got an unexpected keyword argument 'project_dir'`
```

This was introduced given the changes in how Cosmos operators subclass,
that was introduced to allow to dynamically chose which Airflow operator
is run during DAG rendering time.

Closes: #1546

To validate it, we introduced a new small dbt project and an example
DAG, and it can be tested by running:
```
airflow dags test example_task_mapping
```

Co-authored-by: Ash Berlin-Taylor <ash@astronomer.io>
(cherry picked from commit c8c148b)
pankajkoti added a commit that referenced this pull request Mar 13, 2025
@tatiana tatiana mentioned this pull request Mar 13, 2025
tatiana added a commit that referenced this pull request Mar 17, 2025
Bug Fixes

* Fix import error in dbt bigquery adapter mock for ``dbt-bigquery<1.8``
for ``ExecutionMode.AIRFLOW_ASYNC`` by @pankajkoti in #1548
* Fix ``operator_args`` override configuration by @ghjklw in #1558
* Fix missing ``install_dbt_deps`` in ``ProjectConfig`` ``__init__``
method by @ghjklw in #1556
* Fix dbt project parsing ``dbt_vars`` behavior passed via
``operator_args`` by @AlexandrKhabarov in #1543
* Avoid reading the connection during DAG parsing of the async BigQuery
operator by @joppevos in #1582
* Fix: Workaround to incorrectly raised ``gcsfs.retry.HttpError``
(Invalid Credentials, 401) by @tatiana in #1598
* Fix the async execution mode read sql files for dbt packages by
@pankajastro in #1588
* Improve BQ async error handling by @tatiana in #1597
* Fix path selector when ``manifest.json`` is created using MS Windows
by @tatiana in #1601
* Fix log that prints 'Total filtered nodes' by @tatiana in #1603
* Fix select behaviour using ``LoadMode.MANIFEST`` and a path with star
by @tatiana in #1602
* Support ``on_warning_callback`` with ``TestBehavior.BUILD`` and
``ExecutionMode.LOCAL`` by @corsettigyg in #1571
* Fix ``DbtRunLocalOperator.partial()`` support by @tatiana @ashb in
#1609
* fix: ``container_name`` is null for ecs integration by @nicor88 in
#1592

Docs

* Improve MWAA getting-started docs by removing unused imports by
@jx2lee in #1562

Others

* Disable ``example_cosmos_dbt_build.py`` DAG in CI by @pankajastro in
#1567
* Upgrade GitHub Actions Ubuntu version by @tatiana in #1561
* Update GitHub bug issue template by @pankajastro in #1586
* Enable DAG ``example_cosmos_dbt_build.py`` in CI by @pankajastro in
#1573
* Run async DAG in DAG without setup/teardown task by @pankajastro in
#1599
* Add test case that fully covers recent select issue by @tatiana in
#1604
* Add CI job to test multiple dbt versions for the async DAG by
@pankajkoti in #1535
* Improve unit tests speed from 89s to 14s by @tatiana in #1600
* Pre-commit updates: #1560, #1583, #1596


Closes: #1550

Mergeable version of
#1607

Co-authored-by: Pankaj Singh
<98807258+pankajastro@users.noreply.github.com>
Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

customer request An Astronomer customer made requested this dbt:run Primarily related to dbt run command or functionality execution:local Related to Local execution environment size:M This PR changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] DbtRunLocalOperator.partial() got an unexpected keyword argument 'project_dir'

2 participants