Skip to content

Improve BQ async error handling#1597

Merged
tatiana merged 3 commits into
mainfrom
bq-import
Mar 13, 2025
Merged

Improve BQ async error handling#1597
tatiana merged 3 commits into
mainfrom
bq-import

Conversation

@tatiana
Copy link
Copy Markdown
Collaborator

@tatiana tatiana commented Mar 11, 2025

Currently, if someone attempts to run simple_dag_async without previously installing apache-airflow-providers-google, they will face this very ugly error:

Traceback (most recent call last):
  File "/Users/tati/Code/cosmos-fresh/astronomer-cosmos/venv/lib/python3.9/site-packages/airflow/models/dagbag.py", line 383, in parse
    loader.exec_module(new_module)
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/Users/tati/Code/cosmos-fresh/astronomer-cosmos/dags/simple_dag_async.py", line 21, in <module>
    simple_dag_async = DbtDag(
  File "/Users/tati/Code/cosmos-fresh/astronomer-cosmos/cosmos/airflow/dag.py", line 26, in __init__
    DbtToAirflowConverter.__init__(self, *args, **specific_kwargs(**kwargs))
  File "/Users/tati/Code/cosmos-fresh/astronomer-cosmos/cosmos/converter.py", line 328, in __init__
    self.tasks_map = build_airflow_graph(
  File "/Users/tati/Code/cosmos-fresh/astronomer-cosmos/cosmos/airflow/graph.py", line 591, in build_airflow_graph
    task_or_group = conversion_function(  # type: ignore
  File "/Users/tati/Code/cosmos-fresh/astronomer-cosmos/cosmos/airflow/graph.py", line 379, in generate_task_or_group
    task = create_airflow_task(task_meta, dag, task_group=model_task_group)
  File "/Users/tati/Code/cosmos-fresh/astronomer-cosmos/cosmos/core/airflow.py", line 36, in get_airflow_task
    airflow_task = Operator(
  File "/Users/tati/Code/cosmos-fresh/astronomer-cosmos/venv/lib/python3.9/site-packages/airflow/models/baseoperator.py", line 506, in apply_defaults
    result = func(self, **kwargs, default_args=default_args)
  File "/Users/tati/Code/cosmos-fresh/astronomer-cosmos/cosmos/operators/airflow_async.py", line 75, in __init__
    super().__init__(
  File "/Users/tati/Code/cosmos-fresh/astronomer-cosmos/venv/lib/python3.9/site-packages/airflow/models/baseoperator.py", line 506, in apply_defaults
    result = func(self, **kwargs, default_args=default_args)
  File "/Users/tati/Code/cosmos-fresh/astronomer-cosmos/cosmos/operators/_asynchronous/base.py", line 50, in __init__
    async_operator_class = self.create_async_operator()
  File "/Users/tati/Code/cosmos-fresh/astronomer-cosmos/cosmos/operators/_asynchronous/base.py", line 69, in create_async_operator
    async_class_operator = _create_async_operator_class(profile_type, "DbtRun")
  File "/Users/tati/Code/cosmos-fresh/astronomer-cosmos/cosmos/operators/_asynchronous/base.py", line 34, in _create_async_operator_class
    raise ImportError(f"Error in loading class: {class_path}. Unable to find the specified operator class.") from e
ImportError: Error in loading class: cosmos.operators._asynchronous.bigquery.DbtRunAirflowAsyncBigqueryOperator. Unable to find the specified operator class.

The goal with this ticket is to give the same error handling as other parts of Cosmos by raising a more graceful error message.

@netlify
Copy link
Copy Markdown

netlify Bot commented Mar 11, 2025

Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name Link
🔨 Latest commit 4e3b74f
🔍 Latest deploy log https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/67d26b8e2c062d00082eb322

Comment thread cosmos/operators/_asynchronous/bigquery.py Outdated
Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>
@cloudflare-workers-and-pages
Copy link
Copy Markdown

cloudflare-workers-and-pages Bot commented Mar 13, 2025

Deploying astronomer-cosmos with  Cloudflare Pages  Cloudflare Pages

Latest commit: 4e3b74f
Status: ✅  Deploy successful!
Preview URL: https://7d474549.astronomer-cosmos.pages.dev
Branch Preview URL: https://bq-import.astronomer-cosmos.pages.dev

View logs

@codecov
Copy link
Copy Markdown

codecov Bot commented Mar 13, 2025

Codecov Report

Attention: Patch coverage is 57.14286% with 3 lines in your changes missing coverage. Please review.

Project coverage is 97.40%. Comparing base (b309dac) to head (4e3b74f).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
cosmos/operators/_asynchronous/bigquery.py 57.14% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1597      +/-   ##
==========================================
- Coverage   97.46%   97.40%   -0.06%     
==========================================
  Files          80       80              
  Lines        4937     4942       +5     
==========================================
+ Hits         4812     4814       +2     
- Misses        125      128       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@tatiana tatiana marked this pull request as ready for review March 13, 2025 06:10
@dosubot dosubot Bot added the size:S This PR changes 10-29 lines, ignoring generated files. label Mar 13, 2025
@tatiana tatiana added this to the Cosmos 1.9.1 milestone Mar 13, 2025
@dosubot dosubot Bot added the profile:bigquery Related to BigQuery ProfileConfig label Mar 13, 2025
@tatiana tatiana requested a review from pankajkoti March 13, 2025 06:14
@tatiana tatiana merged commit 09bcb55 into main Mar 13, 2025
@tatiana tatiana deleted the bq-import branch March 13, 2025 07:02
pankajkoti pushed a commit that referenced this pull request Mar 13, 2025
Currently, if someone attempts to run `simple_dag_async` without
previously installing `apache-airflow-providers-google`, they will face
this very ugly error:

```
Traceback (most recent call last):
  File "/Users/tati/Code/cosmos-fresh/astronomer-cosmos/venv/lib/python3.9/site-packages/airflow/models/dagbag.py", line 383, in parse
    loader.exec_module(new_module)
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/Users/tati/Code/cosmos-fresh/astronomer-cosmos/dags/simple_dag_async.py", line 21, in <module>
    simple_dag_async = DbtDag(
  File "/Users/tati/Code/cosmos-fresh/astronomer-cosmos/cosmos/airflow/dag.py", line 26, in __init__
    DbtToAirflowConverter.__init__(self, *args, **specific_kwargs(**kwargs))
  File "/Users/tati/Code/cosmos-fresh/astronomer-cosmos/cosmos/converter.py", line 328, in __init__
    self.tasks_map = build_airflow_graph(
  File "/Users/tati/Code/cosmos-fresh/astronomer-cosmos/cosmos/airflow/graph.py", line 591, in build_airflow_graph
    task_or_group = conversion_function(  # type: ignore
  File "/Users/tati/Code/cosmos-fresh/astronomer-cosmos/cosmos/airflow/graph.py", line 379, in generate_task_or_group
    task = create_airflow_task(task_meta, dag, task_group=model_task_group)
  File "/Users/tati/Code/cosmos-fresh/astronomer-cosmos/cosmos/core/airflow.py", line 36, in get_airflow_task
    airflow_task = Operator(
  File "/Users/tati/Code/cosmos-fresh/astronomer-cosmos/venv/lib/python3.9/site-packages/airflow/models/baseoperator.py", line 506, in apply_defaults
    result = func(self, **kwargs, default_args=default_args)
  File "/Users/tati/Code/cosmos-fresh/astronomer-cosmos/cosmos/operators/airflow_async.py", line 75, in __init__
    super().__init__(
  File "/Users/tati/Code/cosmos-fresh/astronomer-cosmos/venv/lib/python3.9/site-packages/airflow/models/baseoperator.py", line 506, in apply_defaults
    result = func(self, **kwargs, default_args=default_args)
  File "/Users/tati/Code/cosmos-fresh/astronomer-cosmos/cosmos/operators/_asynchronous/base.py", line 50, in __init__
    async_operator_class = self.create_async_operator()
  File "/Users/tati/Code/cosmos-fresh/astronomer-cosmos/cosmos/operators/_asynchronous/base.py", line 69, in create_async_operator
    async_class_operator = _create_async_operator_class(profile_type, "DbtRun")
  File "/Users/tati/Code/cosmos-fresh/astronomer-cosmos/cosmos/operators/_asynchronous/base.py", line 34, in _create_async_operator_class
    raise ImportError(f"Error in loading class: {class_path}. Unable to find the specified operator class.") from e
ImportError: Error in loading class: cosmos.operators._asynchronous.bigquery.DbtRunAirflowAsyncBigqueryOperator. Unable to find the specified operator class.
```

The goal with this ticket is to give the same error handling as other
parts of Cosmos by raising a more graceful error message.

(cherry picked from commit 09bcb55)
@pankajkoti pankajkoti mentioned this pull request Mar 13, 2025
@tatiana tatiana mentioned this pull request Mar 13, 2025
tatiana added a commit that referenced this pull request Mar 17, 2025
Bug Fixes

* Fix import error in dbt bigquery adapter mock for ``dbt-bigquery<1.8``
for ``ExecutionMode.AIRFLOW_ASYNC`` by @pankajkoti in #1548
* Fix ``operator_args`` override configuration by @ghjklw in #1558
* Fix missing ``install_dbt_deps`` in ``ProjectConfig`` ``__init__``
method by @ghjklw in #1556
* Fix dbt project parsing ``dbt_vars`` behavior passed via
``operator_args`` by @AlexandrKhabarov in #1543
* Avoid reading the connection during DAG parsing of the async BigQuery
operator by @joppevos in #1582
* Fix: Workaround to incorrectly raised ``gcsfs.retry.HttpError``
(Invalid Credentials, 401) by @tatiana in #1598
* Fix the async execution mode read sql files for dbt packages by
@pankajastro in #1588
* Improve BQ async error handling by @tatiana in #1597
* Fix path selector when ``manifest.json`` is created using MS Windows
by @tatiana in #1601
* Fix log that prints 'Total filtered nodes' by @tatiana in #1603
* Fix select behaviour using ``LoadMode.MANIFEST`` and a path with star
by @tatiana in #1602
* Support ``on_warning_callback`` with ``TestBehavior.BUILD`` and
``ExecutionMode.LOCAL`` by @corsettigyg in #1571
* Fix ``DbtRunLocalOperator.partial()`` support by @tatiana @ashb in
#1609
* fix: ``container_name`` is null for ecs integration by @nicor88 in
#1592

Docs

* Improve MWAA getting-started docs by removing unused imports by
@jx2lee in #1562

Others

* Disable ``example_cosmos_dbt_build.py`` DAG in CI by @pankajastro in
#1567
* Upgrade GitHub Actions Ubuntu version by @tatiana in #1561
* Update GitHub bug issue template by @pankajastro in #1586
* Enable DAG ``example_cosmos_dbt_build.py`` in CI by @pankajastro in
#1573
* Run async DAG in DAG without setup/teardown task by @pankajastro in
#1599
* Add test case that fully covers recent select issue by @tatiana in
#1604
* Add CI job to test multiple dbt versions for the async DAG by
@pankajkoti in #1535
* Improve unit tests speed from 89s to 14s by @tatiana in #1600
* Pre-commit updates: #1560, #1583, #1596


Closes: #1550

Mergeable version of
#1607

Co-authored-by: Pankaj Singh
<98807258+pankajastro@users.noreply.github.com>
Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

profile:bigquery Related to BigQuery ProfileConfig size:S This PR changes 10-29 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants