Skip to content

Disable example_cosmos_dbt_build.py DAG in CI#1567

Merged
tatiana merged 1 commit into
mainfrom
fix_ci
Feb 27, 2025
Merged

Disable example_cosmos_dbt_build.py DAG in CI#1567
tatiana merged 1 commit into
mainfrom
fix_ci

Conversation

@pankajastro
Copy link
Copy Markdown
Contributor

@pankajastro pankajastro commented Feb 27, 2025

closes: #1564
Fix unit test error

tests/operators/_asynchronous/test_bigquery.py:6: in <module>
    from airflow.providers.google.cloud.operators.bigquery import BigQueryInsertJobOperator
../../../.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.12-2.9/lib/python3.12/site-packages/airflow/providers/google/cloud/operators/bigquery.py:32: in <module>
    from airflow.providers.common.sql.operators.sql import (  # type: ignore[attr-defined] # for _parse_boolean
../../../.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.12-2.9/lib/python3.12/site-packages/airflow/providers/common/sql/operators/sql.py:29: in <module>
    from airflow.providers.common.sql.hooks.sql import DbApiHook, fetch_all_handler, return_single_query_results
../../../.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.12-2.9/lib/python3.12/site-packages/airflow/providers/common/sql/hooks/sql.py:37: in <module>
    from methodtools import lru_cache
E   ModuleNotFoundError: No module named 'methodtools'

Disable DAG example_cosmos_dbt_build.py in CI because of error

[2025-02-26 13:13:41,578] {taskinstance.py:1851} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/runner/work/astronomer-cosmos/astronomer-cosmos/cosmos/operators/base.py", line 278, in execute
    self.build_and_run_cmd(context=context, cmd_flags=self.add_cmd_flags())
  File "/home/runner/work/astronomer-cosmos/astronomer-cosmos/cosmos/operators/local.py", line 708, in build_and_run_cmd
    result = self.run_command(
  File "/home/runner/work/astronomer-cosmos/astronomer-cosmos/cosmos/operators/local.py", line 556, in run_command
    self.handle_exception(result)
  File "/home/runner/work/astronomer-cosmos/astronomer-cosmos/cosmos/operators/local.py", line 229, in handle_exception_dbt_runner
    return dbt_runner.handle_exception_if_needed(result)
  File "/home/runner/work/astronomer-cosmos/astronomer-cosmos/cosmos/dbt/runner.py", line 113, in handle_exception_if_needed
    raise CosmosDbtRunError(f"dbt invocation completed with errors: {error_message}")
cosmos.exceptions.CosmosDbtRunError: dbt invocation completed with errors: relationships_orders_customer_id__customer_id__ref_customers_: Database Error in test relationships_orders_customer_id__customer_id__ref_customers_ (models/schema.yml)
  relation "public.orders" does not exist
  LINE 13:     from "***"."public"."orders"
                    ^
  compiled code at target/run/altered_jaffle_shop/models/schema.yml/relationships_orders_customer_id__customer_id__ref_customers_.sql

Created a follow-up issue: #1568 to enable DAG example_cosmos_dbt_build.py

@netlify
Copy link
Copy Markdown

netlify Bot commented Feb 27, 2025

Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name Link
🔨 Latest commit d6613ec
🔍 Latest deploy log https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/67c0041b24ec3a0008c95dd1

@cloudflare-workers-and-pages
Copy link
Copy Markdown

cloudflare-workers-and-pages Bot commented Feb 27, 2025

Deploying astronomer-cosmos with  Cloudflare Pages  Cloudflare Pages

Latest commit: d6613ec
Status: ✅  Deploy successful!
Preview URL: https://f5595502.astronomer-cosmos.pages.dev
Branch Preview URL: https://fix-ci-nyia.astronomer-cosmos.pages.dev

View logs

Pin dbt-core

Disable example_cosmos_dbt_build.py DAG in CI
@pankajastro pankajastro changed the title Fix the CI failure Disable example_cosmos_dbt_build.py DAG in CI Feb 27, 2025
@pankajastro pankajastro changed the title Disable example_cosmos_dbt_build.py DAG in CI Disable example_cosmos_dbt_build.py DAG in CI Feb 27, 2025
@codecov
Copy link
Copy Markdown

codecov Bot commented Feb 27, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 97.30%. Comparing base (4f793f1) to head (d6613ec).
Report is 2 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1567   +/-   ##
=======================================
  Coverage   97.30%   97.30%           
=======================================
  Files          80       80           
  Lines        4902     4902           
=======================================
  Hits         4770     4770           
  Misses        132      132           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@pankajastro pankajastro marked this pull request as ready for review February 27, 2025 06:43
@dosubot dosubot Bot added size:XS This PR changes 0-9 lines, ignoring generated files. area:ci Related to CI, Github Actions, or other continuous integration tools area:testing Related to testing, like unit tests, integration tests, etc labels Feb 27, 2025
@pankajastro pankajastro requested a review from tatiana February 27, 2025 06:43
Comment thread pyproject.toml
Copy link
Copy Markdown
Collaborator

@tatiana tatiana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing the CI, @pankajastro , hopefully we can fix #1568 soon.

I'm approving in advance, but it would be great if you could answer the question.

@dosubot dosubot Bot added the lgtm This PR has been approved by a maintainer label Feb 27, 2025
@tatiana tatiana merged commit 8630cae into main Feb 27, 2025
@tatiana tatiana deleted the fix_ci branch February 27, 2025 14:17
@tatiana tatiana added this to the Cosmos 1.9.1 milestone Feb 28, 2025
pankajkoti pushed a commit that referenced this pull request Mar 13, 2025
closes: #1564
**Fix unit test error**
```
tests/operators/_asynchronous/test_bigquery.py:6: in <module>
    from airflow.providers.google.cloud.operators.bigquery import BigQueryInsertJobOperator
../../../.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.12-2.9/lib/python3.12/site-packages/airflow/providers/google/cloud/operators/bigquery.py:32: in <module>
    from airflow.providers.common.sql.operators.sql import (  # type: ignore[attr-defined] # for _parse_boolean
../../../.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.12-2.9/lib/python3.12/site-packages/airflow/providers/common/sql/operators/sql.py:29: in <module>
    from airflow.providers.common.sql.hooks.sql import DbApiHook, fetch_all_handler, return_single_query_results
../../../.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.12-2.9/lib/python3.12/site-packages/airflow/providers/common/sql/hooks/sql.py:37: in <module>
    from methodtools import lru_cache
E   ModuleNotFoundError: No module named 'methodtools'
```
**Disable DAG example_cosmos_dbt_build.py in CI because of error**
```
[2025-02-26 13:13:41,578] {taskinstance.py:1851} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/runner/work/astronomer-cosmos/astronomer-cosmos/cosmos/operators/base.py", line 278, in execute
    self.build_and_run_cmd(context=context, cmd_flags=self.add_cmd_flags())
  File "/home/runner/work/astronomer-cosmos/astronomer-cosmos/cosmos/operators/local.py", line 708, in build_and_run_cmd
    result = self.run_command(
  File "/home/runner/work/astronomer-cosmos/astronomer-cosmos/cosmos/operators/local.py", line 556, in run_command
    self.handle_exception(result)
  File "/home/runner/work/astronomer-cosmos/astronomer-cosmos/cosmos/operators/local.py", line 229, in handle_exception_dbt_runner
    return dbt_runner.handle_exception_if_needed(result)
  File "/home/runner/work/astronomer-cosmos/astronomer-cosmos/cosmos/dbt/runner.py", line 113, in handle_exception_if_needed
    raise CosmosDbtRunError(f"dbt invocation completed with errors: {error_message}")
cosmos.exceptions.CosmosDbtRunError: dbt invocation completed with errors: relationships_orders_customer_id__customer_id__ref_customers_: Database Error in test relationships_orders_customer_id__customer_id__ref_customers_ (models/schema.yml)
  relation "public.orders" does not exist
  LINE 13:     from "***"."public"."orders"
                    ^
  compiled code at target/run/altered_jaffle_shop/models/schema.yml/relationships_orders_customer_id__customer_id__ref_customers_.sql
```

Created a follow-up issue:
#1568 to enable
DAG example_cosmos_dbt_build.py

(cherry picked from commit 8630cae)
@pankajkoti pankajkoti mentioned this pull request Mar 13, 2025
@tatiana tatiana mentioned this pull request Mar 13, 2025
tatiana added a commit that referenced this pull request Mar 17, 2025
Bug Fixes

* Fix import error in dbt bigquery adapter mock for ``dbt-bigquery<1.8``
for ``ExecutionMode.AIRFLOW_ASYNC`` by @pankajkoti in #1548
* Fix ``operator_args`` override configuration by @ghjklw in #1558
* Fix missing ``install_dbt_deps`` in ``ProjectConfig`` ``__init__``
method by @ghjklw in #1556
* Fix dbt project parsing ``dbt_vars`` behavior passed via
``operator_args`` by @AlexandrKhabarov in #1543
* Avoid reading the connection during DAG parsing of the async BigQuery
operator by @joppevos in #1582
* Fix: Workaround to incorrectly raised ``gcsfs.retry.HttpError``
(Invalid Credentials, 401) by @tatiana in #1598
* Fix the async execution mode read sql files for dbt packages by
@pankajastro in #1588
* Improve BQ async error handling by @tatiana in #1597
* Fix path selector when ``manifest.json`` is created using MS Windows
by @tatiana in #1601
* Fix log that prints 'Total filtered nodes' by @tatiana in #1603
* Fix select behaviour using ``LoadMode.MANIFEST`` and a path with star
by @tatiana in #1602
* Support ``on_warning_callback`` with ``TestBehavior.BUILD`` and
``ExecutionMode.LOCAL`` by @corsettigyg in #1571
* Fix ``DbtRunLocalOperator.partial()`` support by @tatiana @ashb in
#1609
* fix: ``container_name`` is null for ecs integration by @nicor88 in
#1592

Docs

* Improve MWAA getting-started docs by removing unused imports by
@jx2lee in #1562

Others

* Disable ``example_cosmos_dbt_build.py`` DAG in CI by @pankajastro in
#1567
* Upgrade GitHub Actions Ubuntu version by @tatiana in #1561
* Update GitHub bug issue template by @pankajastro in #1586
* Enable DAG ``example_cosmos_dbt_build.py`` in CI by @pankajastro in
#1573
* Run async DAG in DAG without setup/teardown task by @pankajastro in
#1599
* Add test case that fully covers recent select issue by @tatiana in
#1604
* Add CI job to test multiple dbt versions for the async DAG by
@pankajkoti in #1535
* Improve unit tests speed from 89s to 14s by @tatiana in #1600
* Pre-commit updates: #1560, #1583, #1596


Closes: #1550

Mergeable version of
#1607

Co-authored-by: Pankaj Singh
<98807258+pankajastro@users.noreply.github.com>
Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:ci Related to CI, Github Actions, or other continuous integration tools area:testing Related to testing, like unit tests, integration tests, etc lgtm This PR has been approved by a maintainer size:XS This PR changes 0-9 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fix red CI (integration tests are failing in a weird way)

2 participants