Skip to content

Fix mypy issues related to ObjectStoragePath in main branch#2012

Merged
tatiana merged 12 commits into
mainfrom
fix-mypy
Oct 1, 2025
Merged

Fix mypy issues related to ObjectStoragePath in main branch#2012
tatiana merged 12 commits into
mainfrom
fix-mypy

Conversation

@tatiana
Copy link
Copy Markdown
Collaborator

@tatiana tatiana commented Oct 1, 2025

The solution utilises forward references (string literals) for the ObjectStoragePath type, allowing mypy to understand the types without requiring the actual class to be available at runtime. This is the standard approach for handling optional dependencies in type annotations.

Cosmos' main branch is red due to mypy checks failing, as observed in this sample GitHub Actions job:
https://github.com/astronomer/astronomer-cosmos/actions/runs/18122118077/job/51706664314

mypy-python..............................................................Failed
- hook id: mypy
- exit code: 1

cosmos/cache.py:76: error: Incompatible types in assignment (expression has type "ObjectStoragePath", variable has type "Path | None")  [assignment]
cosmos/cache.py:78: error: Item "None" of "Path | None" has no attribute "exists"  [union-attr]
cosmos/cache.py:78: note: Error code "union-attr" not covered by "type: ignore" comment
cosmos/cache.py:80: error: Item "None" of "Path | None" has no attribute "mkdir"  [union-attr]
cosmos/config.py:233: error: Incompatible types in assignment (expression has type "ObjectStoragePath", variable has type "Path | None")  [assignment]
cosmos/operators/local.py:325: error: Incompatible return value type (got "tuple[ObjectStoragePath, str]", expected "tuple[Path, str] | tuple[None, None]")  [return-value]
Found 5 errors in 3 files (checked 35 source files)

The same job was working without failures two days ago:
https://github.com/astronomer/astronomer-cosmos/actions/runs/18111649257/job/51539065565

Changes introduced

  • Added TYPE_CHECKING imports: Added conditional imports for ObjectStoragePath in all affected files using the TYPE_CHECKING pattern to avoid runtime import errors when the class is not available.
  • Updated type annotations: Used string literals (forward references) for ObjectStoragePath in type annotations to avoid mypy errors:
  Path | "ObjectStoragePath" | None instead of Path | ObjectStoragePath | None

Fixed specific issues:

  • cache.py: Updated _configured_cache_dir variable and _configure_remote_cache_dir() function return type
  • config.py: Updated manifest_path field and mandatory_paths dictionary type annotations
  • operators/local.py: Updated _configure_remote_target_path() return type and _construct_dest_file_path() parameter type
  • dbt/graph.py: Updated _get_dbt_ls_remote_cache() parameter type

Verification

I used the same command as the CI to confirm that the introduced changes fix the original issue:

hatch run tests.py3.10-2.10-1.9:type-check
mypy-python..............................................................Passed

And I also confirmed in the CI:
Screenshot 2025-10-01 at 16 29 55

https://github.com/astronomer/astronomer-cosmos/actions/runs/18167212785/job/51712529044?pr=2012

@netlify
Copy link
Copy Markdown

netlify Bot commented Oct 1, 2025

Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name Link
🔨 Latest commit 6774903
🔍 Latest deploy log https://app.netlify.com/projects/sunny-pastelito-5ecb04/deploys/68dd67cb88cde20008739bcb

@tatiana tatiana marked this pull request as ready for review October 1, 2025 15:27
Copilot AI review requested due to automatic review settings October 1, 2025 15:27
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR fixes mypy type checking issues that were causing CI failures on the main branch. The errors were related to incompatible type assignments involving ObjectStoragePath from Airflow's I/O module.

  • Updated type annotations to use forward references for ObjectStoragePath to handle optional dependencies
  • Added conditional imports using TYPE_CHECKING pattern to avoid runtime import errors
  • Fixed specific type annotation issues in cache, config, and operator modules

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 7 comments.

File Description
cosmos/operators/local.py Added conditional import and updated return/parameter types for remote path functions
cosmos/dbt/graph.py Added conditional import and updated parameter type for remote cache function
cosmos/config.py Added conditional import and updated manifest_path and mandatory_paths type annotations
cosmos/cache.py Added conditional import and updated return/variable types for remote cache directory configuration

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

Comment thread cosmos/operators/local.py
Comment thread cosmos/operators/local.py
Comment thread cosmos/dbt/graph.py
Comment thread cosmos/config.py
Comment thread cosmos/config.py
Comment thread cosmos/cache.py
Comment thread cosmos/cache.py
@tatiana tatiana changed the title Fix mypy issues Fix mypy issues related to ObjectStoragePath in main branch Oct 1, 2025
tatiana and others added 4 commits October 1, 2025 16:32
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
tatiana and others added 4 commits October 1, 2025 16:33
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Copy link
Copy Markdown
Collaborator

@corsettigyg corsettigyg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

apart from the import issue with airflow 3.x, everything looks good to me 💯

a possible (and clean fix) would be

from packaging.version import Version
AIRFLOW_VERSION = Version(airflow.__version__)

if AIRFLOW_VERSION >= Version("3.0"):
    try:
        from airflow.sdk import ObjectStoragePath
    except ImportError:
        pass
else:
    try:
        from airflow.io.path import ObjectStoragePath
    except ImportError:
        pass

Comment thread cosmos/cache.py Outdated
Comment thread cosmos/config.py
@codecov
Copy link
Copy Markdown

codecov Bot commented Oct 1, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.81%. Comparing base (53e1eb9) to head (6774903).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #2012   +/-   ##
=======================================
  Coverage   97.80%   97.81%           
=======================================
  Files          87       87           
  Lines        5525     5526    +1     
=======================================
+ Hits         5404     5405    +1     
  Misses        121      121           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment thread cosmos/cache.py Outdated
@tatiana tatiana merged commit ba9b869 into main Oct 1, 2025
96 checks passed
@tatiana tatiana deleted the fix-mypy branch October 1, 2025 18:17
@tatiana tatiana mentioned this pull request Oct 7, 2025
@tatiana tatiana added this to the Cosmos 1.11.0 milestone Oct 28, 2025
tatiana added a commit that referenced this pull request Oct 29, 2025
**Features**

* Introduce ``ExecutionMode.WATCHER`` to reduce DAG run time by 1/5 in
several PRs. Learn more about it
[here](https://astronomer.github.io/astronomer-cosmos/getting_started/watcher-execution-mode.html#watcher-execution-mode).
This feature was implemented via multiple PRs, including:
* Expose new execution mode by @tatiana @pankajastro @pankajkoti in
#1999
* Add ``DbtProducerWatcherOperator`` for the proposed
``ExecutionMode.WATCHER`` by @pankajkoti in #1982
* Add ``DbtConsumerWatcherSensor`` for the proposed
``ExecutionMode.WATCHER`` by @pankajastro in #1998
* Push producer's task completion status to XCOM by @pankajkoti in #2000
* Add default priority_weight for ``DbtProducerWatcherOperator`` by
@pankajkoti in #1995
* Add sample dbt events for the dbt watcher execution mode by
@pankajkoti in #1952
* Add ``compiled_sql`` as a template fields on
```ExecutionMode.WATCHER``` when using ``run_results.json`` by
@pankajastro in #2070
* Set ``push_run_results_to_xcom`` kwargs correctly for invocation mode
subprocess and Watcher mode by @pankajastro in #2067
* Store compiled SQL as template field for dbt callback events in
``ExecutionMode.WATCHER`` by @pankajkoti in #2068
* Add initial documentation for ``ExecutionMode.WATCHER`` by @tatiana in
#2046
* Support running ``State.UPSTREAM_FAILED`` tasks when WATCHER consumer
upstream tasks fail by @tatiana in #2062
* Fail sensor tasks immediately if the ``ExecutionMode.WATCHER``
producer task fails by @pankajastro in #2040
  * Add ``WATCHER``` to GitHub issue template by @tatiana in #2056
* Add support for ``TestBehavior.AFTER_ALL`` with
``ExecutionMode.WATCHER`` by @pankajastro in #2049
* Add support for ``TestBehavior.NONE`` with ``ExecutionMode.WATCHER``
by @pankajastro in #2047
* Fix ``ExecutionMode.WATCHER`` behaviour with ``DbtTaskGroup`` by
@tatiana in #2044
* Fix Cosmos behaviour when using watcher with
``InvocationMode.DBT_RUNNER`` by @tatiana in #2048

* Add Airflow 3 plugin for dbt docs with multiple dbt projects support
by @pankajkoti in #2009, check the
[documentation](https://astronomer.github.io/astronomer-cosmos/configuration/hosting-docs.html).
* Initial support to ``dbt Fusion`` by @tatiana in #1803. More details
[here](https://astronomer.github.io/astronomer-cosmos/configuration/dbt-fusion).
* Support to prune sources without downstream references in dbt projects
by @corsettigyg in #1988
* Allow to set task display name as a user-defined function by
@corsettigyg in #1761
* Add dbt project's hash to dag docs to support dag versioning in
Airflow 3 by @pankajkoti in #1907
* feat: Add Jinja templating support for ``dbt_cmd_flags`` by
@skillicinski in #1899
* Add Scarf metric to collect the execution mode uses by @pankajastro in
#1981
* Support Airflow 3.1 by @tatiana in #1980
* Add MySQL profile mapping by @Lee2532 in #1977
* Add sqlserver profile mapping by @pankajastro in #1737

**Enhancement**

* Use XCom to store sql when using ``ExecutionMode.AIRFLOW_ASYNC`` by
@pankajastro in #1934
* Refactor ``AIRFLOW_ASYNC`` teardown so it doesn't install the
virtualenv by @pankajastro in #1938
* Reuse the virtual env for ``AIRFLOW_ASYNC`` setup task by @pankajastro
in #1939
* Improve dataset/asset experience in Cosmos by @tatiana in #2030
* Add ``downstreams`` to ``DbtNode`` by @wornjs in #2028

**Bug fixes**

* Fix tags extraction by @ms32035 in #1915
* Fix task flow operator args by @anyapriya in #2024

**Documentation**

* Add documentation for Airflow 3 Plugin supporting dbt docs for
multiple dbt projects by @pankajkoti in #2063
* Add Cosmos Deferrable Operator Guide by @pankajastro in #1922
* Add dbt Fusion documentation by @tatiana in #1824 #1830
* Update dbt-fusion.rst to explicitly highlight it is in alpha by
@tatiana in #1838
* Fix a bunch of docs build errors and warnings by @pankajkoti in
#1886
* Add docs note for param virtualenv_dir for async execution mode by
@pankajastro in #1969
* Use pepy.tech downloads badge in README by @pankajkoti in #1920
* Correct the default value of ``cache_dir`` by @seokyun.ha in #2027

**Others**

* Promote @corsettigyg to committer by @tatiana in #1985
* Add @pankajkoti and @pankajastro to ``contributors.rst`` by @tatiana
in #1983
* Update setup script for airflow3 script by @dwreeves in #2023
* Prevent pytest from trying to test classes that aren't actually tests
by @anyapriya in #2032
* Fix ``dag.test()`` for Airflow 3.1+ by syncing DAG to database bby
@kaxil in #2037
* Disable Scarf in CI by @pankajastro in #2016
* Fix failing dbt Fusion tests when run in parallel in CI by @pankajkoti
in #1896
* Fix MyPy issues related to ``ObjectStoragePath`` in main branch by
@tatiana in #2012
* Cleanup example dbt event JSON dictionaries kept for XCOM referencby
@pankajkoti in #1997
* Bump min hatch version that includes fixes for click>=8.3.0 by
@pankajkoti in #1996
* Use official postgres image from Docker hub for kubernetes setup by
@pankajkoti in #1986
* Use click<8.3.0 for hatch as click 8.3 breaks hatch by @pankajkoti in
#1987
* Pin Airflow version in type check CI job by @pankajastro in #2003
* Improve comments after feedback on #1948 by @tatiana in #1963
* Fix running tests with dbt Fusion 2.0.0 preview versions by @tatiana
in #1948
* Test hardening of dbt node having tags as unset or missing by
@pankajkoti in #1918
* Fix Sphinx issue in the main branch by @tatiana in #2064
* pre-commit autoupdate in #2065, #2043, #2033, #2019, #1990, #2019,
#2008, #1941, #1935, #1924
* GitHub dependabot update in #2051, #2050, #2038, #2022, #1947, #1955,
#1946, #1944, #1945, #1928, #1921, #1917


Co-authored-by: Pankaj Koti <pankaj.koti@astronomer.io>
Co-authored-by: Pankaj Singh <pankaj.singh@astronomer.io>
Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants