Skip to content

Use click<8.3.0 for hatch as click 8.3 breaks hatch#1987

Merged
pankajkoti merged 3 commits into
mainfrom
fix-click-hatch
Sep 19, 2025
Merged

Use click<8.3.0 for hatch as click 8.3 breaks hatch#1987
pankajkoti merged 3 commits into
mainfrom
fix-click-hatch

Conversation

@pankajkoti
Copy link
Copy Markdown
Contributor

@pankajkoti pankajkoti commented Sep 19, 2025

click 8.3.0 release broke hatch run.

To fix this issue, we're pinning click<8.3.0 when installing hatch in the CI pipeline. We will have to wait for hatch to release a fix for the breaking change introduced by click<8.3.0 (Tracking issue: pypa/hatch#2050)

Failure run: https://github.com/astronomer/astronomer-cosmos/actions/runs/17852869985?pr=1987
Success run post restricting click<8.3: https://github.com/astronomer/astronomer-cosmos/actions/runs/17852830814?pr=1987

We're seeing the following error in CI:

 + zstandard==0.25.0
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /opt/hostedtoolcache/Python/3.11.13/x64/lib/python3.11/site-packages/hatch/c │
│ li/__init__.py:221 in main                                                   │
│                                                                              │
│   218                                                                        │
│   219 def main():  # no cov                                                  │
│   220 │   try:                                                               │
│ ❱ 221 │   │   hatch(prog_name='hatch', windows_expand_args=False)            │
│   222 │   except Exception:  # noqa: BLE001                                  │
│   223 │   │   import sys                                                     │
│   224                                                                        │
│                                                                              │
│ /opt/hostedtoolcache/Python/3.11.13/x64/lib/python3.11/site-packages/click/c │
│ ore.py:1462 in __call__                                                      │
│                                                                              │
│ /opt/hostedtoolcache/Python/3.11.13/x64/lib/python3.11/site-packages/click/c │
│ ore.py:1383 in main                                                          │
│                                                                              │
│ /opt/hostedtoolcache/Python/3.11.13/x64/lib/python3.11/site-packages/click/c │
│ ore.py:1850 in invoke                                                        │
│                                                                              │
│ /opt/hostedtoolcache/Python/3.11.13/x64/lib/python3.11/site-packages/click/c │
│ ore.py:1246 in invoke                                                        │
│                                                                              │
│ /opt/hostedtoolcache/Python/3.11.13/x64/lib/python3.11/site-packages/click/c │
│ ore.py:814 in invoke                                                         │
│                                                                              │
│ /opt/hostedtoolcache/Python/3.11.13/x64/lib/python3.11/site-packages/click/d │
│ ecorators.py:34 in new_func                                                  │
│                                                                              │
│ /opt/hostedtoolcache/Python/3.11.13/x64/lib/python3.11/site-packages/hatch/c │
│ li/run/__init__.py:147 in run                                                │
│                                                                              │
│   144 │   elif not env_name:                                                 │
│   145 │   │   env_name = 'system'                                            │
│   146 │                                                                      │
│ ❱ 147 │   ctx.invoke(                                                        │
│   148 │   │   run_command,                                                   │
│   149 │   │   args=[command, *final_args],                                   │
│   150 │   │   env_names=[env_name],                                          │
│                                                                              │
│ /opt/hostedtoolcache/Python/3.11.13/x64/lib/python3.11/site-packages/click/c │
│ ore.py:814 in invoke                                                         │
│                                                                              │
│ /opt/hostedtoolcache/Python/3.11.13/x64/lib/python3.11/site-packages/click/d │
│ ecorators.py:46 in new_func                                                  │
│                                                                              │
│ /opt/hostedtoolcache/Python/3.11.13/x64/lib/python3.11/site-packages/hatch/c │
│ li/env/run.py:123 in run                                                     │
│                                                                              │
│   120 │   if filter_json:                                                    │
│   121 │   │   import json                                                    │
│   122 │   │                                                                  │
│ ❱ 123 │   │   filter_data = json.loads(filter_json)                          │
│   124 │   │   if not isinstance(filter_data, dict):                          │
│   125 │   │   │   app.abort('The --filter/-f option must be a JSON mapping') │
│   126                                                                        │
│                                                                              │
│ /opt/hostedtoolcache/Python/3.11.13/x64/lib/python3.11/json/__init__.py:339  │
│ in loads                                                                     │
│                                                                              │
│   336 │   │   │   │   │   │   │   │     s, 0)                                │
│   337 │   else:                                                              │
│   338 │   │   if not isinstance(s, (bytes, bytearray)):                      │
│ ❱ 339 │   │   │   raise TypeError(f'the JSON object must be str, bytes or by │
│   340 │   │   │   │   │   │   │   f'not {s.__class__.__name__}')             │
│   341 │   │   s = s.decode(detect_encoding(s), 'surrogatepass')              │
│   342                                                                        │
╰──────────────────────────────────────────────────────────────────────────────╯
TypeError: the JSON object must be str, bytes or bytearray, not Sentinel
Error: Process completed with exit code 1.

related: pypa/hatch#2050

@netlify
Copy link
Copy Markdown

netlify Bot commented Sep 19, 2025

Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name Link
🔨 Latest commit e2d6c15
🔍 Latest deploy log https://app.netlify.com/projects/sunny-pastelito-5ecb04/deploys/68cd1f4814012c0008d23268

Comment thread .github/workflows/test.yml Outdated
@codecov
Copy link
Copy Markdown

codecov Bot commented Sep 19, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.67%. Comparing base (b51d57e) to head (eeb7628).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1987      +/-   ##
==========================================
- Coverage   97.99%   97.67%   -0.32%     
==========================================
  Files          87       87              
  Lines        5327     5369      +42     
==========================================
+ Hits         5220     5244      +24     
- Misses        107      125      +18     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR temporarily pins the click package to versions below 8.3.0 to resolve a breaking change that prevents hatch from functioning correctly in CI pipelines.

  • Adds "click<8.3.0" version constraint to all hatch installations in CI workflows
  • Includes explanatory comments referencing the upstream issue for future removal
  • Adds a test branch to the trigger configuration for testing the fix

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

Comment thread .github/workflows/test.yml Outdated
Comment thread .github/workflows/test.yml
Copy link
Copy Markdown
Collaborator

@tatiana tatiana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing this, @pankajkoti !

@pankajkoti pankajkoti merged commit 83b6c21 into main Sep 19, 2025
24 of 91 checks passed
@pankajkoti pankajkoti deleted the fix-click-hatch branch September 19, 2025 09:16
pankajkoti added a commit that referenced this pull request Sep 25, 2025
Hatch released
[v1.14.2](https://github.com/pypa/hatch/releases/tag/hatch-v1.14.2) that
includes fixes for click==8.3.0,
hence bump min hatch version to that in the CI installations.

related: #1987 
related: pypa/hatch#2050
@tatiana tatiana mentioned this pull request Oct 7, 2025
@tatiana tatiana added this to the Cosmos 1.11.0 milestone Oct 28, 2025
tatiana added a commit that referenced this pull request Oct 29, 2025
**Features**

* Introduce ``ExecutionMode.WATCHER`` to reduce DAG run time by 1/5 in
several PRs. Learn more about it
[here](https://astronomer.github.io/astronomer-cosmos/getting_started/watcher-execution-mode.html#watcher-execution-mode).
This feature was implemented via multiple PRs, including:
* Expose new execution mode by @tatiana @pankajastro @pankajkoti in
#1999
* Add ``DbtProducerWatcherOperator`` for the proposed
``ExecutionMode.WATCHER`` by @pankajkoti in #1982
* Add ``DbtConsumerWatcherSensor`` for the proposed
``ExecutionMode.WATCHER`` by @pankajastro in #1998
* Push producer's task completion status to XCOM by @pankajkoti in #2000
* Add default priority_weight for ``DbtProducerWatcherOperator`` by
@pankajkoti in #1995
* Add sample dbt events for the dbt watcher execution mode by
@pankajkoti in #1952
* Add ``compiled_sql`` as a template fields on
```ExecutionMode.WATCHER``` when using ``run_results.json`` by
@pankajastro in #2070
* Set ``push_run_results_to_xcom`` kwargs correctly for invocation mode
subprocess and Watcher mode by @pankajastro in #2067
* Store compiled SQL as template field for dbt callback events in
``ExecutionMode.WATCHER`` by @pankajkoti in #2068
* Add initial documentation for ``ExecutionMode.WATCHER`` by @tatiana in
#2046
* Support running ``State.UPSTREAM_FAILED`` tasks when WATCHER consumer
upstream tasks fail by @tatiana in #2062
* Fail sensor tasks immediately if the ``ExecutionMode.WATCHER``
producer task fails by @pankajastro in #2040
  * Add ``WATCHER``` to GitHub issue template by @tatiana in #2056
* Add support for ``TestBehavior.AFTER_ALL`` with
``ExecutionMode.WATCHER`` by @pankajastro in #2049
* Add support for ``TestBehavior.NONE`` with ``ExecutionMode.WATCHER``
by @pankajastro in #2047
* Fix ``ExecutionMode.WATCHER`` behaviour with ``DbtTaskGroup`` by
@tatiana in #2044
* Fix Cosmos behaviour when using watcher with
``InvocationMode.DBT_RUNNER`` by @tatiana in #2048

* Add Airflow 3 plugin for dbt docs with multiple dbt projects support
by @pankajkoti in #2009, check the
[documentation](https://astronomer.github.io/astronomer-cosmos/configuration/hosting-docs.html).
* Initial support to ``dbt Fusion`` by @tatiana in #1803. More details
[here](https://astronomer.github.io/astronomer-cosmos/configuration/dbt-fusion).
* Support to prune sources without downstream references in dbt projects
by @corsettigyg in #1988
* Allow to set task display name as a user-defined function by
@corsettigyg in #1761
* Add dbt project's hash to dag docs to support dag versioning in
Airflow 3 by @pankajkoti in #1907
* feat: Add Jinja templating support for ``dbt_cmd_flags`` by
@skillicinski in #1899
* Add Scarf metric to collect the execution mode uses by @pankajastro in
#1981
* Support Airflow 3.1 by @tatiana in #1980
* Add MySQL profile mapping by @Lee2532 in #1977
* Add sqlserver profile mapping by @pankajastro in #1737

**Enhancement**

* Use XCom to store sql when using ``ExecutionMode.AIRFLOW_ASYNC`` by
@pankajastro in #1934
* Refactor ``AIRFLOW_ASYNC`` teardown so it doesn't install the
virtualenv by @pankajastro in #1938
* Reuse the virtual env for ``AIRFLOW_ASYNC`` setup task by @pankajastro
in #1939
* Improve dataset/asset experience in Cosmos by @tatiana in #2030
* Add ``downstreams`` to ``DbtNode`` by @wornjs in #2028

**Bug fixes**

* Fix tags extraction by @ms32035 in #1915
* Fix task flow operator args by @anyapriya in #2024

**Documentation**

* Add documentation for Airflow 3 Plugin supporting dbt docs for
multiple dbt projects by @pankajkoti in #2063
* Add Cosmos Deferrable Operator Guide by @pankajastro in #1922
* Add dbt Fusion documentation by @tatiana in #1824 #1830
* Update dbt-fusion.rst to explicitly highlight it is in alpha by
@tatiana in #1838
* Fix a bunch of docs build errors and warnings by @pankajkoti in
#1886
* Add docs note for param virtualenv_dir for async execution mode by
@pankajastro in #1969
* Use pepy.tech downloads badge in README by @pankajkoti in #1920
* Correct the default value of ``cache_dir`` by @seokyun.ha in #2027

**Others**

* Promote @corsettigyg to committer by @tatiana in #1985
* Add @pankajkoti and @pankajastro to ``contributors.rst`` by @tatiana
in #1983
* Update setup script for airflow3 script by @dwreeves in #2023
* Prevent pytest from trying to test classes that aren't actually tests
by @anyapriya in #2032
* Fix ``dag.test()`` for Airflow 3.1+ by syncing DAG to database bby
@kaxil in #2037
* Disable Scarf in CI by @pankajastro in #2016
* Fix failing dbt Fusion tests when run in parallel in CI by @pankajkoti
in #1896
* Fix MyPy issues related to ``ObjectStoragePath`` in main branch by
@tatiana in #2012
* Cleanup example dbt event JSON dictionaries kept for XCOM referencby
@pankajkoti in #1997
* Bump min hatch version that includes fixes for click>=8.3.0 by
@pankajkoti in #1996
* Use official postgres image from Docker hub for kubernetes setup by
@pankajkoti in #1986
* Use click<8.3.0 for hatch as click 8.3 breaks hatch by @pankajkoti in
#1987
* Pin Airflow version in type check CI job by @pankajastro in #2003
* Improve comments after feedback on #1948 by @tatiana in #1963
* Fix running tests with dbt Fusion 2.0.0 preview versions by @tatiana
in #1948
* Test hardening of dbt node having tags as unset or missing by
@pankajkoti in #1918
* Fix Sphinx issue in the main branch by @tatiana in #2064
* pre-commit autoupdate in #2065, #2043, #2033, #2019, #1990, #2019,
#2008, #1941, #1935, #1924
* GitHub dependabot update in #2051, #2050, #2038, #2022, #1947, #1955,
#1946, #1944, #1945, #1928, #1921, #1917


Co-authored-by: Pankaj Koti <pankaj.koti@astronomer.io>
Co-authored-by: Pankaj Singh <pankaj.singh@astronomer.io>
Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>
tatiana added a commit that referenced this pull request Feb 11, 2026
**Context**

Over the past few months, we have experienced recurring CI failures
caused by dependency conflicts. These issues are not triggered by
changes in Cosmos itself, but rather by unpinned and dynamically
resolved dependencies.

These conflicts have been disruptive, time-consuming to debug, and
difficult to reproduce locally.

Some recent examples:

- #2268
- #2172
- #2076
- #1987
- #1909
- #1902

**Current Situation**

We have tried several approaches to mitigate dependency issues:
- Dynamically resolving dependencies while restricting requirements as
little as possible
- Handling Airflow version–specific dependencies individually
- Sometimes relying on Airflow constraints (which do not include
provider versions), sometimes not
- Attempting to isolate dependencies in `pre-install-airflow.sh`

More recently, however, Airflow began being reinstalled via
`integration-setup.sh`, leading to overlapping installations and Python
virtual environments with conflicting packages.

Additionally, if we try to recreate the environment locally using the
exact dependency list printed by the CI at the end of the “Install
packages and dependencies” step, the environment cannot be created due
to dependencies conflicts.

This results in:
- Non-deterministic Python package installation
- CI failures that are difficult (or impossible) to reproduce locally
- Significant frustration and lost engineering time

**Proposed Approach**

This PR proposes pinning dependencies more strictly to make our CI
environments deterministic and reproducible.

I do not believe this is a perfect solution. Some trade-offs include:
- We will no longer automatically test against newly released dependency
versions
- Pinned versions may require manual updates, including potential
security updates

This PR also does not introduce pip constraints files (since we
currently rely on pip install -r). However, it can be viewed as a step
toward the approach discussed in:
#967

I discussed this proposal with @pankajastro and @pankajkoti, and we
agreed it is worth trying. If this strategy does not work well in
practice, we can revert to the current approach.

**Known Side Effects**

The changes in this PR introduce a few limitations:

- Tests no longer run the cross-project example DAG (dbt Loom) using the
LoadMode.DBT_LS approach, as dbt Loom is now installed in a separate
virtual environment due to dependency conflicts.
- The S3 load manifest task is no longer executed in the corresponding
DAG for Airflow 2.9, due to conflicts between aiobotocore and other
dependencies.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants