Skip to content

Move to uv for package management#419

Merged
jlaneve merged 5 commits into
mainfrom
uv
Jun 11, 2025
Merged

Move to uv for package management#419
jlaneve merged 5 commits into
mainfrom
uv

Conversation

@jlaneve

@jlaneve jlaneve commented Jun 6, 2025

Copy link
Copy Markdown
Contributor

This pull request introduces significant improvements to the CI/CD workflows, development environment setup, and documentation by transitioning to the uv tool for faster and more efficient package management. Key changes include updates to workflows, Makefile, Dockerfile, and scripts, as well as enhancements to the contributing documentation. Below is a summary of the most important changes:

CI/CD Workflow Updates

  • Replaced pip and hatch commands with uv commands for installing dependencies, running tests, and publishing packages in .github/workflows/cicd.yaml. [1] [2] [3]
  • Added caching for ~/.cache/uv to improve workflow efficiency. [1] [2]

Development Environment Enhancements

  • Updated the Makefile to use uv for creating virtual environments, installing dependencies, and building wheels. [1] [2]
  • Modified dev/Dockerfile to install uv and use it for package installation.
  • Updated various scripts to replace pip with uv pip for dependency management. [1] [2] [3] [4]

Documentation Improvements

  • Enhanced the contributing guide (docs/contributing/howto.md) to document the use of uv for faster, reproducible builds and dependency management. Added instructions for uv commands and clarified the setup process. [1] [2] [3]

Configuration Updates

  • Introduced a [tool.uv] section in pyproject.toml to define uv configurations, including dev dependencies, constraint dependencies, and workspace settings.

Miscellaneous Changes

  • Updated license configuration in pyproject.toml to use a simplified format.

@codecov-commenter

codecov-commenter commented Jun 6, 2025

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.00%. Comparing base (6ad01ad) to head (f685049).
⚠️ Report is 126 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #419   +/-   ##
=======================================
  Coverage   95.00%   95.00%           
=======================================
  Files          11       11           
  Lines         921      921           
=======================================
  Hits          875      875           
  Misses         46       46           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@jlaneve jlaneve marked this pull request as ready for review June 6, 2025 16:34
@jlaneve jlaneve requested a review from a team as a code owner June 6, 2025 16:34

@pankajastro pankajastro left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR. I'm Curious if the main goal was improved install speed, or something else. Just want to make sure we understand the trade-offs and align on tooling in the future.

Comment thread scripts/airflow3/setup.sh Outdated
Comment thread .github/workflows/cicd.yaml Outdated
jlaneve and others added 2 commits June 10, 2025 15:40
Co-authored-by: Josh Fell <48934154+josh-fell@users.noreply.github.com>
Co-authored-by: Pankaj Singh <98807258+pankajastro@users.noreply.github.com>
@jlaneve

jlaneve commented Jun 10, 2025

Copy link
Copy Markdown
Contributor Author

Thanks for the PR. I'm Curious if the main goal was improved install speed, or something else. Just want to make sure we understand the trade-offs and align on tooling in the future.

uv is becoming the standard for Python package management, both at Astronomer and the broader Python community. Here are a couple articles that help explain why it's different / better:

@jlaneve jlaneve requested review from Copilot and pankajastro June 10, 2025 19:43

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This pull request migrates the project’s package management from pip/hatch to the uv tool, improving CI/CD workflows, development environment setup, and documentation. Key changes include:

  • Replacing direct pip and hatch commands with uv alternatives across shell scripts, Makefile, Dockerfile, and CI workflows.
  • Adding a new [tool.uv] configuration section to pyproject.toml for improved dependency management.
  • Updating the contributing guide to document the usage of uv.

Reviewed Changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated no comments.

Show a summary per file
File Description
scripts/test/pre-install-airflow.sh Replaced pip with uv pip commands for installing dependencies.
scripts/test/integration.sh Updated dependency installation command from pip to uv pip.
scripts/airflow3/setup.sh Changed virtual environment creation and package installation from pip to uv; updated package extra name.
scripts/airflow3/install_from_main.sh Switched uninstall/install commands from pip to uv pip.
pyproject.toml Added a new [tool.uv] section and adjusted license-files syntax.
docs/contributing/howto.md Updated local development instructions to include uv commands.
dev/Dockerfile Adapted Dockerfile to install uv and use it for installing wheels.
Makefile Updated virtual environment setup and build commands to use uv.
.github/workflows/cicd.yaml Revised CI workflows to replace pip/hatch with uv tool commands and add uv cache directories.
Comments suppressed due to low confidence (1)

scripts/airflow3/setup.sh:11

  • The change from '.[test]' to '.[tests]' may introduce an inconsistency with the package extras defined in the project configuration. Please confirm that the extra name 'tests' is correct and in line with the project setup.
-pip3 install ".[test]"

Comment thread pyproject.toml
@jlaneve jlaneve merged commit e39928f into main Jun 11, 2025
64 checks passed
@jlaneve jlaneve deleted the uv branch June 11, 2025 13:47
@pankajastro pankajastro added this to the DAG Factory 0.23.0 milestone Jun 25, 2025
@pankajastro pankajastro mentioned this pull request Jun 27, 2025
pankajastro added a commit that referenced this pull request Jul 14, 2025
## [0.23.0] - 2025-07-14

### Breaking Change

- Drop Airflow 2.2 Support by @pankajastro in
[#388](#388)
- Drop Python 3.8 support by @pankajastro in
[#435](#435)
- Drop Airflow 2.3 Support by @pankajastro in
[#456](#456)

### Airflow 3 Support

- Add tools for Dag-factory Airflow3 testing by @pankajastro in
[#395](#395)
- Fix schedule args for Airflow 3 by @pankajastro in
[#413](#413)
- Fix import path for BranchPythonOperator, PythonOperator and
PythonSensor by @pankajastro in
[#414](#414)
- Add scheduling docs for Airflow 3 by @pankajastro in
[#424](#424)
- Enable Airflow 3 tests in CI by @pankajastro in
[#436](#436)
- Add env AUTO_CONVERT_TO_AF3 in Dockerfile by @pankajastro in
[#455](#455)
- Validate DAG's on Airflow 3 by @pankajastro in
[#457](#457)
- Refactor schedule to use the Python object @pankajastro in
[#458](#458)
- Fix CI and import issues for Airflow 3 compatibility @pankajastro in
[#463](#463)

### Added

- Add support for defining inlets by @IvanSviridov in
[#380](#380)
- Add HttpOperator JSON serialization support with tests by @a-chumagin
in [#382](#382)
- Add support for custom Python object by @pankajastro in
[#444](#444)
- Support env var in default by @gyli in
[#452](#452)
- Pass default arguments via dictionary in .py file by @jroachgolf84 in
[#465](#465)

### Fixed

- Remediated ``default`` behavior, added documentation by @jroachgolf84
in [#378](#378)
- Upgrade `apache-airflow-providers-cncf-kubernetes` provider by
@pankajastro in
[#407](#407)
- Include error message trace in exception by @pankajastro in
[#408](#408)

### Docs

- Remove Unreleased heading section from the CHANGELOG.md by @pankajkoti
in [#365](#365)
- Add Documentation for Conditional Dataset Scheduling with dag-factory
by @ErickSeo in
[#367](#367)
- Add copy right in docs footer by @pankajastro in
[#371](#371)
- Updating docs for callbacks by @jroachgolf84 in
[#375](#375)
- Add stable/latest version in docs by @pankajastro in
[#391](#391)
- Migrate old content to new documentation structure by @pankajastro in
[#393](#393)
- Update Airflow supported version 2.3+ in docs by @pankajastro in
[#412](#412)
- Doc: Add step to fork repo in contributing guide by @pankajastro in
[#427](#427)
- Add setting CONFIG_ROOT_DIR in the contribution doc by @gyli in
[#432](#432)
- Add DAG example showcasing runtime params usage by @pankajastro in
[#449](#449)
- Add jinja2 template example by @pankajastro in
[#450](#450)

### Others

- Add Scraf Pixels for telemetry by @pankajastro in
[#373](#373)
- feat: bumped http provider versions to 2.0+ by @a-chumagin in
[#389](#389)
- Add --verbosity debug in astro-cli cmd by @pankajastro in
[#390](#390)
- Add missing Python file for dynamic task example by @pankajastro in
[#392](#392)
- Pin apache-airflow-providers-cncf-kubernetes<10.4.2 by @pankajastro in
[#400](#400)
- Add script to check version and tag by @pankajastro in
[#395](#394)
- Assert DagRunState in integration test by @pankajastro in
[#415](#415)
- Move to uv for package management by @jlaneve in
[#419](#419)
- Install uv in CI by @jlaneve in
[#421](#421)
- Bump astral-sh/setup-uv from 5 to 6 by @dependabot in
[#423](#423)
- Add Airflow 2.11 in test matrix by @pankajastro in
[#425](#425)
- fix example_dag_factory.yml typo causing catchup: false to not be
respected by @RNHTTR in
[#431](#431)
- Remove expandvars in utils.get_python_callable by @gyli in
[#440](#440)
- Delete unused img folder by @pankajastro in
[#446](#446)
- Clean print statement by @pankajastro
[#447](#447)
- Add hatch to uv dev dependencies by @gyli in
[#453](#453)
- Add Authorize Job in CI by @pankajastro in
[#460](#460)
- Remove PyPI token for releasing packages by @tatiana in
[#461](#461)
- Change CI on trigger event to pull_request from pull_request_target by
@pankajkoti in
[#464](#464)
- Update cicd.yaml: Use pull_request for authorize as we don't have
pull_request_target event configured by @pankajkoti in
[#466](#466)
- Add environment for pypi publish job by @pankajastro in
[#478](#478)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants