Upgrade `apache-airflow-providers-cncf-kubernetes` provider by pankajastro · Pull Request #407 · astronomer/dag-factory

pankajastro · 2025-05-26T11:25:15Z

Previously, DAG Factory was pinning the dependency "apache-airflow-providers-cncf-kubernetes<10.4.2", which imposed several limitations on the project, including not allowing us to support Airflow 3.

This PR removes this pin and makes DAG Factory compatible with apache-airflow-providers-cncf-kubernetes>=4.x

The PR introduces several updates and compatibility improvements for KPO integration in dag-factory:

Remove outdated imports from airflow.contrib.kubernetes.*
Fix version fetch compatibility for Kubernetes provider
Fix Kubernetes provider version fetching compatibility
Resolve Secret object compatibility issues with newer KPO
Add support for missing parameters in KubernetesPodOperator
Unpin the Kubernetes provider version to allow more flexibility with updates

I have created a pre-release https://pypi.org/project/dag-factory/0.23.0a4/ from this branch and validated with end-users that DAG Factory YAML that on the Kubernetes Operator work as expected, by running the DAG:

dag-factory/tests/fixtures/dag_factory_kubernetes_pod_operator.yml

Line 1 in 346fcf1

default:

closes: #397
closes: #94
closes: #98
closes: #180
closes: #288
closes: #75
closes: #405

codecov-commenter · 2025-05-26T13:23:43Z

Codecov Report

❌ Patch coverage is 98.07692% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 95.82%. Comparing base (09bc1e5) to head (16cda2e).
⚠️ Report is 162 commits behind head on main.

Files with missing lines	Patch %	Lines
dagfactory/dagbuilder.py	98.03%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #407      +/-   ##
==========================================
+ Coverage   94.43%   95.82%   +1.38%     
==========================================
  Files          11       11              
  Lines         917      910       -7     
==========================================
+ Hits          866      872       +6     
+ Misses         51       38      -13

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

jlaneve

I wonder if there's a way we could make the Kubernetes provider an optional dependency - that'd be nice for folks who don't want to use it

pankajastro · 2025-05-27T07:23:23Z

I wonder if there's a way we could make the Kubernetes provider an optional dependency - that'd be nice for folks who don't want to use it

Good point! I believe this should be possible. I've created an issue to track it: #409

Upgrade KPO

tatiana

This looks great, @pankajastro , it's impressive how many tickets we're closing with this change.

With this PR, are there any versions of Airflow and the K8s provider that we stop supporting? It would be great to highlight this in the PR description and release notes.

Rob-murphys · 2025-05-28T06:40:23Z

I am trying this pre-release but now get issues similar to #288 even though I am using apache-airflow-providers-cncf-kubernetes==10.4.1

Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1324, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'airflow.providers.cncf.kubernetes.operators.kubernetes_pod'

pankajastro · 2025-05-28T07:03:34Z

I am trying this pre-release but now get issues similar to #288 even though I am using apache-airflow-providers-cncf-kubernetes==10.4.1
Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1324, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'airflow.providers.cncf.kubernetes.operators.kubernetes_pod'

Could you please check your YAML configuration? I suspect it might contain the path airflow.providers.cncf.kubernetes.operators.kubernetes_pod. If that's the case, could you please try replacing it with airflow.providers.cncf.kubernetes.operators.pod and test again. If the issue persists, feel free to share your YAML config so I can reproduce the problem.

Rob-murphys · 2025-05-28T08:51:10Z

I am trying this pre-release but now get issues similar to #288 even though I am using apache-airflow-providers-cncf-kubernetes==10.4.1
Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1324, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'airflow.providers.cncf.kubernetes.operators.kubernetes_pod'
Could you please check your YAML configuration? I suspect it might contain the path airflow.providers.cncf.kubernetes.operators.kubernetes_pod. If that's the case, could you please try replacing it with airflow.providers.cncf.kubernetes.operators.pod and test again. If the issue persists, feel free to share your YAML config so I can reproduce the problem.

Opps you are 100% correct, my bad!

pankajastro · 2025-05-28T15:12:57Z

With this PR, are there any versions of Airflow and the K8s provider that we stop supporting? It would be great to highlight this in the PR description and release notes.

This should work for version >= 4.x

tatiana

Excellent work, @pankajastro , thank you very much for addressing all the feedback! I'm happy for the further refactor to happen in a follow up PR.

@pankajastro

## [0.23.0] - 2025-07-14 ### Breaking Change - Drop Airflow 2.2 Support by @pankajastro in [#388](#388) - Drop Python 3.8 support by @pankajastro in [#435](#435) - Drop Airflow 2.3 Support by @pankajastro in [#456](#456) ### Airflow 3 Support - Add tools for Dag-factory Airflow3 testing by @pankajastro in [#395](#395) - Fix schedule args for Airflow 3 by @pankajastro in [#413](#413) - Fix import path for BranchPythonOperator, PythonOperator and PythonSensor by @pankajastro in [#414](#414) - Add scheduling docs for Airflow 3 by @pankajastro in [#424](#424) - Enable Airflow 3 tests in CI by @pankajastro in [#436](#436) - Add env AUTO_CONVERT_TO_AF3 in Dockerfile by @pankajastro in [#455](#455) - Validate DAG's on Airflow 3 by @pankajastro in [#457](#457) - Refactor schedule to use the Python object @pankajastro in [#458](#458) - Fix CI and import issues for Airflow 3 compatibility @pankajastro in [#463](#463) ### Added - Add support for defining inlets by @IvanSviridov in [#380](#380) - Add HttpOperator JSON serialization support with tests by @a-chumagin in [#382](#382) - Add support for custom Python object by @pankajastro in [#444](#444) - Support env var in default by @gyli in [#452](#452) - Pass default arguments via dictionary in .py file by @jroachgolf84 in [#465](#465) ### Fixed - Remediated ``default`` behavior, added documentation by @jroachgolf84 in [#378](#378) - Upgrade `apache-airflow-providers-cncf-kubernetes` provider by @pankajastro in [#407](#407) - Include error message trace in exception by @pankajastro in [#408](#408) ### Docs - Remove Unreleased heading section from the CHANGELOG.md by @pankajkoti in [#365](#365) - Add Documentation for Conditional Dataset Scheduling with dag-factory by @ErickSeo in [#367](#367) - Add copy right in docs footer by @pankajastro in [#371](#371) - Updating docs for callbacks by @jroachgolf84 in [#375](#375) - Add stable/latest version in docs by @pankajastro in [#391](#391) - Migrate old content to new documentation structure by @pankajastro in [#393](#393) - Update Airflow supported version 2.3+ in docs by @pankajastro in [#412](#412) - Doc: Add step to fork repo in contributing guide by @pankajastro in [#427](#427) - Add setting CONFIG_ROOT_DIR in the contribution doc by @gyli in [#432](#432) - Add DAG example showcasing runtime params usage by @pankajastro in [#449](#449) - Add jinja2 template example by @pankajastro in [#450](#450) ### Others - Add Scraf Pixels for telemetry by @pankajastro in [#373](#373) - feat: bumped http provider versions to 2.0+ by @a-chumagin in [#389](#389) - Add --verbosity debug in astro-cli cmd by @pankajastro in [#390](#390) - Add missing Python file for dynamic task example by @pankajastro in [#392](#392) - Pin apache-airflow-providers-cncf-kubernetes<10.4.2 by @pankajastro in [#400](#400) - Add script to check version and tag by @pankajastro in [#395](#394) - Assert DagRunState in integration test by @pankajastro in [#415](#415) - Move to uv for package management by @jlaneve in [#419](#419) - Install uv in CI by @jlaneve in [#421](#421) - Bump astral-sh/setup-uv from 5 to 6 by @dependabot in [#423](#423) - Add Airflow 2.11 in test matrix by @pankajastro in [#425](#425) - fix example_dag_factory.yml typo causing catchup: false to not be respected by @RNHTTR in [#431](#431) - Remove expandvars in utils.get_python_callable by @gyli in [#440](#440) - Delete unused img folder by @pankajastro in [#446](#446) - Clean print statement by @pankajastro [#447](#447) - Add hatch to uv dev dependencies by @gyli in [#453](#453) - Add Authorize Job in CI by @pankajastro in [#460](#460) - Remove PyPI token for releasing packages by @tatiana in [#461](#461) - Change CI on trigger event to pull_request from pull_request_target by @pankajkoti in [#464](#464) - Update cicd.yaml: Use pull_request for authorize as we don't have pull_request_target event configured by @pankajkoti in [#466](#466) - Add environment for pypi publish job by @pankajastro in [#478](#478)

pankajastro force-pushed the upgrade_cncf_provider branch from fc6edd3 to ac2c135 Compare May 26, 2025 11:41

jlaneve reviewed May 27, 2025

View reviewed changes

pankajastro mentioned this pull request May 27, 2025

Make apache-airflow-providers-cncf-kubernetes an optional dependency #409

Closed

Upgrade KPO

af4f651

Upgrade KPO

pankajastro force-pushed the upgrade_cncf_provider branch from bd167ad to af4f651 Compare May 27, 2025 08:14

pankajastro marked this pull request as ready for review May 27, 2025 10:42

pankajastro requested a review from a team as a code owner May 27, 2025 10:42

tatiana reviewed May 27, 2025

View reviewed changes

Comment thread dagfactory/dagbuilder.py

tatiana reviewed May 27, 2025

View reviewed changes

Comment thread dagfactory/dagbuilder.py Outdated

tatiana reviewed May 27, 2025

View reviewed changes

Address review feedback

07387ca

pankajastro mentioned this pull request May 28, 2025

Check possibility to capture a more specific exception #410

Closed

Test for multiple version provider

16cda2e

pankajastro requested a review from tatiana May 28, 2025 15:14

pankajastro assigned pankajastro and unassigned pankajastro May 28, 2025

pankajastro mentioned this pull request Jun 3, 2025

[Bug] Import error using latest Airflow Kubernetes provider package #288

Closed

1 task

tatiana changed the title ~~Upgrade KPO~~ Upgrade KubernetesPodOperator Jun 4, 2025

tatiana approved these changes Jun 4, 2025

View reviewed changes

tatiana merged commit 95f8ce4 into main Jun 4, 2025
64 checks passed

tatiana deleted the upgrade_cncf_provider branch June 4, 2025 08:10

pankajastro mentioned this pull request Jun 5, 2025

Release 0.23.0 #386

Merged

pankajastro changed the title ~~Upgrade KubernetesPodOperator~~ Upgrade apache-airflow-providers-cncf-kubernetes provider Jul 8, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade `apache-airflow-providers-cncf-kubernetes` provider#407

Upgrade `apache-airflow-providers-cncf-kubernetes` provider#407
tatiana merged 3 commits into
mainfrom
upgrade_cncf_provider

pankajastro commented May 26, 2025 •

edited by tatiana

Loading

Uh oh!

codecov-commenter commented May 26, 2025 •

edited

Loading

Uh oh!

jlaneve left a comment

Uh oh!

pankajastro commented May 27, 2025

Uh oh!

Uh oh!

Uh oh!

tatiana left a comment

Uh oh!

Rob-murphys commented May 28, 2025

Uh oh!

pankajastro commented May 28, 2025

Uh oh!

Rob-murphys commented May 28, 2025

Uh oh!

pankajastro commented May 28, 2025 •

edited

Loading

Uh oh!

tatiana left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

pankajastro commented May 26, 2025 • edited by tatiana Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented May 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

jlaneve left a comment

Choose a reason for hiding this comment

Uh oh!

pankajastro commented May 27, 2025

Uh oh!

Uh oh!

Uh oh!

tatiana left a comment

Choose a reason for hiding this comment

Uh oh!

Rob-murphys commented May 28, 2025

Uh oh!

pankajastro commented May 28, 2025

Uh oh!

Rob-murphys commented May 28, 2025

Uh oh!

pankajastro commented May 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tatiana left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

pankajastro commented May 26, 2025 •

edited by tatiana

Loading

codecov-commenter commented May 26, 2025 •

edited

Loading

pankajastro commented May 28, 2025 •

edited

Loading