Add support for IPv6 and iceberg with spark >= 3.4 #3206

julianStreibel · 2025-03-21T16:46:40Z

Why are the changes needed?

The upgrade to spark >= 3.4 is needed to support IPv6 and iceberg. This is very useful for k8s deployments and is currently breaking our pipelines. We implemented an ugly fix overwriting arguments with ImageSpecs.
Without this we are seeing issues where the ip is not wrapped in [] fixed in
apache/spark#36868

What changes were proposed in this pull request?

Upgrade from spark 3.2.1 to 3.5.5

How was this patch tested?

Ran test of spark plugin successfully

Check all the applicable boxes

I updated the documentation accordingly.
All new and existing tests passed.
All commits are signed-off.

Summary by Bito

This PR upgrades Flytekit Spark integration to support Spark 3.4+ with IPv6 and Iceberg support for Kubernetes deployments. Updates include a new Spark base image, revised hadoop-aws dependencies, and modified installation scripts. The PR also fixes file permissions for spark jars directory, locks pyspark version to prevent compatibility issues, and resolves pipeline issues in Kubernetes deployments.

Unit tests added: False

Estimated effort to review (1-5, lower is better): 1

welcome · 2025-03-21T16:46:43Z

Thank you for opening this pull request! 🙌

These tips will help get your PR across the finish line:

Most of the repos have a PR template; if not, fill it out to the best of your knowledge.
Sign off your commits (Reference: DCO Guide).

flyte-bot · 2025-03-21T16:46:49Z

Code Review Agent Run #3a1fdf

Actionable Suggestions - 1

plugins/flytekit-spark/scripts/flytekit_install_spark3.sh - 1
- Incorrect SHA-512 checksum verification format · Line 26-26

Review Details

Files reviewed - 1 · Commit Range: a0694d5..a0694d5
- plugins/flytekit-spark/scripts/flytekit_install_spark3.sh
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful

Bito Usage Guide

Commands

Type the following command in the pull request comment and save the comment.

/review - Manually triggers a full AI review.

Refer to the documentation for additional commands.

Configuration

This repository uses code_review_bito You can customize the agent settings here or contact your Bito workspace admin at [email protected].

Documentation & Help

AI Code Review powered by

flyte-bot · 2025-03-21T16:49:33Z

Changelist by Bito

This pull request implements the following key changes.

Key Change	Files Impacted
New Feature - Spark & Iceberg Support Enhancements	- `Dockerfile` - Updated base image to apache/spark-py:v3.4.0, revised download commands to fetch hadoop-aws 3.4.0 and added Iceberg jars, and inserted a chown command to adjust permissions for the spark jars directory. - `flytekit_install_spark3.sh` - Replaced the spark distribution URL and checksum to reference spark-3.4.0 and updated download links for hadoop-aws and AWS SDK bundles to the new versions. - `setup.py` - Modified the plugin dependency to require pyspark>=3.4.0, aligning the package with the upgraded Spark version.

plugins/flytekit-spark/scripts/flytekit_install_spark3.sh

Signed-off-by: Julian <[email protected]>

flyte-bot · 2025-03-21T18:22:40Z

Code Review Agent Run #655175

Actionable Suggestions - 0

Review Details

Files reviewed - 1 · Commit Range: 91b3466..f4c0bbe
- plugins/flytekit-spark/scripts/flytekit_install_spark3.sh
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful

Bito Usage Guide

Commands

Type the following command in the pull request comment and save the comment.

/review - Manually triggers a full AI review.

Refer to the documentation for additional commands.

Configuration

This repository uses code_review_bito You can customize the agent settings here or contact your Bito workspace admin at [email protected].

Documentation & Help

AI Code Review powered by

flyte-bot · 2025-03-23T17:24:02Z

Code Review Agent Run #46748b

Actionable Suggestions - 0

Review Details

Files reviewed - 2 · Commit Range: f4c0bbe..d0184c5
- plugins/flytekit-spark/Dockerfile
- plugins/flytekit-spark/scripts/flytekit_install_spark3.sh
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful

Bito Usage Guide

Commands

Type the following command in the pull request comment and save the comment.

/review - Manually triggers a full AI review.

Refer to the documentation for additional commands.

Configuration

This repository uses code_review_bito You can customize the agent settings here or contact your Bito workspace admin at [email protected].

Documentation & Help

AI Code Review powered by

Signed-off-by: Julian <[email protected]>

julianStreibel · 2025-03-24T10:47:22Z

Hi @Future-Outlier, I signed the last commit but one action failed with a timeout on the previous run.

integration (ubuntu-latest, 3.9, integration_test_codecov)
failed 53 minutes ago in 1h 22m 37s

SSH: ssh [email protected]
or: ssh -i <path-to-private-SSH-key> [email protected]
SSH: ssh [email protected]
or: ssh -i <path-to-private-SSH-key> [email protected]
Error: The action 'Setup tmate session' has timed out after 60 minutes.

Future-Outlier

Can you run un a flyte cluster to prove it works?

flyte-bot · 2025-03-24T11:44:17Z

Code Review Agent Run #6b0815

Actionable Suggestions - 0

Review Details

Files reviewed - 2 · Commit Range: 91b3466..5e75d32
- plugins/flytekit-spark/Dockerfile
- plugins/flytekit-spark/scripts/flytekit_install_spark3.sh
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful

Bito Usage Guide

Commands

Type the following command in the pull request comment and save the comment.

/review - Manually triggers a full AI review.

Refer to the documentation for additional commands.

Configuration

This repository uses code_review_bito You can customize the agent settings here or contact your Bito workspace admin at [email protected].

Documentation & Help

AI Code Review powered by

Signed-off-by: Julian <[email protected]>

julianStreibel · 2025-03-25T16:24:11Z

@Future-Outlier, to test this PR I did run spark tasks on k8s submitted with the build docker image from this PR without the ipv6 hack and it worked as expected. I also added jars for iceberg support and gave the spark user access to the jars dir so one can add jars in the spark config to download at runtime. The image is published at https://hub.docker.com/r/juliastreibel/flyte-spark-plugin. The iceberg tasks also run as expected now.

flyte-bot · 2025-03-25T17:14:05Z

Code Review Agent Run #31c59c

Actionable Suggestions - 1

plugins/flytekit-spark/Dockerfile - 1
- Consider matching Hadoop and Spark versions · Line 15-18

Review Details

Files reviewed - 2 · Commit Range: 5e75d32..db4d9e6
- plugins/flytekit-spark/Dockerfile
- plugins/flytekit-spark/setup.py
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful

Bito Usage Guide

Commands

Type the following command in the pull request comment and save the comment.

/review - Manually triggers a full AI review.

Refer to the documentation for additional commands.

Configuration

This repository uses code_review_bito You can customize the agent settings here or contact your Bito workspace admin at [email protected].

Documentation & Help

AI Code Review powered by

plugins/flytekit-spark/Dockerfile

Signed-off-by: Julian <[email protected]>

plugins/flytekit-spark/setup.py

flyte-bot · 2025-03-26T09:44:18Z

Code Review Agent Run #f98abc

Actionable Suggestions - 0

Review Details

Files reviewed - 1 · Commit Range: db4d9e6..f60ee18
- plugins/flytekit-spark/Dockerfile
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful

Bito Usage Guide

Commands

Type the following command in the pull request comment and save the comment.

/review - Manually triggers a full AI review.

Refer to the documentation for additional commands.

Configuration

This repository uses code_review_bito You can customize the agent settings here or contact your Bito workspace admin at [email protected].

Documentation & Help

AI Code Review powered by

Signed-off-by: Julian <[email protected]>

flyte-bot · 2025-03-26T12:34:08Z

Code Review Agent Run #b3690b

Actionable Suggestions - 0

Review Details

Files reviewed - 1 · Commit Range: f60ee18..a2c1565
- plugins/flytekit-spark/setup.py
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful

Bito Usage Guide

Commands

Type the following command in the pull request comment and save the comment.

/review - Manually triggers a full AI review.

Refer to the documentation for additional commands.

Configuration

This repository uses code_review_bito You can customize the agent settings here or contact your Bito workspace admin at [email protected].

Documentation & Help

AI Code Review powered by

julianStreibel · 2025-03-28T07:28:01Z

@Future-Outlier, @pingsutw could you review this? :)

Future-Outlier · 2025-03-28T07:30:19Z

@Future-Outlier, to test this PR I did run spark tasks on k8s submitted with the build docker image from this PR without the ipv6 hack and it worked as expected. I also added jars for iceberg support and gave the spark user access to the jars dir so one can add jars in the spark config to download at runtime. The image is published at https://hub.docker.com/r/juliastreibel/flyte-spark-plugin. The iceberg tasks also run as expected now.

Can you give us a screenshot or a video to prove your code works?

codecov · 2025-03-28T07:32:44Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 46.69%. Comparing base (5503ee5) to head (a2c1565).
Report is 2 commits behind head on master.

❗ There is a different number of reports uploaded between BASE (5503ee5) and HEAD (a2c1565). Click for more details.

HEAD has 49 uploads less than BASE

Flag BASE (5503ee5) HEAD (a2c1565)

51 2

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #3206       +/-   ##
===========================================
- Coverage   81.95%   46.69%   -35.26%     
===========================================
  Files         346      214      -132     
  Lines       27852    22276     -5576     
  Branches     2920     2919        -1     
===========================================
- Hits        22826    10402    -12424     
- Misses       4191    11346     +7155     
+ Partials      835      528      -307

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Signed-off-by: Julian <[email protected]>

julianStreibel · 2025-03-28T09:07:46Z

@Future-Outlier, here is a recording https://drive.google.com/file/d/1IJx-Xk0cdmPDQMBoEbdY3Fp3-jO-tH8z/view?usp=sharing

flyte-bot · 2025-03-28T10:14:16Z

Code Review Agent Run #d6e6be

Actionable Suggestions - 0

Review Details

Files reviewed - 1 · Commit Range: a2c1565..aafe4e5
- plugins/flytekit-spark/Dockerfile
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful

Bito Usage Guide

Commands

Type the following command in the pull request comment and save the comment.

/review - Manually triggers a full AI review.

Refer to the documentation for additional commands.

Configuration

This repository uses code_review_bito You can customize the agent settings here or contact your Bito workspace admin at [email protected].

Documentation & Help

AI Code Review powered by

julianStreibel · 2025-03-31T08:30:43Z

@Future-Outlier, @pingsutw could you review this? :)

julianStreibel · 2025-04-02T10:57:19Z

@Future-Outlier, @pingsutw any updates? :)

welcome · 2025-04-03T02:03:17Z

Congrats on merging your first pull request! 🎉

* Add support for ipv6 with spark >= 3.4 Signed-off-by: Julian <[email protected]> * Integrade flyte-bot suggestion Signed-off-by: Julian <[email protected]> * Change to spark 3.4 for iceberg support and docker image Signed-off-by: Julian <[email protected]> * Add iceberg jars Signed-off-by: Julian <[email protected]> * Upgrade hadoop deps to match spark version Signed-off-by: Julian <[email protected]> * Add pyspark lower bound instead of match in the spark plugin Signed-off-by: Julian <[email protected]> * Trim trailing whitespace Signed-off-by: Julian <[email protected]> --------- Signed-off-by: Julian <[email protected]>

* Add support for ipv6 with spark >= 3.4 Signed-off-by: Julian <[email protected]> * Integrade flyte-bot suggestion Signed-off-by: Julian <[email protected]> * Change to spark 3.4 for iceberg support and docker image Signed-off-by: Julian <[email protected]> * Add iceberg jars Signed-off-by: Julian <[email protected]> * Upgrade hadoop deps to match spark version Signed-off-by: Julian <[email protected]> * Add pyspark lower bound instead of match in the spark plugin Signed-off-by: Julian <[email protected]> * Trim trailing whitespace Signed-off-by: Julian <[email protected]> --------- Signed-off-by: Julian <[email protected]> Signed-off-by: Atharva <[email protected]>

julianStreibel requested review from Future-Outlier, cosmicBboy, eapolinario, kumare3, pingsutw, samhita-alla, thomasjpfan and wild-endeavor as code owners March 21, 2025 16:46

flyte-bot reviewed Mar 21, 2025

View reviewed changes

plugins/flytekit-spark/scripts/flytekit_install_spark3.sh Outdated Show resolved Hide resolved

julianStreibel added 2 commits March 21, 2025 18:28

Add support for ipv6 with spark >= 3.4

91b3466

Signed-off-by: Julian <[email protected]>

Integrade flyte-bot suggestion

f4c0bbe

Signed-off-by: Julian <[email protected]>

julianStreibel force-pushed the master branch from fc162be to f4c0bbe Compare March 21, 2025 17:29

julianStreibel changed the title ~~Add support for ipv6 with spark >= 3.4~~ Add support for IPv6 with spark >= 3.4 Mar 22, 2025

julianStreibel changed the title ~~Add support for IPv6 with spark >= 3.4~~ Add support for IPv6 and iceberg with spark >= 3.4 Mar 23, 2025

Change to spark 3.4 for iceberg support and docker image

5e75d32

Signed-off-by: Julian <[email protected]>

julianStreibel force-pushed the master branch from d0184c5 to 5e75d32 Compare March 24, 2025 10:38

Future-Outlier reviewed Mar 24, 2025

View reviewed changes

Add iceberg jars

db4d9e6

Signed-off-by: Julian <[email protected]>

flyte-bot reviewed Mar 25, 2025

View reviewed changes

plugins/flytekit-spark/Dockerfile Show resolved Hide resolved

Upgrade hadoop deps to match spark version

f60ee18

Signed-off-by: Julian <[email protected]>

julianStreibel requested a review from Future-Outlier March 26, 2025 08:35

pingsutw reviewed Mar 26, 2025

View reviewed changes

plugins/flytekit-spark/setup.py Outdated Show resolved Hide resolved

Add pyspark lower bound instead of match in the spark plugin

a2c1565

Signed-off-by: Julian <[email protected]>

julianStreibel requested a review from pingsutw March 26, 2025 12:19

Trim trailing whitespace

aafe4e5

Signed-off-by: Julian <[email protected]>

julianStreibel force-pushed the master branch from 80d6293 to aafe4e5 Compare March 28, 2025 08:47

EngHabu approved these changes Apr 2, 2025

View reviewed changes

davidmirror-ops enabled auto-merge (squash) April 2, 2025 23:19

Future-Outlier approved these changes Apr 3, 2025

View reviewed changes

davidmirror-ops merged commit 1a8da07 into flyteorg:master Apr 3, 2025
114 checks passed

Add support for IPv6 and iceberg with spark >= 3.4 #3206

Add support for IPv6 and iceberg with spark >= 3.4 #3206

Uh oh!

Conversation

julianStreibel commented Mar 21, 2025 • edited by flyte-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are the changes needed?

What changes were proposed in this pull request?

How was this patch tested?

Check all the applicable boxes

Summary by Bito

Uh oh!

welcome bot commented Mar 21, 2025

Uh oh!

flyte-bot commented Mar 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review Agent Run #3a1fdf

Bito Usage Guide

Uh oh!

flyte-bot commented Mar 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelist by Bito

Uh oh!

Uh oh!

flyte-bot commented Mar 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review Agent Run #655175

Bito Usage Guide

Uh oh!

flyte-bot commented Mar 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review Agent Run #46748b

Bito Usage Guide

Uh oh!

julianStreibel commented Mar 24, 2025

Uh oh!

Future-Outlier left a comment

Choose a reason for hiding this comment

Uh oh!

flyte-bot commented Mar 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review Agent Run #6b0815

Bito Usage Guide

Uh oh!

julianStreibel commented Mar 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

flyte-bot commented Mar 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review Agent Run #31c59c

Bito Usage Guide

Uh oh!

Uh oh!

Uh oh!

flyte-bot commented Mar 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review Agent Run #f98abc

Bito Usage Guide

Uh oh!

flyte-bot commented Mar 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review Agent Run #b3690b

Bito Usage Guide

Uh oh!

julianStreibel commented Mar 28, 2025

Uh oh!

Future-Outlier commented Mar 28, 2025

Uh oh!

codecov bot commented Mar 28, 2025

Codecov Report

Uh oh!

julianStreibel commented Mar 28, 2025

Uh oh!

flyte-bot commented Mar 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review Agent Run #d6e6be

Bito Usage Guide

Uh oh!

julianStreibel commented Mar 31, 2025

Uh oh!

julianStreibel commented Mar 21, 2025 •

edited by flyte-bot

Loading

flyte-bot commented Mar 21, 2025 •

edited

Loading

flyte-bot commented Mar 21, 2025 •

edited

Loading

flyte-bot commented Mar 21, 2025 •

edited

Loading

flyte-bot commented Mar 23, 2025 •

edited

Loading

flyte-bot commented Mar 24, 2025 •

edited

Loading

julianStreibel commented Mar 25, 2025 •

edited

Loading

flyte-bot commented Mar 25, 2025 •

edited

Loading

flyte-bot commented Mar 26, 2025 •

edited

Loading

flyte-bot commented Mar 26, 2025 •

edited

Loading

flyte-bot commented Mar 28, 2025 •

edited

Loading