Add spark sql integration test for Hudi#3194
Merged
gh-yzou merged 2 commits intoapache:mainfrom Jan 27, 2026
Merged
Conversation
Contributor
Author
Contributor
flyrain
reviewed
Dec 3, 2025
| // TODO: extract a polaris-rest module as a thin layer for | ||
| // client to depends on. | ||
| implementation(project(":polaris-core")) { isTransitive = false } | ||
| testImplementation("org.apache.hudi:hudi-spark3.5-bundle_${scalaVersion}:1.1.0") |
Contributor
There was a problem hiding this comment.
Nit: we put versions in the file pluginlibs.versions.toml, refer it as line 35 does.
github-project-automation
bot
moved this from PRs In Progress
to Ready to merge
in Basic Kanban Board
gh-yzou
reviewed
Dec 4, 2025
| // TODO: extract a polaris-rest module as a thin layer for | ||
| // client to depends on. | ||
| implementation(project(":polaris-core")) { isTransitive = false } | ||
| testImplementation("org.apache.hudi:hudi-spark3.5-bundle_${scalaVersion}:1.1.0") |
Contributor
There was a problem hiding this comment.
for the actual spark project, we don't really intend to introduce any table format specific dependency, even for testing. i didn't see any change in the actual spark project, is there a reason that we need this?
Contributor
Author
There was a problem hiding this comment.
let me try removing and see what occurs
| exclude("org.slf4j", "jul-to-slf4j") | ||
| } | ||
|
|
||
| // Add spark-hive for Hudi integration - provides HiveExternalCatalog that Hudi needs |
Contributor
There was a problem hiding this comment.
@rahil-c could you also update the readme to include the support for hudi?
It would be great if we could also have a notebook in the get-started to help people to onboard for hudi, we could do that in follow up, we should also extend the regress test to include actual end to end test for hudi to avoid any potential break of the feature
Contributor
Author
There was a problem hiding this comment.
Will try to follow up in seperate pr for this if possible?
Contributor
|
This PR is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days. |
github-project-automation
bot
moved this from Ready to merge
to Done
in Basic Kanban Board
github-project-automation
bot
moved this from Done
to PRs In Progress
in Basic Kanban Board
rahil-c
force-pushed
the
rahil/polaris-hudi-it-main
branch
from
1d5e8b2 to
d349939
Compare
gh-yzou
approved these changes
Jan 23, 2026
github-project-automation
bot
moved this from PRs In Progress
to Ready to merge
in Basic Kanban Board
flyrain
approved these changes
Jan 23, 2026
github-project-automation
bot
moved this from Ready to merge
to Done
in Basic Kanban Board
snazy
added a commit
to snazy/polaris
that referenced
this pull request
Feb 11, 2026
* Replace custom token-bucket implementation with Guava's `RateLimiter` (apache#3507) Addresses the issues discussed on the dev mailing-list discussion https://lists.apache.org/thread/gkyw7m4fcbjbzhcrlrp4kcq5lr05r0m4, opting to use Guava as the easiest replacement here. * Move idempotency_records schema to v4 and add H2 support (apache#3386) * Move idempotency_records schema to v4 and add H2 support * address comments and fix test failures * fix format * add comment to resource_id * (nit): Getting started examples with mc/s5cmd to aws cli (apache#3526) * Switch mc/s3cmd to aws cli * Switch mc/s3cmd to aws cli * Add support for no KMS with s3-compatible backend (apache#3501) * chore(deps): update amazon/aws-cli docker tag to v2.33.7 (apache#3558) * Update doc for helm around rateLimiter (apache#3562) * Disable renoavte update for python version (apache#3560) * Fix the Keycloak getting-started example for 26.5+ (apache#3568) The example was failing because Keycloak 26.5 introduced stricter validation rules for session lifespan and timeout. * NoSQL: Add to runtime-service (apache#3396) * NoSQL: Add to runtime-service This change adds the NoSQL persistence to polaris-runtime-service. * chore(deps): update amazon/aws-cli docker tag to v2.33.8 (apache#3575) * Add spark sql integration test for Hudi (apache#3194) * Fix ozone getting started example (apache#3574) * Fix Ozone getting started example * Fix Ozone getting started example * Change AWS CLI image to weekly (apache#3578) * fix(deps): update dependency com.diffplug.spotless:spotless-plugin-gradle to v8.2.1 (apache#3576) * chore(deps): update registry.access.redhat.com/ubi9/openjdk-21-runtime docker tag to v1.24-2.1769108682 (apache#3588) * removed references of BEFORE/AFTER_COMMIT_VIEW (apache#3554) * nits - post-merge fixes * Last merged commit 2b0ca21 --------- Co-authored-by: Huaxin Gao <huaxin.gao11@gmail.com> Co-authored-by: Yong Zheng <yongzheng0809@gmail.com> Co-authored-by: Mend Renovate <bot@renovateapp.com> Co-authored-by: Alexandre Dutra <adutra@apache.org> Co-authored-by: Rahil C <32500120+rahil-c@users.noreply.github.com> Co-authored-by: Innocent Djiofack <djiofack007@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
SparkDeltaITChecklist
CHANGELOG.md(if needed)site/content/in-dev/unreleased(if needed)