Initial commit for Iceberg catalog migrator #1

ajantha-bhat · 2025-03-28T05:59:46Z

Please feel free to checkout this PR and run locally.
Verified gradle clean build and all the tests are passing.

Changes:

Copied the latest main branch code form https://github.com/projectnessie/iceberg-catalog-migrator for donation
Replace headers to use apache headers
rename packages to org.apache.polaris.iceberg.catalog.migrator
update security.md
remove the automated workflow related to projectnessie accounts
clean up dremio keyword
Adopt to polaris tools repo

TODO: (In the follow up PRs)

Once Polaris release available, add tests for polaris
Update readme to use polaris examples
Finalize release version
Add automated release workflow
Double check LICENSE and NOTICE (add a separate one for CLI runtime jar distribution)
Bump versions (like Iceberg, Nessie etc)
Migrate open issues/proposals from original repo
Build enhancements (like dependency between javadoc and jandex tasks)

This PR also contains the contributions from

Co-authored-by: Robert Stupp [email protected]
Co-authored-by: Christopher Lambert [email protected]
Co-authored-by: Harsh Maheshwari [email protected]

ajantha-bhat · 2025-03-28T06:00:29Z

cc: @jbonofre, @snazy, @dimas-b

mansehajsingh · 2025-03-28T17:17:23Z

Should there be a gradle-wrapper.jar in gradle/wrapper/? Trying to run a gradle build fails unlike in the original iceberg-catalog-migrator project, where gradle/wrapper/gradle-wrapper.jar exists.

Failure:

Error: Could not find or load main class org.gradle.wrapper.GradleWrapperMain
Caused by: java.lang.ClassNotFoundException: org.gradle.wrapper.GradleWrapperMain

.gitignore

settings.gradle.kts

iceberg-catalog-migrator/cli/build.gradle.kts

ajantha-bhat · 2025-03-29T01:50:05Z

@mansehajsingh: Thanks for taking a look. I have addressed the comments.

snazy

Overall the PR LGTM.
My request for changes is due to the addition of a .jar file and mention of the donation.

I also think, that the migrator should actually live in its own directory, as this repository is rather generic ("tools"). I propose to either move everything to iceberg-catalog-migrator/ or to have a dedicated Git repo for it.

snazy · 2025-03-31T08:53:04Z

gradle/wrapper/gradle-wrapper.jar

The gradle-wrapper.jar should be removed. Binary files, especially those having code, are not good practice.
There's a mechanism in the main repo to work around that.

snazy · 2025-03-31T08:54:58Z

.github/workflows/main.yml

+        uses: gradle/actions/setup-gradle@v4
+
+      - name: Build & Check
+        run: ./gradlew --rerun-tasks assemble ${{ env.ADDITIONAL_GRADLE_OPTS }} check publishToMavenLocal --scan


Suggested change

run: ./gradlew --rerun-tasks assemble ${{ env.ADDITIONAL_GRADLE_OPTS }} check publishToMavenLocal --scan

run: ./gradlew --rerun-tasks assemble ${{ env.ADDITIONAL_GRADLE_OPTS }} check publishToMavenLocal

No scan yet - see apache/polaris#475

snazy · 2025-03-31T08:55:32Z

.github/workflows/main.yml

+        # since the `nessieQuarkusApp` gradle plugin expects the below variable
+        env:
+          JDK17_HOME: ${{ env.JAVA_HOME_17_X64 }}


This should be superfluous

.gitignore

snazy · 2025-03-31T08:56:31Z

NOTICE

+The Apache Software Foundation (http://www.apache.org/).
+
+The initial code for the Polaris project was donated
+to the ASF by Snowflake Inc. (https://www.snowflake.com/) copyright 2024.


This needs to be adopted

good catch. Updated.

snazy · 2025-03-31T08:58:47Z

gradlew.bat

@@ -0,0 +1,94 @@
+@rem


The gradlew.bat file needs to be removed to work with the non-gradle-wrapper-jar mechanism

Please remove this file

gradlew

ajantha-bhat · 2025-03-31T12:56:23Z

@snazy: thanks for the review. I have updated the gradle stuff to be similar to polaris main repo and verified ./gradlew clean build and handled other comments.

ajantha-bhat · 2025-03-31T13:01:15Z

And regarding this

I also think, that the migrator should actually live in its own directory, as this repository is rather generic ("tools"). I propose to either move everything to iceberg-catalog-migrator/ or to have a dedicated Git repo for it.

The modules related to catalog migrator is under iceberg-catalog-migrator/ already. Do you mean move other folders like bom, buildsrc, codestyle also to iceberg-catalog-migrator/? I think these can be common for other tools that will be hosted in this repository.

jbonofre

First quick pass, overall ok, a couple of nits.

I'm also checking build and generated artifacts (in terms of legal).

CODE_OF_CONDUCT.md

LICENSE

jbonofre · 2025-03-31T13:06:31Z

By the way, I would propose to have it in a dedicated repo (as we will have more tools coming in the repo, e.g. benchmark).

buildSrc/build.gradle.kts

buildSrc/src/main/kotlin/Jandex.kt

gradle/contributors.csv

gradle/libs.versions.toml

jbonofre · 2025-03-31T13:20:04Z

Overall the PR LGTM. My request for changes is due to the addition of a .jar file and mention of the donation.

I also think, that the migrator should actually live in its own directory, as this repository is rather generic ("tools"). I propose to either move everything to iceberg-catalog-migrator/ or to have a dedicated Git repo for it.

We agreed on polaris-tools repo. If we change the plan, it has to be discussed again.
I propose to move forward as the approved plan and move later if needed.

mansehajsingh · 2025-04-01T20:50:43Z

And regarding this

I also think, that the migrator should actually live in its own directory, as this repository is rather generic ("tools"). I propose to either move everything to iceberg-catalog-migrator/ or to have a dedicated Git repo for it.

The modules related to catalog migrator is under iceberg-catalog-migrator/ already. Do you mean move other folders like bom, buildsrc, codestyle also to iceberg-catalog-migrator/? I think these can be common for other tools that will be hosted in this repository.

+1 to this - would be super helpful if other additions to the repo were able to share these and deduplicate the work needed to bootstrap a new tool.

ajantha-bhat · 2025-04-02T03:12:13Z

Hi everyone, Thanks for the reviews.

I have addressed all the comments.
As per TODO listed in PR description, we can have follow up PRs.

I would like to squash the PR once I have approval to add coauthored by to the commit message.
Please check.

ajantha-bhat · 2025-04-02T04:59:51Z

Also merging this can help for #2 to avoid duplicate work of introducing build and infra related stuffs.

mansehajsingh

Accidentally submitted approval on my other account.

ajantha-bhat · 2025-04-03T02:25:44Z

I squashed the commit to add co-author by in the commit message.
( I have a backup. If people want to still review it individually again).

ajantha-bhat · 2025-04-04T14:55:55Z

@snazy: Could you please take another look?

README.md

snazy

Given that the polaris-tools repository is meant to host multiple and independent projects, I think that all those sub-projects should live in their own subfolder and be completely independent.

Therefore, everything, except the .github/ folder and the README.md + SECURITY.md files should go under the iceberg-catalog-migrator/ folder.

snazy · 2025-04-10T08:02:55Z

gradlew.bat

@@ -0,0 +1,94 @@
+@rem


Please remove this file

.github/workflows/main.yml

ide-name.txt

- Copied Main branch of nessie catalog migrator #bca310f - Replace headers - Rename packages - Update security.md - Remove automated release workflow related to projectnessie accounts - Clean up Dremio keyword - Adopt to polaris tools repo This PR also contains the contributions from Co-authored-by: Robert Stupp [email protected] Co-authored-by: Christopher Lambert [email protected] Co-authored-by: Harsh Maheshwari [email protected]

ajantha-bhat · 2025-04-10T17:29:00Z

Given that the polaris-tools repository is meant to host multiple and independent projects, I think that all those sub-projects should live in their own subfolder and be completely independent.
Therefore, everything, except the .github/ folder and the README.md + SECURITY.md files should go under the iceberg-catalog-migrator/ folder.

@snazy: Thanks again for the review. I was under the impression that each tool can share the same gradle and build stuffs. But if we are going with this approach is also fine for me as it can help in having individual release versions. I have updated the PR now. Thanks.

snazy

Hm, the build's failing:

FAILURE: Build failed with an exception.

* Where:
Settings file '/home/snazy/devel/polaris/polaris-tools/iceberg-catalog-migrator/settings.gradle.kts' line: 20

* What went wrong:
/home/snazy/devel/polaris/polaris-tools/iceberg-catalog-migrator/version.txt (No such file or directory)

That and a last name change and we should be good!

Don't forget to open an issue about add support for releases.

snazy · 2025-04-11T10:41:23Z

iceberg-catalog-migrator/settings.gradle.kts

+
+val baseVersion = file("version.txt").readText().trim()
+
+rootProject.name = "polaris-tools"


Suggested change

rootProject.name = "polaris-tools"

rootProject.name = "polaris-iceberg-catalog-migrator"

snazy · 2025-04-11T11:03:27Z

Issue #5 created

ajantha-bhat · 2025-04-12T00:55:58Z

Hm, the build's failing:

I verified locally before pushing that builds are passing. But due to .gitignore file, version.txt was not committed to the repo. Forcefully committed it and verified the build by checking out to a fresh repo.

I think PR is ready.

snazy

LGTM!

ajantha-bhat mentioned this pull request Mar 28, 2025

[WIP] Prepare branch for donation to apache projectnessie/iceberg-catalog-migrator#263

Closed

7 tasks

mansehajsingh suggested changes Mar 28, 2025

View reviewed changes

.gitignore Show resolved Hide resolved

mansehajsingh reviewed Mar 28, 2025

View reviewed changes

settings.gradle.kts Show resolved Hide resolved

mansehajsingh reviewed Mar 28, 2025

View reviewed changes

iceberg-catalog-migrator/cli/build.gradle.kts Outdated Show resolved Hide resolved

ajantha-bhat force-pushed the catalog-migrator branch from e1fdcb1 to 904ea26 Compare March 29, 2025 01:48

snazy requested changes Mar 31, 2025

View reviewed changes

ajantha-bhat force-pushed the catalog-migrator branch from 904ea26 to a149ccf Compare March 31, 2025 12:51

jbonofre reviewed Mar 31, 2025

View reviewed changes

CODE_OF_CONDUCT.md Outdated Show resolved Hide resolved

LICENSE Show resolved Hide resolved

dimas-b reviewed Mar 31, 2025

View reviewed changes

buildSrc/build.gradle.kts Show resolved Hide resolved

buildSrc/src/main/kotlin/Jandex.kt Show resolved Hide resolved

gradle/contributors.csv Outdated Show resolved Hide resolved

gradle/libs.versions.toml Show resolved Hide resolved

ajantha-bhat force-pushed the catalog-migrator branch from a149ccf to 1ef40ee Compare March 31, 2025 13:14

ajantha-bhat force-pushed the catalog-migrator branch from 1ef40ee to 8ec369a Compare April 1, 2025 01:58

ajantha-bhat mentioned this pull request Apr 2, 2025

Add Polaris benchmarks #2

Merged

dimas-b approved these changes Apr 2, 2025

View reviewed changes

sfc-gh-mansingh approved these changes Apr 2, 2025

View reviewed changes

mansehajsingh approved these changes Apr 2, 2025

View reviewed changes

ajantha-bhat force-pushed the catalog-migrator branch from 8ec369a to 535d4b6 Compare April 3, 2025 02:22

jbonofre reviewed Apr 4, 2025

View reviewed changes

README.md Show resolved Hide resolved

ajantha-bhat force-pushed the catalog-migrator branch from 535d4b6 to 4292ef4 Compare April 4, 2025 15:35

This was referenced Apr 4, 2025

Add Polaris Sync tool to polaris-tools #3

Closed

Add Polaris synchronization and migration tool to polaris-tools. ajantha-bhat/polaris-tools#1

Closed

Add Polaris synchronization and migration tool to polaris-tools. #4

Merged

snazy reviewed Apr 10, 2025

View reviewed changes

ajantha-bhat force-pushed the catalog-migrator branch from 920518c to 09e2a1f Compare April 10, 2025 17:26

ajantha-bhat force-pushed the catalog-migrator branch 3 times, most recently from 80ed633 to e7e3123 Compare April 10, 2025 18:25

snazy reviewed Apr 11, 2025

View reviewed changes

Address leftover comments

d8b5101

ajantha-bhat force-pushed the catalog-migrator branch from e7e3123 to d8b5101 Compare April 12, 2025 00:52

snazy approved these changes Apr 12, 2025

View reviewed changes

snazy merged commit ac337ab into apache:main Apr 12, 2025

mansehajsingh mentioned this pull request Apr 23, 2025

Add filter for catalog name to Polaris synchronizer/migrator #10

Merged

adam-christian-software mentioned this pull request Nov 14, 2025

Add MCP server #39

Merged

flyrain added a commit to flyrain/polaris-tools that referenced this pull request Dec 14, 2025

UI 85 (apache#1)

1c9b520

	run: ./gradlew --rerun-tasks assemble ${{ env.ADDITIONAL_GRADLE_OPTS }} check publishToMavenLocal --scan
	run: ./gradlew --rerun-tasks assemble ${{ env.ADDITIONAL_GRADLE_OPTS }} check publishToMavenLocal


		val baseVersion = file("version.txt").readText().trim()

		rootProject.name = "polaris-tools"

	rootProject.name = "polaris-tools"
	rootProject.name = "polaris-iceberg-catalog-migrator"

Initial commit for Iceberg catalog migrator #1

Initial commit for Iceberg catalog migrator #1

Uh oh!

Conversation

ajantha-bhat commented Mar 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ajantha-bhat commented Mar 28, 2025

Uh oh!

mansehajsingh commented Mar 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ajantha-bhat commented Mar 29, 2025

Uh oh!

snazy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ajantha-bhat commented Mar 31, 2025

Uh oh!

ajantha-bhat commented Mar 31, 2025

Uh oh!

jbonofre left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jbonofre commented Mar 31, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jbonofre commented Mar 31, 2025

Uh oh!

mansehajsingh commented Apr 1, 2025

Uh oh!

ajantha-bhat commented Apr 2, 2025

Uh oh!

ajantha-bhat commented Apr 2, 2025

Uh oh!

mansehajsingh left a comment

Choose a reason for hiding this comment

Uh oh!

ajantha-bhat commented Apr 3, 2025

Uh oh!

ajantha-bhat commented Apr 4, 2025

Uh oh!

Uh oh!

snazy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ajantha-bhat commented Apr 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ajantha-bhat commented Mar 28, 2025 •

edited

Loading

mansehajsingh commented Mar 28, 2025 •

edited

Loading

ajantha-bhat commented Apr 10, 2025 •

edited

Loading