Unify bootstrap credentials format across docker-compose and Gradle #1905

sclee01 · 2025-06-17T06:52:11Z

This PR addresses an inconsistency in bootstrap credential settings across various parts of the project.
It is originated from the following issue: #1869

Previously:

docker-compose.yml used:

POLARIS_BOOTSTRAP_CREDENTIALS: default-realm,root,s3cr3t

whereas other configurations (e.g., gradle.properties, README examples) used:

-Dpolaris.bootstrap.credentials=POLARIS,root,secret

This inconsistency caused errors when running commands like ./regtests/run_spark_sql.sh.

Summary of Changes

Unified all credential formats to use POLARIS,root,s3cr3t

Modified:

gradle.properties
docker-compose.yml
README.md
run_spark.sql.sh

This change ensures consistent local development setup across the project.

adutra · 2025-06-17T12:53:01Z

plugins/spark/v3.5/regtests/docker-compose.yml

@sclee01 I wonder if POLARIS isn't a better default, because that is the default realm defined in Polaris configuration:

polaris/quarkus/defaults/src/main/resources/application.properties

Line 107 in 23f6480

polaris.realm-context.realms=POLARIS

@adutra Updated to POLARIS for consistency — take a look when you can.

dimas-b

Thanks for your contribution, @sclee01 ! The docker compose changes LGTM 👍

Could you, please, double check whether any related README files need to be adjusted because of the default credentials changes?

dimas-b

What about getting-started/jdbc/docker-compose-bootstrap-db.yml?

sclee01 · 2025-06-17T15:00:20Z

Thanks for your notice. I will check it again.

adutra · 2025-06-17T15:08:22Z

What about getting-started/jdbc/docker-compose-bootstrap-db.yml?

It seems all the Jupyter notebooks need to be changed (search for SparkPolaris.ipynb).

plugins/spark/v3.5/getting-started/notebooks/SparkPolaris.ipynb

client/python/integration_tests/conftest.py

dimas-b · 2025-06-17T19:21:17Z

getting-started/jdbc/README.md

nit: I'm fine with this change, however the old s3cr3t string was quite unique and easy to search for in text ;)

I agreed with @dimas-b for the uniqueness, but the string secret was used in regression tests. To change it to something else will have a bigger impact on downstream projects.

sclee01 · 2025-06-18T00:51:49Z

I’ve rechecked for consistency.

flyrain · 2025-06-18T01:25:11Z

There are strings of s3cr3t in BootstrapCommandTestBase, but I think it's fine. We don't have to unify them.

flyrain

Thanks a lot for doing this, @sclee01 !

snazy

The PR doesn't do what's mentioned in the PR summary:

Unified all credential formats to use default-realm,root,s3cr3t

I'd prefer to stick with default-realm,root,s3cr3t. POLARIS is IMHO a bad name - it's clear that it's Polaris - whereas "realm" is telling.

sclee01 · 2025-06-18T11:32:39Z

Thanks for your comment, @snazy . You're absolutely right that default-realm is more generic and descriptive.

I originally used default-realm,root,s3cr3t, but updated it to POLARIS,root,secret following reviewer feedback — particularly the point that "POLARIS" is set as the default realm in Polaris configuration.

I am okay with keeping it this way if it makes sense, but also open to switching back if there's a strong preference for default-realm. Appreciate your thoughts.

snazy · 2025-06-18T11:42:32Z

Thanks for your comment, @snazy . You're absolutely right that default-realm is more generic and descriptive.

Ah, bummer. Right, it's mentioned in the default configuration that way. Guess we have to live with POLARIS then.

But mind changing secret back to s3cr3t - I suspect that's better to figure out (code search).

sclee01 · 2025-06-18T11:47:12Z

Got it. I'll change secret back to s3cr3t so it's easier to search for. Thanks for pointing that out!.

sclee01 · 2025-06-18T13:03:39Z

All feedback has been applied.

dimas-b

Changes LGTM overall, but please check CI errors 😅

- unified formatting across docker, gradle - reverted secret to s3cr3t - updated docker-compose, README, conftest.py fix: unify bootstrap credentials across docker-compose, Gradle, and docs use POLARIS for consistency across docker, gradle and others. Applied changes to ../docker-compose-bootstrap-db.yml and other locations. Applied change 'default-realm' to 'POLARIS' Applied change 'default-realm' to 'POLARIS' in conftest.py Applied changes to 'polaris_root_credential' Applied changes to README files changing secret back to s3cr3t Fix missing part from previous commit: reverted secret to s3cr3t fixed missing part

sclee01 · 2025-06-19T03:34:26Z

Fixed CI failure caused by credential format mismatch. PTAL

eric-maynard · 2025-06-20T02:56:32Z

I'm having trouble figuring out the state of the PR based on what appear to be two conflicting comments:

Unify bootstrap credentials format across docker-compose and Gradle #1905 (comment)
Unify bootstrap credentials format across docker-compose and Gradle #1905 (comment)

I don't have a preference between the two, but are we aligned on using s3cr3t?

sclee01 · 2025-06-20T04:01:49Z

We’ve aligned on using POLARIS,root,s3cr3t for consistency and searchability. All files now reflect this.

dimas-b · 2025-06-20T14:30:52Z

I don't have a preference between the two, but are we aligned on using s3cr3t?

This PR moves all default / example credentials to s3cr3t as far as I can tell. I'm fine with this approach.

* Cleanup unnecessary files in client/python (apache#1878) Cleanup unnecessary files in `client/python` * Bump version in version.txt With the release/1.0.0 branch being cut, we should bump this to reflect the current state of main * JDBC: Refactor DatabaseOps (apache#1843) * removes the databaseType computation from JDBCMetastoreManagerFactory to DbOperations * wraps the bootstrap in a transaction ! * refactor Production Readiness checks for Postgres * Fix two wrong links in README.md (apache#1879) * Avoid using org.testcontainers.shaded.** (apache#1876) * main: Update dependency io.smallrye.config:smallrye-config-core to v3.13.2 (apache#1888) * main: Update registry.access.redhat.com/ubi9/openjdk-21-runtime Docker tag to v1.22-1.1749462970 (apache#1887) * main: Update dependency boto3 to v1.38.36 (apache#1886) * fix(build): Fix deprecation warnings in PolarisIntegrationTestExtension (apache#1895) * Enable patch version updates for maintained Polaris version (apache#1891) Polaris 1.x will be a supported/maintained release. It is crucial to apply bug and security fixes to such release branches. Therefore, this change enables patch-version updates for Polaris 1.* * Add Polaris community meeting record for 2025-06-12 (apache#1892) * Do not use relative path inside CLI script Issue apache#1868 reported that the Polaris script can fail when it's run from an unexpected path. The recent addition of a reference to `./gradlew` looks incorrect here, and should be changed to use an absolute path. Fixes apache#1868 * feat(build): Add Checkstyle plugin and an IllegalImport rule (apache#1880) * Python CI: pin mypy version to avoid CI failure due to new release (apache#1903) Mypy did a new release 1.16.1 and it cause our CI to fail for about 20 minutes due to missing wheel (upload not completed) ``` | Unable to find installation candidates for mypy (1.16.1) | | This is likely not a Poetry issue. | | - 14 candidate(s) were identified for the package | - 14 wheel(s) were skipped as your project's environment does not support the identified abi tags | | Solutions: | Make sure the lockfile is up-to-date. You can try one of the following; | | 1. Regenerate lockfile: poetry lock --no-cache --regenerate | 2. Update package : poetry update --no-cache mypy | | If neither works, please first check to verify that the mypy has published wheels available from your configured source that are compatible with your environment- ie. operating system, architecture (x86_64, arm64 etc.), python interpreter. | ``` This PR temporarily restrict the mypy version to avoid the similar issue. We may consider bring poetry.lock back to git tracking so we won't automatically update test dependencies all the time * Remove `.github/CODEOWNERS` (apache#1902) As per this [dev-ML discussion](https://lists.apache.org/thread/jjr5w3hslk755yvxy8b3z45c7094cxdn) * Rename quarkus as runtime (apache#1695) * Rename runtime/test-commons to runtime/test-common (for consistency with module name) (apache#1906) * docs: Add `Polaris Evolution` page (apache#1890) --------- Co-authored-by: Eric Maynard <[email protected]> * feat(ci): Split Java Gradle CI in many jobs to reduce execution time (apache#1897) * Add webpage for Generic Table support (apache#1889) * add change * add comment * address feedback * update limitations * update docs * update doc * address feedback * Improve the parsing and validation of UserSecretReferenceUrns (apache#1840) This change addresses all the TODOs found the org.polaris.core.secrets package. Main changes: - Create a helper to parse, validate and build the URN strings. - Use Regex instead of `String.split()`. - Add Precondition checks to ensure that the URN is valid and the UserSecretManager matches the expected type. - Remove the now unused `GLOBAL_INSTANCE` of the UnsafeInMemorySecretsManager. Testing - Existing `UnsafeInMemorySecretsManagerTest` captures most of the functional changes. - Added `UserSecretReferenceUrnHelperTest` to capture the utilities exposed. * Reuse shadowJar for spark client bundle jar maven publish (apache#1857) * fix spark client * fix test failure and address feedback * fix error * update regression test * update classifier name * address comment * add change * update doc * update build and readme * add back jr * udpate dependency * add change * update * update tests * remove merge service file * update readme * update readme * fix(ci): Remove dummy "build" job from Gradle CI (apache#1911) Since apache#1897, the jobs in gradle.yaml changed and the "build" job was split into many smaller jobs. But since it was a required job, it couldn't be removed immediately. * main: Update Quarkus Platform and Group to v3.23.3 (apache#1797) * main: Update Quarkus Platform and Group to v3.23.3 * Adopt polaris-admin test invocation --------- Co-authored-by: Robert Stupp <[email protected]> * Feature: Rollback compaction on conflict (apache#1285) Intention is make the catalog smarter, to revert the compaction commits in case of crunch to let the writers who are actually adding or removing the data to the table succeed. In a sense treating compaction as always a lower priority process. Presently the rest catalog client creates the snapshot and asks the Rest Server to apply the snapshot and gives this in a combination of requirement and update. Polaris could apply some basic inference and generate some updates to metadata given a property is enabled at a table level, by saying that It will revert back the commit which was created by compaction and let the write succeed. I had this PR in OSS, which was essentially doing this at the client end, but we think its best if we do this as server end. to support more such clients. How to use this Enable a catalog level configuration : polaris.config.rollback.compaction.on-conflicts.enabled when this is enabled polaris will apply the intelligence of rollbacking those REPLACE ops snapshot which have the property of polaris.internal.rollback.compaction.on-conflict in their snapshot summary to resolve conflicts at the server end ! a sample use case is there is a deployment of a Polaris where this config is enabled and there is auto compaction (maintenance job) which is updating the table state, it adds the snapshot summary that polaris.internal.rollback.compaction.on-conflict is true now when a backfill process running for 8 hours want to commit but can't because the compaction job committed before so in this case it will reach out to Polaris and Polaris will see if the snapshot of compation aka replace snapshot has this property if yes roll it back and let the writer succeed ! Devlist: https://lists.apache.org/thread/8k8t77dgk1vc124fnb61932bdp9kf1lc * NoSQL: nits * `AutoCloseable` for `PersistenceTestExtension` * checkstyle adoptions * fix: unify bootstrap credentials and standardize POLARIS setup (apache#1905) - unified formatting across docker, gradle - reverted secret to s3cr3t - updated docker-compose, README, conftest.py use POLARIS for consistency across docker, gradle and others. * Add doc for rollback config (apache#1919) * Revert "Reuse shadowJar for spark client bundle jar maven publish (apache#1857)" (apache#1921) …857)" This reverts commit 1f7f127. The shadowJar plugin actually stops publish the original jar, which is not what spark client intend to publish for the --package usage. Revert it for now, will follow up with a better way to reuse the shadow jar plugin, likely with a separate bundle project * fix(build): Gradle caching effectively not working (apache#1922) Using a `custom()` spotless formatter check effectively disables caching, see `com.diffplug.gradle.spotless.FormatExtension#custom(java.lang.String, com.diffplug.spotless.FormatterFunc)` using `globalState`, which is a `NeverUpToDateBetweenRuns`. This change refactors this to be cachable. We also already have a errorprone rule, so we can get rid entirely of the spotless step. * Update spark client to use the shaded iceberg-core in iceberg-spark-runtime to avoid spark compatibilities issue (apache#1908) * add change * add comment * update change * add comment * add change * add tests * add comment * clean up style check * update build * Revert "Reuse shadowJar for spark client bundle jar maven publish (apache#1857)" This reverts commit 1f7f127. * Reuse shadowJar for spark client bundle jar maven publish (apache#1857) * fix spark client * fix test failure and address feedback * fix error * update regression test * update classifier name * address comment * add change * update doc * update build and readme * add back jr * udpate dependency * add change * update * update tests * remove merge service file * update readme * update readme * update checkstyl * rebase with main * Revert "Reuse shadowJar for spark client bundle jar maven publish (apache#1857)" This reverts commit 40f4d36. * update checkstyle * revert change * address comments * trigger tests * Last merged commit 93938fd --------- Co-authored-by: Honah (Jonas) J. <[email protected]> Co-authored-by: Eric Maynard <[email protected]> Co-authored-by: Prashant Singh <[email protected]> Co-authored-by: Yufei Gu <[email protected]> Co-authored-by: Dmitri Bourlatchkov <[email protected]> Co-authored-by: Mend Renovate <[email protected]> Co-authored-by: Alexandre Dutra <[email protected]> Co-authored-by: JB Onofré <[email protected]> Co-authored-by: Eric Maynard <[email protected]> Co-authored-by: Yun Zou <[email protected]> Co-authored-by: Pooja Nilangekar <[email protected]> Co-authored-by: Seungchul Lee <[email protected]>

github-project-automation bot added this to Basic Kanban Board Jun 17, 2025

github-project-automation bot moved this to PRs In Progress in Basic Kanban Board Jun 17, 2025

sclee01 mentioned this pull request Jun 17, 2025

Make the POLARIS_BOOTSTRAP_CREDENTIALS consistent across docker image, docker composer and gradle run #1869

Closed

adutra reviewed Jun 17, 2025

View reviewed changes

dimas-b reviewed Jun 17, 2025

View reviewed changes

adutra reviewed Jun 17, 2025

View reviewed changes

plugins/spark/v3.5/getting-started/notebooks/SparkPolaris.ipynb Outdated Show resolved Hide resolved

sclee01 force-pushed the fix/unify-bootstrap-crendetials branch from 4e8b012 to a127614 Compare June 17, 2025 16:32

HonahX reviewed Jun 17, 2025

View reviewed changes

client/python/integration_tests/conftest.py Outdated Show resolved Hide resolved

dimas-b reviewed Jun 17, 2025

View reviewed changes

flyrain previously approved these changes Jun 18, 2025

View reviewed changes

github-project-automation bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board Jun 18, 2025

snazy reviewed Jun 18, 2025

View reviewed changes

sclee01 dismissed flyrain’s stale review via 89ccfdd June 18, 2025 12:53

sclee01 force-pushed the fix/unify-bootstrap-crendetials branch from 89ccfdd to 7e04813 Compare June 18, 2025 13:01

adutra previously approved these changes Jun 18, 2025

View reviewed changes

sclee01 dismissed adutra’s stale review via fb2ca77 June 18, 2025 13:43

dimas-b reviewed Jun 18, 2025

View reviewed changes

sclee01 force-pushed the fix/unify-bootstrap-crendetials branch from fb2ca77 to 1b0ee0c Compare June 19, 2025 03:30

adutra approved these changes Jun 19, 2025

View reviewed changes

dimas-b approved these changes Jun 19, 2025

View reviewed changes

flyrain approved these changes Jun 20, 2025

View reviewed changes

HonahX approved these changes Jun 20, 2025

View reviewed changes

dimas-b merged commit cd59302 into apache:main Jun 20, 2025
12 checks passed

github-project-automation bot moved this from Ready to merge to Done in Basic Kanban Board Jun 20, 2025

sclee01 deleted the fix/unify-bootstrap-crendetials branch June 22, 2025 23:45

Unify bootstrap credentials format across docker-compose and Gradle #1905

Unify bootstrap credentials format across docker-compose and Gradle #1905

Uh oh!

Conversation

sclee01 commented Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Previously:

docker-compose.yml used:

whereas other configurations (e.g., gradle.properties, README examples) used:

Summary of Changes

Uh oh!

adutra Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

sclee01 Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dimas-b left a comment

Choose a reason for hiding this comment

Uh oh!

dimas-b left a comment

Choose a reason for hiding this comment

Uh oh!

sclee01 commented Jun 17, 2025

Uh oh!

adutra commented Jun 17, 2025

Uh oh!

Uh oh!

Uh oh!

dimas-b Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

flyrain Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

sclee01 commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

flyrain commented Jun 18, 2025

Uh oh!

flyrain left a comment

Choose a reason for hiding this comment

Uh oh!

snazy left a comment

Choose a reason for hiding this comment

Uh oh!

sclee01 commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

snazy commented Jun 18, 2025

Uh oh!

sclee01 commented Jun 18, 2025

Uh oh!

sclee01 commented Jun 18, 2025

Uh oh!

dimas-b left a comment

Choose a reason for hiding this comment

Uh oh!

sclee01 commented Jun 19, 2025

Uh oh!

eric-maynard commented Jun 20, 2025

Uh oh!

sclee01 commented Jun 20, 2025

Uh oh!

dimas-b commented Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

sclee01 commented Jun 17, 2025 •

edited

Loading

sclee01 Jun 17, 2025 •

edited

Loading

dimas-b Jun 17, 2025 •

edited

Loading

sclee01 commented Jun 18, 2025 •

edited

Loading

sclee01 commented Jun 18, 2025 •

edited

Loading

dimas-b commented Jun 20, 2025 •

edited

Loading