Support additional configurations for Nessie catalog by ajantha-bhat · Pull Request #17725 · trinodb/trino

ajantha-bhat · 2023-06-01T12:49:13Z

Description

Nessie server can be deployed with Auth mode like keycloak.
So, need to expose the Nessie client configurations to handle the Auth.
Along with that, some common Nessie server configurations like read-timeout-ms,
connect-timeout-ms, and compression-enabled properties are exposed to have finer
control over the Nessie commits.

Additional context and related issues

https://projectnessie.org/tools/client_config/

Release notes

(x) Release notes are required, with the following suggested text:

# Iceberg
* Add support for bearer authentication in the Nessie catalog. ({issue}`17725`)

docs/src/main/sphinx/connector/iceberg.rst

dimas-b

Nice improvement 👍

...iceberg/src/main/java/io/trino/plugin/iceberg/catalog/nessie/IcebergNessieCatalogConfig.java

...iceberg/src/main/java/io/trino/plugin/iceberg/catalog/nessie/IcebergNessieCatalogModule.java

ebyhr

Please add tests (config test is insufficient especially for auth) and fix the commit title https://github.com/trinodb/trino/blob/master/.github/DEVELOPMENT.md#format-git-commit-messages

...n/trino-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/nessie/AuthenticationType.java

...iceberg/src/main/java/io/trino/plugin/iceberg/catalog/nessie/IcebergNessieCatalogConfig.java

docs/src/main/sphinx/connector/iceberg.rst

ajantha-bhat · 2023-06-02T07:49:24Z

@ebyhr: Thanks for an awesome review.

Please add tests (config test is insufficient especially for auth)

testDefaults checks for the NONE auth and testExplicitPropertyMapping checks for BEARER auth now.
If you meant an end-to-end integration test for auth. I think it is possible with Nessie container and keycloak docker. I am figuring it out.

.../java/io/trino/plugin/iceberg/catalog/nessie/TestIcebergNessieCatalogConnectorSmokeTest.java

ajantha-bhat · 2023-06-08T12:10:03Z

@ebyhr: The PR is ready. Can you please take another look? thanks.

ajantha-bhat · 2023-06-16T06:24:28Z

Gentle reminder @ebyhr, @electrum

EternalDeiwos · 2023-06-21T06:56:41Z

@ajantha-bhat thanks for adding this. BEARER auth is strictly necessary to use the Nessie catalog.

ajantha-bhat · 2023-06-21T07:13:51Z

@ajantha-bhat thanks for adding this. BEARER auth is strictly necessary to use the Nessie catalog.

True. Nessie is usually deployed with Auth in production and this PR is definitely a high priority for Using Nessie + Trino in production environment.

ajantha-bhat · 2023-06-27T06:37:39Z

@findinpath: Thanks for self requesting the review.
Looking forward to the review and help in merging this.

ajantha-bhat · 2023-07-05T11:10:57Z

I just rebased the PR since there was a conflict in pom file.

docs/src/main/sphinx/connector/iceberg.rst

...iceberg/src/main/java/io/trino/plugin/iceberg/catalog/nessie/IcebergNessieCatalogConfig.java

plugin/trino-iceberg/pom.xml

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergPlugin.java

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/containers/NessieContainer.java

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/containers/KeycloakContainer.java

...g/src/test/java/io/trino/plugin/iceberg/catalog/nessie/TestIcebergNessieCatalogWithAuth.java

...iceberg/src/main/java/io/trino/plugin/iceberg/catalog/nessie/IcebergNessieCatalogConfig.java

ajantha-bhat · 2023-10-23T05:47:18Z

Just pushed the rebase (no conflicts). I will work on comments on the latest code.

ajantha-bhat · 2023-10-23T11:13:01Z

@ebyhr : Addressed all the comments except #17725 (comment) and #17725 (comment). (Both related to test refactoring)

Let me know how to proceed further.

ajantha-bhat · 2023-10-24T17:21:04Z

I just did a rebase since there was conflict in pom.xml

Nessie server configurations like read-timeout-ms, connect-timeout-ms and enable-compression properties are exposed to have finer control over the Nessie commits.

ajantha-bhat · 2023-10-25T05:20:23Z

Rebased agian to resolve conflict from Iceberg 1.4.1 version bump.

ajantha-bhat · 2023-11-01T05:17:39Z

ebyhr force-pushed the config branch from 688d637 to 602e03b
19 hours ago

LGTM.

ajantha-bhat · 2023-11-01T11:38:33Z

Added an empty commit to retrigger the build as the testcase failure seems to be flaky (unrelated to Nessie)

ebyhr · 2023-11-02T04:34:39Z

CI hit #18134

Nessie server can be deployed with Authentication mode with keycloak. So, need to expose the Nessie client configurations to handle the Auth.

ajantha-bhat · 2023-11-02T09:00:12Z

Thanks @ebyhr for the thorough review and the guidance.

It generates unnecessary clutter when running tests. Follow up to trinodb#17725

It generates unnecessary clutter when running tests. Follow up to #17725

mosabua · 2023-11-02T22:59:41Z

Congratulations @ajantha-bhat .. great job. Thank you so much to @ebyhr for all the help and guidance.

cla-bot bot added the cla-signed label Jun 1, 2023

github-actions bot added docs iceberg Iceberg connector labels Jun 1, 2023

ajantha-bhat commented Jun 1, 2023

View reviewed changes

docs/src/main/sphinx/connector/iceberg.rst Outdated Show resolved Hide resolved

dimas-b reviewed Jun 1, 2023

View reviewed changes

...iceberg/src/main/java/io/trino/plugin/iceberg/catalog/nessie/IcebergNessieCatalogConfig.java Outdated Show resolved Hide resolved

...iceberg/src/main/java/io/trino/plugin/iceberg/catalog/nessie/IcebergNessieCatalogModule.java Outdated Show resolved Hide resolved

ebyhr reviewed Jun 1, 2023

View reviewed changes

This comment was marked as duplicate.

Sign in to view

ebyhr reviewed Jun 1, 2023

View reviewed changes

ajantha-bhat changed the title ~~Iceberg: Support additional configurations for Nessie catalog~~ Support additional configurations for Nessie catalog for Iceberg connector Jun 1, 2023

ajantha-bhat changed the title ~~Support additional configurations for Nessie catalog for Iceberg connector~~ Support additional configurations for Nessie catalog in Iceberg connector Jun 1, 2023

ajantha-bhat force-pushed the config branch 2 times, most recently from 824df79 to 24b4643 Compare June 2, 2023 07:44

ajantha-bhat force-pushed the config branch 2 times, most recently from ac7f884 to 0f9f61e Compare June 5, 2023 02:04

ajantha-bhat mentioned this pull request Jun 6, 2023

Enhance Trino-Nessie integration projectnessie/nessie#6996

Open

4 tasks

dimas-b reviewed Jun 7, 2023

View reviewed changes

.../java/io/trino/plugin/iceberg/catalog/nessie/TestIcebergNessieCatalogConnectorSmokeTest.java Outdated Show resolved Hide resolved

ajantha-bhat force-pushed the config branch 2 times, most recently from 1a1bfec to a23af13 Compare June 8, 2023 10:41

dimas-b approved these changes Jun 8, 2023

View reviewed changes

ajantha-bhat mentioned this pull request Jun 21, 2023

Trino support for project nessie with Apache Iceberg projectnessie/nessie#5528

Closed

findinpath self-requested a review June 21, 2023 20:28

ajantha-bhat force-pushed the config branch from a23af13 to 93081c7 Compare July 5, 2023 11:10

findinpath reviewed Jul 5, 2023

View reviewed changes

ebyhr reviewed Oct 22, 2023

View reviewed changes

ajantha-bhat force-pushed the config branch from d8bc99d to fb4716e Compare October 23, 2023 05:45

ajantha-bhat force-pushed the config branch 2 times, most recently from eeda12c to da29c08 Compare October 23, 2023 10:59

ajantha-bhat force-pushed the config branch from da29c08 to a41bffd Compare October 24, 2023 17:20

Support additional configurations for Nessie catalog

26e5bce

Nessie server configurations like read-timeout-ms, connect-timeout-ms and enable-compression properties are exposed to have finer control over the Nessie commits.

ajantha-bhat force-pushed the config branch from a41bffd to e264dc4 Compare October 25, 2023 05:19

ajantha-bhat force-pushed the config branch 2 times, most recently from b78b5ae to 688d637 Compare October 31, 2023 09:03

ebyhr force-pushed the config branch from 688d637 to 602e03b Compare October 31, 2023 10:08

ebyhr force-pushed the config branch from 602e03b to 6d38fde Compare November 1, 2023 06:18

Support authentication configurations for Nessie catalog

aee00b9

Nessie server can be deployed with Authentication mode with keycloak. So, need to expose the Nessie client configurations to handle the Auth.

ebyhr force-pushed the config branch from f923075 to aee00b9 Compare November 2, 2023 06:15

ebyhr merged commit 73b1f75 into trinodb:master Nov 2, 2023

github-actions bot added this to the 432 milestone Nov 2, 2023

martint mentioned this pull request Nov 2, 2023

Remove unnecessary logging #19616

Merged

colebow mentioned this pull request Nov 2, 2023

Add Trino 432 release notes #19599

Merged

martint added a commit to martint/trino that referenced this pull request Nov 2, 2023

Remove unnecessary logging

cd85703

It generates unnecessary clutter when running tests. Follow up to trinodb#17725

martint added a commit that referenced this pull request Nov 2, 2023

Remove unnecessary logging

406d0f8

It generates unnecessary clutter when running tests. Follow up to #17725

denodo-research-labs mentioned this pull request Nov 6, 2025

fix: Fix Bearer authentication with Nessie catalog prestodb/presto#26512

Merged

6 tasks

Conversation

ajantha-bhat commented Jun 1, 2023 • edited by ebyhr Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Additional context and related issues

Release notes

Uh oh!

Uh oh!

dimas-b left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ebyhr left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as duplicate.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ajantha-bhat commented Jun 2, 2023

Uh oh!

Uh oh!

ajantha-bhat commented Jun 8, 2023

Uh oh!

ajantha-bhat commented Jun 16, 2023

Uh oh!

EternalDeiwos commented Jun 21, 2023

Uh oh!

ajantha-bhat commented Jun 21, 2023

Uh oh!

ajantha-bhat commented Jun 27, 2023

Uh oh!

ajantha-bhat commented Jul 5, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ajantha-bhat commented Oct 23, 2023

Uh oh!

ajantha-bhat commented Oct 23, 2023

Uh oh!

ajantha-bhat commented Oct 24, 2023

Uh oh!

ajantha-bhat commented Oct 25, 2023

Uh oh!

ajantha-bhat commented Nov 1, 2023

Uh oh!

ajantha-bhat commented Nov 1, 2023

Uh oh!

ebyhr commented Nov 2, 2023

Uh oh!

ajantha-bhat commented Nov 2, 2023

Uh oh!

mosabua commented Nov 2, 2023

Uh oh!

Reviewers

Assignees

Labels

ajantha-bhat commented Jun 1, 2023 •

edited by ebyhr

Loading

ebyhr left a comment •

edited

Loading