Robustify boto3 session handling (DynamoDB, RestCatalog) #2071

jayceslesar · 2025-06-07T14:46:57Z

3 changes here...

Basically a mirror of Adds support for creating a GlueCatalog with own client #1920 but for dynamo
Allow users to pass in an existing client/pass initialize client with properties in RestCatalog with sigv4 enabled. (closes RestCatalog with sigv4 enabled doesn't pick up creds provided as arguments #2070, closes Allow to pass botocore session to REST Catalog (S3 Tables) like Glue Catalog does #2008)
Re-use of the client in add_headers in RestCatalog with sigv4. (closes RestCatalog with sigv4 enabled instantiates a boto session per each client call #2069 )

jayceslesar · 2025-06-08T14:39:05Z

pyiceberg/catalog/rest/__init__.py

            def __init__(self, **properties: str):
                super().__init__()
                self._properties = properties
+                self._boto_session = boto3.Session(
+                    region_name=get_first_property_value(self._properties, AWS_REGION),
+                    botocore_session=self._properties.get(BOTOCORE_SESSION),
+                    aws_access_key_id=get_first_property_value(self._properties, AWS_ACCESS_KEY_ID),
+                    aws_secret_access_key=get_first_property_value(self._properties, AWS_SECRET_ACCESS_KEY),
+                    aws_session_toxken=get_first_property_value(self._properties, AWS_SESSION_TOKEN),
+                )

            def add_headers(self, request: PreparedRequest, **kwargs: Any) -> None:  # pylint: disable=W0613
-                boto_session = boto3.Session()
-                credentials = boto_session.get_credentials().get_frozen_credentials()
-                region = self._properties.get(SIGV4_REGION, boto_session.region_name)


@Fokko @kevinjqliu probably worth refactoring this class out to get some proper test coverage?

Hey @jayceslesar yes, I think that makes sense. Let's do that in a separate PR 👍

rudolfix · 2025-06-09T18:50:23Z

thx for working on this :)

pyiceberg/catalog/rest/__init__.py

Fokko · 2025-06-10T09:28:53Z

pyproject.toml

 glue = ["boto3", "mypy-boto3-glue"]
 adlfs = ["adlfs"]
-dynamodb = ["boto3"]
+dynamodb = ["boto3", "mypy-boto3-dynamodb"]


Shouldn't this be a dev-dependency?

probably -- the glue one isnt but I will move both over

Fokko · 2025-06-10T09:30:32Z

pyiceberg/catalog/dynamodb.py

+            )
+            self.dynamodb = session.client(DYNAMODB_CLIENT)
+            self.dynamodb_table_name = self.properties.get(DYNAMODB_TABLE_NAME, DYNAMODB_TABLE_NAME_DEFAULT)
+            self._ensure_catalog_table_exists_or_create()


Do we want to execute this when passing in an existing client?

Fokko · 2025-06-10T09:31:28Z

pyiceberg/catalog/dynamodb.py

+                aws_session_token=get_first_property_value(properties, DYNAMODB_SESSION_TOKEN, AWS_SESSION_TOKEN),
+            )
+            self.dynamodb = session.client(DYNAMODB_CLIENT)
+            self.dynamodb_table_name = self.properties.get(DYNAMODB_TABLE_NAME, DYNAMODB_TABLE_NAME_DEFAULT)


I think we also want to assign this one when passing in an existing client. Adding the methods to the class for completeness may also be a good thing to do as part of this PR.

I think they do exist (in the MetastoreCatalog) and were just never updated into this class. I will take a look

After a second look I dont know if we need to make a change _ensure_catalog_table_exists_or_create makes the dynamo table which is needed to make + interact with iceberg tables

Co-authored-by: Marko Grujic <[email protected]>

CharlesAttend · 2025-06-10T12:17:35Z

Thank you for fixing this!

Fokko

Thanks again @jayceslesar for working on this, one minor comment that needs to be addressed, but apart from that, this looks great 🥳

pyiceberg/catalog/dynamodb.py

pyiceberg/catalog/glue.py

jayceslesar · 2025-06-11T14:19:10Z

poetry.lock

-version = "2.44.0"
+version = "2.46.0"


@Fokko possible causes of CI failing?

Maybe need to add

"ignore::ResourceWarning", "ignore::pytest.PytestUnraisableExceptionWarning",

To filterwarnings? idrk

ray-project/ray#10279

Ideally, we don't want to bump Ray as part of this PR. Maybe add the <=2.44.0 constraint to Ray for now?

Fokko · 2025-06-11T19:15:52Z

Finally green, thanks @jayceslesar for working on this 🙌

3 changes here... 1. Basically a mirror of apache#1920 but for dynamo 2. Allow users to pass in an existing client/pass initialize client with properties in `RestCatalog` with sigv4 enabled. (closes apache#2070, closes apache#2008) 3. Re-use of the client in `add_headers` in `RestCatalog` with sigv4. (closes apache#2069 ) --------- Co-authored-by: Marko Grujic <[email protected]>

jayceslesar added 2 commits June 7, 2025 10:46

Adds support for creating a DynamoDB with own client

e81cd9c

allow custom session, re-use session if it exists

51f1e0b

jayceslesar changed the title ~~Adds support for creating a DynamoDB with own client~~ Robustify boto3 session handling (DynamoDB, RestCatalog) Jun 8, 2025

jayceslesar commented Jun 8, 2025

View reviewed changes

Merge branch 'main' into dynamodb_client

eb0e1f9

rudolfix mentioned this pull request Jun 9, 2025

Allow to pass botocore session to REST Catalog (S3 Tables) like Glue Catalog does #2008

Closed

gruuya reviewed Jun 10, 2025

View reviewed changes

pyiceberg/catalog/rest/__init__.py Outdated Show resolved Hide resolved

Fokko reviewed Jun 10, 2025

View reviewed changes

jayceslesar and others added 3 commits June 10, 2025 07:13

Update pyiceberg/catalog/rest/__init__.py

b7ac8ae

Co-authored-by: Marko Grujic <[email protected]>

make dev deps

457bc4a

fix init

cc28012

Fokko approved these changes Jun 10, 2025

View reviewed changes

pyiceberg/catalog/dynamodb.py Outdated Show resolved Hide resolved

jayceslesar added 2 commits June 10, 2025 16:58

clean up type imports for glue and dynamo catalogs

9819074

Merge branch 'main' into dynamodb_client

26f0767

Fokko reviewed Jun 10, 2025

View reviewed changes

pyiceberg/catalog/dynamodb.py Outdated Show resolved Hide resolved

quote optional type imports

6584f25

Fokko reviewed Jun 10, 2025

View reviewed changes

pyiceberg/catalog/glue.py Outdated Show resolved Hide resolved

last quote I was missing oops

81ed8c6

jayceslesar commented Jun 11, 2025

View reviewed changes

restrict ray

d89610a

Fokko merged commit 38ebb19 into apache:main Jun 11, 2025
10 checks passed

Flogue mentioned this pull request Oct 23, 2025

REST catalog, S3tables with botocore session #2657

Open

3 tasks

geruh mentioned this pull request Oct 25, 2025

Add python 3.13 support #2658

Open

		version = "2.44.0"
		version = "2.46.0"

Robustify boto3 session handling (DynamoDB, RestCatalog) #2071

Robustify boto3 session handling (DynamoDB, RestCatalog) #2071

Uh oh!

Conversation

jayceslesar commented Jun 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rudolfix commented Jun 9, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jayceslesar Jun 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CharlesAttend commented Jun 10, 2025

Uh oh!

Fokko left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Fokko commented Jun 11, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jayceslesar commented Jun 7, 2025 •

edited

Loading

jayceslesar Jun 10, 2025 •

edited

Loading