[Epic][S3FileIo] Turn S3 Analytics Accelerator on by default #14350
Description
Proposed Change
We have developed an opensource library that aims to accelerate read access to S3. (will be referred to as AAL)
https://github.com/awslabs/analytics-accelerator-s3
It is currently merged in behind a feature flag that can be toggled with:
--conf "spark.sql.catalog.<CATALOG_NAME>.s3.analytics-accelerator.enabled=true"
This epic is to track the work needed to turn it on by default for all Iceberg and S3 customers.
If you are a Iceberg User and would like to test please let us know and we will be happy to schedule some time with you.
work needed
- Integrate default OFF: AWS: Integrate S3 analytics accelerator library #12299
- Add read vector support to core iceberg core: Adding read vector to range readable interface and adding mappe… #13997
- Add read vector support to S3FileIO feat: adding s3fileio vector reader #14352
- Add read vector support to AAL AWS: swapping s3 aal to the sync client and adding aal vector path #14247
- Add sync client support for AAL and default to it AWS: swapping s3 aal to the sync client and adding aal vector path #14247
- blocked: feature Parity including:
- Support SSE-C
- All retry logic in default stream is compatible with AAL
- All integration tests pass with AAL - Customer testing and success stories started testing data will be shared later
- Default On
- Iceberg Release
Proposal document
https://docs.google.com/document/d/13shy0RWotwfWC_qQksb95PXdi-vSUCKQyDzjoExQEN0/edit?usp=sharing
Specifications
- Table
- REST
- View
- Puffin
- Encryption
- Other