Skip to content

refactor(obj): Implement hierarchical object storage client architecture (DNM) - #1187

Closed
aranadive wants to merge 13 commits into
ai-dynamo:pull-request/1127from
aranadive:s3reorg
Closed

aranadive wants to merge 13 commits into
ai-dynamo:pull-request/1127from
aranadive:s3reorg

Conversation

@aranadive

Copy link
Copy Markdown
Contributor

What?

Introduce a modular, inheritance-based architecture for object storage
clients to support multiple S3-compatible backends and acceleration paths.

The new hierarchy organizes clients as follows:

+------------------+
| Object Base Type|
+------------------+
          |
          v
+------------------+
| S3 Vanilla Object|
+------------------+
          |
      ----------------------------
      |                          |
      v                          v
+------------------+       +----------------+
|  S3 CRT Object   |       | S3 Accel Object|
+------------------+       +----------------+
                                    |
                                    v
                           +----------------+
                           | Dell OBS Object|
                           +----------------+

Why?

The previous flat architecture made it difficult to extend the object plugin
with new vendor-specific backends and acceleration methods. This refactoring
provides:

  • Clean separation of concerns with distinct client types
  • Easy addition of new vendor-specific backends (Dell, future vendors)
  • Flexible runtime client selection via custom parameters
  • Polymorphic interface for simplified transfer logic
  • Better code organization matching the inheritance relationships

How?

  • Created base/client.h with iObjClient interface defining common operations
    (setExecutor, putObjectAsync, getObjectAsync, checkObjectExists)

  • Reorganized S3 clients into subdirectories reflecting inheritance:

    • s3/ - Standard S3 client (awsS3Client) inheriting from iObjClient
    • s3_crt/ - S3 CRT client (awsS3CrtClient) inheriting from awsS3Client
    • s3_accel/ - S3 Accelerated client using CRT implementation
    • s3_accel/dell_obs/ - Dell OBS client with vendor-specific optimizations
  • Updated obj_backend to dynamically instantiate clients based on parameters:

    • Default: S3 Vanilla client
    • accelerated=true: S3 Accel client
    • accelerated=true,type=dell: Dell OBS client
    • S3 CRT client now optional (only when crtMinLimit is configured)
  • Refactored client member types to use polymorphic iObjClient interface

Lets use S3 CRT Client as an additional client type when creating the
object plugin. The CRT Client generally performs better for large
objects by supporting multi-parting automatically and ranged GETs.
This increases throughput and will help large KV transfers to objects.

Both the S3Client and S3CRTClient are used based on a new
backend parameter called crtMinLimit. This determines the client type
used of the object size to be transfered is larger than this value, then
the S3CRTClient is used.

Since S3 and S3CRT are very similar API wise, most of the common code is
refactored into separate files. Also, Dockerfiles are modified to
compile the s3crt client as well.

Signed-off-by: Adit Ranadive <aranadive@nvidia.com>
Signed-off-by: Adit Ranadive <aranadive@nvidia.com>
Signed-off-by: Adit Ranadive <aranadive@nvidia.com>
Signed-off-by: Adit Ranadive <aranadive@nvidia.com>
Signed-off-by: Adit Ranadive <aranadive@nvidia.com>
Signed-off-by: Adit Ranadive <aranadive@nvidia.com>
By default the tests use small object sizes, I modified the transfer
handler to create larger objects based on parameters passed. The CRT
tests will use these when testing.

Signed-off-by: Adit Ranadive <aranadive@nvidia.com>
Move the common utilities for S3 and S3CRT to the utils folder. Other
S3-based object plugins can link to the utils lib as required.

Signed-off-by: Adit Ranadive <aranadive@nvidia.com>
Some more reorg of the object plugin to let each object storage api have
its own directory. Should make it easier to onboard other S3-type
plugins.

Signed-off-by: Adit Ranadive <aranadive@nvidia.com>
Signed-off-by: Adit Ranadive <aranadive@nvidia.com>
Signed-off-by: Adit Ranadive <aranadive@nvidia.com>
Signed-off-by: Adit Ranadive <aranadive@nvidia.com>
obj): Implement hierarchical object storage client architecture

Introduce a modular, inheritance-based architecture for object storage
clients to support multiple S3-compatible backends and acceleration paths.

Architecture Overview:
  iObjClient (base interface)
  └── awsS3Client (S3 Vanilla)
      ├── awsS3CrtClient (S3 CRT for large objects)
      └── awsS3AccelClient (S3 Accelerated)
          └── awsDellOBSClient (Dell ObjectScale)

Key Changes:
- Create base/client.h with iObjClient interface defining common operations
  (setExecutor, putObjectAsync, getObjectAsync, checkObjectExists)

- Reorganize S3 clients into subdirectories:
  * s3/ - Standard S3 client (awsS3Client) inheriting from iObjClient
  * s3_crt/ - S3 CRT client (awsS3CrtClient) inheriting from awsS3Client
  * s3_accel/ - S3 Accelerated client using CRT implementation
  * s3_accel/dell_obs/ - Dell OBS client with vendor-specific optimizations

- Update obj_backend to dynamically instantiate clients based on parameters:
  * Default: S3 Vanilla client
  * accelerated=true: S3 Accel client
  * accelerated=true,type=dell: Dell OBS client
  * S3 CRT client now optional (only when crtMinLimit is configured)

- Refactor client member types to use polymorphic iObjClient interface
  for simplified transfer logic and extensibility

Benefits:
- Clean separation of concerns with distinct client types
- Easy addition of new vendor-specific backends
- Flexible runtime client selection via custom parameters
- Maintains backward compatibility with existing configurations"

Signed-off-by: Adit Ranadive <aranadive@nvidia.com>
@github-actions

github-actions Bot commented Jan 9, 2026

Copy link
Copy Markdown

👋 Hi aranadive! Thank you for contributing to ai-dynamo/nixl.

Your PR reviewers will review your contribution then trigger the CI to test your changes.

🚀

@aranadive
aranadive changed the base branch from main to pull-request/1127 January 14, 2026 22:21
@copy-pr-bot
copy-pr-bot Bot deleted the branch ai-dynamo:pull-request/1127 January 20, 2026 17:25
@copy-pr-bot copy-pr-bot Bot closed this Jan 20, 2026
@aranadive
aranadive deleted the s3reorg branch February 6, 2026 07:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant