Remote storage pipeline #899

tstamler · 2025-10-14T15:08:16Z

What?

Update the remote storage example with pipelining code and more details in README.

copy-pr-bot · 2025-10-14T15:08:19Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

github-actions · 2025-10-14T15:08:29Z

👋 Hi tstamler! Thank you for contributing to ai-dynamo/nixl.

Your PR reviewers will review your contribution then trigger the CI to test your changes.

🚀

Signed-off-by: Timothy Stamler <[email protected]>

examples/python/remote_storage_example/storage_pipeline.png

roiedanino · 2025-10-19T07:01:01Z

examples/python/remote_storage_example/nixl_p2p_storage_example.py

+    end_time = time.time()
+
+    elapsed = end_time - start_time


Suggested change

end_time = time.time()

elapsed = end_time - start_time

elapsed = time.time() - start_time

roiedanino · 2025-10-19T07:01:23Z

examples/python/remote_storage_example/nixl_p2p_storage_example.py

+    end_time = time.time()
+
+    elapsed = end_time - start_time


Suggested change

end_time = time.time()

elapsed = end_time - start_time

elapsed = time.time() - start_time

roiedanino · 2025-10-19T07:02:12Z

examples/python/remote_storage_example/nixl_p2p_storage_example.py

+    mem = "DRAM"
+
+    if args.role == "client":
+        mem = "VRAM"


Don't we have some constants for "DRAM", "VRAM", "GDS_MT" and so on?

We pretty consistently expect python applications to just use strings, its a little clearer than constant definitions in the bindings. The python API will do these translations.

I meant constant strings

Something like:

class MemoryType(StrEnum): DRAM = "DRAM" VRAM = "VRAM" ...

Then usage will be:

MemoryType.DRAM

We have those enums in Rust and C++, thought we had something similar here, but if there isn't, it requires a separate PR as it's unrelated

roiedanino · 2025-10-19T07:05:22Z

examples/python/remote_storage_example/nixl_storage_utils/common.py

    nixl_mem_reg_list = []
    nixl_file_reg_list = []

+    if mem == "VRAM":


Maybe it would be better to use constants for memory/storage types

roiedanino · 2025-10-19T07:08:52Z

examples/python/remote_storage_example/nixl_p2p_storage_example.py

+
+    elapsed = end_time - start_time
+
+    logger.info(f"Time for {iterations} READ iterations: {elapsed} seconds")


Suggested change

logger.info(f"Time for {iterations} READ iterations: {elapsed} seconds")

logger.info("Time for %s READ iterations: %s seconds", iterations, elapsed)

We shouldn't use f-strings in loggers, it's not optimized, pylint would warn about it

roiedanino · 2025-10-19T07:12:16Z

examples/python/remote_storage_example/nixl_p2p_storage_example.py

+                s += 1
+                continue
+
+            if s == iterations:


How can this flow happen with the s < iterations while condition?

good catch, while loop should be an or

roiedanino · 2025-10-19T07:13:42Z

examples/python/remote_storage_example/nixl_p2p_storage_example.py

+            if s == 0:
+                futures.append(
+                    executor.submit(
+                        execute_transfer,
+                        my_agent,
+                        my_mem_descs,
+                        my_file_descs,
+                        my_agent.name,
+                        "READ",
+                    )
+                )
+                s += 1
+                continue


Maybe we can move this line before the loop, initiating s to 1?

I think it would simplify the loop and help avoid a branch

roiedanino · 2025-10-19T07:16:18Z

examples/python/remote_storage_example/nixl_p2p_storage_example.py


+    end_time = time.time()
+
+    logger.info(f"Time for {iterations} iterations: {end_time - start_time} seconds")


f-string -> %s logger formatting

vvenkates27 · 2025-10-24T15:41:43Z

examples/python/remote_storage_example/README.md

+### Pipelining
+
+To improve performance of the remote storage server, we can pipeline operations to network and storage. This pipelining allows multiple threads to handle each request. However, in order to maintain correctness, the order of network and storage must happen in order for each individual remote storage operation. To do this, we implemented a simple pipelining scheme.
+


@tstamler please can you add some more description on how this is implemented with NIXL in this example?

added a brief description

vvenkates27 · 2025-10-24T15:42:36Z

examples/python/remote_storage_example/README.md

+
+### Performance Tips
+
+For high-speed storage and network hardware, you may need to tweak performance with a couple of environment variables.


Please provide suggested list of env - vars, (like providing example configuration with CX-7, SPX and providing what UCX tuning/GDS tuning that is seen to be beneficial?

The next two lines are these suggestions. I don't know what other example configuration options you might be talking about.

vvenkates27 · 2025-10-24T15:43:21Z

examples/python/remote_storage_example/README.md

+
+For high-speed storage and network hardware, you may need to tweak performance with a couple of environment variables.
+
+First, for optimal GDS performance, ensure you are using the GDS_MT backend with default concurrency. Additionally, you can use the cufile options described in the [GDS README](https://github.com/ai-dynamo/nixl/blob/main/src/plugins/cuda_gds/README.md).


We should also highlight the difference between the GDS backend and that GDS plugins can work in compat mode by default, and what is required for true GDS support.

This sounds GDS specific and should be in the GDS README. I can at least add a reminder here.

examples/python/remote_storage_example/storage_pipeline.png

pull-request-size bot added the size/XL label Oct 14, 2025

github-actions bot added the external-contribution label Oct 14, 2025

tstamler force-pushed the remote_storage_pipeline branch from 1111e18 to c3f3cdc Compare October 14, 2025 15:13

pull-request-size bot added size/L and removed size/XL labels Oct 14, 2025

tstamler requested review from aranadive and vvenkates27 October 14, 2025 17:41

tstamler marked this pull request as ready for review October 17, 2025 14:20

tstamler requested review from a team, mkhazraee, ovidiusm and roiedanino as code owners October 17, 2025 14:20

PYTHON: new remote storage example

a800b56

Signed-off-by: Timothy Stamler <[email protected]>

tstamler force-pushed the remote_storage_pipeline branch from b2c6c33 to 805df1e Compare October 17, 2025 14:20

PYTHON: updated remote storage example with pipeline and better README

93ab2a6

Signed-off-by: Timothy Stamler <[email protected]>

tstamler force-pushed the remote_storage_pipeline branch from cb625ac to 93ab2a6 Compare October 17, 2025 14:41

roiedanino reviewed Oct 19, 2025

View reviewed changes

vvenkates27 reviewed Oct 24, 2025

View reviewed changes

tstamler added 7 commits November 3, 2025 13:08

update diagram

b7b3d68

update readme

b4640a3

one more update

4f83249

python format changes

5f4fc62

update other diagram

1cc674c

pre-commit

43c0cd8

black

0126efc


		elapsed = end_time - start_time

		logger.info(f"Time for {iterations} READ iterations: {elapsed} seconds")

	logger.info(f"Time for {iterations} READ iterations: {elapsed} seconds")
	logger.info("Time for %s READ iterations: %s seconds", iterations, elapsed)


		end_time = time.time()

		logger.info(f"Time for {iterations} iterations: {end_time - start_time} seconds")

		### Pipelining

		To improve performance of the remote storage server, we can pipeline operations to network and storage. This pipelining allows multiple threads to handle each request. However, in order to maintain correctness, the order of network and storage must happen in order for each individual remote storage operation. To do this, we implemented a simple pipelining scheme.


		### Performance Tips

		For high-speed storage and network hardware, you may need to tweak performance with a couple of environment variables.


		For high-speed storage and network hardware, you may need to tweak performance with a couple of environment variables.

		First, for optimal GDS performance, ensure you are using the GDS_MT backend with default concurrency. Additionally, you can use the cufile options described in the [GDS README](https://github.com/ai-dynamo/nixl/blob/main/src/plugins/cuda_gds/README.md).

Remote storage pipeline #899

Are you sure you want to change the base?

Remote storage pipeline #899

Conversation

tstamler commented Oct 14, 2025

What?

Uh oh!

copy-pr-bot bot commented Oct 14, 2025

Uh oh!

github-actions bot commented Oct 14, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants