add support in nixlbench for O_DIRECT

[cheese-head]

What?

This PR provides support for O_DIRECT in NIXLBench.

Why?

To provide a more accurate representation of storage access performance when using the POSIX Plugin

How?

It aligns memory to the underlying page size using posix_memalign

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

👋 Hi cheese-head! Thank you for contributing to ai-dynamo/nixl.

Your PR reviewers will review your contribution then trigger the CI to test your changes.

🚀

[cheese-head]

A quick example of running NIXLBench without O_DIRECT

**********************************************************************
NIXLBench Configuration
**********************************************************************
Runtime (--runtime_type=[etcd])                             : ETCD
ETCD Endpoint                                               : http://localhost:2379
Worker type (--worker_type=[nixl,nvshmem])                  : nixl
Backend (--backend=[UCX,UCX_MO,GDS,POSIX])                  : POSIX
Enable pt (--enable_pt=[0,1])                               : 0
Device list (--device_list=dev1,dev2,...)                   : all
Enable VMM (--enable_vmm=[0,1])                             : 0
POSIX API type (--posix_api_type=[AIO,URING])               : AIO
POSIX filepath (--posix_filepath=path)                      : /data
POSIX enable direct (--storage_enable_direct=[0,1])         : 0
Number of files (--num_files=N)                             : 1
Initiator seg type (--initiator_seg_type=[DRAM,VRAM])       : DRAM
Target seg type (--target_seg_type=[DRAM,VRAM])             : DRAM
Scheme (--scheme=[pairwise,manytoone,onetomany,tp])         : pairwise
Mode (--mode=[SG,MG])                                       : SG
Op type (--op_type=[READ,WRITE])                            : READ
Check consistency (--check_consistency=[0,1])               : 0
Total buffer size (--total_buffer_size=N)                   : 35168256
Num initiator dev (--num_initiator_dev=N)                   : 1
Num target dev (--num_target_dev=N)                         : 1
Start block size (--start_block_size=N)                     : 36864
Max block size (--max_block_size=N)                         : 36864
Start batch size (--start_batch_size=N)                     : 954
Max batch size (--max_batch_size=N)                         : 954
Num iter (--num_iter=N)                                     : 16
Warmup iter (--warmup_iter=N)                               : 0
Num threads (--num_threads=N)                               : 1
--------------------------------------------------------------------------------

Block Size (B)      Batch Size     Avg Lat. (us)  B/W (MiB/Sec)  B/W (GiB/Sec)  B/W (GB/Sec)   
--------------------------------------------------------------------------------
36864               954            2.34205        15010.9        14.6591        15.7401  

And with O_DIRECT

**********************************************************************
NIXLBench Configuration
**********************************************************************
Runtime (--runtime_type=[etcd])                             : ETCD
ETCD Endpoint                                               : http://localhost:2379
Worker type (--worker_type=[nixl,nvshmem])                  : nixl
Backend (--backend=[UCX,UCX_MO,GDS,POSIX])                  : POSIX
Enable pt (--enable_pt=[0,1])                               : 0
Device list (--device_list=dev1,dev2,...)                   : all
Enable VMM (--enable_vmm=[0,1])                             : 0
POSIX API type (--posix_api_type=[AIO,URING])               : AIO
POSIX filepath (--posix_filepath=path)                      : /data
POSIX enable direct (--storage_enable_direct=[0,1])         : 1
Number of files (--num_files=N)                             : 1
Initiator seg type (--initiator_seg_type=[DRAM,VRAM])       : DRAM
Target seg type (--target_seg_type=[DRAM,VRAM])             : DRAM
Scheme (--scheme=[pairwise,manytoone,onetomany,tp])         : pairwise
Mode (--mode=[SG,MG])                                       : SG
Op type (--op_type=[READ,WRITE])                            : READ
Check consistency (--check_consistency=[0,1])               : 0
Total buffer size (--total_buffer_size=N)                   : 35168256
Num initiator dev (--num_initiator_dev=N)                   : 1
Num target dev (--num_target_dev=N)                         : 1
Start block size (--start_block_size=N)                     : 36864
Max block size (--max_block_size=N)                         : 36864
Start batch size (--start_batch_size=N)                     : 954
Max batch size (--max_batch_size=N)                         : 954
Num iter (--num_iter=N)                                     : 16
Warmup iter (--warmup_iter=N)                               : 0
Num threads (--num_threads=N)                               : 1
--------------------------------------------------------------------------------

Block Size (B)      Batch Size     Avg Lat. (us)  B/W (MiB/Sec)  B/W (GiB/Sec)  B/W (GB/Sec)   
--------------------------------------------------------------------------------
36864               954            14.0126        2508.91        2.4501         2.63078

[vvenkates27]

Overall LGTM, just some minor comments.

[vvenkates27]

O_DIRECT is not necessarily and API type, I would say AIO/LIBURING are.
O_DIRECT IMO should be a separate option for NIXLBench,

[vvenkates27]

This is reduced to 16 in latest master.

[vvenkates27]

Ok you are setting defaults here.. depending on the number of IOs this can be tweaked.
With GDS_MT plugin, this limiting issue also goes away

[vvenkates27]

+1 - O_DIRECT and memalign ensures there is p2p gpu direct storage.

[aranadive]

PR #557 add this