Skip to content

KVBENCH: Fixed filepath option - 0.4.0 #556

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pvijayakrish merged 1 commit into from
Jul 9, 2025
Merged

KVBENCH: Fixed filepath option - 0.4.0 #556

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 13 additions & 28 deletions benchmark/kvbench/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,12 +69,11 @@ Options:
--help Show this message and exit.

Commands:
ct-perftest Run custom traffic performance test using patterns...
io-size Display IO size information
kvcache Display kvcache information
plan Display the recommended configuration for nixlbench
profile Run nixlbench
sequential-ct-perftest Run sequential custom traffic performance test...
ct-perftest Run custom traffic performance test using...
kvcache Display kvcache information
plan Display the recommended configuration for...
profile Run nixlbench
sequential-ct-perftest Run sequential custom traffic performance test...
```

## Command Line Arguments
Expand Down Expand Up @@ -141,8 +140,8 @@ These arguments are used by both `plan` and `profile` commands:
| `--device_list` | Comma-separated device names (default: all) |
| `--runtime_type` | Type of runtime to use [ETCD] (default: ETCD) |
| `--etcd-endpoints` | ETCD server URL for coordination (default: http://localhost:2379) |
| `--storage_enable_direct` | Enable direct I/O for GDS operations |
| `--gds_filepath` | File path for GDS operations |
| `--storage_enable_direct` | Enable direct I/O for storage operations |
| `--filepath` | File path for storage operations |
| `--enable_vmm` | Enable VMM memory allocation when DRAM is requested |

### CTP Command Arguments
Expand All @@ -165,40 +164,26 @@ Specific to CTP (Custom Traffic Performance) commands:
The `plan` command generates and displays recommended `nixlbench` command configurations based on your model architecture and parameters. It helps you prepare optimal benchmark settings without running the benchmark itself.

```bash
python main.py plan --model ./examples/model_deepseek_r1.yaml --model_config "./examples/block-tp1-pp8.yaml"
python main.py plan --model ./examples/model_deepseek_r1.yaml --model_config "./examples/block-tp1-pp8.yaml" --backend POSIX
```

#### Profile Command

The `profile` command actually runs the benchmark with the specified configuration using `nixlbench`, collecting performance data across various KV cache operations and access patterns.

```bash
python main.py profile --model ./examples/model_deepseek_r1.yaml --model_config "./examples/block-tp1-pp8.yaml"
python main.py profile --model ./examples/model_deepseek_r1.yaml --model_config "./examples/block-tp1-pp8.yaml" --backend POSIX
```

#### KVCache Command

The `kvcache` command analyzes and displays detailed information about the KV cache for a specified model configuration, including model type, sequence lengths, batch sizes, and I/O sizes.

```bash
python main.py kvcache --model ./examples/model_deepseek_r1.yaml --model_config "./examples/block-tp1-pp8.yaml"
Model : DEEPSEEK_R1
Input Sequence Length : 10000
Batch Size : 298
IO Size : 1.12 MB
```

#### IO-Size Command

The `io-size` command displays information about the I/O size requirements for a specified model configuration, helping you understand memory usage and data transfer needs.

```bash
python main.py io-size --model ./examples/model_deepseek_r1.yaml --model_config "./examples/block-tp1-pp8.yaml"
Model: DEEPSEEK_R1
Page Size: 256.0 B
Input Sequence Length: 10000
Batch Size: 298
IO Size: 1.12 MB
python main.py kvcache --model ./examples/model_deepseek_r1.yaml --model_config "./examples/block-tp1-pp8.yaml" --isl 10000 --page_size 512
Model ISL Num Requests Batch Size IO Size TP PP Page Size Access
----------- ----- -------------- ------------ --------- ---- ---- ----------- --------
DEEPSEEK_R1 10000 10 1490 2.25 MB 1 8 512 block
```

### CTP Commands
Expand Down
19 changes: 13 additions & 6 deletions benchmark/kvbench/commands/args.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,9 @@

def common_args(func):
"""Decorator for common model arguments"""
func = click.option("--model", type=str, help="Model name (e.g., 'llama3.1-8b')")(
func
)
func = click.option(
"--model", type=str, help="Path to a model architecture YAML file"
)(func)
func = click.option(
"--model_config", type=str, help="Path to a single model config YAML file"
)(func)
Expand Down Expand Up @@ -70,7 +70,9 @@ def nixl_bench_args(func):
help="Destination of the nixl descriptors [file, memory, gpu] (default: memory)",
)(func)
func = click.option(
"--backend", type=str, help="Communication backend [UCX, UCX_MO] (default: UCX)"
"--backend",
type=str,
help="Communication backend [POSIX, GDS] (default: POSIX)",
)(func)
func = click.option(
"--worker_type",
Expand Down Expand Up @@ -156,11 +158,16 @@ def nixl_bench_args(func):
func = click.option(
"--storage_enable_direct",
type=bool,
help="Enable direct I/O for storage operations (only used with POSIX backend)",
help="Enable direct I/O for storage operations",
default=False,
)(func)
func = click.option(
"--gds_filepath", type=str, help="File path for GDS operations"
"--filepath", type=str, help="File path for storage operations"
)(func)
func = click.option(
"--posix_api_type",
type=str,
help="API type for POSIX operations [AIO, URING] (only used with POSIX backend",
)(func)
func = click.option(
"--enable-vmm",
Expand Down
95 changes: 74 additions & 21 deletions benchmark/kvbench/commands/nixlbench.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,17 +36,21 @@ def __init__(
enable_pt=False,
etcd_endpoints="http://localhost:2379",
storage_enable_direct=False,
gds_filepath="",
filepath="",
gds_batch_pool_size=32,
gds_batch_limit=128,
initiator_seg_type="DRAM",
enable_vmm=False,
max_batch_size=None,
max_block_size=None,
mode="SG",
num_files=1,
num_initiator_dev=1,
num_iter=1000,
num_target_dev=1,
num_threads=1,
op_type="WRITE",
posix_api_type="AIO",
runtime_type="ETCD",
scheme="pairwise",
start_batch_size=None,
Expand All @@ -68,17 +72,21 @@ def __init__(
enable_pt (bool, optional): Whether to enable peer-to-peer transfer. Defaults to False.
etcd_endpoints (str, optional): ETCD endpoints for runtime. Defaults to "http://localhost:2379".
storage_enable_direct (bool, optional): Whether to enable direct I/O for storage operations. Defaults to False.
gds_filepath (str, optional): Path for GDS file. Defaults to "".
enable_vmm (bool, optional): Whether to use VMM memory allocation. Defaults to False.
filepath (str, optional): Path for GDS and POSIX operations. Defaults to "".
gds_batch_pool_size (int, optional): Batch pool size for GDS operations. Defaults to 32.
gds_batch_limit (int, optional): Batch limit for GDS operations. Defaults to 128.
initiator_seg_type (str, optional): Type of initiator segment. Defaults to "DRAM".
enable_vmm (bool, optional): Whether to use VMM memory allocation. Defaults to False.
max_batch_size (int, optional): Maximum batch size for testing. Defaults to model_config value.
max_block_size (int, optional): Maximum block size for testing. Defaults to tp_size * isl.
mode (str, optional): Benchmarking mode. Defaults to "SG".
num_files (int, optional): Number of files. Defaults to 1.
num_initiator_dev (int, optional): Number of initiator devices. Defaults to 1.
num_iter (int, optional): Number of iterations. Defaults to 1000.
num_target_dev (int, optional): Number of target devices. Defaults to 1.
num_threads (int, optional): Number of threads. Defaults to 1.
op_type (str, optional): Operation type. Defaults to "WRITE".
posix_api_type (str, optional): POSIX API type. Defaults to "AIO".
runtime_type (str, optional): Runtime type. Defaults to "ETCD".
scheme (str, optional): Communication scheme. Defaults to "pairwise".
start_batch_size (int, optional): Starting batch size. Defaults to 1.
Expand All @@ -96,17 +104,21 @@ def __init__(
self.enable_pt = enable_pt
self.etcd_endpoints = etcd_endpoints
self.storage_enable_direct = storage_enable_direct
self.gds_filepath = gds_filepath
self.filepath = filepath
self.enable_vmm = enable_vmm
self.gds_batch_pool_size = gds_batch_pool_size
self.gds_batch_limit = gds_batch_limit
self.initiator_seg_type = initiator_seg_type
self.max_batch_size = max_batch_size
self.max_block_size = max_block_size
self.mode = mode
self.num_files = num_files
self.num_initiator_dev = num_initiator_dev
self.num_iter = num_iter
self.num_target_dev = num_target_dev
self.num_threads = num_threads
self.op_type = op_type
self.posix_api_type = posix_api_type
self.runtime_type = runtime_type
self.scheme = scheme
self.start_batch_size = start_batch_size
Expand All @@ -121,25 +133,54 @@ def set_io_size(self, io_size: int):
self.start_block_size = io_size
self.max_block_size = io_size

def _configure_gds(self, source: str, destination: str):
if source == "file":
# this is a READ from GDS to GPU
self.op_type = "READ"
self.target_seg_type = "VRAM"
elif source == "gpu":
# this is a WRITE from GPU to GDS
self.op_type = "WRITE"
self.target_seg_type = "VRAM"
else:
raise ValueError(f"Invalid source for GDS: {source}")

def _configure_posix(self, source: str, destination: str):
if source == "file":
self.op_type = "READ"
self.target_seg_type = "DRAM"
elif source == "memory":
self.op_type = "WRITE"
self.initiator_seg_type = "DRAM"
else:
raise ValueError(f"Invalid source for POSIX: {source}")

def configure_segment_type(self, backend: str, source: str, destination: str):
if backend == "GDS":
if source == "file":
# this is a READ from GDS to GPU
self.op_type = "READ"
self.target_seg_type = "VRAM"
elif source == "gpu":
# this is a WRITE from GPU to GDS
self.op_type = "WRITE"
self.target_seg_type = "VRAM"

elif source == "memory":
# this is a WRITE from memory to GDS
self.op_type = "WRITE"
self.initiator_seg_type = "DRAM"
self.target_seg_type = "DRAM"
if backend.lower() == "gds":
self._configure_gds(source, destination)
elif backend.lower() == "posix":
self._configure_posix(source, destination)
else:
raise ValueError(f"Invalid backend: {backend}")

# if backend == "GDS" or backend == "POSIX":
# if source == "file":
# # this is a READ from GDS to GPU
# self.op_type = "READ"
# self.target_seg_type = "VRAM"
# elif source == "gpu":
# # this is a WRITE from GPU to GDS
# self.op_type = "WRITE"
# self.target_seg_type = "VRAM"

# elif source == "memory":
# # this is a WRITE from memory to GDS
# self.op_type = "WRITE"
# self.initiator_seg_type = "DRAM"
# self.target_seg_type = "DRAM"
# else:
# raise ValueError(f"Invalid backend: {backend}")

def configure_scheme(self, scheme: str = "pairwise", direction: str = "isl"):
"""
Configure the scheme based on the model configuration.
Expand All @@ -154,6 +195,10 @@ def configure_scheme(self, scheme: str = "pairwise", direction: str = "isl"):
self.num_target_dev = 1

def set_batch_size(self, batch_size: int):
"""
Set the batch size for benchmarking.
"""

self.start_batch_size = batch_size
self.max_batch_size = batch_size

Expand Down Expand Up @@ -184,17 +229,21 @@ def _params(self):
"enable_pt": self.enable_pt,
"etcd_endpoints": self.etcd_endpoints,
"storage_enable_direct": self.storage_enable_direct,
"gds_filepath": self.gds_filepath,
"filepath": self.filepath,
"enable_vmm": self.enable_vmm,
"gds_batch_pool_size": self.gds_batch_pool_size,
"gds_batch_limit": self.gds_batch_limit,
"initiator_seg_type": self.initiator_seg_type,
"max_batch_size": self.max_batch_size,
"max_block_size": self.max_block_size,
"mode": self.mode,
"num_files": self.num_files,
"num_initiator_dev": self.num_initiator_dev,
"num_iter": self.num_iter,
"num_target_dev": self.num_target_dev,
"num_threads": self.num_threads,
"op_type": self.op_type,
"posix_api_type": self.posix_api_type,
"runtime_type": self.runtime_type,
"scheme": self.scheme,
"start_batch_size": self.start_batch_size,
Expand Down Expand Up @@ -223,17 +272,21 @@ def defaults():
"enable_pt": False,
"etcd_endpoints": "http://localhost:2379",
"storage_enable_direct": False,
"gds_filepath": "",
"filepath": "",
"enable_vmm": False,
"gds_batch_pool_size": 32,
"gds_batch_limit": 128,
"initiator_seg_type": "DRAM",
"max_batch_size": 1, # ios per gpu
"max_block_size": 67108864, # io size
"mode": "SG",
"num_files": 1,
"num_initiator_dev": 1,
"num_iter": 1000,
"num_target_dev": 1,
"num_threads": 1,
"op_type": "WRITE",
"posix_api_type": "AIO",
"runtime_type": "ETCD",
"scheme": "pairwise",
"start_batch_size": 1,
Expand Down
12 changes: 4 additions & 8 deletions benchmark/kvbench/docs/tutorial-gds.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,10 +34,6 @@ system:

## Profiling

> [!NOTE]
> When using KVBench/NIXLBench, you should perform a WRITE operation before a READ.
> This is because the files need to be created and populated with data before you can read from them.

### Step 1: Perform a WRITE operation first (GPU to Storage)

```bash
Expand All @@ -47,15 +43,15 @@ python main.py profile \
--backend GDS \
--source gpu \
--etcd-endpoints "http://localhost:2379" \
--gds_filepath /path/to/your/directory \
--filepath /path/to/your/directory \
--num_requests 1
```

This command:
- Uses the DeepSeek model configuration
- Sets up GDS as the backend
- Sets the source as GPU memory (`--source gpu`)
- Specifies the directory for GDS files (`--gds_filepath`)
- Specifies the directory for GDS files (`--filepath`)
- Specifies the number of concurrent user requests to simulate (`--num_requests`)

### Step 2: Perform a READ operation (Storage to GPU)
Expand All @@ -69,7 +65,7 @@ python main.py profile \
--backend GDS \
--source file \
--etcd-endpoints "http://localhost:2379" \
--gds_filepath /path/to/your/directory \
--filepath /path/to/your/directory \
--num_requests 1
```

Expand All @@ -87,7 +83,7 @@ The profiling results will show:
## Troubleshooting

- If you encounter "File not found" errors during READ operations, ensure you've performed a WRITE operation first
- Check that the directory specified in `--gds_filepath` exists and has proper permissions
- Check that the directory specified in `--filepath` exists and has proper permissions
- Verify that your system has GPU Direct Storage support enabled
- Make sure the ETCD server is running at the specified endpoint

Expand Down
Loading