[aranadive]

(This is essentially extending the NIXLKVBench benchmark to cover KV Cache transfers over network which was being added separately in #263). Fixes NIX-144.

NIXL KVBench

NIXL KVBench is a benchmarking utility for testing KVCache transfers in large language models (LLMs) using the NIXL runtime. It supports various model architectures, parallelism strategies, and custom traffic simulation for asymmetric workloads.

---

🚀 Features

🔧 KVBench Commands

Benchmark KVCache transfers with support for:

• Block and Layer access patterns
• Tensor and Pipeline parallelism
• Integration with nixlbench

Commands:

• plan: Generate recommended nixlbench command configs
• profile: Run actual benchmarks
• kvcache, io-size: Inspect model memory/transfer characteristics

📶 CTP (Custom Traffic Performance)

Simulate and analyze asymmetric data transfers using traffic matrices:

• YAML-configurable traffic patterns
• Sequential or single-pattern execution
• Reports total/isolated latency, bandwidth, and congestion response
• JSON output and buffer verification supported

---

✅ Supported Architectures

• DeepSeek R1
• LLaMA 3.1
• More supported via custom config YAMLs

---

🧪 Example: KVBench Profile

Run benchmark for DeepSeek R1 with block access pattern:

python main.py profile \
  --model ./examples/model_deepseek_r1.yaml \
  --model_config ./examples/block-tp1-pp8.yaml \
  --backend GDS \
  --source gpu \
  --etcd-endpoints "http://localhost:2379"

nixlbench \
  --backend GDS \
  --source gpu \
  --etcd-endpoints http://localhost:2379 \
  --max_batch_size 5958 \
  --start_batch_size 5958 \
  --max_block_size 589824 \
  --start_block_size 589824 \
  --target_seg_type VRAM

🧪 Example: CTP Sequential Test

python main.py sequential-ct-perftest ./config.yaml \
  --verify-buffers \
  --json-output-path ./results.json

⚙️ Getting Started

• Python or Docker setup supported
• Commands:
  • plan, profile, kvcache, io-size
  • ct-perftest, sequential-ct-perftest

📚 Documentation

• See the /docs directory for:
  • Setup tutorials
  • Creating model config files

[github-actions]

👋 Hi aranadive! Thank you for contributing to ai-dynamo/nixl.

Your PR reviewers will review your contribution then trigger the CI to test your changes.

🚀

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[x41lakazam]

I have updated the CT perftest files with the latest working version

@aranadive For my part, it is ready to be merged

[aranadive]

/ok to test d912d04

[aranadive]

/build

[aranadive]

/build

[aranadive]

/build

[aranadive]

/ok to test b5608aa

[aranadive]

/build

[aranadive]

/ok to test 4db2ff0

[barneuman]

/build

[barneuman]

/ok to test 4dd3d2c

[barneuman]

/ok to test 4dd3d2c

[roiedanino]

/build

[roiedanino]

/ok to test 4dd3d2c