Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion benchmarks/frontend/scripts/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,7 @@ When `--num-models` is 1, the served model name matches the HF model path (e.g.,
|------|----------|---------|
| etcd | Yes | `apt install etcd` or [releases](https://github.com/etcd-io/etcd/releases) |
| nats-server | Yes | `apt install nats-server` or [nats.io](https://nats.io/download/) |
| aiperf | Yes | `uv pip install "aiperf==0.8.0"` (in dynamo venv) |
| aiperf | Yes | `uv pip install "aiperf==0.10.0"` (in dynamo venv) |
| jq | Yes | `apt install jq` |
| perf | Optional | `apt install linux-tools-$(uname -r)` |
| bpftrace | Optional | `apt install bpftrace` (needs root or CAP_BPF + CAP_PERFMON) |
Expand Down
4 changes: 2 additions & 2 deletions benchmarks/frontend/scripts/sweep_k8s/aiperf.py
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ def _build_aiperf_script(

return f"""set -e
apt-get update -qq && apt-get install -y -qq curl jq git procps 2>/dev/null
# aiperf 0.8.0 pulls crick==0.0.8 which has no manylinux aarch64 wheel; on
# aiperf 0.10.0 pulls crick==0.0.8 which has no manylinux aarch64 wheel; on
# arm64 nodes pip falls back to sdist and needs a C compiler. python:3.12-slim
# bundles Python headers at /usr/local/include/python3.12/, but libc headers
# (stdlib.h, etc.) come from libc6-dev — and on Debian libc6-dev is only a
Expand All @@ -71,7 +71,7 @@ def _build_aiperf_script(
if [ "$(uname -m)" = "aarch64" ]; then
apt-get install -y -qq gcc libc6-dev 2>/dev/null
fi
pip install --quiet aiperf==0.8.0
pip install --quiet aiperf==0.10.0
echo "aiperf installed"

# Wait for model
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ classifiers = [

dependencies = [
"aiconfigurator[webapp]>=0.9.0",
"aiperf==0.8.0",
"aiperf==0.10.0",
"matplotlib",
"networkx",
"numpy",
Expand Down
2 changes: 1 addition & 1 deletion container/deps/requirements.benchmark.txt
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,4 @@

# Benchmark and profiling tool dependencies.

aiperf==0.8.0
aiperf==0.10.0
2 changes: 1 addition & 1 deletion container/templates/frontend.Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
FROM ${EPP_IMAGE} AS epp

# Build `crick` as a wheel in an isolated stage so the C toolchain never
# reaches the final frontend image. aiperf 0.8.0 depends on crick==0.0.8,
# reaches the final frontend image. aiperf 0.10.0 depends on crick==0.0.8,
# which publishes no manylinux aarch64 wheel — without this, arm64 builds
# fall back to sdist and fail in the final stage where gcc is intentionally
# absent. amd64 has a prebuilt manylinux_x86_64 wheel on PyPI, so the build
Expand Down
2 changes: 1 addition & 1 deletion docs/benchmarks/deepseek-v3-2-wideep-routing.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ The benchmark runs as a Kubernetes Job; tail it with `kubectl logs -f -l job-nam
## Notes

- The source publishes the comparison as a results video plus the dataset statistics above, not a numeric results table; run both configurations to produce TTFT/ITL/goodput deltas for your cluster.
- `perf.yaml` pins `transformers==4.57.6` alongside `aiperf==0.6.0` — older transformers cannot load the `deepseek_v32` tokenizer and AIPerf surfaces it as "Failed to load tokenizer".
- `perf.yaml` pins `transformers==4.57.6` alongside `aiperf==0.10.0` — the trtllm-runtime base ships transformers 4.55.0, and aiperf's `transformers>=4.56.0` floor would otherwise pull 5.x, which cannot load the `deepseek_v32` tokenizer (surfaces as "Failed to load tokenizer").
- Multi-node GB200 deployments need the ComputeDomain CR so the DRA scheduler co-locates worker pods on MNNVL-connected nodes; if you rename it, mirror the change in each `deploy.yaml` under `extraPodSpec.resourceClaims` and `resources.claims`.
- Background on the underlying optimizations: [Optimizing DeepSeek-V3.2 on NVIDIA Blackwell GPUs](https://nvidia.github.io/TensorRT-LLM/blogs/tech_blog/blog15_Optimizing_DeepSeek_V32_on_NVIDIA_Blackwell_GPUs.html).
- Source: [recipes/deepseek-v32-fp4](https://github.com/ai-dynamo/dynamo/tree/main/recipes/deepseek-v32-fp4)
Expand Down
2 changes: 1 addition & 1 deletion docs/benchmarks/kv-router-ab-testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -450,7 +450,7 @@ spec:
- -lc
- |
apt-get update -qq && apt-get install -y -qq tmux > /dev/null 2>&1
pip install -q aiperf==0.8.0
pip install -q aiperf==0.10.0
echo "Benchmark pod ready (tmux + aiperf installed)."
sleep infinity
imagePullPolicy: IfNotPresent
Expand Down
2 changes: 1 addition & 1 deletion docs/recipes/deepseek-v3-2-nvfp4.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ Because the replay is `--fixed-schedule`, throughput is fixed by the trace — t
## Notes

- The aggregated round-robin configuration (`trtllm/agg-round-robin/`) is the benchmark control arm; it appears on the Feature Benchmark page, not as a recipe target here.
- The benchmark job pins `transformers==4.57.6`: the trtllm-runtime base ships 4.55.0, and aiperf 0.6.0's `transformers>=4.56.0` floor would otherwise pull 5.x, which cannot load the `deepseek_v32` tokenizer (surfaces as "Failed to load tokenizer").
- The benchmark job pins `transformers==4.57.6`: the trtllm-runtime base ships 4.55.0, and aiperf's `transformers>=4.56.0` floor would otherwise pull 5.x, which cannot load the `deepseek_v32` tokenizer (surfaces as "Failed to load tokenizer").
- Decode workers run the WIDEEP MoE backend with `moe_expert_parallel_size: 8` and FP8 KV cache; prefill enables chunked prefill and KV block reuse (`tokens_per_block: 64`, matching `--kv-block-size 64`).
- The trace tops out at 109,459 input tokens; both engine configs set `max_seq_len: 121000` to accommodate it.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ spec:
- -c
- |
apt-get update && apt-get install -y curl jq procps git && apt-get clean
pip install aiperf==0.6.0;
pip install aiperf==0.10.0;
echo "aiperf installation completed";
sysctl -w net.ipv4.ip_local_port_range="1024 65000"
cat /proc/sys/net/ipv4/ip_local_port_range
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ spec:
# native support for model_type=deepseek_v32 (huggingface/transformers#41251)
# and fails AutoTokenizer.from_pretrained() before reading tokenizer.json.
# aiperf surfaces this as "Failed to load tokenizer". DYN-2878.
pip install "aiperf==0.6.0" "transformers==4.57.6"
pip install "aiperf==0.10.0" "transformers==4.57.6"
echo "aiperf installation completed"
sysctl -w net.ipv4.ip_local_port_range="1024 65000" 2>/dev/null || true
export COLUMNS=200
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ spec:
# native support for model_type=deepseek_v32 (huggingface/transformers#41251)
# and fails AutoTokenizer.from_pretrained() before reading tokenizer.json.
# aiperf surfaces this as "Failed to load tokenizer". DYN-2878.
pip install "aiperf==0.6.0" "transformers==4.57.6"
pip install "aiperf==0.10.0" "transformers==4.57.6"
echo "aiperf installation completed"
sysctl -w net.ipv4.ip_local_port_range="1024 65000" 2>/dev/null || true
export COLUMNS=200
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ spec:
# No apt-get: cluster runs us as non-root, and we don't actually need
# curl/jq — Python stdlib handles the readiness probe. Install aiperf
# into HOME=/tmp/.local via --user so it works under runAsNonRoot.
pip install --user -q aiperf==0.6.0 transformers==4.57.3 tokenizers==0.22.2
pip install --user -q aiperf==0.10.0 transformers==4.57.3 tokenizers==0.22.2

echo "Waiting for model at http://$ENDPOINT/v1/models..."
python3 - <<'PY'
Expand Down
2 changes: 1 addition & 1 deletion recipes/glm-5-nvfp4/sglang/disagg/efa/perf.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ spec:
# GLM-5 NVFP4 tokenizer uses the `TokenizersBackend` auto-tokenizer
# class introduced in transformers 5.x — pin to match the runtime
# image (see `pip show transformers` inside the sglang decode pod).
pip install aiperf==0.6.0 transformers==5.3.0 tokenizers==0.22.2
pip install aiperf==0.10.0 transformers==5.3.0 tokenizers==0.22.2

# Pre-save tokenizer to a flat directory once, then point aiperf at
# that path. Avoids each of aiperf's N record-processor subprocesses
Expand Down
2 changes: 1 addition & 1 deletion recipes/glm-5-nvfp4/sglang/disagg/perf.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ spec:
# stdlib is enough for the readiness probe.
# GLM-5's tokenizer_config.json uses tokenizer_class=TokenizersBackend,
# which is supported by Transformers v5.
pip install --user -q aiperf==0.6.0 transformers==5.8.0 tokenizers==0.22.2
pip install --user -q aiperf==0.10.0 transformers==5.8.0 tokenizers==0.22.2
Comment thread
pvijayakrish marked this conversation as resolved.

echo "Waiting for model at http://$ENDPOINT/v1/models..."
python3 - <<'PY'
Expand Down
2 changes: 1 addition & 1 deletion recipes/gpt-oss-120b/trtllm/agg/perf.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ spec:
- -c
- |
apt-get update && apt-get install -y curl jq procps git && apt-get clean
pip install git+https://github.com/ai-dynamo/aiperf.git@54cd6dc820bff8bfebc875da104e59d745e14f75;
pip install "aiperf==0.10.0";

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '--- perf.yaml context ---\n'
sed -n '1,120p' recipes/gpt-oss-120b/trtllm/agg/perf.yaml

printf '\n--- aiperf references in repo ---\n'
rg -n "aiperf|crick|build-essential|gcc|python:3.12-slim|arm64|aarch64" recipes/gpt-oss-120b -S || true

Repository: ai-dynamo/dynamo

Length of output: 245


🌐 Web query:

aiperf 0.10.0 PyPI Linux aarch64 crick sdist build toolchain install notes

💡 Result:

When installing AIPerf version 0.10.0 on Linux aarch64 (arm64) systems, you must install the system build toolchain before running pip install because the dependency crick is provided only as a source distribution (sdist) and requires a C compiler at installation time [1][2][3]. For Debian or Ubuntu-based distributions, install the necessary tools using [1][2][3]: sudo apt install build-essential For RHEL or CentOS-based distributions, use [1][2][3]: sudo yum groupinstall "Development Tools" Once the build toolchain is present, you can proceed with the standard installation in a virtual environment [1][2][3]: python3 -m venv venv source venv/bin/activate pip install aiperf This requirement is specific to Linux aarch64 environments. Other platforms, such as Linux x86_64, macOS, and Windows, typically install from pre-built wheels and do not require a system build toolchain [1][2][3].

Citations:


🏁 Script executed:

#!/bin/bash
set -euo pipefail

sed -n '1,120p' recipes/gpt-oss-120b/trtllm/agg/perf.yaml

printf '%s\n' '--- nearby aiperf references ---'
rg -n "aiperf|python:3.12-slim|build-essential|gcc|arm64|aarch64" recipes/gpt-oss-120b/trtllm/agg/perf.yaml recipes/gpt-oss-120b -S || true

Repository: ai-dynamo/dynamo

Length of output: 6733


Install a build toolchain before pip install aiperf on arm64.

This recipe targets ARM64 nodes, and aiperf==0.10.0 still needs a system compiler there because crick installs from sdist. python:3.12-slim doesn’t include gcc/build-essential, so the runtime install will fail unless the toolchain is added or aiperf is baked into the image.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@recipes/gpt-oss-120b/trtllm/agg/perf.yaml` at line 32, The perf recipe
installs aiperf directly in the runtime image, but on arm64 this can fail
because crick is built from sdist and the base image lacks a compiler toolchain.
Update the setup around the pip install in the perf.yaml recipe to ensure
gcc/build-essential (or an equivalent build toolchain) is installed before
invoking pip install aiperf, or otherwise use a prebuilt image with aiperf
already baked in; keep the change localized to the aiperf install step in this
recipe.

echo "aiperf installation completed";
sysctl -w net.ipv4.ip_local_port_range="1024 65000"
cat /proc/sys/net/ipv4/ip_local_port_range
Expand Down
2 changes: 1 addition & 1 deletion recipes/gpt-oss-120b/trtllm/disagg/perf.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ spec:
- -c
- |
apt-get update && apt-get install -y curl jq procps git && apt-get clean
pip install git+https://github.com/ai-dynamo/aiperf.git@4d3fa29403c8f75da22a14f1f7b3aeb27db9288f;
pip install "aiperf==0.10.0";
Comment thread
pvijayakrish marked this conversation as resolved.
echo "aiperf installation completed";
sysctl -w net.ipv4.ip_local_port_range="1024 65000"
cat /proc/sys/net/ipv4/ip_local_port_range
Expand Down
2 changes: 1 addition & 1 deletion recipes/kimi-k2.5/trtllm/agg-eagle-kv-router/perf.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ spec:
set -euo pipefail
ulimit -n 600000
apt-get update && apt-get install -y curl jq procps git && apt-get clean
pip install "aiperf==0.7.0" protobuf "transformers==4.57.3"
pip install "aiperf==0.10.0" protobuf "transformers==4.57.3"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== perf.yaml context =="
sed -n '1,140p' recipes/kimi-k2.5/trtllm/agg-eagle-kv-router/perf.yaml | cat -n

echo
echo "== search for aiperf/crick references =="
rg -n "aiperf|crick|arm64|aarch64|python:3.12-slim" recipes/kimi-k2.5/trtllm/agg-eagle-kv-router -S || true

Repository: ai-dynamo/dynamo

Length of output: 8115


🌐 Web query:

aiperf 0.10.0 PyPI Linux aarch64 crick sdist build toolchain arm64

💡 Result:

AIPerf version 0.10.0 is available on PyPI [1]. When installing aiperf on Linux aarch64 (ARM64) systems, you must install a system build toolchain before running pip install aiperf [2][3]. This requirement exists because one of the package's dependencies, crick, does not provide pre-built wheels for the aarch64 architecture and instead ships only a source distribution (sdist) [2][3]. Consequently, pip must compile the extension at install time, which requires a C compiler and development headers present on the host system [2][3]. To prepare your environment, install the appropriate build toolchain for your distribution: - Debian/Ubuntu: sudo apt install build-essential [2][3] - RHEL/CentOS: sudo yum groupinstall "Development Tools" [2][3] In contrast, Linux x86_64, macOS, and Windows environments typically install aiperf using pre-built wheels, which do not require a local build toolchain [2][3].

Citations:


Install a build toolchain before pip install on arm64. aiperf==0.10.0 pulls in crick, which only ships an sdist on Linux aarch64, so this python:3.12-slim job will fail during startup on arm64 unless you add build-essential/compiler headers first or pin the pod to amd64.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@recipes/kimi-k2.5/trtllm/agg-eagle-kv-router/perf.yaml` at line 54, The
startup command in the perf job installs a package set that can fail on arm64
because aiperf==0.10.0 brings in crick, which needs to build from source on
Linux aarch64. Update the job setup around the pip install step to either
install the required build toolchain and compiler headers before running pip, or
constrain this workload to amd64 via the relevant job/pod configuration. Use the
pip install command in the perf.yaml startup sequence as the anchor for the fix.

sysctl -w net.ipv4.ip_local_port_range="1024 65000" 2>/dev/null || true
export COLUMNS=200
EPOCH=$(date +%s)
Expand Down
2 changes: 1 addition & 1 deletion recipes/kimi-k2.5/trtllm/agg-eagle-round-robin/perf.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ spec:
set -euo pipefail
ulimit -n 600000
apt-get update && apt-get install -y curl jq procps git && apt-get clean
pip install "aiperf==0.7.0" protobuf "transformers==4.57.3"
pip install "aiperf==0.10.0" protobuf "transformers==4.57.3"
sysctl -w net.ipv4.ip_local_port_range="1024 65000" 2>/dev/null || true
export COLUMNS=200
EPOCH=$(date +%s)
Expand Down
2 changes: 1 addition & 1 deletion recipes/kimi-k2.5/trtllm/agg-round-robin/perf.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ spec:
set -euo pipefail
ulimit -n 600000
apt-get update && apt-get install -y curl jq procps git && apt-get clean
pip install "aiperf==0.7.0" protobuf "transformers==4.57.3"
pip install "aiperf==0.10.0" protobuf "transformers==4.57.3"
sysctl -w net.ipv4.ip_local_port_range="1024 65000" 2>/dev/null || true
export COLUMNS=200
EPOCH=$(date +%s)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ spec:
set -euo pipefail
ulimit -n 600000
apt-get update && apt-get install -y curl jq procps git && apt-get clean
pip install "aiperf==0.7.0" protobuf "transformers==4.57.3"
pip install "aiperf==0.10.0" protobuf "transformers==4.57.3"
sysctl -w net.ipv4.ip_local_port_range="1024 65000" 2>/dev/null || true
export COLUMNS=200
EPOCH=$(date +%s)
Expand Down
2 changes: 1 addition & 1 deletion recipes/kimi-k2.6/perf/perf.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ spec:
set -euo pipefail
ulimit -n 600000
apt-get update && apt-get install -y curl jq procps git && apt-get clean
pip install "aiperf==0.7.0" "protobuf==7.35.0" "transformers==4.57.3" "tiktoken==0.13.0"
pip install "aiperf==0.10.0" "protobuf==7.35.0" "transformers==4.57.3" "tiktoken==0.13.0"
sysctl -w net.ipv4.ip_local_port_range="1024 65000" 2>/dev/null || true
export COLUMNS=200
EPOCH=$(date +%s)
Expand Down
2 changes: 1 addition & 1 deletion recipes/llama-3-70b/vllm/agg/perf.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ spec:
- -c
- |
apt-get update && apt-get install -y curl jq procps git && apt-get clean
pip install git+https://github.com/ai-dynamo/aiperf.git@54cd6dc820bff8bfebc875da104e59d745e14f75;
pip install "aiperf==0.10.0";
echo "aiperf installation completed";
sysctl -w net.ipv4.ip_local_port_range="1024 65000"
cat /proc/sys/net/ipv4/ip_local_port_range
Expand Down
2 changes: 1 addition & 1 deletion recipes/llama-3-70b/vllm/disagg-multi-node/perf.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ spec:
- -c
- |
apt-get update && apt-get install -y curl jq procps git && apt-get clean
pip install git+https://github.com/ai-dynamo/aiperf.git@54cd6dc820bff8bfebc875da104e59d745e14f75;
pip install "aiperf==0.10.0";
echo "aiperf installation completed";
sysctl -w net.ipv4.ip_local_port_range="1024 65000"
cat /proc/sys/net/ipv4/ip_local_port_range
Expand Down
2 changes: 1 addition & 1 deletion recipes/llama-3-70b/vllm/disagg-single-node/perf.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ spec:
- -c
- |
apt-get update && apt-get install -y curl jq procps git && apt-get clean
pip install git+https://github.com/ai-dynamo/aiperf.git@54cd6dc820bff8bfebc875da104e59d745e14f75;
pip install "aiperf==0.10.0";
echo "aiperf installation completed";
sysctl -w net.ipv4.ip_local_port_range="1024 65000"
cat /proc/sys/net/ipv4/ip_local_port_range
Expand Down
2 changes: 1 addition & 1 deletion recipes/nemotron-3-super/perf/perf.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ spec:
set -euo pipefail
ulimit -n 600000
apt-get update && apt-get install -y curl jq procps git && apt-get clean
pip install "aiperf==0.7.0" "protobuf==7.35.0" "transformers==4.57.3" "tiktoken==0.13.0"
pip install "aiperf==0.10.0" "protobuf==7.35.0" "transformers==4.57.3" "tiktoken==0.13.0"
sysctl -w net.ipv4.ip_local_port_range="1024 65000" 2>/dev/null || true
export COLUMNS=200
EPOCH=$(date +%s)
Expand Down
2 changes: 1 addition & 1 deletion recipes/nemotron-3-ultra/perf/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -147,7 +147,7 @@ Edit the `env` block on the Job:
| `CONCURRENCY` | `18` | Single value; see [Running a Concurrency Sweep](#running-a-concurrency-sweep) |
| `TARGET_MODEL` | `nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4` | Must match the served model name on the DGD frontend |
| `TOKENIZER_PATH` | `/opt/models/patched/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4` | Used by AIPerf tokenization |
| `AIPERF_VERSION` | `0.8.0` | Matches the saved Ultra benchmark profile exports |
| `AIPERF_VERSION` | `0.10.0` | aiperf version used for benchmarking |
| `ROOT_ARTIFACT_DIR` | `/opt/models/perf` | Shared PVC artifact root |

## Artifacts
Expand Down
2 changes: 1 addition & 1 deletion recipes/nemotron-3-ultra/perf/perf.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -183,7 +183,7 @@ spec:
- name: AIPERF_HTTP_CONNECTION_LIMIT
value: "200"
- name: AIPERF_VERSION
value: "0.8.0"
value: "0.10.0"
- name: AIPERF_HTTP_SO_RCVTIMEO
value: "120"
- name: AIPERF_SERVICE_PROFILE_CONFIGURE_TIMEOUT
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ spec:
- -c
- |
apt-get update && apt-get install -y curl jq procps git && apt-get clean
pip install git+https://github.com/ai-dynamo/aiperf.git@54cd6dc820bff8bfebc875da104e59d745e14f75;
pip install "aiperf==0.10.0";
echo "aiperf installation completed";
sysctl -w net.ipv4.ip_local_port_range="1024 65000"
cat /proc/sys/net/ipv4/ip_local_port_range
Expand Down
2 changes: 1 addition & 1 deletion recipes/qwen3-235b-a22b-fp8/trtllm/agg/hopper/perf.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ spec:
- -c
- |
apt-get update && apt-get install -y curl jq procps git && apt-get clean
pip install git+https://github.com/ai-dynamo/aiperf.git@54cd6dc820bff8bfebc875da104e59d745e14f75;
pip install "aiperf==0.10.0";
echo "aiperf installation completed";
sysctl -w net.ipv4.ip_local_port_range="1024 65000"
cat /proc/sys/net/ipv4/ip_local_port_range
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ spec:
- -c
- |
apt-get update && apt-get install -y curl jq procps git && apt-get clean
pip install git+https://github.com/ai-dynamo/aiperf.git@54cd6dc820bff8bfebc875da104e59d745e14f75;
pip install "aiperf==0.10.0";
echo "aiperf installation completed";
sysctl -w net.ipv4.ip_local_port_range="1024 65000"
cat /proc/sys/net/ipv4/ip_local_port_range
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ spec:
- -c
- |
apt-get update && apt-get install -y curl jq procps git && apt-get clean
pip install git+https://github.com/ai-dynamo/aiperf.git@54cd6dc820bff8bfebc875da104e59d745e14f75;
pip install "aiperf==0.10.0";
echo "aiperf installation completed";
sysctl -w net.ipv4.ip_local_port_range="1024 65000"
cat /proc/sys/net/ipv4/ip_local_port_range
Expand Down
2 changes: 1 addition & 1 deletion recipes/qwen3-32b-fp8/trtllm/agg/perf.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ spec:
- -c
- |
apt-get update && apt-get install -y curl jq procps git && apt-get clean
pip install git+https://github.com/ai-dynamo/aiperf.git@54cd6dc820bff8bfebc875da104e59d745e14f75;
pip install "aiperf==0.10.0";
echo "aiperf installation completed";
sysctl -w net.ipv4.ip_local_port_range="1024 65000"
cat /proc/sys/net/ipv4/ip_local_port_range
Expand Down
2 changes: 1 addition & 1 deletion recipes/qwen3-32b-fp8/trtllm/disagg/perf.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ spec:
- -c
- |
apt-get update && apt-get install -y curl jq procps git && apt-get clean
pip install git+https://github.com/ai-dynamo/aiperf.git@54cd6dc820bff8bfebc875da104e59d745e14f75;
pip install "aiperf==0.10.0";
echo "aiperf installation completed";
sysctl -w net.ipv4.ip_local_port_range="1024 65000"
cat /proc/sys/net/ipv4/ip_local_port_range
Expand Down
2 changes: 1 addition & 1 deletion recipes/qwen3-32b-fp8/vllm/disagg/perf.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ spec:
- -c
- |
apt-get update && apt-get install -y curl jq procps git && apt-get clean
pip install git+https://github.com/ai-dynamo/aiperf.git@54cd6dc820bff8bfebc875da104e59d745e14f75;
pip install "aiperf==0.10.0";

@coderabbitai coderabbitai Bot Jun 29, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== perf.yaml =="
cat -n recipes/qwen3-32b-fp8/vllm/disagg/perf.yaml

echo
echo "== file stats =="
wc -l recipes/qwen3-32b-fp8/vllm/disagg/perf.yaml

echo
echo "== nearby references to aiperf =="
rg -n "aiperf|crick|aarch64|arm64|gcc|build-essential" recipes/qwen3-32b-fp8 -S || true

Repository: ai-dynamo/dynamo

Length of output: 7872


🌐 Web query:

aiperf 0.10.0 aarch64 crick sdist build toolchain pip install

💡 Result:

When installing AIPerf version 0.10.0 on Linux aarch64 (arm64) systems, you must install the system build toolchain before running pip install because the dependency crick ships only as an sdist (source distribution) and requires a C compiler to build during the installation process [1][2][3]. For common Linux distributions, you can install the necessary build toolchain using the following commands: - Debian/Ubuntu: sudo apt install build-essential [1][3] - RHEL/CentOS: sudo yum groupinstall "Development Tools" [1][3] This requirement is specific to Linux aarch64 architectures. On Linux x86_64, macOS, and Windows, AIPerf is installed via pre-built wheels and does not require a system build toolchain [1][2][4].

Citations:


🌐 Web query:

PyPI aiperf 0.10.0 crick aarch64 build dependencies

💡 Result:

When installing aiperf 0.10.0 on aarch64 (ARM64) Linux, the dependency crick does not provide a pre-built wheel, forcing an installation from the source distribution (sdist) [1][2][3]. As a result, you must have a C compiler and the system build toolchain installed on your host system before running pip install aiperf [1][2][4]. To satisfy these build requirements, you should install the necessary development tools for your specific Linux distribution: - For Debian or Ubuntu-based systems: sudo apt install build-essential [1][3] - For RHEL, CentOS, or compatible systems: sudo yum groupinstall "Development Tools" [1][3] For other platforms, such as Linux x86_64, macOS, and Windows, aiperf and its dependencies typically install from pre-built wheels, meaning no additional system-level C compiler is required [1][4].

Citations:


Install a build toolchain before pip install aiperf aiperf==0.10.0 pulls in crick, which builds from source on Linux arm64. python:3.12-slim only installs curl/jq/procps/git here, so this job will fail on arm64 unless you add build-essential or pin the job to amd64.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@recipes/qwen3-32b-fp8/vllm/disagg/perf.yaml` at line 22, The perf job
installs aiperf in the Qwen disagg setup, but the base image lacks a native
build toolchain needed when crick is built from source on Linux arm64. Update
the setup step around the pip install in the disagg perf workflow to install
build-essential (or equivalent compiler tooling) before invoking aiperf, or
otherwise constrain the job to amd64 so the install path does not require source
compilation.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added a comment
#NOTE: aiperf>=0.8.0 depends on crick (no prebuilt arm64 wheel); on arm64
# nodes the apt-get step above must include gcc and libc6-dev so pip can
# compile crick from source.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pvijayakrish, thanks for adding the note — that helps document the caveat.

I don’t think this fully resolves the issue, though. At Line 22 the job still installs aiperf==0.10.0 after only installing curl jq procps git, so on an arm64 node it would still fail unless the command also installs the compiler headers/toolchain.

If this job can run on arm64, please update the apt-get install line to include the required packages (for example gcc and libc6-dev, or build-essential). If it is guaranteed to run only on amd64, then constraining the job to amd64 would also address it.


🧠 Learnings used
Learnt from: karen-sy
Repo: ai-dynamo/dynamo PR: 7915
File: recipes/qwen3-32b-fp8/vllm/disagg/deploy.yaml:13-25
Timestamp: 2026-04-09T05:39:40.946Z
Learning: For vLLM disaggregated deployment recipes under recipes/**/vllm/disagg, avoid setting `envFromSecret: hf-token-secret` (HuggingFace token) on the Frontend service. The HuggingFace token secret should only be provided to worker components (e.g., `VllmPrefillWorker`, `VllmDecodeWorker` or any component that actually pulls/loads model weights). Since the Frontend only routes requests, it should not receive the HF token secret. In reviews, flag Frontend/HPA/frontend pods that include `envFromSecret: hf-token-secret` unless there is a clear, automated need to access model assets in the Frontend.

Learnt from: karen-sy
Repo: ai-dynamo/dynamo PR: 7915
File: recipes/qwen3-32b-fp8/vllm/disagg/deploy.yaml:13-25
Timestamp: 2026-04-09T05:39:40.946Z
Learning: For vLLM disaggregated deployment recipes in ai-dynamo/dynamo (files under recipes/**/vllm/disagg/), do not add the `--disaggregation-mode decode` flag to the `VllmDecodeWorker` args unless there is a confirmed routing/handler behavior change. Dynamo routes requests to the same decode handler regardless of whether the flag is explicitly set, so omitting the flag in these recipes is intentional and should not be treated as a bug during review.

Learnt from: tanmayv25
Repo: ai-dynamo/dynamo PR: 8584
File: lib/backend-common/examples/rotating_tokens/docker-compose.yml:41-48
Timestamp: 2026-04-23T01:22:51.416Z
Learning: In this repository, `bitnamilegacy/etcd` is an intentional, repo-wide convention for deployments. When reviewing YAML files (e.g., docker-compose, Helm `values.yaml`, ECS task/compose manifests), do not flag occurrences of `bitnamilegacy/etcd` as issues or deviations from the standard. Treat any future migration away from it as a holistic, coordinated change documented in `deploy/helm/charts/platform/README.md`, not as isolated per-file replacements.

Learnt from: ritazh
Repo: ai-dynamo/dynamo PR: 8355
File: recipes/qwen3.6-35b-a3b/vllm/agg/deploy.yaml:43-55
Timestamp: 2026-05-03T17:54:20.684Z
Learning: In this repository, vLLM deployment recipe YAMLs that reference the vLLM runtime image should use the planned stable image tag `nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.1.0` (which bundles vLLM 0.19.0+), not the dev tag `nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.1.0-dev.1`. Update image references accordingly in recipes under `recipes/qwen3.6-35b-a3b/` and other similar recipe paths once `1.1.0` is published.

Learnt from: hutm
Repo: ai-dynamo/dynamo PR: 8927
File: recipes/deepseek-r1/trtllm/agg/b200/perf.yaml:35-35
Timestamp: 2026-05-29T00:27:15.634Z
Learning: For benchmark/performance Kubernetes Job manifests named `perf.yaml` in the ai-dynamo/dynamo repo, reviewers should treat using `sysctl -w net.ipv4.ip_local_port_range="1024 65000"` inside the container’s shell command/entrypoint together with `securityContext.privileged: true` as an intentional repo-wide convention. Do not flag or suggest changing either setting on a per-file basis, since using pod-level `securityContext.sysctls` for `net.*` would require kubelet `--allowed-unsafe-sysctls` allowlisting that may not be available on target clusters. If you want to harden this behavior, propose it as a coordinated change across all `perf.yaml` files holistically (not a single-file deviation).

echo "aiperf installation completed";
sysctl -w net.ipv4.ip_local_port_range="1024 65000"
cat /proc/sys/net/ipv4/ip_local_port_range
Expand Down
2 changes: 1 addition & 1 deletion recipes/qwen3-32b/vllm/agg-round-robin/perf.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ spec:
apt update && apt install tmux wget curl jq -y

# Install benchmarking tool
pip install aiperf==0.6.0
pip install aiperf==0.10.0

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift

🧩 Analysis chain

🏁 Script executed:

sed -n '1,80p' recipes/qwen3-32b/vllm/agg-round-robin/perf.yaml

Repository: ai-dynamo/dynamo

Length of output: 2939


🌐 Web query:

aiperf 0.10.0 Linux aarch64 crick sdist build toolchain install notes

💡 Result:

When installing AIPerf 0.10.0 on a Linux aarch64 (ARM64) system, you must manually install a C build toolchain before running the installation command. This requirement exists because one of AIPerf's dependencies, crick, is distributed only as a source distribution (sdist) and requires compilation at install time [1][2][3]. To prepare your environment, install the appropriate build toolchain for your distribution: For Debian/Ubuntu systems: sudo apt install build-essential For RHEL/CentOS systems: sudo yum groupinstall "Development Tools" Once the toolchain is installed, you can proceed with the standard installation of AIPerf using pip: pip install aiperf Please note that this step is specific to Linux aarch64 environments. Other platforms, such as Linux x86_64, macOS, and Windows, receive pre-built wheels and do not require a local C compiler toolchain during the installation process [1][2][4].

Citations:


Add a build toolchain or pin this job to amd64

aiperf 0.10.0 needs a local C toolchain on Linux aarch64 because crick is source-only there. This pod only installs tmux, wget, curl, and jq into python:3.11, so arm64 runs will fail at pip install aiperf==0.10.0 unless you bake in the compiler tools or restrict the manifest to amd64.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@recipes/qwen3-32b/vllm/agg-round-robin/perf.yaml` at line 23, The perf job
setup in the aiperf install step assumes aarch64 can build source-only
dependencies, but the container currently only installs runtime tools and will
fail on arm64. Update the job definition around the pip install aiperf==0.10.0
step by either adding the required Linux build toolchain to the image/setup or
constraining this manifest to amd64 so the install path is only used on
supported architecture.


# Wait for model to be ready
echo "Waiting for model '${MODEL_NAME}' at http://${FRONTEND}:8000/v1/models..."
Expand Down
2 changes: 1 addition & 1 deletion recipes/qwen3-32b/vllm/disagg-kv-router/perf.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ spec:
apt update && apt install tmux wget curl jq -y

# Install benchmarking tool
pip install aiperf==0.6.0
pip install aiperf==0.10.0

# Wait for model to be ready
echo "Waiting for model '${MODEL_NAME}' at http://${FRONTEND}:8000/v1/models..."
Expand Down
2 changes: 1 addition & 1 deletion recipes/qwen3-vl-30b/vllm/agg-embedding-cache/perf.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ spec:
ulimit -u 65536

apt update && apt install -y tmux curl jq
pip install aiperf==0.6.0
pip install aiperf==0.10.0

echo "Waiting for model '${MODEL_NAME}' at http://${FRONTEND}:8000/v1/models..."
until curl -s "http://${FRONTEND}:8000/v1/models" | jq -e --arg model "${MODEL_NAME}" '.data[]? | select(.id == $model)' >/dev/null 2>&1; do
Expand Down
2 changes: 1 addition & 1 deletion recipes/qwen3-vl-32b-fp8/benchmark/perf.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ spec:
set -e
apt-get update && apt-get install -y curl jq procps git && apt-get clean

pip install aiperf==0.9.0
pip install aiperf==0.10.0

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '\n== perf.yaml relevant lines ==\n'
sed -n '1,260p' recipes/qwen3-vl-32b-fp8/benchmark/perf.yaml | nl -ba | sed -n '1,120p'

printf '\n== sweep_k8s/aiperf.py relevant lines ==\n'
sed -n '1,220p' benchmarks/frontend/scripts/sweep_k8s/aiperf.py | nl -ba | sed -n '1,140p'

printf '\n== frontend.Dockerfile relevant lines ==\n'
sed -n '1,120p' container/templates/frontend.Dockerfile | nl -ba | sed -n '1,80p'

printf '\n== search for aiperf / crick references ==\n'
rg -n "aiperf|crick|libc6-dev|gcc" recipes/qwen3-vl-32b-fp8/benchmark/perf.yaml benchmarks/frontend/scripts/sweep_k8s/aiperf.py container/templates/frontend.Dockerfile .ai -S || true

Repository: ai-dynamo/dynamo

Length of output: 226


🏁 Script executed:

#!/bin/bash
set -euo pipefail

python3 - <<'PY'
import json, urllib.request
urls = [
    "https://pypi.org/pypi/aiperf/0.10.0/json",
    "https://pypi.org/pypi/crick/0.0.8/json",
]
for url in urls:
    try:
        with urllib.request.urlopen(url, timeout=15) as r:
            data = json.load(r)
        print(f"== {url} ==")
        print("name:", data["info"]["name"], "version:", data["info"]["version"])
        files = data["releases"].get(data["info"]["version"], [])
        for f in files:
            print(f["filename"], f.get("packagetype"), f.get("python_version"))
    except Exception as e:
        print(f"FAILED {url}: {e}")
PY

Repository: ai-dynamo/dynamo

Length of output: 518


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '\n== perf.yaml relevant lines ==\n'
sed -n '1,260p' recipes/qwen3-vl-32b-fp8/benchmark/perf.yaml | cat -n | sed -n '1,120p'

printf '\n== sweep_k8s/aiperf.py relevant lines ==\n'
sed -n '1,220p' benchmarks/frontend/scripts/sweep_k8s/aiperf.py | cat -n | sed -n '1,140p'

printf '\n== frontend.Dockerfile relevant lines ==\n'
sed -n '1,120p' container/templates/frontend.Dockerfile | cat -n | sed -n '1,80p'

printf '\n== search for aiperf / crick / arm64 install deps ==\n'
rg -n "aiperf|crick|libc6-dev|gcc|aarch64|arm64" recipes/qwen3-vl-32b-fp8/benchmark/perf.yaml benchmarks/frontend/scripts/sweep_k8s/aiperf.py container/templates/frontend.Dockerfile .ai -S || true

Repository: ai-dynamo/dynamo

Length of output: 21825


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '\n== perf.yaml container/spec tail ==\n'
sed -n '120,260p' recipes/qwen3-vl-32b-fp8/benchmark/perf.yaml | cat -n

printf '\n== perf.yaml architecture/node scheduling hints ==\n'
rg -n "nodeSelector|affinity|arch|arm64|amd64|kubernetes.io/arch|node-name|tolerations" recipes/qwen3-vl-32b-fp8/benchmark/perf.yaml -S || true

printf '\n== PyPI simple pages for crick==0.0.8 and aiperf==0.10.0 (wheel filenames) ==\n'
python3 - <<'PY'
import ssl, urllib.request, re
ctx = ssl._create_unverified_context()
for url in [
    "https://pypi.org/simple/aiperf/",
    "https://pypi.org/simple/crick/",
]:
    try:
        with urllib.request.urlopen(url, context=ctx, timeout=20) as r:
            html = r.read().decode("utf-8", "replace")
        print(f"URL: {url}")
        for needle in ["aiperf-0.10.0", "crick-0.0.8"]:
            if needle in html:
                print(f"  contains {needle}")
        # print matching wheel links around versions of interest
        for pattern in [r'href="([^"]*aiperf-0\.10\.0[^"]*)"', r'href="([^"]*crick-0\.0\.8[^"]*)"' ]:
            hits = re.findall(pattern, html, flags=re.I)
            for h in hits[:20]:
                print(" ", h)
    except Exception as e:
        print(f"FAILED {url}: {e}")
PY

Repository: ai-dynamo/dynamo

Length of output: 8518


🏁 Script executed:

#!/bin/bash
set -euo pipefail

python3 - <<'PY'
import ssl, urllib.request, json, re
ctx = ssl._create_unverified_context()
for pkg, ver in [("aiperf","0.10.0"), ("crick","0.0.8")]:
    url = f"https://pypi.org/pypi/{pkg}/{ver}/json"
    try:
        with urllib.request.urlopen(url, context=ctx, timeout=20) as r:
            data = json.load(r)
        print(f"== {pkg} {ver} ==")
        files = data.get("releases", {}).get(ver, [])
        if not files:
            files = data.get("urls", [])
        for f in files:
            fn = f.get("filename", "")
            if fn:
                print(fn)
    except Exception as e:
        print(f"FAILED {url}: {e}")
PY

Repository: ai-dynamo/dynamo

Length of output: 1790


Install gcc/libc6-dev on arm64 before pip install aiperf==0.10.0.

aiperf is a pure-Python wheel, but its crick==0.0.8 dependency has no manylinux aarch64 wheel, so an unconstrained pod on python:3.12-slim can fail during the install step on arm64. Add the arm64 build deps here or pin the Job to amd64.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@recipes/qwen3-vl-32b-fp8/benchmark/perf.yaml` at line 31, The benchmark setup
in perf.yaml installs aiperf without ensuring arm64 build dependencies are
present, which can break on python:3.12-slim when crick==0.0.8 falls back to a
source build. Update the install step around the aiperf command to add gcc and
libc6-dev for arm64 before pip install, or otherwise constrain the job to amd64;
use the existing benchmark setup block to keep the change localized.

echo "aiperf installation completed"

export COLUMNS=200
Expand Down
18 changes: 8 additions & 10 deletions recipes/qwen3.6-35b/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -159,16 +159,14 @@ nodes live in a different AZ.

## aiperf install

We install aiperf from source pinned to a recent `main` SHA that
includes [PR 824](https://github.com/ai-dynamo/aiperf/pull/824)
(`feat(dataset): add session_id to single-turn for causal ordering`).
Our sliding-window dataset writes one row per `(user, turn)` with
`session_id=user_<N>`; PR 824 is what makes aiperf's `single_turn`
mode honor that ordering so prefix-cache hits across turns are real.

The pin lives in the shared `perf.yaml` (`AIPERF_GIT_REF` env var) at
the recipe root, applied identically across all three configs. Bump
when you want newer aiperf fixes.
We install `aiperf==0.10.0` from PyPI. This release includes
[PR 824](https://github.com/ai-dynamo/aiperf/pull/824)
(`feat(dataset): add session_id to single-turn for causal ordering`),
which makes `single_turn` mode honor `session_id` ordering so
prefix-cache hits across the 8 turns of a given user are real.

The version is pinned in `perf.yaml` and applies identically across
all three configs. Bump the `aiperf==` pin there to roll forward.

## Naming & ownership

Expand Down
Loading
Loading