fix(recipes): add build tools for ARM recipes on aiperf>=0.8.0 (#11278)

[karen-sy]

Fixes DYN-3396 and #11061 by making ARM-advertised benchmark recipes able to install aiperf==0.10.0 on ARM nodes for relevant recipes.

Cherry picks #11278

---

[devin-ai-integration]

Devin Review found 3 potential issues.

View 2 additional findings in Devin Review.

[devin-ai-integration]

🔴 Tokenizer version pin removed for DeepSeek-V3.2, likely breaking tokenizer loading

The required version pin for the tokenizer library is removed (pip install "aiperf==0.10.0" at recipes/deepseek-v32-fp4/trtllm/agg-round-robin/perf.yaml:53) without addressing the known incompatibility, so the benchmark will fail to load the tokenizer.

Impact: The DeepSeek-V3.2 aggregated benchmark job will crash at startup with a "Failed to load tokenizer" error.

Detailed mechanism: transformers 5.x lacks deepseek_v32 model_type support

The previous code at both recipes/deepseek-v32-fp4/trtllm/agg-round-robin/perf.yaml and recipes/deepseek-v32-fp4/trtllm/disagg-kv-router/perf.yaml explicitly pinned transformers==4.57.6 with a detailed comment explaining the issue (see the deleted lines in the diff):

Pin transformers==4.57.6 (verified to load the deepseek_v32 tokenizer). The trtllm-runtime base ships transformers 4.55.0; aiperf 0.6.0's transformers>=4.56.0 floor would otherwise upgrade to 5.x, which lacks native support for model_type=deepseek_v32 (huggingface/transformers#41251) and fails AutoTokenizer.from_pretrained() before reading tokenizer.json. aiperf surfaces this as "Failed to load tokenizer".

The new code removes this pin entirely, doing just pip install "aiperf==0.10.0". Since aiperf has a transformers>=4.56.0 floor dependency, pip will resolve to the latest available transformers (likely 5.x), which lacks native support for model_type=deepseek_v32.

The tokenizer is loaded from HuggingFace (--tokenizer "nvidia/DeepSeek-V3.2-NVFP4"), meaning AutoTokenizer.from_pretrained() is called and the model_type issue applies.

Notably, other recipes in this same PR still pin their transformers versions (e.g., kimi-k2.5 pins 4.57.3, glm-5 pins 5.3.0/5.8.0, dsv4-pro pins 4.57.3), suggesting this removal was unintentional.

Suggested change

	pip install "aiperf==0.10.0"
	pip install "aiperf==0.10.0" "transformers==4.57.6"

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

🔴 Tokenizer version pin removed for DeepSeek-V3.2 disaggregated recipe, likely breaking tokenizer loading

The required version pin for the tokenizer library is removed (pip install "aiperf==0.10.0" at recipes/deepseek-v32-fp4/trtllm/disagg-kv-router/perf.yaml:53) without addressing the known incompatibility, so the benchmark will fail to load the tokenizer.

Impact: The DeepSeek-V3.2 disaggregated benchmark job will crash at startup with a "Failed to load tokenizer" error.

Detailed mechanism: same transformers 5.x incompatibility as the agg recipe

This is the same issue as in the agg-round-robin recipe. The previous code pinned transformers==4.57.6 with a comment explaining that transformers 5.x lacks native support for model_type=deepseek_v32 (huggingface/transformers#41251). The pin was removed, so pip will resolve to 5.x, causing AutoTokenizer.from_pretrained("nvidia/DeepSeek-V3.2-NVFP4") to fail.

Suggested change

	pip install "aiperf==0.10.0"
	pip install "aiperf==0.10.0" "transformers==4.57.6"

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

🔍 Non-root security constraints removed to enable apt-get

Two recipes (recipes/deepseek-v4/deepseek-v4-pro/vllm/disagg/gb200/perf.yaml:24-26 and recipes/glm-5-nvfp4/sglang/disagg/perf.yaml:16-18) had their runAsNonRoot: true, runAsUser: 1000, and runAsGroup: 1000 pod security context removed. The old code explicitly ran as a non-root user and used pip install --user to work within those constraints. The new code runs as root to enable apt-get install build-essential. This is a deliberate trade-off to support aarch64 builds, but it weakens the pod's security posture. If the cluster enforces PodSecurity admission (e.g., restricted profile), these pods will now be rejected. Consider whether a node affinity to x86_64 nodes (where crick has prebuilt wheels) would be preferable for environments with strict security policies, or whether a custom base image with build tools pre-installed could avoid needing root.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[datadog-official]

⚠️ Warnings

🚦 1 Pipeline job failed

Docs link check | lychee     

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 0fcb9d1 | Docs | Give us feedback!}