ci(buildkit): probe pod ordinals up to the KEDA replica ceiling (OPS-7960)

[dmitry-tokarev-nv]

Summary

get_active_indices() in .github/scripts/route_buildkit.sh discovers builder pods by resolving buildkit-<arch>-<i> over DNS for i in [0, MAX_POD_CHECK). That bound was 10, matching the KEDA maxReplicaCount at the time it was written.

Raising the builder pool above 10 without raising this leaves the extra pods undiscovered: they start, hold a Karpenter node, and never receive a build, while the pods below the bound absorb the entire queue. This bumps the default to 16 and makes it environment-overridable.

Prerequisite for ai-dynamo/velonix#545, which takes the pool from 9 to 15 pods per arch. Raising this first is safe on its own — ordinals with no pod behind them fail to resolve and are skipped, so on today's 9-pod fleet the behavior is unchanged apart from a few extra NXDOMAIN lookups.

Why it matters beyond the pod count

compute_group_pools() splits discovered pods into three cache groups — vllm, sglang, and general+trtllm — via pool_size=$(( (count + 2) / 3 )). The split is driven by count, so a stale MAX_POD_CHECK silently pins the group sizes too.

Running the real algorithm over contiguous fleets (it reproduces the pools seen in production logs at n=9 — {6,7,8} amd64, {0,2,8} arm64):

pods	vllm	sglang	general+trtllm	overlap
9	3	3	3	none
15	5	5	5	none
16	6	6	6	yes — 18 slots over 16 pods

Note 16 is not a better 15: pool_size rounds up, so three pools of 6 over 16 pods forces two pods into two cache domains at once. Multiples of 3 stay disjoint while the group count is 3.

Measured motivation

First-attempt build outcomes, stratified sample of 14 PR runs/day over 2026-07-24..07-29 (670 build jobs; runs cancelled by a superseding push excluded, since those cancel long builds before they can time out):

day	vllm	sglang	general	trtllm
07-24	0%	0%	5%	0%
07-25	0%	0%	5%	0%
07-26	11%	4%	33%	12%
07-27	0%	0%	5%	0%
07-28	0%	0%	32%	17%
07-29	0%	0%	28%	23%

Timeouts are confined to the shared general+trtllm group. Successful builds in that group have a median of 9–17 min against 30/45/60 min caps, so the failures are contention, not work that outgrew its budget.

Validation

• bash -n .github/scripts/route_buildkit.sh passes.
• MAX_POD_CHECK unset resolves to 16; exported value takes precedence.
• Extracted compute_group_pools verbatim and ran it over contiguous fleets; at n=9 it reproduces the exact pools observed in production build logs, confirming the harness matches the deployed algorithm.
• No behavior change on the current 9-pod fleet: ordinals 9–15 do not resolve and are skipped by the existing nslookup guard.

Linear: https://linear.app/nvidia/issue/OPS-7960/increase-dynamo-builder-buildkit-warm-pool-from-9-to-15-pods-per-arch

🤖 Generated with Claude Code

---

Summary by CodeRabbit

• Bug Fixes
  • Improved BuildKit pod discovery by allowing DNS probing across a larger range of pod ordinals.
  • Added clearer guidance around pod discovery limits.

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

[coderabbitai]

Walkthrough

The BuildKit routing script increases the default DNS pod probing limit from 10 to 16 and documents that the value is an exclusive upper bound for pod ordinals.

Changes

BuildKit DNS probing

Layer / File(s)	Summary
Update pod probe boundary .github/scripts/route_buildkit.sh	Raises MAX_POD_CHECK from 10 to 16 and documents the exclusive probing limit.

Estimated code review effort: 1 (Trivial) | ~2 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately summarizes the main change: raising BuildKit pod ordinal probing to the KEDA ceiling.
Description check	✅ Passed	The description is detailed and covers the change, rationale, validation, and issue context, even if it doesn't follow the template exactly.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/scripts/route_buildkit.sh:
- Around line 149-152: Validate MAX_POD_CHECK immediately after its default
assignment and before the probe loop, rejecting malformed or non-positive values
with a clear error and nonzero exit. Ensure the accepted upper bound remains
aligned with the buildkit StatefulSet’s KEDA maxReplicaCount so discovery covers
every permitted pod ordinal.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 5a619722-a467-41c3-8abd-566303896be2

📥 Commits

Reviewing files that changed from the base of the PR and between ce54943 and ba9567c.

📒 Files selected for processing (1)

• .github/scripts/route_buildkit.sh

[datadog-official]

⚠️ Warnings

🚦 3 Pipeline jobs failed

Docs link check | lychee     

PR | backend-status-check     

PR | frontend / Build multi-arch cpu     

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: ba9567c | Docs | Datadog PR Page | Give us feedback!}