fix(planner): bound Kubernetes API calls with a request timeout

[aylonsk]

Overview:

None of the planner's Kubernetes calls set a request timeout. Most run synchronously on the planner's event loop, so an apiserver connection that is accepted but never answered blocks the planner, including its control API, forever. This PR bounds every call with a (connect, read) timeout of (5, 30) seconds. It mirrors what #13245 did for Prometheus.

Summary

• Add a module constant REQUEST_TIMEOUT = (5, 30) in connectors/clients/kubernetes_api.py. Pass it as _request_timeout= to all six calls there and to the worker-metadata list in connectors/kubernetes.py.
• kubernetes-client only honors an int or a 2-tuple; a bare float is silently ignored. So the new test also pins the tuple shape.
• urllib3 is added to STUB_MODULES so the marker-report pre-commit hook can collect test_kube.py, the same as fix(planner): keep control APIs responsive during Prometheus collection #14377 did for aiohttp.test_utils.

Details:

With the timeout, a request that never gets an answer now raises instead of blocking:

• GET: MaxRetryError after 4 × 30 s = 120 s, because urllib3's default Retry(3) retries reads (measured 120.2 s).
• PATCH: raw ReadTimeoutError after 30 s, because urllib3 does not retry writes. The apiserver may already have applied the write.

How each caller reacts, unchanged by this PR:

• Startup and per-tick refresh: the error propagates out of the run loop, which only catches GPUShapeUnavailableError. The planner exits and the pod restarts, just as for other Kubernetes API errors today.
• Scaling path (_apply_effects): the error reaches _mark_effect_submission_unknown, which disables automatic scaling until the planner restarts. This is the existing response to a scaling call with an unknown outcome. Previously a stalled call never got that far, because the loop was blocked. Note the trade-off:
  • A stall that eventually recovers, such as a two-minute partition with TCP still alive, previously resumed by itself. It now leaves automatic scaling off until the planner restarts, with an ERROR log line.
  • A connection that never answers previously hung the planner and its control API forever.

Values and policy:

• urllib3's retry policy is left as is. fix(kubernetes): add client-go retry semantics #13247 is defining retry semantics for this client.
• 30 s is below the apiserver's 60 s request deadline. An apiserver whose first byte takes 30-60 s now fails after 4 abandoned requests, where it previously succeeded. (5, 60) would avoid that at the cost of a 240 s stall; I'm happy to switch if you prefer.
• This conflicts with fix(kubernetes): add client-go retry semantics #13247 at the touched call sites. After a merge, its new scale pre-read, the scale PATCH closure and the relocated worker-metadata list would also need the timeout.

Validation

• New test_stalled_apiserver_raises_instead_of_blocking in test_kube.py. It points a real client at a socket that accepts the connection and never answers, and expects MaxRetryError within 10 s. Six existing exact-kwargs assertions gain _request_timeout=REQUEST_TIMEOUT.
• KubernetesAPI.get_graph_deployment against that socket: still blocked after 20 s on main; MaxRetryError after 120.2 s with this change.
• pytest components/src/dynamo/planner/tests in a Linux container with ai-dynamo-runtime 1.6.0.dev20261006: 1498 passed, 2 failed. The same 2 tests (test_wait_for_unacknowledged_decision) fail on main because the container has no etcd binary. components/src/dynamo/global_planner/tests: 75 passed.
• mypy --explicit-package-bases components/src/dynamo/planner (the CI invocation, scoped to the planner): no issues.
• pre-commit hooks on the commit, including pytest-marker-report: passed.

Where should the reviewer start?

components/src/dynamo/planner/connectors/clients/kubernetes_api.py: REQUEST_TIMEOUT and its call sites.

Related Issues

• Closes [BUG]: Planner Kubernetes API calls have no request timeout, so a stalled apiserver connection hangs the planner #15759

Summary by CodeRabbit

• Bug Fixes
  • Added bounded connection and read timeouts to Kubernetes operations, including deployment and pod queries, scaling changes, and worker metadata retrieval. This helps prevent requests from hanging indefinitely.
• Tests
  • Added coverage to verify Kubernetes requests use the configured timeout and that stalled API calls fail within a set limit.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

👋 Hi aylonsk! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository: ai-dynamo/dynamo/.coderabbit.yaml
• Review profile: CHILL
• Plan: Enterprise
• Run ID: eb0c0a0a-5438-4eca-ba0a-b17fe3b75680

📥 Commits

Reviewing files that changed from the base of the PR and between 7d82c10 and 9218187.

📒 Files selected for processing (4)

• components/src/dynamo/planner/connectors/clients/kubernetes_api.py
• components/src/dynamo/planner/connectors/kubernetes.py
• components/src/dynamo/planner/tests/unit/test_kube.py
• tests/report_pytest_markers.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 9 remain after this review.

---

Walkthrough

Planner Kubernetes API calls now use a shared (5, 30) second connect/read timeout. The worker-metadata list request uses the same timeout. Tests check timeout propagation and verify that a stalled API server request raises MaxRetryError.

Changes

Planner Kubernetes Request Timeouts

Layer / File(s)	Summary
Define and apply the shared request timeout components/src/dynamo/planner/connectors/clients/kubernetes_api.py, components/src/dynamo/planner/connectors/kubernetes.py, components/src/dynamo/planner/tests/unit/test_kube.py, tests/report_pytest_markers.py	KubernetesAPI defines and passes a shared (5, 30) timeout to its Kubernetes requests. The worker-metadata custom-object list request also uses it. Tests check timeout propagation and stalled-server behavior. The marker test stubs include urllib3.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Severity of issue fixed: Medium

Merge Risk: ⚪ Minimal · up to 92181

The scoped Kubernetes requests now have finite timeouts, and no unresolved merge-blocking concern is identified.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: adding bounded timeouts to planner Kubernetes API calls.
Description check	✅ Passed	The description covers the required Overview, Details, reviewer starting point, and Related Issues sections. It includes the required issue reference, Closes #15759, and provides relevant implementati…
Linked Issues check	✅ Passed	Issue #15759 requires bounded timeouts for DGD get/list/patch, DGDSA scale get/patch, pod list, and the worker-metadata list. The PR passes REQUEST_TIMEOUT = (5, 30) to each of these calls. The sock…
Out of Scope Changes check	✅ Passed	The changes stay within issue #15759. The urllib3 marker-report stub supports collection of the new test that imports urllib3; the remaining changes add or verify the requested request timeouts.
Docstring Coverage	✅ Passed	Docstring coverage is 87.50% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 16 functions across 4 files.

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

_{Comment @coderabbitai help to get the list of available commands.}