Skip to content

Fix NVIDIA workload startup settings and Docker readiness - #81

Merged
khluu merged 1 commit into
mainfrom
codex/fix-nvidia-startup-failures-567
Sep 16, 2026
Merged

khluu merged 1 commit into
mainfrom
codex/fix-nvidia-startup-failures-567

Conversation

@khluu

@khluu khluu commented Sep 15, 2026

Copy link
Copy Markdown
Member

Fix two NVIDIA workload startup settings and stop holding GPU hosts after a Docker server has exited.

This PR was authored with assistance from OpenAI Codex (GPT-6).

Changes

  • Cap GLM-5.3 on 8x H200 at 65,536 tokens. In build #567, its default 1,048,576-token context required 53.45 GiB of KV cache with only 18.94 GiB available. The cap covers the configured 8K-input/1K-output benchmark and GSM8K sequence length. It changes the server context limit; full BFCL behavior under the cap still needs GPU validation.
  • Replace DeepSeek-V4-Flash B200's removed use_fp4_indexer_cache=True flag with --attention-config.indexer_kv_dtype=mxfp4, preserving the old setting's meaning. Build #568 reached a GPU host but failed argument validation on the old flag.
  • Track the Docker server container during readiness checks and return failure when it stops or disappears under --rm. In #567, GLM's engine failed around 13:40 UTC but readiness held the host until 14:20 UTC. Keep the native-process check, reset runtime tracking at startup, and try to print the last container logs on failure.
  • Add regression coverage for stopped/removed containers, a running container becoming healthy, and native readiness without Docker. Run the lifecycle suite in the CPU pipeline-generation step and document readiness behavior.

Validation

  • Server lifecycle tests: 7/7 passed (Docker operations mocked; existing health test uses localhost HTTP).
  • Pipeline generation: 21/21 passed.
  • Benchmark repetitions: 8/8 passed.
  • Ingestion authentication: 2/2 passed.
  • Parser smoke checks passed for both modified workloads with a stubbed lm-eval task registry.
  • shellcheck lib/server.sh, shell syntax checks, git diff --check, and Buildkite's full online pipeline-schema validation passed.
  • GPU performance, GSM8K, and BFCL runs have not been launched for this PR.

Remaining failures from #567

This addresses perf-eval configuration and lifecycle behavior. Completing NVIDIA recovery also requires:

  • Capacity for DeepSeek's single-node 8-GPU request; #567's pod expired while unschedulable.
  • An image containing vLLM #56825 for GLM-5.3-Flash's CUDA graph/event startup crash.
  • The MSA compatibility revision from MSA #12, integrated by the still-open vLLM #56545, for MiniMax-M3's missing QuACK symbol.

AMD workload definitions and GPU scheduling policy are unchanged.

Cap GLM-5.3 H200 context, migrate the DeepSeek FP4 indexer flag, and fail readiness promptly for stopped or removed Docker containers. Run lifecycle regression tests in the CPU pipeline step.

Co-Authored-By: GPT-6 <noreply@openai.com>
Signed-off-by: khluu <khluu000@gmail.com>
@khluu
khluu merged commit 7ee0fae into main Sep 16, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant