Repository navigation
fix(warmup): cover long-prefill metadata kernels in boot shape warmup - #170
Conversation
The only kernel that JIT-compiled during live traffic in a 5 h agent serve was BuildPrefillChunkMetadataKernel (jit_monitor WARN, hours after boot): the warmup ladder only exercised <=248-token prompts, so the first long request paid the compile on the user's path. - PREFILL_S=(3584 7168 14336 65536) long-prefill rungs; prefill does not affect the DFlash decode BLOCK shapes (decode is one query). - stage tokenize/completions payloads in files: 64k-token prompts exceed ARG_MAX when passed as a curl argument. - ignore the staged .json payload files in the end-of-run tally. Verified on 2x DGX Spark GB10: prior serve's only mid-serve JIT disappears with the new rungs; boot warmup reports 24/24 requests.
609af28 to
3d070f8
Compare
plotarmordev
added
P2
Review follow-up —
|
plotarmordev
removed
the
awaiting-test-evidence
mk_ladder_prompt appended to a growing string once per rung. The s=65536 rung made that quadratic — ~2 GiB of copies — and the three CPU-only smoke runs each spent 111-119s building/exercising prompts with the inert WARMUP_CURL seam and no model latency. Format reuse in a single printf emits the identical bytes in 0.04s at s=65536. Same prompt either way: n copies of "hello", single-space separated, no trailing space (checked against the old builder for n<=16384 and by sha256 at 65536; a trailing-space variant is rejected by the new test). tests/test_boot_shape_warmup.py runs the shipped script end-to-end with WARMUP_CURL stubbed (no network, GPU or timing thresholds): the 9 ladder/prefill prompts arrive byte-exact at the API incl. s=65536, the tally holds at 24/24, a one-rung /tokenize mismatch and a small-context refusal still warm the other 23 shapes before exiting 1.
plotarmordev
left a comment
plotarmordev
left a comment
There was a problem hiding this comment.
Exact-head approval: 2cdf21e. Independently reviewed all carried changes and Main CPU evidence: byte-exact prompts through65536, 24/24 tally, mismatch/refused64K each23/24 exit1 with continuation. Fixed rungs and nonfatal launcher-WARN policy accepted; printf/seq removes quadratic construction without changing prompt bytes. No new GPU/live-boot or specialization-completeness qualification claimed. Publication worktree and pushed commit range scope/secret/deployment-content checks found no issues; no source edits by SIX.
… contrasts Review follow-up on 81c83fc. Documentation scope only: the recorded tables, the probes and every measured number are unchanged (B3-B7 table and the explicit 0/0 rows verified byte-identical by hash). - Scope note: this PR adds the document and the three probes only; the launcher, overlay, .env, README and warmup changes named below are local-checkout or separate unmerged-PR work. - 12.3: replace "B6 isolates the cause" with whole-stack wording; the B3-vs-B4/B5/B6 contrast does not by itself identify the responsible mechanism and is not a basis for replacing the replay guard main ships (MiaAI-Lab#130's versioned DFlash SWA replay clamp). - 11 / 12.1: label each component's status (PR MiaAI-Lab#83 implementation vs main's MiaAI-Lab#130 rework, unmerged PR MiaAI-Lab#84, local vLLM #54163 cherry-pick, local launcher/.env/README edits). - 5 + TL;DR row 3: the long-prefill warmup rungs are separate, unmerged PR MiaAI-Lab#170, not main; main's script still has no PREFILL_S rungs. - 4.4 / 12: present "the cliff stopped reproducing" as a whole-stack observation rather than a root cause. No new run requested or performed; no gates changed; no new performance claims.
Summary
The only kernel that JIT-compiled during live traffic in a 5 h agent serve was
BuildPrefillChunkMetadataKernel(jit_monitorWARN, hours after boot): the boot warmup ladder only exercised ≤ 248-token prompts, so the first long request paid the compile on the user's path.PREFILL_S=(3584 7168 14336 65536)long-prefill rungs; prefill does not affect the DFlash decode BLOCK shapes (decode is one query).ARG_MAXwhen passed as a curl argument (curl: Argument list too long)..jsonpayload files in the end-of-run outcome tally.Historical author verification
BuildPrefillChunkMetadataKernelat 06:17 was the only compile on the request path.BuildPrefillChunkMetadataKernelcompile during subsequent long-prompt probes.bash -nclean.Notes
The author reported that the 64 k rung takes ~35 s at boot on this kit. > 128 k prompts can still compile one more specialization; lower/raise
PREFILL_Sif a deployment wants a smaller warmup budget.Maintainer review and CPU qualification (2026-09-15)
Final head
2cdf21e1e452f6bb1daa41d2a9eb85766c280bedretains the fixed prefill rungs and existing nonfatal launcher-WARN policy, including for deployments that reject the 64K rung. The prompt builder now uses the existing printf/seq idiom rather than repeated growing-string appends.Main ran
python3 tests/test_boot_shape_warmup.pyat this exact head: exit 0 in 3.63 s across the three inert-curl scenarios; normal 24/24 with all nine rung prompts byte-exact through 65536, tokenizer mismatch 23/24 with exit 1 and expected/reported counts 7168/7169, and refused 64K 23/24 with exit 1 while other requests continue. The original unmodified script took 111–119 s in concurrent inert CPU runs; those were not a single-threaded live-boot benchmark.SIX independently reviewed the full carried script, regression and documentation changes and the exact-head receipt; scope, secret and deployment-content scans found no issues. This establishes prompt construction and failure/tally behavior, not new proof of runtime specialization coverage, complete warmup, GPU latency or the author-reported boot timing. No GPU/live-server run was performed for this landing.