[krunkosaurus]

Reviewed source publication — HOLD, not merge-ready

Exact head 4ffef088edb18797ba2763a84e38fd520d39a0f7 was fast-forward published to the original author branch, preserving history. Integrated base: a9bbbadc4c7253870d0d519c3d445e4ee7b97540. This is older-base source publication, not current-main integration qualification. Later public main changes, including the TP=3/TP=4 default decisions, must be preserved in a subsequent normal integration; this publication does not overwrite main or establish current default policy.

Current source scope

Retain the single canonical installer, policy-progress checks, existing concurrency canary and concurrency documentation. The installer is byte-identical to #80, accepting the supported v6 opt-in helper without introducing a second policy implementation. This branch does not change launcher defaults or land the proposed TP=2 CHUNK=0 default. The concurrency canary is not a completed contention study; no new live result is claimed.

Strict installer migration and recovery requirement

One shared installer authenticates published v1/v2/v5 helper bytes, validates exact helper/gate regions before replacement, requires round-trip consistency and verifies current v6 idempotence. Main independently matched the three registry SHA-256/length entries against public producer source, not only the test fixtures. Drifted, partial, duplicated, decorated and marker-only inputs are refused without a write.

v3/v4 migration is unsupported and fails closed without rewriting the source. No authenticated producer for those helper bodies was recovered; method names or invented fixtures do not establish canonical provenance. An operator encountering these markers must restore/reinstall the pinned pristine or an authenticated supported scheduler through an approved maintenance procedure before applying v6; do not erase markers or bypass validation. This is a deliberate narrowed support contract, not a claim that every historical variant migrates.

CPU evidence and remaining hold

Main supplied the pinned-scheduler installer plus 30 fair-policy checks and five corrected concurrency-policy checks. Independent HELP review accepted the shared strict installer delta. CPU policy progress is not live fairness, throughput or latency qualification. This final reviewer read the supplied evidence and focused deltas; no tests, live endpoints, model/GPU/node/service operations were run here. Historical runtime receipts remain tied to their recorded source/image and are not qualification of this head.

Next-action owner: maintainer plotarmordev. Complete normal current-main integration while retaining current public defaults, obtain exact-head cumulative source/CPU clearance, and schedule the predeclared supported protocol/resource window. #80 live opt-in thresholds and #180 long-prefill/active-decode contention, separate correctness and fairness/latency/throughput admission remain unqualified. The prospective study must retain failed/partial captures and unchanged controls; the earlier maintainer wrapper HTTP 404 produced zero measured waves and is not a scheduler regression or a passed gate. Main owns wrapper readiness and protocol scheduling, not the author.

No unqualified TP=2 CHUNK=0 production default may land through #149 or #180. No immediate author rebase or new benchmark is requested. merge-hold remains and #180 remains draft; this is a COMMENT source review, not approval or merge authorization.

Prior PR description and historical proposal — preserved; superseded by the current scope above

Description and Motivation

With GLM53_MIXED_PREFILL_CHUNK=skip, an agent processing a new prompt can receive zero prefill tokens for the entire time another agent generates. This also affects requests already admitted to the running list. A client can exhaust its idle timeout despite available sequence slots and KV memory.

Default the launcher, example configuration, and scheduler overlay to stock chunked prefill (0) so new prompts can use the remaining step budget during generation. Preserve explicit skip / -1 and positive caps. Reapplying the overlay also upgrades the known legacy default in existing images. The launcher propagates the same setting to both TP=2 ranks.

This favors concurrent prompt progress and can reduce an individual stream's decode throughput during mixed work. The docs explain that tradeoff, migration of existing .env files, the existing --long-prefill-token-threshold option for sharing a small batch budget, and the difference between a per-request context limit and total resident KV capacity.

Add CPU regressions and a single live canary that requires a short request to answer before a long response finishes. The canary checks completed streams, the expected short answer, and overlapping content timestamps; serialized HTTP 200 responses do not pass.

Related Issues

Related to #6. The mixed-prefill throughput tradeoff remains available as explicit decode protection; this change addresses the starvation caused by making that policy the default.

---

Testing

Validated the exact published head, 14b6a9f6d4b6256ef4a51dade72581c4cf4a3175:

• Eight CPU concurrency regressions pass, including installed-scheduler upgrade and idempotence checks on a temporary copy of the deployed scheduler.
• Existing scheduler-overlay and launcher-override checks pass.
• bash -n passes separately for start.sh, stop.sh, and download.sh; changed Python files compile; git diff --check passes.

Live runtime validation used two DGX Sparks at clean upstream commit f906ee990596486e10ddbe381efa6f0e496f77e3, built with the original Dockerfile into image sha256:232573f48556c301b516ebc86a651501ed4f5c69aa72ef957168ff5a6ecd4fdf. The runtime applies this PR's concurrent-prefill policy through explicit configuration; the published PR source is covered by the regressions above.

The recipe uses GLM53_MIXED_PREFILL_CHUNK=0, LONG_PREFILL_TOKEN_THRESHOLD=512, a 1,024-token batch budget, a 1,000,000-token per-request limit, four sequence slots, GPU memory utilization 0.87, FP8 KV, DFlash k=7 with draft TP=2, and the upstream FAT/grouped/temp-row/rightsize performance settings. The engine reports 2,070,680 total KV-cache tokens. A separate LiteLLM stream-lifetime backport is installed, and GLM-route, proxy, and Hermes stale/read timeouts are 1,800 seconds.

• Decode overlap: A completed 1,024 output tokens. B received a distinct 8,037-token prompt and answered OK in 9.013 seconds, with first content at 8.837 seconds. B answered 13.014 seconds before A's final content.
• Long-prefill overlap: a 999,095-input-token request returned all three exact head/middle/tail markers in 858.149 seconds, with first content at 856.319 seconds. Its short peer returned OK in 1.455 seconds while the large prompt was still processing.
• Native endurance: two persistent Hermes CLI processes ran from 1:00 AM to 6:17 AM PT on September 13, 2026, for 5 hours 16 minutes 57 seconds. They completed 251 and 249 independently validated file jobs, including eight and seven automatic compressions. Both lanes performed useful work for more than 5.25 hours. Their peak observed prompts were 494,550 and 494,443 tokens. Final duration, receipt, compression-continuation, terminal-error, and observation audits passed; serving and gateway container identities remained unchanged with zero restarts or OOM events. Main-agent thinking remained enabled; only compression used thinking disabled.
• Dual near-1M capacity: two distinct prompts containing 999,090 and 999,094 input tokens were submitted together. Both streams completed with every head/middle/tail marker correct. First output arrived at 1,572.371 and 1,448.644 seconds; completion took 1,572.876 and 1,452.146 seconds. Two active backend requests were observed, with zero preemptions and zero prefix-cache hits. Peak active KV usage was 87.09%; this establishes concurrent completion, not both full contexts resident at the same instant.

These are functional canaries and an operational endurance qualification of the whole deployed recipe, not a controlled throughput comparison or a guarantee of bounded latency under arbitrary load. Hermes normally compresses around 500K tokens; the separate capacity canary tests near-1M API requests.

---

Checklist:

• I have read the README and kept my changes consistent with it.
• My pull request has a sound title and description (not something vague like Update README.md).
• My change is reproducible and verified (a boot and, for behavior changes, measurements).
• Defaults still work out of the box; a new knob has a sane fallback consistent with the existing ones.
• I updated the README (and docs/ where relevant) for any knob, default, or measured number I changed.

[plotarmordev]

Reviewed — changes/evidence requested; do not merge yet

Maintainer source fixes are now pushed to this PR, preserving author history: 587b39e05806. The integration includes main b3bd1f6928930727b0e9521478340dfd074b16c8 (including #184 and #51). Scoped CPU checks passed on this source. No fresh GPU/model qualification is claimed for this refreshed head; earlier runtime receipts remain tied to their recorded candidate and runtime identities.

Incorporated source fixes

The scheduler patch now validates its canonical helper and both gates, rejects missing/drifted/duplicated definitions without rewriting the target, and compiles successful rewrites. Current-main/TP3 changes are integrated without broadening TP4 policy. Refreshed prefill-concurrency, pinned-scheduler, mixed-policy, long-prefill-threshold, launcher override, numeric and rank-parity checks passed.

Why performance is still unqualified

The failed acquisition was my harness error, not a demonstrated scheduler regression. My wrapper supplied the server origin instead of the full completion URL; the first A warmup returned HTTP 404. There were zero measured waves. The old operator also failed to propagate capture rejection and began B loading; it was interrupted and restored safely before any B capture. A2 was not captured. No replacement acquisition was used to turn this into a pass.

Next actions — maintainer harness correction, then scheduled qualification

1. Maintainer (plotarmordev) owns this wrapper correction. Use the full completion-resource URL and stop the orchestration when the collector reports invalid/incomplete acquisition; process exit success alone must not advance the study or establish qualification. The current tests/check_concurrent_agents.py already constructs the completion-resource path and fails closed on invalid/non-overlapping streams. This is not outstanding author source work or an alleged product bug.
2. Provide a prospectively bounded, relevant long-prefill plus active-decode contention study. Exercise suitable C1/C2/C4 cells and record per-request progress/starvation, TTFT, settlement latency and aggregate throughput; report tail observations without claiming statistically unsupported production p95/p99 values.
3. Compare stock/skip/bounded-chunk settings while holding unrelated configuration fixed. Include unchanged controls, functional concurrent-request checks and all failed/partial receipts. A uniform short-request C1 result alone does not validate this feature's fairness claim.
4. Submit the exact-head reports against declared fairness/latency/throughput acceptance criteria. Missing performance evidence—not the old HTTP 404—is the remaining qualification blocker.

Required qualification before marking ready

Use the published TheGrill v0.2.0 experimental prerelease, from source 4166e9f0c4467ce516c8d35ae363a80abe7b6494. Install the native-target archive and verify that release's checksums and receipts. The published binaries were rebuilt by CI: earlier local staging hashes are not their identities. The release page is authoritative for downloads; the source-pinned installation guide retains pre-publication wording and older v0.1.0 examples.

Follow the measurement/evidence scope, claim-specific recipe guidance, and report template. Pin the executable, workload/producer, actual backend/runtime and model revisions, and declared settings. Qualify supported request controls before the prospective acquisition; use the full /v1/chat/completions endpoint. Keep unchanged-deployment controls and every failed/partial capture. Do not strip fields, replace rejected samples, change gates after seeing results, or treat missing usage as zero. Share a sanitized summary and evidence identities, not credentials, private prompts or deployment configuration.

For this PR, please keep it draft and mark it ready only after the requested changes and separate correctness/numerical checks pass, and the relevant TheGrill 0.2.0 reports complete and satisfy the predeclared PR-specific acceptance gates. Then request maintainer re-review of the exact final head. Exit 0 or DESCRIPTIVE alone is not a performance PASS; default C1 does not establish fairness, cache-state safety, capacity or kernel correctness. An inconclusive comparison is not a no-regression certificate, and an inconclusive unchanged-deployment performance control does not by itself prove a broken server. Investigate variation that prevents the claimed conclusion rather than discarding the control.

TheGrill publication is not blanket backend support or persisted-model-KV/eviction proof; unsupported claims need separate real-backend evidence or must remain explicitly unqualified. This is the review gate for this PR, not a claim that every project must adopt TheGrill.

Disposition: reviewed; changes/evidence requested; merge hold. Tooling availability is now resolved, so only awaiting-tooling-release is being removed. Draft status, merge-hold, changes-requested, reviewed and awaiting-test-evidence remain.

Coordinated scheduler scope and explicit unblock condition

The maintainer selects stock CHUNK=0 as the eventual direction across #149/#180, but that default must not bypass this PR's missing contention qualification by landing through #149. Preserve the unique validator, canary and concurrency documentation in one eventual integration. Retain #80 as opt-in warm/deadline work, with both policies inactive unless explicitly enabled, reuse main's caller-overrides, and extend canonical helper/gate validation for supported v2 forms without weakening fail-closed rejection.

Next-action owner: maintainer (plotarmordev) for the acquisition-wrapper correction, integration and prospective protocol/resource scheduling. The author is not being asked for another immediate run or to fix already-correct canary/source behavior. The study outline above is a qualification requirement, not a fully specified executable experiment: the maintainer must first pin supported controls, workload, limits and acceptance criteria. After valid acquisition and separate correctness checks meet those predeclared gates on the final head, the maintainer performs exact-head re-review before readiness or any production default landing. Until then the draft/evidence/merge hold remains; no gates have passed by virtue of this correction.

[plotarmordev]

Changes/evidence requested on 587b39e058069a87bf39d02eb2cc8ea32bdf0a7e. Maintainer fixes are pushed and scoped CPU checks passed; this is not approval of the remaining runtime claims. Specific remaining changes, acknowledged positive/negative evidence, and exact readiness gates are in the updated maintainer handoff.

Use TheGrill v0.2.0 experimental prerelease with its published CI-built artifact checksums/receipts, not staging hashes. Keep the PR draft until its separate correctness/numerical checks and relevant predeclared performance gates are satisfied; then request exact-head re-review. Tooling availability is resolved, but the merge hold remains. No additional GPU qualification is claimed by this review.

[plotarmordev]

Source-only final review at 4ffef08: normal fast-forward verified; strict authenticated v1/v2/v5 migration and v6 reapplication accepted against Main CPU/public-producer receipts and independent HELP review. Unsupported v3/v4 refuse without mutation and require approved restore/reinstall, not marker removal. Gate knobs remain opt-in 0/0, deadline means eligibility only, and warm observations are exploratory. Older-base publication does not qualify current main or replace later TP3/TP4 defaults. Maintainer plotarmordev owns current-main integration, exact-head cumulative checks and authorized runtime/contended qualification; no author homework, numerical approval or merge. HOLD remains.

[plotarmordev]

Hold update

Public head checked: 4ffef088edb18797ba2763a84e38fd520d39a0f7.

Why it is held: The live contention check did not establish the progress we need. Request B produced no content before request A finished its 1,024-token output, about 21 seconds later. B was cancelled. Both HTTP responses were 200; that alone was not a fairness pass.

What we tested: 30 CPU scheduler checks passed. The bounded live run used a private integrated candidate and the earlier scheduler settings. It was not a test of the newer defaults from #194, and it does not by itself identify the cause.

Next action — me, @plotarmordev: Integrate the selected scheduler work and define the exact contention comparison before scheduling it. Use overlapping long-prefill and active-decode requests, unchanged controls, fixed limits, and per-request progress, time to first output and completion time. Count an unfinished request as incomplete, not success.

Author action now: None. Your source/migration fixes were accepted in the current-head review. I am clearing the old native change request; this remains a maintainer-owned qualification hold, not merge approval.

Please keep the hold until the conditions above are met. When a new version is ready for review, include its commit ID, the checks actually run, their results, and anything not tested. Keep failed or incomplete runs in the report. Share only sanitized results—not private prompts, credentials, or machine configuration.

Status correction: source changes have been accepted; the remaining hold is maintainer-owned protocol/integration/qualification work. No new author run or rebase is requested. See the latest current-head hold update. This dismissal is not merge approval.

[plotarmordev]

Maintainer hold update — original #180; tracking #283

Reviewed public head: 4ffef088edb18797ba2763a84e38fd520d39a0f7 (draft, conflicts with current main). The prior source-only acceptance of strict authenticated helper migration and policy/canary remains scoped to that head and its recorded base. It is not a current-main runtime, throughput, numerical or final performance qualification. Preserve the existing unsupported-v3/v4 fail-closed contract and published defaults during integration with #246 and #221.

A private upstream-intended combined candidate received independent source-only review and CPU checks (379 passed / 65 skipped); both baseline and candidate TP2 images booted. In the first frozen baseline natural-pressure run, native evidence observed a selected WAITING allocator refusal, but all three trials were invalid (one truncated trace, two W stream timeouts). Neither candidate C nor the return-to-baseline control was scored. No TheGrill performance repetitions were run. The historical contention hold therefore remains; HTTP 200, boot success and CPU checks do not constitute its fairness/latency/throughput pass.

Next owner: maintainer @plotarmordev until @hecisaza accepts the invitation to be single integration lead in #283. Unblock: integrate strict migration/current-main policy without unintended default changes, obtain valid published-recipe native correctness and claim-specific TheGrill v0.2.0 performance evidence under a prospective protocol, then independent exact-final-head review and normal original-PR merge. Compatible contributor hardware is fine. The failed acquisition is retained, not replaced or blamed on this PR's author. No immediate author rebase or benchmark request; draft/merge hold stays in force.

[hecisaza]

Landed through #309: the strict migration contract, the concurrency canary and its CPU regressions. The canary passed 5/5 in every arm. Thanks @krunkosaurus. The TP=2 CHUNK=0 default stays out, as agreed. Performance cells that the baseline's own run-to-run spread left inconclusive are tracked in #308. Closing as landed.