[mmizutani]

Description and Motivation

The hybrid coordinator includes KpoolTailManager in its fine-grained lookup veto even though that scratch group opts out of prefix caching. This PR lets participating cache groups use 64-token lookup boundaries when the runtime layout supports them, instead of always falling back to the 3,584-token hybrid page.

GLM53_FINEGRAINED_APC=0 is the example, launcher, and runtime default. Set 1 and restart to opt in. Current evidence demonstrates a conditional benefit, but does not qualify default enablement or resolve earlier draft-acceptance concerns.

• Scope lookup compatibility to participating groups and verify scratch alignment from live manager/spec attributes. On this layout, fine enablement requires hash_block_size % index_kpool == 0. Participating coarse-only managers retain the fallback; unsafe or unverifiable scratch layouts refuse initialization when fine hits would otherwise be enabled.
• Validate existing patch-owned helper/gate regions canonically, including AST scope and owned-name bindings. Drift fails without rewriting the target. Remove the connector receipt and speculative upstream-fixed silent no-op detector.
• Retain exact 0/1 validation, explicit-empty rejection, kill switch, pre-stop artifact checks, both-rank wiring, and Dockerfile integration. Caller overrides use main's existing generic mechanism.
• Preserve APC: opt-in per-group retention, draft eviction priority, and safe DFlash replay #130 retention/replay behavior, including the 2,048-token DFlash replay requirement. The README documents the knob and tests; the 99-line feature note covers the contract, safety invariant, verification, and limitations.

Based on upstream main d8ad1831182412dfc7323d90f4ad91b8d636451e.

When enabling is worthwhile

A promising workload to test is a long, unchanged prefix that remains cached, followed by substantial new text—for example, an existing conversation followed by a large tool result. A usable fine-grained boundary must survive the retention and DFlash replay checks; a long conversation alone does not ensure a benefit.

Keep GLM53_FINEGRAINED_APC=1 only after repeated on/off comparisons on representative requests show a worthwhile improvement: more actual cached tokens and reduced time to first output and/or total response time, without regressions in answer correctness or draft acceptance. Check completed-request cache counters and timings; an enabled 64-token alignment or an intermediate replay-clamp target is insufficient evidence.

Leave 0 for mostly unrelated/cold requests or workloads without a demonstrated benefit. The core short-follow-up tests gained no additional reuse. The successful long-continuation probe had enough new input to exceed the tested 2,048-token replay window, but 2,048 is not a universal enablement threshold, and one enabled observation does not qualify long agentic workflows generally.

Related Issues

Successor to #84; preserves its A/B evidence and acceptance assessment. Addresses the implementation requests in the #171 review; broader runtime qualification remains open. Preserves #130.

---

Testing

Current local checks after diff reduction:

• 188 APC checks on both host and image, covering transactional drift rejection, runtime gate/flag handling, idempotence, and all six fine/hybrid/retention application orders.
• 199 launcher parity/order/pre-stop checks, plus existing caller-override and numeric-validation suites.
• Executable AST comparison of the overlay and actual generated coordinator against 18daea4: identical after excluding documentation. The launcher preamble exactly matches main. Shell syntax, diff whitespace, and relative documentation links passed.

The live verification record at 18daea4 contains the complete configuration, fixed payload protocol, metrics, and provenance deviations. Two CX7-connected Sparks ran TP=2, DFlash2 k=7/draft TP=2, adaptive-k off, FP8 KV, context 131,072, batch 7,168, max sequences 4, global retention unset, SWA retention 0, and replay window 2,048.

• 60 core requests, four fresh BAAB boots: both settings had 43.426% warm cache hits and no hits off the 3,584-token grid. Median warm TTFT was 3.985 s off / 3.936 s on. Acceptance-length pair differences had opposite signs (+0.088 / −0.026).
• Separate six-request AAB diagnostic: a 15,493-token continuation reused 11,904 tokens with fine hits, versus 7,168 in two controls; TTFT was 2.770 s versus 6.100/6.097 s. This is one enabled observation, excluded from BAAB aggregates.
• 66/66 output checks passed, with corresponding outputs byte-identical. Actual cache hits use isolated per-request Prometheus deltas, not intermediate replay-clamp targets.

These are local observations, not CI results or a general speedup guarantee. The cleanup adds no new GPU measurements. Internal KV/KDA/CoW equivalence, eviction pressure, KV-transfer connectors, long agentic-loop acceptance, and default-on qualification remain unproven by this run.

---

Checklist:

• Changes follow the README's default APC and retention guidance.
• Title and description describe the final opt-in implementation.
• Isolated boots and bounded behavior measurements are recorded with their limits.
• The new knob defaults to 0 with strict validation and a runtime kill switch.
• Updated the README, .env.example, and feature documentation.

[plotarmordev]

Holding this draft as the single tracked successor for the fine-grained prefix-cache work in #84. Independent source review and direct content comparison confirm that the overlay and design document are byte-identical across the two PRs; this branch carries the mechanism on the post-#130 base. Closing the older duplicate will not discard its measurements or credit.

Please carry forward the outstanding #84 change requests:

1. Default off at every layer: .env.example, launcher fallback, and overlay runtime fallback must all use 0. Retain strict 0/1 validation, explicit-empty rejection, and the runtime kill switch.
2. Preserve #130's per-group retention behavior and the runtime hash/kpool compatibility check. Keep both-rank wiring and pre-stop validation.
3. Validate patch-owned helper/gate regions canonically before accepting an already-patched file; token/sub-string presence alone is insufficient. Anchor drift must fail without modifying the target.
4. Remove the connector receipt and the upstream-fixed silent no-op detector requested for removal in #84. Retain meaningful host/in-image regression coverage and rebase onto current main after the independent fixes land.

The reviewed opt-in variant completed 292 observational requests across four conditions: each had the same 63/73 strict checker result and the same ten failures. This found no fine-only checker regression, but is not a fresh-boot BAAB qualification or an internal KV/KDA/CoW/replay numerical oracle. The earlier acceptance-length concerns remain unresolved; fine-grained APC is not approved as a default. Preserve #84's live receipts and A/B results as reference evidence, rather than resetting that review history.

[plotarmordev]

Reviewed — changes/evidence requested; do not merge yet

Maintainer source fixes are now pushed to this PR, preserving author history: 4ee3f3d04fb7. The integration includes main b3bd1f6928930727b0e9521478340dfd074b16c8 (including #184 and #51). Scoped CPU checks passed on this source. No fresh GPU/model qualification is claimed for this refreshed head; earlier runtime receipts remain tied to their recorded candidate and runtime identities.

Incorporated source fixes and author work

The integration preserves the author's 18daea4 safety corrections and 70106a9 simplification rather than restoring older duplicate implementations. Default-off behavior, strict flags, canonical/AST source validation and current replay/retention guards remain. The still-useful local addition is the Docker CPU build gate on pristine runtime source. Refreshed CPU checks passed, including all six fine/hybrid/retention overlay orders, retention/replay behavior, launcher validation and token defaults.

Evidence acknowledged; what remains open

Our earlier short-tail probe required 31,616 cached tokens but observed 28,672, after the replay clamp and reconciliation; we retained the guard and stopped that acquisition at its failed prerequisite.

The author's new 66-request report adds important evidence: the 60-request core BAAB study found no additional cache reuse; a separate six-request AAB diagnostic included an enabled long continuation with 11,904 cached tokens versus 7,168 in its controls. All 66 output checks passed as reported by the author. We have not independently reproduced that study. One enabled supplemental observation establishes a conditional lead, not a repeatable general or default-on benefit; it must stay separate from the core protocol.

Specific author changes/evidence requested

1. Do not weaken replay safety or try to force an unsafe fine hit for the old short-tail case. Keep such cases as negative controls, including completed-request counters after clamping/reconciliation.
2. Predeclare a representative eligible long-prefix/long-continuation workload and repeat balanced on/off controls to establish the conditional benefit. Preserve the supplemental study as exploratory evidence, not retroactive core membership.
3. Reconcile actual completed-request hit/query counters and draft-acceptance counters. An enabled 64-token alignment or intermediate clamp target is not a successful-hit receipt. Retain no-benefit cases too.
4. Supply targeted KV/KDA numerical and state-safety evidence, CoW/eviction pressure, kpool-remainder and drafter safety at the contexts being claimed. State unsupported distributed-connector, capacity and long-agentic-loop scope explicitly; a performance report cannot fill those gaps.
5. Keep the option default-off. Pin the final source/image and qualify the supported opt-in scope—not default enablement—before requesting re-review.

Required qualification before marking ready

Use the published TheGrill v0.2.0 experimental prerelease, from source 4166e9f0c4467ce516c8d35ae363a80abe7b6494. Install the native-target archive and verify that release's checksums and receipts. The published binaries were rebuilt by CI: earlier local staging hashes are not their identities. The release page is authoritative for downloads; the source-pinned installation guide retains pre-publication wording and older v0.1.0 examples.

Follow the measurement/evidence scope, claim-specific recipe guidance, and report template. Pin the executable, workload/producer, actual backend/runtime and model revisions, and declared settings. Qualify supported request controls before the prospective acquisition; use the full /v1/chat/completions endpoint. Keep unchanged-deployment controls and every failed/partial capture. Do not strip fields, replace rejected samples, change gates after seeing results, or treat missing usage as zero. Share a sanitized summary and evidence identities, not credentials, private prompts or deployment configuration.

For this PR, please keep it draft and mark it ready only after the requested changes and separate correctness/numerical checks pass, and the relevant TheGrill 0.2.0 reports complete and satisfy the predeclared PR-specific acceptance gates. Then request maintainer re-review of the exact final head. Exit 0 or DESCRIPTIVE alone is not a performance PASS; default C1 does not establish fairness, cache-state safety, capacity or kernel correctness. An inconclusive comparison is not a no-regression certificate, and an inconclusive unchanged-deployment performance control does not by itself prove a broken server. Investigate variation that prevents the claimed conclusion rather than discarding the control.

TheGrill publication is not blanket backend support or persisted-model-KV/eviction proof; unsupported claims need separate real-backend evidence or must remain explicitly unqualified. This is the review gate for this PR, not a claim that every project must adopt TheGrill.

Disposition: reviewed; changes/evidence requested; merge hold. Tooling availability is now resolved, so only awaiting-tooling-release is being removed. Draft status, merge-hold, changes-requested, reviewed and awaiting-test-evidence remain.

[mmizutani]

Pushed the rebased branch and review fixes. The PR remains an opt-in proposal, with GLM53_FINEGRAINED_APC=0 at every default layer. The updated title and description reflect the final implementation and current measurements.

Commit map

• 26332fae0cc6927fbda9df76125d7dad4fe15975: original PR mechanism rebased from c4c04ae929b9c1b056e9d4dc9baaea8336e676aa onto upstream main d8ad183.
• 18daea462848947dbd8999e121ca8ae9f7c56dcf: review corrections, regression coverage, and design-document clarification. This is the new PR head.

How the review requests were addressed

All implementation corrections below are in 18daea4; current-main behavior comes from the rebase above.

Review request	Change and reason
Default off everywhere	Changed .env.example, both launcher fallbacks, and the coordinator fallback to 0. Retained exact 0/1 validation, explicit-empty rejection, and the runtime kill switch. The evidence does not justify enabling this for everyone. Existing explicit =1 settings remain explicit opt-ins.
Preserve #130 and runtime safety checks	Kept current-main retention/replay/free-order behavior, including the 2,048-token replay requirement; retained runtime hash/kpool compatibility checks, participating-manager fallback, both-rank wiring, and pre-stop validation. No replay guard was weakened to produce finer hits.
Canonical validation of already-patched source	Replaced marker/sub-string acceptance with exact owned helper/gate regions plus AST scope/definition checks. Missing, changed, duplicated, or rebound owned names are rejected without rewriting the file. Regression cases cover altered helper/gate bodies, constants, imports, and rebinding. This is source-drift validation, not a Python security sandbox.
Remove obsolete detection/receipts and retain regression coverage	Removed the log-only connector receipt and speculative upstream-fixed silent no-op path. A changed upstream veto is now unsupported source drift. Added six-order composition coverage for fine/hybrid/per-group overlays, AST-equivalent coordinator/pool/manager results, and byte-idempotent reapplication. Fixed duplicate os import insertion exposed by composition order.
Preserve #84 history	Updated the design record to distinguish historical results from qualification of this candidate and retain the original A/B evidence and acceptance concerns.

Focused live verification, completed 2026-09-15 JST

Each label identifies a separate test run, starting from a fresh boot of the inference service across both CX7-connected DGX Sparks together:

• A means GLM53_FINEGRAINED_APC=0: the disabled control.
• B means GLM53_FINEGRAINED_APC=1: fine-grained hits enabled.
• 1 and 2 distinguish the first and second runs of that setting. They are run numbers, not machine or rank identifiers.

The actual execution order was:

Execution order	Run label	Setting
1	B1	First enabled run (1)
2	A1	First disabled run (0)
3	A2	Second disabled run (0)
4	B2	Second enabled run (1)

This is the BAAB (on/off/off/on) sequence referenced below. In the core results table, A1+A2 pools the two disabled runs and B1+B2 pools the two enabled runs.

The runs used isolated containers, launcher staging, API endpoint, and compile caches. Fixed TP=2, EXL3/TR3 4bpw target, DFlash2 k=7 and draft TP=2, adaptive-k off, FP8 KV, context 131,072, batch 7,168, max sequences 4, utilization 0.85, global retention unset, SWA retention 0, replay window 2,048. Only the fine-grained flag differs across the core arms. This does not qualify the full 850K configuration.

Each boot received the same 15 serialized requests: three roughly 9K–12K-token documents, each tested cold, repeated, with two fixed branches, and returned to the original prompt. Temperature 0, seed 171, thinking disabled, maximum output 256. Formatting/checksum checks are deterministic, and generated output is not fed back into the branch prompts.

Core warm metric, 24 requests per setting	Off: A1+A2	On: B1+B2
Median TTFT	3.985 s	3.936 s
Median wall time	6.054 s	5.896 s
Cached / queried tokens	114,688 / 264,100	114,688 / 264,100
Hit fraction	43.426%	43.426%
Drafts-weighted acceptance length	6.156	6.187
Actual hits off the 3,584-token grid	0	0

Acceptance length is 1 + sum(accepted_tokens) / sum(drafts). Enabled-minus-disabled boot-pair differences were +0.088 and −0.026; their opposite signs do not establish a repeatable improvement or resolve the earlier agentic acceptance concern.

The replay clamp explains why an enabled 64-token alignment is insufficient evidence of an actual fine hit. For example, a 9,005-token repeat logged a candidate of 8,960, clamped to 6,912, but the completed request recorded zero cached tokens. Roughly 12K short-tail requests ultimately reused 7,168 tokens. Intermediate clamp targets are not hit receipts. This agrees with the distinction raised in the earlier bounded qualification comment.

After B1 exposed this behavior, a separate fill/long-continuation pair was added for A1, A2, and B2 only. It is an AAB (off/off/on) diagnostic, excluded from the core BAAB results: an 11,966-token fill followed by a 15,493-token continuation, with enough new input to exceed the replay window.

Supplemental continuation	A1 off	A2 off	B2 on
Actual cached tokens	7,168	7,168	11,904
TTFT	6.100 s	6.097 s	2.770 s
Wall time	8.290 s	8.175 s	4.826 s
Acceptance length	4.789	5.056	5.056

11,904 is divisible by 64 but not by 3,584. The enabled observation reused 4,736 additional tokens, reducing fresh prefill from 8,325 to 3,589 tokens and TTFT by 54.6% relative to the mean controls. All three outputs were byte-identical. This demonstrates a conditional benefit, with only one enabled observation against two controls.

When enabling is worthwhile

A promising workload to test is a long, unchanged prefix that remains cached, followed by substantial new text—for example, an existing conversation followed by a large tool result. A usable fine-grained boundary must survive the retention and DFlash replay checks; a long conversation alone does not ensure a benefit.

Keep GLM53_FINEGRAINED_APC=1 only after repeated on/off comparisons on representative requests show a worthwhile improvement: more actual cached tokens and reduced time to first output and/or total response time, without regressions in answer correctness or draft acceptance. Check completed-request cache counters and timings; an enabled 64-token alignment or an intermediate replay-clamp target is insufficient evidence.

Leave 0 for mostly unrelated/cold requests or workloads without a demonstrated benefit. The core short-follow-up tests gained no additional reuse. The successful long-continuation probe had enough new input to exceed the tested 2,048-token replay window, but 2,048 is not a universal enablement threshold, and one enabled observation does not qualify long agentic workflows generally.

Verification receipts and limits

• Recorded final checks: 217 host APC, 208 in-image APC, 199 launcher parity/order/pre-stop checks; numeric validation, per-group retention, and in-image retention/hybrid regressions passed. The host's additional pristine-source leg accounts for the APC count difference. Shell syntax and diff-whitespace checks passed.
• 66/66 strict request checks passed. Corresponding core and supplemental outputs matched byte-for-byte across settings; core cold/repeat/return outputs also matched. No serving error/traceback/CUDA illegal-access/NCCL-error pattern was found in the eight rank logs.
• The API omits cached-token usage, so hits are actual per-request vllm:prefix_cache_hits_total deltas. Every query delta matched prompt usage, every success delta was exactly one, and running/waiting scheduler counters were checked before each request. Per-position draft acceptance counters reconciled with total accepted tokens.
• Both ranks used the same derived image and identical runtime coordinator/pool/manager bytes. A late patcher-only syntax-error-handling edit initially left B1's mounted patcher sources different; they were synchronized without changing loaded runtime. Later boots used identical final patchers. The measured coordinator contains one redundant import os; after removing it, its AST matches the final freshly generated coordinator, including all helpers, gates, and class bodies.
• Original environment files were backed up and preserved, original image tags were preserved, and test containers were removed, returning the pair to its initial stopped state. Raw payloads, streams, counters, source hashes, logs, and protocol deviations are retained locally; the aggregate results above are the public record of this run.

Conclusion: retain opt-in default 0. The toggle and a real fine-grid reuse benefit were demonstrated. These observations do not prove internal KV/KDA/CoW numerical equivalence, eviction-pressure behavior, distributed KV-transfer safety, long agentic-loop acceptance, or downside-free default enablement. The #84 A/B history and acceptance assessment remain relevant evidence.

[plotarmordev]

Changes/evidence requested on 4ee3f3d04fb7a61961af36ecb177a848d6e9ddff. Maintainer fixes are pushed and scoped CPU checks passed; this is not approval of the remaining runtime claims. Specific remaining changes, acknowledged positive/negative evidence, and exact readiness gates are in the updated maintainer handoff.

Use TheGrill v0.2.0 experimental prerelease with its published CI-built artifact checksums/receipts, not staging hashes. Keep the PR draft until its separate correctness/numerical checks and relevant predeclared performance gates are satisfied; then request exact-head re-review. Tooling availability is resolved, but the merge hold remains. No additional GPU qualification is claimed by this review.

[plotarmordev]

Hold update

Public head checked: 4ee3f3d04fb7a61961af36ecb177a848d6e9ddff.

Why it is held: The cache feature has a conditional benefit, not a general win. In our bounded integrated comparison, long append reused 12,032 rather than 10,752 tokens; medium append reused 24,000 rather than 21,504; short append fell to 7,168 rather than 10,752. The reference answers were correct. This is not proof of internal state or numerical safety.

What is already addressed: Default-off behavior, strict flags, both-rank wiring, and replay/retention safeguards were accepted in the source review. Your separate core BAAB study and exploratory long-continuation result remain distinct evidence; we are not relabeling one as the other.

Next action — me, @plotarmordev: Settle the supported scope and competing implementations (#125/#195), then pin the numerical/state-safety and performance checks. The outstanding qualification request is not cleared by the cache gains above.

Author action now: No immediate rebase or new hardware run. Keep the option off by default and retain the short/no-benefit cases. Once the plan is agreed, compare balanced on/off runs with completed-request counters and answer checks; separately check the claimed replay, eviction and state-safety behavior. Do not weaken safety guards to make a hit count pass.

Please keep the hold until the conditions above are met. When a new version is ready for review, include its commit ID, the checks actually run, their results, and anything not tested. Keep failed or incomplete runs in the report. Share only sanitized results—not private prompts, credentials, or machine configuration.

[plotarmordev]

Maintainer priority triage — 2026-09-26

Reviewed head: 4ee3f3d04fb7a61961af36ecb177a848d6e9ddff.

P2, retained for a maintainer scope/qualification decision—not for merging duplicate lookup code. Main #251 now implements participating-group fine-grained lookup plus replay/state handling, so the original opt-in implementation is no longer an independent feature to land as-is. Outstanding state-safety/quality questions are not dismissed by that overlap or by closing the lean duplicate #125. Next owner: @plotarmordev decides whether to move the remaining qualification work into a distinct current-main task and archive this PR, or define a genuinely non-overlapping delta. That explicit disposition and preserved safety gates unblock further work. No author rebase/benchmark request now.

This is prioritization only; no merge or implementation work is authorized by it.

[plotarmordev]

Archiving this PR, per the 2026-09-26 maintainer triage. Main #251 now implements participating-group fine-grained lookup with replay/state handling, so this PR's lookup code would duplicate it. The unresolved state-safety and qualification questions, and your evidence, move to #300 so they aren't lost. Thank you @mmizutani for the mechanism, the safety corrections and the 66-request study; they shaped what landed.