Conversation
Re-adds the per-tier cache hit breakdown that was previously reverted in feat/umbp-pr (commit 001e9ce): PrefillAdder tracks L1 (GPU device), L2 (host DRAM), and L3 (storage) hit tokens plus cache misses, reported via SchedulerStats and exported as sglang:cache_hit_tokens_l{1,2,3}_total / cache_miss_tokens_total Prometheus counters. Adapted to current code: uses req.storage_hit_length directly (no getattr) and req.extend_range instead of the since-removed req.extend_input_len. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Visualizes the L1/L2/L3 per-tier cache hit metrics from the previous commit: 'Cache Hit Rate (Per-Tier)' (percentage view split by tier) and 'Cached Tokens Rate (hit tokens/s)' (existing sglang:cached_tokens_total counter, restored alongside it as a natural companion). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
TianDi101
requested review from
Ying1123,
fzyzcjy,
hnyls2002,
merrymercy,
sufeng-buaa and
xiezhq-hermann
as code owners
July 21, 2026 04:21
Contributor
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
21 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Split out of #31879 (general per-tier hit counters only, no UMBP-specific or eviction/load-back changes).
Re-adds the per-tier cache hit breakdown that was previously reverted in
feat/umbp-pr(001e9ce) and never carried forward:PrefillAddertracks L1 (GPU device), L2 (host DRAM), and L3 (storage) hit tokens plus cache misses, reported viaSchedulerStatsand exported assglang:cache_hit_tokens_l{1,2,3}_total/cache_miss_tokens_totalPrometheus counters. Adapted to current code: usesreq.storage_hit_lengthdirectly (nogetattr) andreq.extend_rangeinstead of the since-removedreq.extend_input_len.Backend-agnostic —
req.storage_hit_lengthis set generically byscheduler.pop_prefetch_loaded_tokens()regardless of which L3 storage backend (mooncake, hf3fs, file, UMBP, ...) served the prefetch.Also restores the corresponding Grafana panels: "Cache Hit Rate (Per-Tier)" (new metric, percentage view split by tier) and "Cached Tokens Rate (hit tokens/s)" (existing
sglang:cached_tokens_totalcounter, restored alongside it as a natural companion). Uses"datasource": {"default": true}rather than a hardcoded UID so it works on any fresh Grafana instance.Test plan
--enable-metrics --enable-hierarchical-cacheand confirmsglang:cache_hit_tokens_l1/l2/l3_totalandcache_miss_tokens_totalpopulate correctly, and the two Grafana panels renderCI States
Latest PR Test (Base): ❌ Run #29801015638
Latest PR Test (Extra): ❌ Run #29801015521