Conversation
Populate prefetch_bandwidth (batch_get) and backup_bandwidth (batch_set) StorageMetrics from UMBPStore, drained by HiRadixCache under --enable-metrics, so sglang exports sglang:prefetch_bandwidth / sglang:backup_bandwidth for the leaderboard L3 get/put bandwidth cards. Aggregate-then-drain pattern (append per-op, clear on get_stats) matches mooncake/hf3fs; guarded by enable_storage_metrics so no overhead when metrics are off. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Re-adds the per-tier cache hit breakdown that was previously reverted in feat/umbp-pr (commit 001e9ce): PrefillAdder tracks L1 (GPU device), L2 (host DRAM), and L3 (storage) hit tokens plus cache misses, reported via SchedulerStats and exported as sglang:cache_hit_tokens_l{1,2,3}_total / cache_miss_tokens_total Prometheus counters. Adapted to current code: uses req.storage_hit_length directly (no getattr) and req.extend_range instead of the since-removed req.extend_input_len. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds back the panels that were dropped in feat/umbp-pr (001e9ce) and never carried forward: per-tier Cache Hit Rate (percentage view), HiCache Host Utilization, HiCache Host Used/Total Capacity, L3 Prefetch/Writeback Rate, L1<->L2 Eviction/Load-back Rate, Cached Tokens Rate, GPU KV Cache Utilization, and the L1<->L2 / L3 PCIe bandwidth panels. Also fixes the datasource.yaml URL, which was pinned to a stale host from a prior run. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Re-adds the HiCache observability that was reverted in feat/umbp-pr (001e9ce) and never carried forward, now adapted to the current async write/load pipeline: - sglang:evicted_backuped_tokens_total / evicted_regular_tokens_total: split of evicted_tokens_total by whether the node had already been backed up to host (cheap detach) vs dropped with no host copy. - sglang:gpu_kv_cache_occupancy: alias gauge of token_usage, kept for HiCache dashboards that query it under the old name. - sglang:eviction_bandwidth_gb_s / load_back_bandwidth_gb_s: real per-batch PCIe bandwidth. HiCacheAck already carried CUDA start/finish events bracketing the D2H/H2D copy in cache_controller.py's start_writing()/start_loading(); they just weren't created with enable_timing=True and had no token count attached. Restored both (LayerLoadingEvent too, since load-back reuses its start/finish pair) and added a num_tokens field to HiCacheAck so writing_check()/ loading_check() can derive GB/s from start_event.elapsed_time(finish_event) and mem_pool_host.get_size_per_token() at ack-drain time. - sglang:backuped_bytes_total / prefetched_bytes_total: byte-scaled counterparts of the existing backuped_tokens_total/prefetched_tokens_total. HiCacheAck gaining a 4th field required widening the ack-queue unpacking in unified_radix_cache.py, hi_mamba_radix_cache.py, and decode_kvcache_offload_manager.py, which share the same queue but don't need the new metrics. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… template vars) Same fix as pushed to fix/grafana-dashboard-datasource-and-vars — applying directly on this branch since it's the shared checkout every deploy/bench skill (deploy-1p1d-deepseek, bench-batch-direct-repro, etc.) mounts via NFS_BASE=/apps/ditian12 for Grafana provisioning. Leaving this as an uncommitted working-tree diff meant any git checkout/pull/reset on this shared path would silently revert it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Point Prometheus datasource at the correct node IP and register a separate UMBP dashboards provider so mori dashboards load alongside the default sglang ones. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
TianDi101
requested review from
Ying1123,
alphabetc1,
fzyzcjy,
hanming-lu,
hnyls2002,
hzh0425,
ispobock,
merrymercy,
sufeng-buaa,
xiezhq-hermann and
yizhang2077
as code owners
July 21, 2026 03:44
Contributor
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
This was referenced Jul 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Re-adds HiCache observability that was reverted in
feat/umbp-pr(001e9ce) and never carried forward, adapted to the current async write/load pipeline and reconciled withtiming_enabled/make_timing_event_pair()(the portability guard main already added for backends without CUDA/HIP event-timing support):PrefillAddernow tracks L1 (GPU device), L2 (host DRAM), and L3 (storage) hit tokens plus misses, exported assglang:cache_hit_tokens_l{1,2,3}_total/cache_miss_tokens_total.sglang:evicted_backuped_tokens_total/evicted_regular_tokens_total(split by whether the node was already backed up before eviction),sglang:gpu_kv_cache_occupancy(alias gauge for dashboards querying the old name),sglang:eviction_bandwidth_gb_s/load_back_bandwidth_gb_s(real per-batch PCIe bandwidth derived fromHiCacheAck.num_tokens+ CUDA/HIP event timing, gated ontiming_enabled), andsglang:backuped_bytes_total/prefetched_bytes_total(byte-scaled counterparts of the existing token counters).cache_controller.py,hybrid_cache_controller.py) to also usemake_timing_event_pair()instead of unguardedenable_timing=True, matching the load path's existing portability handling, and to carrytiming_enabledthroughHiCacheAckso the new bandwidth metrics degrade gracefully on backends without event-timing support.UMBPStorenow populatesprefetch_bandwidth/backup_bandwidth, drained byHiRadixCacheunder--enable-metricsand exported assglang:prefetch_bandwidth/sglang:backup_bandwidth.Deliberately excludes the
sglang:prefetch_wait_stall_mshistogram (separate concern, not included in this PR).Test plan
--enable-metrics --enable-hierarchical-cacheand confirm the new Prometheus counters/histograms populate and the restored Grafana panels renderCI States
Latest PR Test (Base): ❌ Run #29799315685
Latest PR Test (Extra): ❌ Run #29799315587