[TianDi101]

What this does

A UMBP peer measured everything and, without a master, published none of it. This branch makes a masterless node report what it is doing — to Prometheus, and to its own log.

Three commits, each usable on its own:

1. 2c5d3871 — serve metrics from a masterless node

MasterClient was the only metric destination in the process: PoolClient::CountMetric returned early when it was null, and the component tick was registered only through MasterClient::AddMetricsProvider. The standalone server — the deployment sglang's direct external linker actually runs — therefore had no /metrics endpoint, and no flag could give it one.

MetricSink becomes the seam between measuring and publishing. MasterClient implements it unchanged, so with a master nothing about the behaviour changes. PrometheusMetricSink is the masterless counterpart: it owns a MetricsServer in-process and stamps the node label master used to add on ingest, so a dashboard query written for a master-backed cluster works against a standalone server.

Port comes from UMBP_STANDALONE_METRICS_PORT, default 9092 rather than the master's 9091, so a node running both does not have two processes contending for one port.

2. b9973589 — reject a mistyped metrics port, and make the endpoint reachable

Verifying the above end to end turned up one defect and two missing pieces of deployment glue.

• The defect. ResolveMetricsPortFromEnv used std::stol, which parses a numeric prefix and reports success for what it read. UMBP_STANDALONE_METRICS_PORT=9O92 (letter O) resolved to port 9 — privileged, so the bind fails, so the operator gets no metrics and an ERROR about a port they never typed. That is exactly the "typo discovered as a blank dashboard days later" the resolver was written to prevent. It now requires the whole string to be consumed. This is what the existing GarbageIsRejectedNotSilentlyDisabled test was already asserting, and the only one of its nine that failed.
• Nothing told Prometheus the endpoint exists. examples/monitoring shipped one scrape fragment, for the master's 9091, so a masterless node published on 9092 and was never scraped. Adds umbp_standalone.scrape.yml beside it, and a README table for which deployment wants which. The dashboards do not change — they are written against the series.
• UMBP_STANDALONE_METRICS_PORT was absent from runtime-env-vars.md.

3. 35a5fe18 — one periodic INFO line for offload / load / evict / promote / demote

Answering "is this node evicting?" meant turning on DEBUG, which prints a line per key: enough volume under steady offload to change the timing of the thing being measured, and it drags every other DEBUG line along with it. The numbers were all being measured already — they just had nowhere to go in a run with no Prometheus attached, which is every run that is hard to debug: a ctest, a one-shot benchmark, an sglang rank someone is bisecting, a customer node.

8.0s: offload 400.00 MiB/400 keys (50.0 MiB/s) · load 192.00 MiB/192 keys (24.0 MiB/s) miss=208
      · evict 208.00 MiB/208 keys (26.0 MiB/s) · promote 0 B · demote 0 B
  DRAM resident 192.00 MiB/256.00 MiB (75%) 192 keys

ActivitySummary owns no thread, reads no clock and calls no logger: it takes a cumulative snapshot and returns text, which is what makes the deltas, the units and the idle case testable without standing up a pool. PoolClient drives it on its own timer, in every deployment — master, masterless or embedded — because it is precisely the no-destination case that needs it. UMBP_SUMMARY_INTERVAL_MS, default 60000, 0 disables.

An idle window still logs, carrying resident bytes and key count: a node holding 40 000 keys that moved none of them for a minute is the stall this is for, and is indistinguishable from a healthy quiet node unless something says how much is sitting there.

Local eviction was invisible to metrics entirely

Worth calling out separately, because it is a pre-existing hole this branch closes rather than a consequence of the new log.

A page-backed medium reclaiming to its low watermark frees pages inline instead of calling its own Evict(). No interface call is ever made, so InstrumentedBackend never sees it and op="evict" stayed at zero however much a masterless node reclaimed. The per-round DEBUG line was the only witness there has ever been — and this branch removes that line.

So PageBackend now publishes it as a medium event. Measured on a 256 MiB tier taking 400 × 1 MiB:

$ grep 'op="evict"' /metrics
>>> nothing — op="evict" does not appear at all

mori_umbp_backend_medium_events_total{tier="DRAM",event="local_evict"}  208
mori_umbp_backend_medium_bytes_total{tier="DRAM",event="local_evict"}   218103808   (= 208 MiB)

Every byte of the evict 208.00 MiB in the summary line above comes from that new event. This also means local eviction reaches Grafana for the first time — it was missing from both the log and the dashboards before.

SSD is the other way round: its local eviction does go through the interface, so it lands in op="evict" normally. CollectActivitySnapshot adds the two together; they are disjoint, so there is no double-counting, and both medium types report eviction in the same field.

DEBUG logs removed

Nine sites, replaced by counters the summary aggregates — eight on the re-cache path in pool_client.cpp (admission rejected, staging alloc failed, queue full ×2, fragmented slot, prefetched, re-cached, install failed) and the per-round eviction line in page_backend.cpp.

The give-up reasons became counters rather than being dropped, which is the part DEBUG was worst at: every one of those paths is a best-effort return that leaves the read it was helping perfectly correct, so a dropped re-cache was invisible unless you were already printing a line per key. Each reason has a different fix — a bigger queue, a wider admission policy, a larger tier — so they stay distinguishable:

recache: 90 installed / 100 admitted (already=0, prefetched=0); dropped 13
         (rejected=3 queue_full=7 no_memory=2 install_failed=1 fragmented=0)

Left alone deliberately: master-side eviction, routing decisions and slot-lifecycle housekeeping are not the five actions this summarises. SsdCopyPipeline's three per-key drop logs look in scope, but that class is referenced nowhere outside its own file and unit test — dead code from a removed wiring — so they cannot fire in production. Its counters are labelled "Prometheus-only observability" and nothing publishes them; that is a separate cleanup, not this one.

Design note

The summary does not go through Prometheus. Both are parallel readers of the same source — the components' cumulative counters, via the one SampleMetrics() seam:

                    backend counters (atomics)
                              │
                     SampleMetrics()
                    ┌─────────┴─────────┐
        MetricPublisher              CollectActivitySnapshot
        (own baseline)                  (own baseline)
             │                               │
        MetricSink                    ActivitySummary
      (master / :9092)                       │
             │                          MORI_UMBP_INFO
         /metrics

MetricPublisher's baseline is deliberately not shared with the summary. Both need deltas from cumulative counters, so each subtracts the last value it saw; a shared baseline would let whichever ticked first consume the delta and leave the other reading zero. They also run at different rates (1 s vs 60 s).

The upshot is that a new medium lands in both the summary and Grafana with no code in either reader — which is how the local_evict fix above reached both at once.

Testing

• 22 new unit tests, all passing: test_activity_summary (13) and test_prometheus_metric_sink (9).
• End-to-end on smci355-n09-29, masterless standalone server, DRAM tier:
  • /metrics live on 9092; after 64 × 1 MiB Put+Get it reports 67108864 bytes committed and 67108864 resolved, every series carrying node=.
  • 400 × 1 MiB into a 256 MiB tier: 400 offloaded − 208 evicted = 192 resident = 192 loaded, and the 208 evicted are exactly the 208 misses. The summary's numbers close against the probe's return values.
  • Both disable knobs verified (UMBP_SUMMARY_INTERVAL_MS=0, UMBP_STANDALONE_METRICS_PORT=0).

ctest, branch vs. parent commit, same container, same cmake configuration:

	tests	failures
base (2c5d3871)	56	11
this branch	57	10

The single difference is test_prometheus_metric_sink, which commit 2 fixes. No new failures.

The remaining 10 are environmental, not regressions. Every one of their 87 failed assertions had a single cause: adding /dev/infiniband plus the ionic NIC mounts removed the RDMA errors entirely, and what is left dies in SetUp on rsmi_num_monitor_devices reported 0 GPUs. All 8 GPUs on that node were fully committed to another job, so they could not be run there and remain unverified on this branch.

🤖 Generated with Claude Code

[TianDi101]

Merged into #753 together with the eviction convergence work (#741/#742). The PageBackend local-evict counters were dropped in the merge because #741 moves local eviction to the pool, where it is counted as op="evict" and mori_umbp_client_local_evict_total. Leaving this open until #753 lands.

[TianDi101]

End-to-end validation of the eviction PR chain: passed.

Branch test/umbp-evict-e2e-v2 @ 2e1663d91 = main (097bb470f) + merges of #741, #762, #763 and #683, nothing else. (The #683 merge drops PageBackend's local-evict counters, which #741 removed, and adds #763's K dropped summary line.) Workload: InferenceY single-node agentic (DeepSeek-V4-Pro fp4, TP8 + DP attention, MTP, conc 256, CI run 29816's config) on n06-21, 3600 s profiling. UMBP is a standalone server with no master (so default local eviction), with a 300 GB DRAM tier demoting to a 1.5 TB SSD tier on 3 drives.

	earlier e2e on #753	this chain (exact PR heads)
Input throughput	281,015 tok/s	283,993 tok/s
Prefix hit (HBM + UMBP)	92.03%	91.99%
Failed loads / SSD read or write errors / sglang crashes	0 / 0 / 0	0 / 0 / 0
DRAM→SSD demotions / copies dropped	1.48M / not measured	1.47M / 0
Demotion failures retried until they landed (#763)	not measured	5,048 (SSD staging full), all landed
SSD keys deferred to a later round / staging released early (#762)	2.93M / 60.3M	2.83M / 60.3M
Local eviction rounds / stalls (#741)	19 / 0	19 / 0

The log signatures of the 7 bugs this e2e originally found all stay at 0: whole-batch staging failures, UMBP get failed, overdue local holds, SSD write failures, PrepareReadBatch read failed, dropped demotions, and any [error] line. An earlier pass on the pre-trim heads gave the same result (277,392 tok/s, 0 failures).

Notes:

• The harness sets UMBP_STANDALONE_STARTUP_TIMEOUT_MS=1800000. Since fix(umbp): keep the standalone server serving when its GPU-registered DRAM pool is migrated #756 the server reports ready while its 300 GB pool is still being GPU-registered, and under concurrent model loading sglang's default 30 s connect timed out. That is not part of this chain.
• One rerun was lost to an sglang GPU memory fault in one DP rank (its own kernel, no UMBP data being loaded at the time); the rerun passed.

🤖 Generated with Claude Code