Skip to content

docs(guide): multi-tenant quota, priority & saturation (Pub/Sub or Redis SortedSet) - #285

Closed
shimib wants to merge 17 commits into
llm-d:mainfrom
shimib:docs/multitenant-pubsub-demo
Closed

docs(guide): multi-tenant quota, priority & saturation (Pub/Sub or Redis SortedSet)#285
shimib wants to merge 17 commits into
llm-d:mainfrom
shimib:docs/multitenant-pubsub-demo

Conversation

@shimib

@shimib shimib commented Jun 24, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Summary

A runnable guide (docs/guides/multitenant/) for the async processor showcasing team × tier × model multi-tenancy, with the message queue as a pluggable backend (Redis SortedSet or GCP Pub/Sub) and two observability options (self-hosted Prometheus/Grafana or GCP Cloud Monitoring).

The three dimensions

  • Model → pool isolation. Two models = two worker pools (model-a/model-b); 6 queues (team × model). The publisher sets payload.model; one gateway routes to two InferencePools. Each pool has its own workers and its own saturation gate scoped to its model's inference_pool, so a hot model can't starve a cold one.
  • Team → reserved/overflow quota. Per-team redis-quota in classifying mode (per-model prefix quota:a:/quota:b:): within quota → reserved, over → overflow (dispatched, not nacked).
  • Tier → priority. Per-queue tier label; the tier-priority merge policy (Add tier-priority merge policy and gate + refactor merge policies #294) runs per pool, dispatching the 6 (classification, tier) lanes in strict order (all reserved before any overflow) and stamping x-gateway-priority.

Contents

  • values/redis/ + values/pubsub/ overlays (quota-only + saturation): 6 queues → 2 pools, per-model prefixes/result lists, tier labels, per-model saturation gates (POOL_A/POOL_B), tier-priority merge policy.
  • README: topology, publish helper publish <team> <a|b>, scenarios (A reserved/overflow+tier within a model; B overflow deprioritization + model isolation; C per-model saturation back-off), 2-InferencePool prereq.
  • manifests/redis.yaml, GCP scripts (create the 6 team×model topics), observability manifests + Cloud Monitoring dashboard.
  • Animation (diagram/architecture.html + regenerated architecture.gif): two model bands; model A saturates and parks while model B keeps flowing.

How was this tested?

  • helm template + helm lint on all 5 overlays (2 pools, 6 queues, per-model prefixes, per-model saturation scoping, merge policy).
  • Data-plane harness (miniredis + fake Prometheus + mock gateway): per-model saturation isolation — model-a parks while model-b flows, then drains when saturation clears. (Reserved-before-overflow ordering is covered on a live Kind cluster by test(e2e): add 3-team multi-tenant scenario (redis-sortedset) #305.)
  • Animation regenerated via capture.sh (chrome + ffmpeg) and visually verified.

Release note

NONE

shimib added 10 commits June 24, 2026 17:45
@shimib
docs(guide): add multi-tenant Pub/Sub demo guide
26152ca 
A runnable guide under docs/guides/multitenant-pubsub for running the async
processor over GCP Pub/Sub with three teams: per-team quota (redis-quota gate
keyed on a `team` message attribute), priority under saturation (per-team
prometheus-query gates over vllm:num_requests_running), and two interchangeable
observability backends — GCP Cloud Monitoring (GMP + query frontend) or a
self-hosted Prometheus + Grafana stack (the chart's bundled PodMonitor /
PrometheusRule / Grafana dashboard).

Includes values overlays (quota-only, GMP saturation, Prometheus), redis / GMP /
Prometheus manifests, a Cloud Monitoring dashboard, and GCP provisioning
scripts. Requests are published via gcloud (the message contract is documented);
no bundled publisher. All environment-specific values are placeholders
(PROJECT_ID / NAMESPACE / IGW_HOST / model).

Signed-off-by: Shimi Bandiel <shimib@google.com>
@shimib
docs(guide): add animated architecture diagram
3918c18 
Add an animated SVG/HTML architecture diagram (diagram/architecture.html) and a
rendered GIF embedded at the top of the guide README. The loop shows per-team
message flow through the quota + saturation gates and the priority-under-
saturation story (batch gate closes + backlog grows while premium keeps
flowing). capture.sh regenerates the GIF headlessly via Chrome + ffmpeg.

Signed-off-by: Shimi Bandiel <shimib@google.com>
@shimib
docs(guide): polish animated diagram (arrows, overlay, pacing)
7ae6c0e 
Bright arrowed connectors between stages, flow dots rendered beneath the boxes
so they ride exactly on the wires, and a calmer 20s loop. Regenerated GIF.

Signed-off-by: Shimi Bandiel <shimib@google.com>
@shimib
docs(guide): polish diagram (pacing, all connections, clean render)
5dd0302 
Slower dot flow, explicit Redis->gate quota wires (every connection now
visible), flat background + deterministic capture for a clean jitter-free GIF.

Signed-off-by: Shimi Bandiel <shimib@google.com>
@shimib
docs(guide): fix diagram README — mp4 is generated, not committed
fa279b9 
Signed-off-by: Shimi Bandiel <shimib@google.com>
@shimib
docs(guide): re-fit Scenario C to pool-level gates (llm-d#276)
0435533 
Adopt the two-level gate model: per-team redis-quota stays a QUEUE gate
(admission/nack); the saturation priority moves to a POOL-level
wait-on-refuse(prometheus-query) gate on each worker pool, so under saturation
a team's workers park in-memory (ActionWait) instead of nacking (no broker
backlog churn). README documents queue vs pool gates.

Pool gates require the binary from llm-d#276 (pin an image that includes it; v0.7.1
predates it and ignores pool gate_type).

Signed-off-by: Shimi Bandiel <shimib@google.com>
@shimib
docs(guide): pin v0.7.2 (first release with pool gates llm-d#276)
8e6f1bc 
Bump the demo image/chart references to v0.7.2, which includes the pool-level
gates (llm-d#276) the Scenario-C config now relies on; drop the 'predates llm-d#276'
notes. Validated end-to-end on v0.7.2.

Signed-off-by: Shimi Bandiel <shimib@google.com>
@shimib
docs(guide): diagram shows llm-d#276 two gate levels (queue quota vs …
9786e30 
…pool saturation)

Queue gate (quota/nack) between sub and pool; pool gate (saturation) after the
pool with WAIT/park under load (batch parks in-memory, premium keeps flowing).
Capture fix: state toggles use fill-opacity so headless screenshots render them.

Signed-off-by: Shimi Bandiel <shimib@google.com>
@shimib
docs(guide): add gate-metric dashboard panels (llm-d#290/llm-d#291)
e5fca67 
Cloud Monitoring dashboard gains panels for async_dispatch_budget (gate openness
per team), async_gate_decisions_total (by reason / team+reason), and worker
utilization (inflight / pool_worker_limit). README notes they require an image
newer than v0.7.2.

Signed-off-by: Shimi Bandiel <shimib@google.com>
@shimib
Merge upstream/main into docs/multitenant-pubsub-demo
eda7c98 
Incorporate all merged PRs since the branch was last updated, including the
integration-test hang fix (llm-d#296) that was causing the pre-commit job to time out.

Signed-off-by: Shimi Bandiel <shimib@google.com>
@shimib shimib mentioned this pull request Jul 8, 2026
Merged
8 tasks
shimib added 3 commits July 9, 2026 17:55
@shimib
Merge upstream/main into docs/multitenant-pubsub-demo
28a22d8 
Pick up llm-d#295 (gate_params now accepts native typed values) and llm-d#300
(structured ResultMessage) so the guide reflects current config/wire format.

Signed-off-by: Shimi Bandiel <shimib@google.com>
@shimib
docs(guide): document structured result-message fields (llm-d#300)
57663fd 
Note the status_code/error_code/error_message result schema so team consumers
of results-sub can branch on HTTP vs non-HTTP outcomes.

Signed-off-by: Shimi Bandiel <shimib@google.com>
@shimib
docs(guide): refocus multi-tenant guide on the scenario, dual queue b…
a6f9d3d 
…ackends

Rename docs/guides/multitenant-pubsub -> multitenant and retitle to
"Multi-tenant quota, priority & saturation". The scenario (per-team quota +
priority + saturation) is queue-agnostic, so present the message queue as a
pluggable backend fully supported on both Redis SortedSet (portable, featured
first) and GCP Pub/Sub.

- values overlays split into values/redis/ and values/pubsub/; new Redis
  SortedSet overlays (quota-only, saturation-prometheus) mirroring the Pub/Sub
  ones via redis.queuesConfig.
- README rewritten scenario-first with a 'Choose your queue backend' section,
  per-backend publish helpers (ZADD envelope vs gcloud), results/backlog, and
  observability (self-hosted Prometheus for either; GMP for Pub/Sub on GKE).
- Redis manifest now backs both the queues and quota counters (redis-quota
  service -> redis); gate addresses updated.
- diagram/scripts framed as the Pub/Sub example / Pub/Sub-only.

Verified all four AP overlays render and lint with helm template/lint.

Signed-off-by: Shimi Bandiel <shimib@google.com>
@shimib shimib changed the title docs(guide): add multi-tenant Pub/Sub demo guide docs(guide): multi-tenant quota, priority & saturation (Pub/Sub or Redis SortedSet) Jul 9, 2026
shimib added 2 commits July 9, 2026 21:57
@shimib
Merge upstream/main into docs/multitenant-pubsub-demo
db5f23e 
Pick up llm-d#294 (tier-priority merge policy + requestMergePolicyConfig chart support).

Signed-off-by: Shimi Bandiel <shimib@google.com>
@shimib
docs(guide): adopt the tier-priority merge policy (reserved/overflow …
17360ef 
…x tier)

Rework the multi-tenant guide's priority model to use the tier-priority merge
policy merged in llm-d#294:

- All three teams feed ONE shared pool; each team queue carries a tier label
  (premium=interactive, standard=async, batch=batch).
- redis-quota gate switched to classifying mode: within quota -> reserved,
  over quota -> overflow (dispatched, not nacked) — the per-team reserved lane.
- ap.requestMergePolicyConfig: tier-priority — dispatches the 6
  (classification x tier) lanes in strict order (all reserved before any
  overflow; tier-ordered within each) and stamps x-gateway-priority.
- Saturation: one shared-pool wait-on-refuse(prometheus-query) gate; merge
  policy drains highest lanes first as capacity frees.
- README: reworked What-it-shows table + lane table, Scenarios A/B (reserved vs
  overflow) and C (shared-pool saturation), quota/observability notes; diagram
  note updated. All 5 redis/pubsub overlays render + helm lint clean.

Merged upstream/main (llm-d#294) to pick up the chart's requestMergePolicyConfig.

Signed-off-by: Shimi Bandiel <shimib@google.com>
@shimib
shimib marked this pull request as ready for review July 9, 2026 22:49
@shimib
shimib marked this pull request as draft July 9, 2026 23:03
shimib added 2 commits July 9, 2026 23:19
@shimib
docs(guide): extend to team × tier × model (2 models, 2 pools)
8b98ce3 
Add the model dimension: 2 models = 2 worker pools (model-a/model-b), 6 queues
(team × model). Each model pool has its own saturation gate scoped to its
InferencePool, its own tier-priority lane ordering, and per-(team,model) reserved
quotas (redis-quota classifying, per-model prefix quota:a:/quota:b:). Showcases
model isolation + reserved/overflow + tier priority together.

- 5 redis/pubsub overlays reworked to 6 queues / 2 pools with per-model prefixes,
  result lists, tier labels, and per-pool saturation queries (POOL_A/POOL_B).
- README rewritten for the 3 dimensions: topology, publish helper (team+model),
  scenarios (reserved/overflow+tier within a model; model isolation; per-model
  saturation), 2-InferencePool prereq, per-model counters.
- gcp-setup/teardown create the 6 (team × model) topics/subscriptions.

Tested: helm template + lint on all 5 overlays; a miniredis + fake-Prometheus +
mock-gateway data-plane harness confirming per-model saturation isolation
(model-a parks while model-b flows, then drains). Animation update follows.

Signed-off-by: Shimi Bandiel <shimib@google.com>
@shimib
docs(guide): redraw animation for team × tier × model
487c750 
Rework architecture.html into two model bands (model-a / model-b), each with its
three team/tier lanes → reserved/overflow quota gate → tier-priority merge → its
pool → per-model saturation gate → its own vLLM. The loop shows model isolation:
model A saturates (pool gate WAIT, parks in-memory, red meter) while model B
keeps flowing (OPEN, green). Regenerated architecture.gif/.mp4 via capture.sh
(canvas 1300×900; window size updated to match).

Signed-off-by: Shimi Bandiel <shimib@google.com>
@shimib

shimib commented Jul 14, 2026

Copy link
Copy Markdown
Member Author

relocating to llm-d/llm-d guides

@shimib shimib closed this Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ahg-g ahg-g Awaiting requested review from ahg-g ahg-g is a code owner
@jtechapps jtechapps Awaiting requested review from jtechapps jtechapps is a code owner
@RishabhSaini RishabhSaini Awaiting requested review from RishabhSaini RishabhSaini is a code owner
@evacchi evacchi Awaiting requested review from evacchi evacchi is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@shimib