diff --git a/RELEASE-NOTES.md b/RELEASE-NOTES.md index ac4b9047..3a934522 100644 --- a/RELEASE-NOTES.md +++ b/RELEASE-NOTES.md @@ -10,6 +10,14 @@ PR template's `release-note` block — see +RELEASE v0.9.0 2026-07-29 +2026-07-28 https://github.com/llm-d/llm-d-async/pull/370 Fix the `prometheus-budget` gate on a stock llm-d install: its primary source queried `inference_extension_flow_control_queue_size`, which only exists when EPP runs the flow control plugin, so the gate silently served its fallback budget. A source based on `inference_pool_per_pod_queue_size` (part of EPP's base metric set) is now tried between the flow control and vLLM sources. The resolved PromQL for each cascade tier is logged at startup, and the new `llm_d_async_async_gate_metric_source_available` gauge distinguishes a fallback budget from a real reading of the same value. +2026-07-28 https://github.com/llm-d/llm-d-async/pull/371 Fix `async_gate_decisions_total{reason="gate_closed"}`, which could never increment for a budget-based gate: back-pressure is applied pre-dequeue by shrinking the dispatch batch to zero, while the counter was only incremented post-dequeue inside the loop that batch size had just emptied — and a counter series that never increments is absent from `/metrics` entirely, so the query returned an empty vector rather than 0. The decision is now recorded where it is made: the sorted-set path counts each throttled poll whose batch the budget zeroed while the queue is non-empty, and the Pub/Sub path counts each receive window it skips for the same reason. Both backends also pre-create all four `reason` series at 0 when a queue starts. The counter's unit is decisions, not messages; the help text and README metric table now say so. +2026-07-29 https://github.com/llm-d/llm-d-async/pull/372 Fix `pool_name`, which carried three different meanings across the five series an operator needs to diagnose a throttled queue, so none of them joined. `async_gate_metric_value` and `async_gate_metric_threshold` labeled it with the gate's `pool` param — the InferencePool being queried — and carried no queue labels at all; `async_dispatch_budget` and `async_gate_decisions_total` read it from the raw queue config, so a queue that omitted `worker_pool_id` reported `pool_name=""` while its own `async_broker_backlog` and `async_pool_worker_limit` reported `"default"`. `pool_name` now always names the async worker pool that owns the series, the queue config is normalized before it is stored, and the two gate gauges carry the full `queue_id`/`queue_name`/`pool_name` triple so they join with the rest of the queue's metrics. The InferencePool a gate queries moves to its own `inference_pool` label, which also makes two worker pools gating on one InferencePool distinguishable. **Breaking for dashboards on the two gauges:** select `{inference_pool="..."}` where you previously selected `{pool_name="..."}`. +2026-07-29 https://github.com/llm-d/llm-d-async/pull/373 Fixed the end-to-end deploy guide (`docs/guides/e2e-deploy.md`), which referenced llm-d paths and chart flags that no longer exist. Step 5 now installs the `llm-d-router-gateway` chart instead of the retired GAIE `inferencepool` chart (`httpRoute.*` instead of `experimentalHttpRoute.*`, monitoring via the router recipe's `monitoring.values.yaml`, pinned to a new `ROUTER_CHART_VERSION`), the Istio and observability references point at their current locations, and the description of what the install creates now matches the rendered chart. +2026-07-29 https://github.com/llm-d/llm-d-async/pull/374 The `prometheus-budget` gate now logs its resolved closing point when it is created (`"prometheus-budget gate configured" ... closesAtLoadPerReadyPod=95`), making a mis-sized `max_concurrency` visible immediately instead of leaving the gate silently open forever. `max_concurrency` is documented as a **per ready pod** capacity in the README and the e2e deploy guide, with guidance on deriving it from the EPP saturation detector's `MaxConcurrency` or from the observed per-pod peak of `vllm:num_requests_running`. + + RELEASE v0.8.0 2026-07-24 2026-07-09 https://github.com/llm-d/llm-d-async/pull/300 **Breaking:** `ResultMessage` now carries structured result fields so consumers can distinguish HTTP successes, HTTP errors, and non-HTTP failures — added `StatusCode` (int, non-zero means an HTTP response was received), `ErrorCode` (string, e.g. `DEADLINE_EXCEEDED`, `GATE_DROPPED`, `GATE_ERROR`, `INFERENCE_ERROR`, `INVALID_REQUEST`), and `ErrorMessage` (string). Wire-format changes to be aware of: - HTTP-error `Payload` now contains the raw upstream response body instead of a wrapped error string. - Gate-drop `Payload` changed from `{"status":"dropped"}` to `{"error":"Pool gating dropped request"}` and is now unified across all transports via `NewGateDroppedResult`. Detect gate drops via `ErrorCode == "GATE_DROPPED"` rather than parsing the payload body. - Gate-drop results now include `Metadata` (from `req.ReqMetadata()`), which the previous inline construction omitted. - `InferenceClient.SendRequest` now returns `(*InferenceResponse, error)`, where `InferenceResponse` has `StatusCode int` and `Body []byte` fields (previously `([]byte, int, error)`). 2026-07-10 https://github.com/llm-d/llm-d-async/pull/306 The default worker `concurrency` is now **64** (was 8). The processor is I/O-bound, so by Little's Law in-flight concurrency caps throughput; the old default of 8 left real inference pools mostly idle. Tune `concurrency` to your backend's latency/throughput target (see the Async Processor Operations Guide). diff --git a/go.mod b/go.mod index 63997219..0c8e5332 100644 --- a/go.mod +++ b/go.mod @@ -7,9 +7,9 @@ require ( cloud.google.com/go/pubsub/v2 v2.6.1 github.com/alicebob/miniredis/v2 v2.38.0 github.com/go-logr/logr v1.4.4 - github.com/llm-d/llm-d-async/api v0.8.0 - github.com/llm-d/llm-d-async/pipeline v0.8.0 - github.com/llm-d/llm-d-async/producer v0.8.0 + github.com/llm-d/llm-d-async/api v0.9.0 + github.com/llm-d/llm-d-async/pipeline v0.9.0 + github.com/llm-d/llm-d-async/producer v0.9.0 github.com/onsi/ginkgo/v2 v2.32.0 github.com/onsi/gomega v1.42.1 github.com/prometheus/client_golang v1.24.1 diff --git a/pipeline/go.mod b/pipeline/go.mod index c24f2893..c4c06078 100644 --- a/pipeline/go.mod +++ b/pipeline/go.mod @@ -2,6 +2,6 @@ module github.com/llm-d/llm-d-async/pipeline go 1.25.0 -require github.com/llm-d/llm-d-async/api v0.8.0 +require github.com/llm-d/llm-d-async/api v0.9.0 replace github.com/llm-d/llm-d-async/api => ../api diff --git a/producer/go.mod b/producer/go.mod index 0dda436a..08c3faf2 100644 --- a/producer/go.mod +++ b/producer/go.mod @@ -4,7 +4,7 @@ go 1.25.0 require ( github.com/alicebob/miniredis/v2 v2.38.0 - github.com/llm-d/llm-d-async/api v0.8.0 + github.com/llm-d/llm-d-async/api v0.9.0 github.com/redis/go-redis/v9 v9.21.0 github.com/stretchr/testify v1.11.1 ) diff --git a/release-notes.d/unreleased/370.md b/release-notes.d/unreleased/370.md deleted file mode 100644 index a77c34b5..00000000 --- a/release-notes.d/unreleased/370.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -pr: 370 -url: https://github.com/llm-d/llm-d-async/pull/370 -author: shimib -date: 2026-07-28 ---- -Fix the `prometheus-budget` gate on a stock llm-d install: its primary source queried `inference_extension_flow_control_queue_size`, which only exists when EPP runs the flow control plugin, so the gate silently served its fallback budget. A source based on `inference_pool_per_pod_queue_size` (part of EPP's base metric set) is now tried between the flow control and vLLM sources. The resolved PromQL for each cascade tier is logged at startup, and the new `llm_d_async_async_gate_metric_source_available` gauge distinguishes a fallback budget from a real reading of the same value. diff --git a/release-notes.d/unreleased/371.md b/release-notes.d/unreleased/371.md deleted file mode 100644 index 565d2574..00000000 --- a/release-notes.d/unreleased/371.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -pr: 371 -url: https://github.com/llm-d/llm-d-async/pull/371 -author: shimib -date: 2026-07-28 ---- -Fix `async_gate_decisions_total{reason="gate_closed"}`, which could never increment for a budget-based gate: back-pressure is applied pre-dequeue by shrinking the dispatch batch to zero, while the counter was only incremented post-dequeue inside the loop that batch size had just emptied — and a counter series that never increments is absent from `/metrics` entirely, so the query returned an empty vector rather than 0. The decision is now recorded where it is made: the sorted-set path counts each throttled poll whose batch the budget zeroed while the queue is non-empty, and the Pub/Sub path counts each receive window it skips for the same reason. Both backends also pre-create all four `reason` series at 0 when a queue starts. The counter's unit is decisions, not messages; the help text and README metric table now say so. diff --git a/release-notes.d/unreleased/372.md b/release-notes.d/unreleased/372.md deleted file mode 100644 index 237d7bbe..00000000 --- a/release-notes.d/unreleased/372.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -pr: 372 -url: https://github.com/llm-d/llm-d-async/pull/372 -author: shimib -date: 2026-07-29 ---- -Fix `pool_name`, which carried three different meanings across the five series an operator needs to diagnose a throttled queue, so none of them joined. `async_gate_metric_value` and `async_gate_metric_threshold` labeled it with the gate's `pool` param — the InferencePool being queried — and carried no queue labels at all; `async_dispatch_budget` and `async_gate_decisions_total` read it from the raw queue config, so a queue that omitted `worker_pool_id` reported `pool_name=""` while its own `async_broker_backlog` and `async_pool_worker_limit` reported `"default"`. `pool_name` now always names the async worker pool that owns the series, the queue config is normalized before it is stored, and the two gate gauges carry the full `queue_id`/`queue_name`/`pool_name` triple so they join with the rest of the queue's metrics. The InferencePool a gate queries moves to its own `inference_pool` label, which also makes two worker pools gating on one InferencePool distinguishable. **Breaking for dashboards on the two gauges:** select `{inference_pool="..."}` where you previously selected `{pool_name="..."}`. diff --git a/release-notes.d/unreleased/373.md b/release-notes.d/unreleased/373.md deleted file mode 100644 index 44cd5153..00000000 --- a/release-notes.d/unreleased/373.md +++ /dev/null @@ -1,14 +0,0 @@ ---- -pr: 373 -url: https://github.com/llm-d/llm-d-async/pull/373 -author: shimib -date: 2026-07-29 ---- - -Fixed the end-to-end deploy guide (`docs/guides/e2e-deploy.md`), which referenced -llm-d paths and chart flags that no longer exist. Step 5 now installs the -`llm-d-router-gateway` chart instead of the retired GAIE `inferencepool` chart -(`httpRoute.*` instead of `experimentalHttpRoute.*`, monitoring via the router -recipe's `monitoring.values.yaml`, pinned to a new `ROUTER_CHART_VERSION`), the -Istio and observability references point at their current locations, and the -description of what the install creates now matches the rendered chart. diff --git a/release-notes.d/unreleased/374.md b/release-notes.d/unreleased/374.md deleted file mode 100644 index d9cc50ce..00000000 --- a/release-notes.d/unreleased/374.md +++ /dev/null @@ -1,14 +0,0 @@ ---- -pr: 374 -url: https://github.com/llm-d/llm-d-async/pull/374 -author: shimib -date: 2026-07-29 ---- - -The `prometheus-budget` gate now logs its resolved closing point when it is -created (`"prometheus-budget gate configured" ... closesAtLoadPerReadyPod=95`), -making a mis-sized `max_concurrency` visible immediately instead of leaving the -gate silently open forever. `max_concurrency` is documented as a **per ready -pod** capacity in the README and the e2e deploy guide, with guidance on deriving -it from the EPP saturation detector's `MaxConcurrency` or from the observed -per-pod peak of `vllm:num_requests_running`.