feat(responses): pluggable ResponseStore abstraction

[will-deines]

Recreated from #35874, which was closed when the fork was temporarily made private.

Depends on #35740 — review that first. Diff will clean up automatically once it merges.

Summary

Extracts the in-memory response/message dicts from OpenAIServingResponses into a pluggable ResponseStore ABC, addressing RFC #26934.

• New ResponseStore ABC (vllm/entrypoints/openai/responses/store.py) with 5 async methods for get/put responses, get/put messages, and atomic status transitions
• InMemoryResponseStore default implementation wrapping current dict behavior with internal asyncio.Lock (removes external response_store_lock)
• VLLM_RESPONSES_STORE_BACKEND env var — point to a fully-qualified class name (e.g. mypackage.redis_store.RedisResponseStore) to swap backends
• Refactored serving.py — ~15 call sites updated from direct dict access to self.store.* async calls
• Atomic conditional writes (put_response(..., unless_status="cancelled")) replace get-check-put-under-lock patterns
• Atomic status transitions (update_response_status(..., allowed_current_statuses=...)) replace get-check-mutate patterns

What stays local (not pluggable)

• event_store and background_tasks — these use asyncio.Event/asyncio.Task which are inherently process-local

Decisions we made

This section explains the key design choices and how they trace back to the RFCs and discussions.

1. Pluggable ABC + factory, not a simple on/off toggle

RFC #26934 originally proposed VLLM_ENABLE_RESPONSES_API_STORE as a boolean toggle — either vLLM owns state or it doesn't. We chose a pluggable ABC with a factory instead because the RFC discussion converged on the principle that "vLLM should not be opinionated on whether it owns the state management layer" (@qandrew, comment). A toggle only gives two options; an ABC lets each deployment choose the backend that fits its topology — in-memory for dev, Redis/Postgres/DynamoDB for production — without vLLM shipping or maintaining any of those external integrations.

2. Env var with fully-qualified class name (VLLM_RESPONSES_STORE_BACKEND)

RFC #32850 discusses vLLM's policy for Responses API extensions. The agreed direction is to "allow extensions when there is a documented need" and follow existing vLLM patterns (@DanielMe's proposal). vLLM already uses this pattern for other pluggable components (e.g. resolve_obj_by_qualname). An env var keeps the interface zero-config for the common case (in-memory default) while enabling production deployments to swap backends without forking.

3. Atomic operations in the store interface, not external locks

The original serving.py used an external response_store_lock around get-check-put sequences on plain dicts. This pattern cannot work with external stores — a Redis or Postgres backend can't share a Python asyncio.Lock. We pushed atomicity into the ABC itself:

• put_response(..., unless_status="cancelled") — conditional write that skips if the response is already cancelled (replaces the cancel-race-condition guard)
• update_response_status(..., allowed_current_statuses=...) — atomic CAS-style status transition (replaces get-check-mutate patterns)

This lets external implementations use their native atomicity primitives (Redis SETNX, Postgres UPDATE ... WHERE status IN (...), DynamoDB conditional expressions).

4. Pluggable store supersedes LRU eviction (#34738)

PR #34738 proposed OrderedDict + LRU eviction to bound memory. @qandrew's response questioned the production motivation and pointed to #26934 as the longer-term direction. @lisperz agreed that in-memory storage "doesn't survive restarts and doesn't scale across multiple API server instances." Rather than shipping a bounded-but-still-in-memory solution, the pluggable store lets users choose their own eviction/persistence/replication strategy. The in-memory default intentionally ships without LRU — adding it later as a separate concern is straightforward if needed.

5. event_store and background_tasks stay process-local

These use asyncio.Event and asyncio.Task which are inherently bound to a single event loop. Externalizing them would require a fundamentally different design (distributed task queues, pub/sub for event notification). The RFC's priority is decoupling response/message persistence — the process-local coordination primitives are a separate concern for a follow-up if needed.

6. No new wire protocol fields

RFCs #32850 and #33381 both emphasize alignment with the OpenResponses spec and conservative extensions. This PR introduces zero new fields on the request or response wire format. The VLLM_RESPONSES_STORE_BACKEND env var is a server-side configuration knob, not a protocol extension. Clients are completely unaware of which backend is in use.

7. Complementary to the stateless path (#35740), not a replacement

RFC #26934 identifies two user personas: large-scale production (stateless, external state management) and researchers/small-scale (stateful, all-in-one). PR #35740 serves persona 1 with the encrypted_content state carrier. This PR serves both personas by making the store-backed path production-ready: persona 2 gets the in-memory default that just works, and persona 1 can either go fully stateless (#35740) or plug in their own persistent store for features that need server-side state (e.g. background mode, retrieve_responses).

8. Concurrency ownership moves into the store

The external response_store_lock in serving.py was a code smell — it coupled the serving layer to the implementation detail that the store was a plain dict. Each store implementation now owns its own concurrency strategy: InMemoryResponseStore uses an asyncio.Lock, a Redis implementation might use optimistic locking or WATCH/MULTI, a Postgres implementation might use row-level locks. This is a cleaner separation of concerns.

Related Issues & RFCs

#	Title	Relevance
#26934	[RFC] Separating State & Providing Flexibility for serving ResponsesAPI	Primary motivation — proposes decoupling state management from vLLM; pluggable backends for production
#35740	feat(responses): stateless multi-turn via encrypted_content state carrier	Predecessor PR — implements stateless path; this PR adds the pluggable interface for the store-backed path
#33089	[Feature] Support multi-turn conversation for OpenAI Response API	Direct fix — previous_response_id requires store; pluggable backend enables production-grade multi-turn
#34738	Fix memory leak in Responses API store (LRU eviction)	Superseded — LRU eviction is a band-aid; pluggable store lets users choose their own eviction/persistence strategy
#32850	[RFC] Clarify policy for Open Responses API extensions in vLLM	Extension policy — VLLM_RESPONSES_STORE_BACKEND env var is a vLLM extension; aligns with existing extension precedent
#33381	[RFC] Align with the openresponses.org spec	Conformance — pluggable store supports standard state formats; no new wire protocol fields

Context from RFC #26934 (Meta)

The RFC argues that vLLM should not be opinionated about storage — users need to plug in their own backend depending on deployment topology (single-node dev, multi-node production, managed service). The current in-memory dicts (response_store, msg_store) marked # HACK / # FIXME in serving.py:

• Leak memory unboundedly
• Lose all state on restart
• Are incompatible with multi-node / load-balanced deployments

This PR provides the interface layer. vLLM ships only the in-memory default — users bring their own Redis/Postgres/DynamoDB implementation by setting one env var. No external dependencies added.

Why not just LRU eviction (#34738)?

LRU eviction (proposed in #34738) is a band-aid that still loses state on restart and doesn't work across nodes. The pluggable store lets users choose their own eviction/persistence/replication strategy appropriate to their deployment.

Test plan

• pytest tests/entrypoints/openai/responses/test_store.py -v — unit + integration tests for store ABC, InMemoryResponseStore, factory, and serving.py integration (cancel/retrieve)
• pytest tests/entrypoints/openai/responses/test_serving_stateless.py -v — existing stateless tests pass with refactored store
• pre-commit run --all-files — linting passes
• E2e tests against running server (local_test/ harness, gitignored)

[gemini-code-assist]

Code Review

This pull request introduces a significant and well-designed feature: a pluggable ResponseStore abstraction, which correctly separates state management from the serving logic, and adds a stateless multi-turn conversation capability using a signed state carrier. This greatly improves the production-readiness of vLLM's Responses API. However, a critical prompt injection vulnerability has been identified. In the non-Harmony path, the server uses the unsigned output field from the client-provided previous_response even with a verified state carrier, allowing an attacker to spoof assistant messages in the conversation history. The suggested fix is to ensure the unsigned output is only used in the store-backed (stateful) path where previous_response is retrieved from the server's own store. Additionally, an area for improvement was noted in the deserialization logic to enhance its robustness.

[gemini-code-assist]

In stateless multi-turn mode, the conversation history is reconstructed using both the signed state carrier (prev_msg) and the unsigned output field of the previous_response object. Since the output field is provided by the client and not protected by the HMAC, an attacker can modify it to inject arbitrary assistant messages into the prompt. Additionally, because the signed history already contains the previous turn's output, this results in the previous output being duplicated in the prompt.

To remediate this, the prev_response_output should be set to None when prev_messages (the stateless history) is provided, as the history is already complete.

Suggested change

	prev_response_output=prev_response.output if prev_response else None,
	prev_response_output=prev_response.output if prev_response and prev_messages is None else None,

[gemini-code-assist]

The list comprehension to deserialize prev_messages has a branch else m which could allow non-dict elements into prev_msgs. This could lead to a TypeError later if an element is not an OpenAIHarmonyMessage instance. It's safer to rely on OpenAIHarmonyMessage.model_validate(m) for all elements and let it raise a ValidationError for non-dict items. This makes the code more robust against malformed state, even if such state is correctly signed.

                prev_msgs = [OpenAIHarmonyMessage.model_validate(m) for m in prev_messages]

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7ac6fe63bb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Avoid duplicating prior output in stateless follow-up turns

When previous_response is used, prev_msg is populated from the state carrier, and that carrier is built from the full history (messages plus the prior turn’s output in responses_full_generator). Passing prev_response_output again here re-appends the same assistant output a second time, so each non-Harmony stateless turn repeats prior outputs, bloats context, and can skew model behavior.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Gate custom store initialization behind enable_store

create_response_store() is called unconditionally during handler init, so a configured VLLM_RESPONSES_STORE_BACKEND is imported/instantiated even when VLLM_ENABLE_RESPONSES_API_STORE=0. This can fail server startup (or create unnecessary external connections) in deployments that intentionally run stateless mode, because the custom backend is never needed on those code paths.

Useful? React with 👍 / 👎.