feat(responses): pluggable ResponseStore abstraction

[will-deines]

Depends on #35740 — review that first. Diff will clean up automatically once it merges.

Summary

Extracts the in-memory response/message dicts from OpenAIServingResponses into a pluggable ResponseStore ABC, addressing RFC #26934.

• New ResponseStore ABC (vllm/entrypoints/openai/responses/store.py) with 5 async methods for get/put responses, get/put messages, and atomic status transitions
• InMemoryResponseStore default implementation wrapping current dict behavior with internal asyncio.Lock (removes external response_store_lock)
• VLLM_RESPONSES_STORE_BACKEND env var — point to a fully-qualified class name (e.g. mypackage.redis_store.RedisResponseStore) to swap backends
• Refactored serving.py — ~15 call sites updated from direct dict access to self.store.* async calls
• Atomic conditional writes (put_response(..., unless_status="cancelled")) replace get-check-put-under-lock patterns
• Atomic status transitions (update_response_status(..., allowed_current_statuses=...)) replace get-check-mutate patterns

What stays local (not pluggable)

• event_store and background_tasks — these use asyncio.Event/asyncio.Task which are inherently process-local

Decisions we made

This section explains the key design choices and how they trace back to the RFCs and discussions.

1. Pluggable ABC + factory, not a simple on/off toggle

RFC #26934 originally proposed VLLM_ENABLE_RESPONSES_API_STORE as a boolean toggle — either vLLM owns state or it doesn't. We chose a pluggable ABC with a factory instead because the RFC discussion converged on the principle that "vLLM should not be opinionated on whether it owns the state management layer" (@qandrew, comment). A toggle only gives two options; an ABC lets each deployment choose the backend that fits its topology — in-memory for dev, Redis/Postgres/DynamoDB for production — without vLLM shipping or maintaining any of those external integrations.

2. Env var with fully-qualified class name (VLLM_RESPONSES_STORE_BACKEND)

RFC #32850 discusses vLLM's policy for Responses API extensions. The agreed direction is to "allow extensions when there is a documented need" and follow existing vLLM patterns (@DanielMe's proposal). vLLM already uses this pattern for other pluggable components (e.g. resolve_obj_by_qualname). An env var keeps the interface zero-config for the common case (in-memory default) while enabling production deployments to swap backends without forking.

3. Atomic operations in the store interface, not external locks

The original serving.py used an external response_store_lock around get-check-put sequences on plain dicts. This pattern cannot work with external stores — a Redis or Postgres backend can't share a Python asyncio.Lock. We pushed atomicity into the ABC itself:

• put_response(..., unless_status="cancelled") — conditional write that skips if the response is already cancelled (replaces the cancel-race-condition guard)
• update_response_status(..., allowed_current_statuses=...) — atomic CAS-style status transition (replaces get-check-mutate patterns)

This lets external implementations use their native atomicity primitives (Redis SETNX, Postgres UPDATE ... WHERE status IN (...), DynamoDB conditional expressions).

4. Pluggable store supersedes LRU eviction (#34738)

PR #34738 proposed OrderedDict + LRU eviction to bound memory. @qandrew's response questioned the production motivation and pointed to #26934 as the longer-term direction. @lisperz agreed that in-memory storage "doesn't survive restarts and doesn't scale across multiple API server instances." Rather than shipping a bounded-but-still-in-memory solution, the pluggable store lets users choose their own eviction/persistence/replication strategy. The in-memory default intentionally ships without LRU — adding it later as a separate concern is straightforward if needed.

5. event_store and background_tasks stay process-local

These use asyncio.Event and asyncio.Task which are inherently bound to a single event loop. Externalizing them would require a fundamentally different design (distributed task queues, pub/sub for event notification). The RFC's priority is decoupling response/message persistence — the process-local coordination primitives are a separate concern for a follow-up if needed.

6. No new wire protocol fields

RFCs #32850 and #33381 both emphasize alignment with the OpenResponses spec and conservative extensions. This PR introduces zero new fields on the request or response wire format. The VLLM_RESPONSES_STORE_BACKEND env var is a server-side configuration knob, not a protocol extension. Clients are completely unaware of which backend is in use.

7. Complementary to the stateless path (#35740), not a replacement

RFC #26934 identifies two user personas: large-scale production (stateless, external state management) and researchers/small-scale (stateful, all-in-one). PR #35740 serves persona 1 with the encrypted_content state carrier. This PR serves both personas by making the store-backed path production-ready: persona 2 gets the in-memory default that just works, and persona 1 can either go fully stateless (#35740) or plug in their own persistent store for features that need server-side state (e.g. background mode, retrieve_responses).

8. Concurrency ownership moves into the store

The external response_store_lock in serving.py was a code smell — it coupled the serving layer to the implementation detail that the store was a plain dict. Each store implementation now owns its own concurrency strategy: InMemoryResponseStore uses an asyncio.Lock, a Redis implementation might use optimistic locking or WATCH/MULTI, a Postgres implementation might use row-level locks. This is a cleaner separation of concerns.

Related Issues & RFCs

#	Title	Relevance
#26934	[RFC] Separating State & Providing Flexibility for serving ResponsesAPI	Primary motivation — proposes decoupling state management from vLLM; pluggable backends for production
#35740	feat(responses): stateless multi-turn via encrypted_content state carrier	Predecessor PR — implements stateless path; this PR adds the pluggable interface for the store-backed path
#33089	[Feature] Support multi-turn conversation for OpenAI Response API	Direct fix — previous_response_id requires store; pluggable backend enables production-grade multi-turn
#34738	Fix memory leak in Responses API store (LRU eviction)	Superseded — LRU eviction is a band-aid; pluggable store lets users choose their own eviction/persistence strategy
#32850	[RFC] Clarify policy for Open Responses API extensions in vLLM	Extension policy — VLLM_RESPONSES_STORE_BACKEND env var is a vLLM extension; aligns with existing extension precedent
#33381	[RFC] Align with the openresponses.org spec	Conformance — pluggable store supports standard state formats; no new wire protocol fields

Context from RFC #26934 (Meta)

The RFC argues that vLLM should not be opinionated about storage — users need to plug in their own backend depending on deployment topology (single-node dev, multi-node production, managed service). The current in-memory dicts (response_store, msg_store) marked # HACK / # FIXME in serving.py:

• Leak memory unboundedly
• Lose all state on restart
• Are incompatible with multi-node / load-balanced deployments

This PR provides the interface layer. vLLM ships only the in-memory default — users bring their own Redis/Postgres/DynamoDB implementation by setting one env var. No external dependencies added.

Why not just LRU eviction (#34738)?

LRU eviction (proposed in #34738) is a band-aid that still loses state on restart and doesn't work across nodes. The pluggable store lets users choose their own eviction/persistence/replication strategy appropriate to their deployment.

Test plan

• pytest tests/entrypoints/openai/responses/test_store.py -v — unit + integration tests for store ABC, InMemoryResponseStore, factory, and serving.py integration (cancel/retrieve)
• pytest tests/entrypoints/openai/responses/test_serving_stateless.py -v — existing stateless tests pass with refactored store
• pre-commit run --all-files — linting passes
• E2e tests against running server (local_test/ harness, gitignored)

[gemini-code-assist]

Code Review

This pull request introduces a well-designed pluggable ResponseStore abstraction, refactoring the in-memory response handling into a more robust and extensible system. The changes significantly improve thread safety by replacing direct dictionary access with atomic async methods, mitigating potential race conditions. Additionally, the implementation of a stateless multi-turn conversation feature using a signed state carrier is a valuable enhancement. The code is well-structured and accompanied by thorough tests. I've identified one critical issue related to a potential race condition in the InMemoryResponseStore that should be addressed.

[will-deines]

Gemini's critical finding has been addressed in 7ac6fe6 — get_messages and put_messages now acquire self._lock before accessing self._messages, consistent with how self._responses is protected.