[Artifact Connector][R3] Rewrite SHM backend on PR12

[aoshen02]

Stack and review scope

• Base: aoshen02/vllm:codex/mrv2-r3-stacked at a3caf7b0c4, the head of xhx1022/vllm#12.
• Head: aoshen02/vllm:codex/artifact-r3-shm-pr12-rewrite at e64a44b48e.
• This draft supersedes aoshen02/vllm#4. It is a ground-up simplification and revalidation, not a second implementation intended to coexist with [Artifact Connector][R3] Add ordered-key Core and SHM backend #4.

This PR implements the first production-shaped Artifact Connector slice: R3
(routed-expert IDs), Model Runner V2, prefix reuse keyed like KV cache, and a
same-node SHM backend. It deliberately does not include Mooncake,
TransferQueue, prompt-logprobs artifacts, DSA artifacts, multimodal artifacts,
or request-wise top-p token IDs.

Why replace #4

#4 mixed parts of the old physical-slot mmap/offload lifecycle with the new
logical Artifact Connector. That made Artifact behavior influence KV admission
and left too much request/slot cleanup in Scheduler.

This rewrite establishes two strict boundaries:

1. KV cache remains authoritative for prefix hits, allocation, preemption, and
   block release. Artifact code never shortens a KV hit. If KV reports a hit but
   the corresponding immutable Artifact object is missing, the engine fails
   closed instead of recomputing or altering KV behavior.
2. R3 publication uses logical token ranges and KV-compatible block hashes. It
   does not depend on mutable physical KV slots, so CPU KV offload does not need
   to move R3 alongside KV blocks.

Data and control flow

MRV2 MoE routing
  -> stable per-step GPU snapshot
  -> async D2H
  -> remove rejected speculative rows
  -> worker-owned logical request buffer
  -> immutable KV-hash-aligned SHM objects

KV prefix hit
  -> KV cache selects its prefix exactly as before
  -> derive Artifact keys from the same logical block hashes
  -> terminal materialization gets every object or fails closed

terminal request
  -> publish full blocks plus a KV-hashed partial block, if any
  -> worker returns finalize ACK + ordered keys
  -> Scheduler-side Artifact Connector materializes complete R3
  -> terminal HTTP choice returns one encoded ndarray

The last sampled token is excluded because it has not executed a target-model
forward pass.

Non-test changes and rationale

Configuration and compatibility

• vllm/config/artifact.py, vllm/config/__init__.py,
  vllm/config/vllm.py, and vllm/engine/arg_utils.py add one explicit
  ArtifactConfig and normal CLI/config plumbing. enable_routed_experts,
  backend selection, SHM path, capacity, and TTL have one canonical owner in
  ArtifactConfig; ModelConfig no longer carries Artifact state. The legacy
  --enable-return-routed-experts option is translated once in EngineArgs
  for compatibility. The SHM backend defaults to an 8 GiB
  per-engine/per-DP-rank capacity and one-hour inactive-store TTL.
• Configuration requires the SHM root to be under /dev/shm, rejects PP and
  context parallelism whose writer/ordering semantics are not implemented, and
  rejects PD-disaggregated KV transfer. TP, asynchronous scheduling, prefix
  caching, MTP speculative decoding, and kv_role=kv_both remain supported.
• Artifact mode requires Model Runner V2. gpu_worker.py follows the existing
  V1/V2 dynamic-import pattern but calls only the generic init_artifacts()
  entrypoint after KV initialization. Routed-experts capture and backend
  construction remain private ModelRunner/Artifact implementation details.

Unified routed-experts capture

• routed_experts_capture/{__init__,common,state,async_output}.py keeps one
  focused MRV2 capture state. ArtifactWorkerConnector now creates, owns,
  clears, and snapshots that state; GPUModelRunner owns only the generic
  connector and no longer exposes R3 capture internals. Every TP rank creates
  and binds the capture state because router capture participates in TP
  collectives. Only global rank zero constructs the SHM store, logical request
  buffer, and request core, so publication remains single-writer without
  deadlocking multi-rank capture. The no-op
  close() and artificial Optional capturer state are removed; only the
  Artifact Store owns a closeable resource.
• MRV2 attaches request IDs, logical token starts, query boundaries, and
  speculative rejection counts to the same stable async output. Router IDs are
  converted from their compute dtype to the compact schema dtype
  (uint8/uint16) before entering the logical buffer.
• The old manager.py and shared_region.py physical-slot mmap implementation
  is deleted. There is no second routed-experts cache indexed by mutable KV
  slots.
• gpu_model_runner.py, gpu/model_runner.py, gpu/async_utils.py, and
  gpu_worker.py route sync and async MRV2 output through the same Artifact
  connector handoff. The async path depends only on the structural
  ArtifactWriteTask protocol rather than an R3-specific task type. The
  existing V1 runner remains unchanged when Artifact mode is disabled.

Artifact Core and immutable key space

• artifact_connector/buffer.py owns only uncommitted logical request rows. It
  supports non-zero cached starts, overlap replacement after recomputation,
  rejects gaps, and releases committed full-block prefixes.
• artifact_connector/request_core.py owns deterministic keys, object
  envelopes, checksums, full-block/tail encoding, ordered coverage, and
  materialization. Both full blocks and a terminal partial block are addressed
  by KV-compatible block hashes plus the model weight version. The Scheduler
  computes a partial block hash from the same parent hash, token IDs, and KV
  extra keys used by prefix caching; tails are therefore content-addressed and
  are not request-attempt scoped.
• The key intentionally has no ModelConfig.compute_hash() namespace or
  separate Artifact profile ID. This deployment assumes an equal KV-compatible
  hash plus equal model weight version denotes reusable R3; object envelopes
  still validate dtype, shape, token range, identity, and checksum when read.
• artifact_connector/store.py is the backend-neutral immutable-object
  contract (put, exists, get, close). It has no Scheduler, request, or
  R3-specific policy.
• artifact_connector/shm.py implements atomic publish, immutable same-key
  validation, checksums, bounded capacity, writer liveness, and TTL collection
  under a trusted /dev/shm root. Readers consume values, not mmap paths or
  physical-slot handles.

Strict failure semantics

• Store publication and terminal materialization no longer return per-object or
  per-request error strings. Capacity, collision, missing-object, checksum, and
  finalize failures raise their typed exceptions immediately. Artifact mode is
  a correctness mode: the engine must not return a successful response with
  missing or partial R3.
• ArtifactFinalizeResult.keys is mandatory. The optional keys, error, and
  redundant ArtifactConnectorOutput.is_empty() branches are deleted.
• Aborting a request cancels its pending finalize and discards only its
  uncommitted logical request buffer. Already-published immutable blocks remain
  available to later KV prefix hits.
• The forward-only ArtifactRequestCore.close() is deleted. Worker shutdown
  closes the store directly.
• The request-only routed_experts_prompt_start sampling option and its
  Scheduler guard are deleted. Artifact output has one canonical coverage rule:
  every executed target-model token except the final unexecuted sampled token.

Scheduler/worker protocol

• artifact_connector/protocol.py carries commit/finalize operations using
  request-attempt IDs, logical ranges, and block hashes. It contains no physical
  block IDs.
• artifact_connector/connector.py separates Scheduler control state from
  worker publication. It deliberately has no prefix-readiness preflight:
  Artifact availability never changes the KV-selected hit length. The
  Scheduler-side connector validates finalize ACKs and materializes through
  backend get; a missing or corrupt object fails the request/engine closed.
  The rank-zero worker batches immutable puts and returns ordered terminal keys.
• sched/output.py and v1/outputs.py add only the Artifact metadata/output
  fields needed on the existing SchedulerOutput/ModelRunnerOutput path.
• sched/scheduler.py observes the KV-selected cached length, submits accepted
  full-block progress, and starts terminal finalization. It never checks
  Artifact existence during KV admission and does not modify
  KVCacheManager.get_computed_blocks, max_cache_hit_length, allocation,
  preemption, or block-free decisions.
• Frontend stop-string finalization enters the existing finish_requests path
  for request lookup, queue removal, and finished-state update. Normal
  completion and frontend-resolved stop both reuse
  _free_request(..., artifact_token_end=...). Artifact delays only terminal
  delivery until finalize ACK; it does not create a parallel request-removal or
  resource-release lifecycle.
• Artifact does not implement a private has_pending_work engine keep-alive.
  Active requests, existing finished_req_ids metadata scheduling, and the
  normal async batch queue already provide progress; a private flag would create
  an empty-step busy loop after a lost ACK.
• sched/interface.py, engine/{__init__,core,core_client,async_llm,llm_engine, output_processor}.py propagate the exact frontend-resolved token boundary
  and hold only the terminal output until worker ACK materialization.
• The existing get_grammar_bitmask structured-output implementation is
  unchanged; it only appears in large Scheduler diff context.

Remove obsolete KV-offload sidecars

• kv_transfer/.../sidecar.py, offloading/sidecar.py, and the sidecar test are
  deleted; offloading_connector.py drops the corresponding hooks.
• R3 is addressed by immutable KV-compatible keys, not physical offload slots.
  Moving it in every CPU KV transfer would duplicate storage and couple
  Artifact correctness back to one KV connector implementation.

Terminal API delivery

• serial_utils.py adds ndarray .npy base64 encoding.
• Completion, chat, and token-in/token-out protocol/serving files attach actual
  R3 only to the terminal choice/output. HTTP users do not receive SHM paths and
  do not concatenate blocks themselves.
• Streaming R3 responses are intentionally not added by this PR. Artifact R3
  is returned only by the existing non-streaming terminal response paths.
• Stop strings detected in the frontend retain their exact executed-token
  boundary while the terminal response waits for Artifact finalization. A
  terminal routed_experts value is itself the ACK-backed completion signal;
  stale frames without that value cannot complete a pending frontend stop.

Weight-version isolation

• EngineCore propagates the current model weight version into Artifact keys.
  Prefix reuse across in-place model updates therefore cannot return R3 produced
  by an older weight version.

Validation

Current unit and static checks

Run against the exact tree in commit 8d38626b00:

git diff --check

/home/aoshen/vllm/.venv/bin/pre-commit run ruff-check --files <all changed/new Python files>
/home/aoshen/vllm/.venv/bin/pre-commit run ruff-format --files <all changed/new Python files>
/home/aoshen/vllm/.venv/bin/pre-commit run mypy-3.12 \
  --hook-stage manual --files <all changed/new Python files>

/home/aoshen/vllm/.venv/bin/python -m pytest \
  tests/distributed/artifact_connector/test_shm.py \
  tests/model_executor/test_routed_experts_capture.py \
  tests/config/test_artifact_config.py \
  tests/utils_/test_serial_utils.py \
  tests/v1/engine/test_output_processor.py::test_stop_string_waits_for_artifact_terminal_output \
  -q

Results:

• git diff --check: passed.
• ruff check and format: passed.
• Python 3.12 mypy: passed.
• focused unit suite: 98 passed, with only pre-existing/dependency warnings.

GPU E2E status

Earlier TP1/nightly numbers were removed from this description because that
container had received a whole-source overlay on top of a different nightly
build; its Python/native-extension provenance was not trustworthy. They are
not evidence for this commit.

Clean GPU revalidation is in progress under vllm-agent-infra with a dedicated
SLURM claim, task-specific container, and path-identical bind mount of this
worktree. No editable install and no container-source overwrite will be used.
The following runtime matrix remains required before this draft can be marked
ready:

• MRV2 + SHM, TP1 and TP2;
• MTP speculative decoding with target-only R3 capture;
• explicitly enabled prefix caching with repeated-prefix reuse;
• terminal length finish and frontend stop-string finish;
• asynchronous scheduling;
• kv_role=kv_both CPU KV offload compatibility;
• negative startup checks for unsupported PP, DCP/PCP, and disaggregated KV.

Compatibility represented by this PR

• Covered by unit/static checks: immutable SHM objects, KV-compatible full and
  partial keys, weight-version isolation, logical request buffering, terminal
  ACK/materialization, frontend stop clipping, config validation, and TP-rank
  ownership construction.
• Allowed by design but not yet claimed as GPU-validated in this revision:
  MRV2, TP1/TP2, asynchronous scheduling, MTP, prefix caching, SHM, and
  kv_role=kv_both CPU KV offload.
• Fail-closed for now: PP, DCP/PCP, PD-disaggregated KV transfer, non-SHM
  cross-node Artifact storage, and MRV1 Artifact mode.

Duplicate-work and AI assistance

Upstream open-PR searches found no implementation of this immutable logical R3
Artifact Connector. vllm-project/vllm#45635
and PR12 provide the capture/offload baseline; they do not provide this key
space, store boundary, fail-closed prefix semantics, or terminal Artifact ACK
path. This draft supersedes #4 because its cache and lifecycle contracts are
materially different.

AI assistance was used for implementation, simplification, testing, and PR
documentation. The human submitter must review every changed line, rerun the
remaining compatibility matrix, and be able to defend the design end to end
before marking this draft ready.