[FS Offloading] Move FSAsyncLookup into a subprocess

[varun-sundar-rabindranath]

Purpose

TieredOffloading with FS tier on main yields poor performance. This is mostly due to lookup delays triggered by the FSAsyncLookupManager.
We find that the thread backed FSAsyncLookup is severely impacted by the GIL.

PR #45765 Introduced a timeout for this as a stop-gap. This PR attempts to remove the GIL contention by moving the core part of FSAsyncLookup into a subprocess.

Changes:

• Introduce FsIoProcess. This class,

  • Creates a process
  • Destroys the process
  • Uses MessageQueue from vLLM infrastructure for communication.
  • Passes batch lookup commands to the process
  • Retrieves lookup results from the process
    Note that only the core batch lookup (group by requests + os.path.exists) is executed in the subprocess. All other functionality still lives in the main/scheduler thread.

• Refactor AsyncLookupManager.

  • Introduce BaseAsyncLookupManager. This class is responsible for,
    • All the lifecycle logic (create lookup_batch, flush, drain) as exists in main.
    • introduce 2 new abstract methods,
      • send_lookup_batch,
      • try_recv_result_batch
  • Redefine AsyncLookupManager to be thread-backed, like in main, and specialize send_lookup_batch and try_recv_result_batch to simply write and read from a SimpleQueue.
  • Redefine FSAsyncLookupManager to interface with FsIoProcess

Additionally FsIOProcess can be expanded to handle load_blocks, store_blocks in the future.

Test Plan

Test Result

[varun-sundar-rabindranath]

Requesting reviews from @tlrmchlsmth @njhill and @robertgshaw2-redhat for FsIOProcess .

I ended up using MessageQueue from vLLM infra for communication with the main thread.
Gemini tells me multiprocessing.Queue can block on put if there is system back-pressure and that MessageQueue simply switches from shm to zmq. We don't want the scheduler thread blocking and I chose to use MessageQueue.
I also have a feeling that MessageQueue might be a overkill for this.

Please advise 🙌

[tlrmchlsmth]

Nice work @varun-sundar-rabindranath

One downside of doing this in a separate process is a lot of added complexity around errors handling and process cleanup. Looks like there are a few things that aren't handled yet in this PR -- I had claude take a look, here are some potential issues:

Details

Confirmed findings

1. vllm/v1/kv_offload/tiering/fs/io_process.py:75 — wait_until_ready() has no timeout; parent deadlocks if child crashes mid-handshake

If the child process crashes after sending result_handle via the pipe but before completing its wait_until_ready() calls, the parent blocks forever on self._request_queue.wait_until_ready() (line 75) or self._result_queue.wait_until_ready() (line 76).
MessageQueue.wait_until_ready() uses raw socket.recv() with no timeout — it's a collective barrier. The pipe-based timeouts (lines 64, 79) only protect the pipe reads, not the MessageQueue handshake. Trigger: child OOM, segfault, or exception between
sending the handle and completing the barrier.

2. vllm/v1/kv_offload/tiering/fs/io_process.py:88 — No child liveness detection; silent hang if subprocess dies at runtime

Once the handshake completes, the parent has no heartbeat or is_alive() check on the child process. dequeue_lookup_results_nowait() returns None on timeout (line 91), which is indistinguishable from "still computing." If the child is OOM-killed or
segfaults mid-lookup, LookupState.result stays None forever — drain_results() just sees no results and breaks (async_lookup.py:137). The only use of is_alive() is in shutdown(). No staleness timeout exists on pending lookups. Trigger: child killed by OS,
unhandled exception in _fs_lookup_worker_main, or NFS hang causing os.path.exists() to block indefinitely.

---

Plausible findings

3. vllm/v1/kv_offload/tiering/fs/io_process.py:85 — enqueue_lookup_batch() has no guard against post-shutdown calls

enqueue_lookup_batch() calls self._request_queue.enqueue(batch) with no check of self._shutting_down. If shutdown() has already been called (setting _shutting_down = True and calling _request_queue.shutdown()), a subsequent flush() → send_lookup_batch()
→ enqueue_lookup_batch() will call enqueue() on a closed MessageQueue. Both operations happen on the scheduler thread, so this depends on the engine shutdown sequence: if scheduling isn't fully quiesced before tier teardown, this sequence is possible.

---

Full failure mechanism catalog (per your request)

Startup │ Child crashes before sending handle │ Yes — 30s pipe timeout
Startup │ Child crashes after handle, before wait_until_ready() │ No — deadlock (finding 1)
Startup │ Child sends malformed handshake │ Yes — format check + RuntimeError
Runtime │ Child crashes/OOM mid-lookup │ No — silent hang (finding 2)
Runtime │ Filesystem hangs (stale NFS) │ No — os.path.exists() blocks child indefinitely, parent unaware
Runtime │ MessageQueue.enqueue() full/timeout in child │ Partial — child exits, but parent doesn't detect
Shutdown │ Batch in-flight when sentinel sent │ Partial — 5s join then terminate, in-flight results lost
Shutdown │ terminate() doesn't kill child │ No — no fallback to SIGKILL
Shutdown │ enqueue() called after shutdown() │ No (finding 3)
Shutdown │ Double shutdown() │ Yes — _shutting_down guard
Serialization │ FileMapper unpicklable │ Not an issue — simple types, fork context
Resources │ Shared memory leak on unclean exit │ Partial — shutdown() cleans up, but parent crash leaks /dev/shm
Resources │ Zombie child if parent crashes │ Partial — child is daemon=True but blocks in dequeue(indefinite=True)
Ordering │ Results arrive out of order │ Not an issue — keyed by OffloadKey, applied idempotently
Ordering │ Results lost on child crash between dequeue/enqueue │ No — same as runtime crash

I think these might be easier to handle by avoiding the additional process but instead using a lower-level language like Rust to avoid the GIL at a higher granularity. I saw that @orozery had some desire to keep things in Python but I'd like to understand the technical reasons why

[orozery]

@varun-sundar-rabindranath thanks for the quick implementation!
I think we can solve this issue in python (without using C/rust), and without spawning a new process.
See my suggestion here:
#45765 (comment)

@tlrmchlsmth I think developing/maintaining python code has many advantages over compiled languages.
I think that it helps vLLM (and sglang) move and develop faster.
Sometime we hit a performance issues while developing.
I think that for some (or most) people, the easiest way (especially with today's agents) is to "give up on python".
I do think that sometimes moving to C/rust makes sense, but I want to be fully convinced first that "there's no other way".
Right now, with this specific issue, I'm not there yet.

[varun-sundar-rabindranath]

closing in favor of C based approach #46713