fix(mllm_scheduler): drain requests dict and clear metal cache after completions

[kol22]

Summary

• Add self.requests.pop(request_id, None) in abort_request() and _cleanup_finished() to drain the request tracking dict
• Add mx.clear_cache() after _cleanup_finished() in step() to release MLX metal buffer pool memory back to the OS

Context

The MLLMScheduler has two resource leaks that compound under continuous batching with vision requests:

1. self.requests dict never drained. Requests are added at add_request() but never removed during the normal lifecycle. _cleanup_finished() marks requests in finished_req_ids and cleans up running/UID mappings, but leaves self.requests intact. A remove_finished_request() method exists but is only called from EngineCore, which uses the regular Scheduler, the MLLM scheduler's own async loop never calls it. Over time, self.requests grows without bound (7,000+ orphaned entries observed per backend before OOM).

2. MLX metal buffer pool monotonic growth. When KV cache tensors are deallocated after request completion, MLX returns the GPU memory to its internal metal buffer pool rather than the OS. Without an explicit mx.clear_cache() call, the pool grows monotonically. The regular Scheduler already received this fix in PR #44 (merged Feb 9), but the MLLMScheduler was missed.

Together, these leaks cause backends to hit kIOGPUCommandBufferCallbackErrorOutOfMemory after hours of continuous serving with multimodal requests.

Observed impact

Tested on Mac Studio M3 Ultra (96 GB) running two vllm-mlx backends with --continuous-batching and Qwen3.5-35B-A3B (4-bit):

Metric	Before	After
Soak test result	94.4% error rate (1201/1272 failures)	0% error rate (8279/8279 OK)
Failure mode	kIOGPUCommandBufferCallbackErrorOutOfMemory after ~15 min	Stable over 100+ min
self.requests size per backend	7,085 orphaned entries at crash	0 (properly drained)

[waybarrios]

Nice work on this one @kol22 , the OOM fix is legit. Tested numbers look great too. Only thing I'd maybe think about later is doing the cache clearing on a timer like the LLM scheduler does instead of only when requests finish, since a big video request could still pile up memory if nothing completes for a while. But honestly that's a separate thing, this fixes the actual problem.

[waybarrios]

Let me know if you wanna work in the last thing I mention @kol22

[kol22]

@waybarrios I'll work this in