fix(kvbm): prevent scheduler crash from race between abort and request completion

[lg-epic]

Summary

Fixes a race condition in the KVBM connector that crashes the vLLM EngineCore scheduler with assert req_id in self.requests in _update_from_kv_xfer_finished. The crash is deterministic under streaming load in aggregated inference mode with KVBM.

Root Cause

The race is between _process_aborts_queue() and update_from_output() within a single EngineCore.step(). When a request's abort and stop condition coincide in the same step:

1. execute_model() — worker reports finished_sending (KV offload complete)
2. _process_aborts_queue() — abort arrives, calls request_finished(), slot removed, returns true (request stays in self.requests)
3. update_from_output() — same request hit a stop condition, calls request_finished() again, slot is gone, returns false → vLLM deletes from self.requests
4. update_from_output() continues — _update_from_kv_xfer_finished() processes finished_sending → assert req_id in self.requests → crash

This is specific to KVBM because KV blocks are offloaded incrementally during generation via save_kv_layer(). The final offload can complete in the same step the request finishes, so finished_sending contains the request at the same time it's being freed.

Changes

• leader.rs: Return true when slot is missing in request_finished(), keeping the request alive in self.requests so _update_from_kv_xfer_finished() can process finished_sending and free blocks properly. Aligns with the existing TODO comment (L577-587) that the return value should always be true.
• worker.rs: Replace two panic!() calls with tracing::warn!() + signal completion when slots disappear from maybe_finished_offloading/maybe_finished_onboarding sets between iterations.

Error observed

```
AssertionError at scheduler.py:1950
assert req_id in self.requests
in _update_from_kv_xfer_finished
```

Preceded by:
```
WARN: request_finished called for request_id: ... but slot is not found
WARN: finished request received for unknown request_id; assuming never started
```

Environment

Dynamo v0.9.0, vLLM 0.15.1+cu130, KVBM connector in aggregated inference on L40S GPUs.

Related

• [Bug]: update_from_kv_xfer_finished raise KeyError when using NixlConnector vllm-project/vllm#25562 (original KeyError variant, closed as stale)
• [NIXL] Ignore abort on already-finished request vllm-project/vllm#25067 (added the always return true invariant comment)

Test plan

• Existing CI tests pass (Rust compile + unit tests)
• Verify with aggregated inference workload under load that the assert req_id in self.requests crash no longer occurs
• Verify no request leaks (requests stuck in self.requests forever) by monitoring scheduler queue depth under sustained traffic

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

👋 Hi lg-epic! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀