[Bugfix][KV Offload][OBJ] Preserve job completion during cleanup

[mindungil]

Purpose

The OBJ secondary tier removes a transfer from _transfers before releasing its NIXL resources. If release_xfer_handle() raises, no JobResult is published and the OBJ tier has already discarded the only state needed to retry cleanup.

The parent TieringOffloadingManager therefore cannot pop the corresponding _transfer_jobs entry or call primary.complete_read(). For a primary-to-OBJ store, the CPU cache block remains pinned and has_pending_work() remains true.

NIXL documents that releasing an active transfer can fail when the transfer cannot be cancelled; in that case the handle is not freed:
https://github.com/ai-dynamo/nixl/blob/main/src/api/python/_api.py#L705-L714

Fix

• Keep each transfer entry in _transfers until release_xfer_handle() succeeds.
• If releasing the transfer handle raises, continue polling and retry cleanup on the next scheduler step.
• After the transfer handle is released, treat descriptor-list and OBJ registration cleanup as best effort so those exceptions cannot suppress the job completion.
• Publish each JobResult exactly once.

This preserves the safety boundary: the parent is not allowed to reuse primary-tier memory while a NIXL transfer may still be active.

Regression proof

The regression test connects the real TieringOffloadingManager, CPU primary tier, and OBJ secondary tier, and drives the public on_schedule_end() path. It injects:

1. an exception from check_xfer_state(), leaving the transfer state unknown; and
2. one failure from release_xfer_handle(), representing an active transfer that NIXL cannot cancel yet.

On unmodified main, the same test fails with RuntimeError: transfer is still active. A diagnostic assertion immediately after that failure observes:

parent_jobs=1
block_ref_cnt=1
has_pending_work=True
obj_transfers=0

The parent job and CPU pin remain, but the OBJ tier no longer has the transfer entry needed to finish them.

With this change, the first scheduler poll retains all of that state safely. The next poll retries cleanup, after which the test observes an empty parent job map, ref_cnt == 0, no OBJ transfer, and has_pending_work() == False.

Live NIXL/OBJ validation

In addition to the mock-based failure injection, I ran the real ObjectStoreSecondaryTierManager with NIXL 1.3.1's OBJ backend against a local S3-compatible Moto server:

1. The startup connectivity probe reached the test bucket.
2. NIXL asynchronously wrote one 1,024-byte CPU DRAM block to an actual S3 object.
3. The CPU block was zeroed locally.
4. NIXL read the object back into the same CPU block.
5. The restored tensor matched the original byte-for-byte.
6. S3 listing confirmed one .bin object with size 1,024 bytes.
7. Manager shutdown completed normally.

Observed output:

NIXL is available
Backend OBJ was instantiated
Object store tier connectivity probe succeeded
LIVE_OBJ_PASS restored_bytes=1024
LIVE_S3_OBJECT ...0000000000000065.bin size=1024

This validates the normal live WRITE/READ and cleanup path without a GPU. The cleanup exceptions themselves remain deterministic fault injection because reproducing a real transport cancellation failure on demand is not reliable.

Duplicate-work check

No open issue or PR found for lost OBJ job completion during NIXL cleanup.

Related work is not a duplicate:

• [Bugfix] Remove failed transfer from inflight on NIXL poll exception (#49528) #49561 handles a check_xfer_state() exception in the P2P NixlTransport inflight map. It does not touch the OBJ secondary-tier manager or parent job completion.
• [KV Offload] Fix failed-load livelock by marking the lookup verdict as a miss #49328 invalidates cached lookup verdicts after failed promotions and hardens the FS tier. Its OBJ change concerns failed-load lookup invalidation, not cleanup retry or completion delivery.
• [Bugfix][KV Offload] Bound the HIT_PENDING wait so a stalled write cannot defer requests indefinitely #49850 bounds request-level waiting on leaked HIT_PENDING entries. It mitigates a downstream symptom and does not repair stalled transfer bookkeeping.

Searches run:

gh pr list --repo vllm-project/vllm --state open --search '"ObjectStoreSecondaryTierManager"'
gh pr list --repo vllm-project/vllm --state open --search '"release_xfer_handle" OBJ'
gh pr list --repo vllm-project/vllm --state open --search '"cleanup" "JobResult" "KV Offload"'

Test plan

.venv/bin/python -m pytest -q tests/v1/kv_offload/tiering/test_obj_tier.py
# 36 passed

pre-commit run --files \
  vllm/v1/kv_offload/tiering/obj/manager.py \
  tests/v1/kv_offload/tiering/test_obj_tier.py
# all applicable hooks passed

.venv/bin/python tools/pre_commit/mypy.py 3.10 vllm/v1/kv_offload/tiering/obj/manager.py tests/v1/kv_offload/tiering/test_obj_tier.py
.venv/bin/python tools/pre_commit/mypy.py 3.11 vllm/v1/kv_offload/tiering/obj/manager.py tests/v1/kv_offload/tiering/test_obj_tier.py
.venv/bin/python tools/pre_commit/mypy.py 3.12 vllm/v1/kv_offload/tiering/obj/manager.py tests/v1/kv_offload/tiering/test_obj_tier.py
.venv/bin/python tools/pre_commit/mypy.py 3.13 vllm/v1/kv_offload/tiering/obj/manager.py tests/v1/kv_offload/tiering/test_obj_tier.py
# all passed

Model evaluation: N/A. This changes transfer cleanup and bookkeeping only; it does not affect model output or accuracy.

AI assistance

This PR includes AI-assisted code and analysis from OpenAI Codex. I reviewed every changed line, understand the failure mode and fix, and take responsibility for the contribution.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

PRs do not trigger a full CI run by default. Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

Agent Guidelines

IMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban.

🚀

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[orozery]

Thanks for the fix @mindungil !