Skip to content

fix(agent): abort timed-out SGLang requests via router workers - #2340

Merged
zhuzilin merged 5 commits into
THUDM:mainfrom
Arlo-mt:fix/sglang-abort
Sep 3, 2026
Merged

zhuzilin merged 5 commits into
THUDM:mainfrom
Arlo-mt:fix/sglang-abort

Conversation

@Arlo-mt

@Arlo-mt Arlo-mt commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Background

In the agentic rollout path, each agent turn sends a request to SGLang through the SGLang router. When a request is cancelled or hits the client-side read timeout, Slime must explicitly abort the corresponding SGLang request to release its scheduler slot and KV-cache resources.

Problem

sglang_url points to the SGLang router address, but the router does not expose or forward the /abort_request endpoint.

The existing cleanup logic sends:

POST <router>/abort_request

This request returns HTTP 404 and the exception is silently ignored. As a result, the original request may continue decoding on the worker even though the client has already timed out. The retry request can then be routed to another DP worker, causing the old request and the retry request to run concurrently.

This leads to:

  • Orphaned SGLang requests occupying KV-cache resources
  • Duplicate computation after retries
  • Multiple DP workers decoding requests from the same logical agent turn
  • Longer tail latency and possible additional timeout failures

Solution

This change adds a small helper, _abort_sglang_request, in slime/agent/adapters/common.py.

When a request is cancelled or times out:

  1. Query the router's /workers endpoint.
  2. Extract the backend worker URLs.
  3. Send /abort_request with the original request rid to every worker concurrently.
  4. Keep the abort request scoped to the specific rid; abort_all is never used.

Only the worker that owns the request will actually abort it. Requests with other IDs are not affected.

Scope

  • Modify only slime/agent/adapters/common.py
  • No new dependencies
  • No SGLang router changes
  • No public API changes
  • Preserve support for the current router-based SGLang deployment

Validation

  • python3 -m py_compile slime/agent/adapters/common.py
  • git diff --check
  • Manual async test with a fake router and two fake workers
  • Verified that the same rid is sent to all backend workers

@zhuzilin
zhuzilin merged commit 4c193f1 into THUDM:main Sep 3, 2026
73 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants