Skip to content

[PD] Rate limit prefill inflight polling warnings #24967

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ShangmingCai merged 3 commits into from
May 12, 2026
Merged

[PD] Rate limit prefill inflight polling warnings #24967

Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
88a4288
[PD] Rate limit prefill inflight polling warnings
May 11, 2026
aaf3ca6
Update python/sglang/srt/disaggregation/prefill.py
tangcy98 May 11, 2026
bbae058
Use logger.warning_once
May 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 36 additions & 5 deletions python/sglang/srt/disaggregation/prefill.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,10 @@
release_kv_cache,
)
from sglang.srt.mem_cache.memory_pool import HybridLinearKVPool, NSATokenToKVPool
from sglang.srt.observability.req_time_stats import set_schedule_time_batch
from sglang.srt.observability.req_time_stats import (
monotonic_time,
set_schedule_time_batch,
)

if TYPE_CHECKING:
from torch.distributed import ProcessGroup
Expand All @@ -67,6 +70,28 @@
logger = logging.getLogger(__name__)


def _log_prefill_inflight_poll_warning(
self: Scheduler,
req: Req,
message: str,
) -> None:
now = monotonic_time()
warning_state = getattr(self, "_prefill_inflight_poll_warning_state", None)
if warning_state is None:
warning_state = {}
self._prefill_inflight_poll_warning_state = warning_state

warning_interval_s = 1.0
last_log_time, suppressed = warning_state.get(req.rid, (0.0, 0))
if last_log_time > 0 and now - last_log_time < warning_interval_s:
warning_state[req.rid] = (last_log_time, suppressed + 1)
return

suppressed_msg = f", suppressed_repeats={suppressed}" if suppressed else ""
logger.warning(f"{message}{suppressed_msg}")
warning_state[req.rid] = (now, 0)


def release_req_to_metadata_buffer(
req: Req, allocator: ReqToMetadataIdxAllocator
) -> None:
Expand Down Expand Up @@ -613,9 +638,11 @@ def process_disagg_prefill_inflight_queue(
KVPoll.Success,
KVPoll.Failed,
):
logger.warning(
_log_prefill_inflight_poll_warning(
self,
req,
f"PP rank {self.pp_rank}: unexpected poll state {poll} for rid {req.rid} "
f"from consensus; treating as undone"
f"from consensus; treating as undone",
)
undone_reqs.append(req)
continue
Expand Down Expand Up @@ -646,14 +673,18 @@ def process_disagg_prefill_inflight_queue(
if self.enable_metrics:
self.metrics_collector.increment_transfer_failed_reqs()
else:
logger.warning(
_log_prefill_inflight_poll_warning(

@ShangmingCai ShangmingCai May 11, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not just use logger.warning_once?

@tangcy98 tangcy98 May 11, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that logger.warning_once would be more concise, but I'm unsure whether it is needed here to inform the user about the real-time status of this request. Using warning_once would lose track of the request's status. What do you think? I can update it to a more suitable implementation.

@ShangmingCai ShangmingCai May 12, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In fact, in process_disagg_prefill_inflight_queue, the kvpoll state should never beKVPoll.Bootstrapping, I have never seen this warning log before. I assume you are using nixl/mori backend?

@tangcy98 tangcy98 May 12, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am using Dynamo+SGLang+Qwen3.5-122B-A10B+Mooncake in a 4p1d scenario. Here is my start command:

python3 -m dynamo.sglang
--model-path "$PREFILL_MODEL" \
--served-model-name "$MODEL_NAME" \
--trust-remote-code \
--enable-metrics \
--collect-tokens-histogram \
--tp-size 1 \
--quantization w4afp8 \
--kv-cache-dtype fp8_e4m3 \
--mem-fraction-static 0.875 \
--context-length 32768 \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_coder \
--speculative-algorithm NEXTN \
--speculative-num-steps 2 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 3 \
--chunked-prefill-size 16384 \
--max-running-requests 4 \
--max-mamba-cache-size 20 \
--mamba-scheduler-strategy extra_buffer \
--tokenizer-backend fastokens \
--linear-attn-backend flashinfer \
--page-size 64 \
--disable-cuda-graph \
--disaggregation-mode prefill \
--disaggregation-transfer-backend mooncake \
--disaggregation-bootstrap-port "$bootstrap_port" \
"${ib_args[@]}" \
--host 0.0.0.0 \
--port "$sglang_port"

@ShangmingCai ShangmingCai May 12, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think warning_once should be fine. Now we need to figure out why a req could be KVPoll.Bootstrapping status when in process_disagg_prefill_inflight_queue, it shouldn't be at this status anymore when it leaves pop_bootstrapped.

self,
req,
f"Unexpected polling state {poll} for rid {req.rid} in inflight queue; "
f"treating as undone"
f"treating as undone",
)
undone_reqs.append(req)

for req in done_reqs:
req.time_stats.set_completion_time()
if hasattr(self, "_prefill_inflight_poll_warning_state"):
self._prefill_inflight_poll_warning_state.pop(req.rid, None)

for req in done_reqs:
if isinstance(req.finished_reason, FINISH_ABORT):
Expand Down
Loading