[NickCao]

Purpose

This PR fixes the Realtime auto-truncation issues identified in #8339 and aligns our behavior with the OpenAI Realtime GA model.

Previously, automatic truncation could remove items from retained conversation history and submit oversized prompts to the engine. This PR separates the two concepts:

• Conversation history remains complete and retrievable.
• Model context is a suffix of that history and may be pruned automatically.

Key changes:

• Enforce the model context length during speculative Realtime rendering, including output-token reservations and custom truncation limits.
• Treat only input-length validation failures as over-limit errors; unrelated rendering errors now propagate normally.
• Match OpenAI GA behavior: automatic truncation does not emit conversation.item.deleted, and removed items remain available through conversation.item.retrieve.
• Preserve full history while maintaining a monotonic model-context cursor.
• Use binary search over the retained suffix to find the next feasible context, avoiding repeated full-history scans.

Test Plan

vLLM Version: 0.31.0

vLLM-Omni Commit: 9d352ba

pytest tests/entrypoints/openai_api/test_realtime_prompt_preflight.py

Test Result

passed

[vllm-omni-review-bot]

This PR appears to belong to: docs/design/module/profiling.md, docs/design/module/entrypoints.md, docs/design/module/observability.md.

Module owners: @alex-jw-brooks @linyueqian @gcanlin

Routing: @alex-jw-brooks via module of the changed files, semantic router, CODEOWNERS; @linyueqian via module of the changed files, semantic router, CODEOWNERS; @gcanlin via module of the changed files

@NickCao, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[vllm-omni-review-bot]

Omni ReviewBot routing record

Assigned Strict on zcode (GLM-5.3-Flash) under experiment fleet-strict-cursor-grok46-zcode-glm53flash-5050-c5-z10-20261002.

[vllm-omni-review-bot]

Omni ReviewBot: three questions on the performance claim

@NickCao this PR reads as a performance or value claim:

• title: [BugFix] Optimize turn-based /v1/realtime auto truncation

Before the full evidence checklist, three short questions:

1. Bottleneck — what is the current bottleneck, and which profile, trace or per-stage measurement shows it?
2. Value — what does the change buy the user or the system (latency, throughput, memory, cost), and at which workload?
3. A/B or ablation — is there a same-workload, same-head/config comparison that isolates each main claim on its own? For stacked optimizations, one number per item rather than a blended delta.

When you answer, the evidence that settles it is: base and head SHA, hardware, model, workload, warm-up and repeat count, mean or percentiles with their spread, and a correctness/quality-equivalence signal; an end-to-end claim also needs stage attribution.

[NickCao]

Omni ReviewBot: three questions on the performance claim

@NickCao this PR reads as a performance or value claim:

* title: [BugFix] Optimize turn-based /v1/realtime auto truncation

This is primarily a correctness fix (to match the observed openai API behavior), with some added performance improvement (replacing a linear scan with bisect).

Before the full evidence checklist, three short questions:

1. **Bottleneck** — what is the current bottleneck, and which profile, trace or per-stage measurement shows it?

The current bottleneck is in prompt rendering during the scan for finding the cut to make the prompt fit under the budget, previously the linear scan would take O(n) to decide the number of turns to truncate, the bisect would take O(logn).

2. **Value** — what does the change buy the user or the system (latency, throughput, memory, cost), and at which workload?

The gain is on latency, previously there would be a huge latency spike when auto truncation kicks in, with the new implementation the spike would be much flatter (O(n) -> O(logn))

3. **A/B or ablation** — is there a same-workload, same-head/config comparison that isolates each main claim on its own? For stacked optimizations, one number per item rather than a blended delta.

• Base: 65699fe94
• Head: 9d352bae9
• Synthetic alternating user/assistant Realtime items

History items	Base renders	Head renders	Render reduction	Base median	Head median	Speedup
192	109	8	92.7%	2.55 s	125 ms	20.4×
256	173	9	94.8%	5.02 s	158 ms	31.7×
384	301	9	97.0%	12.05 s	186 ms	64.6×
512	429	10	97.7%	21.78 s	239 ms	91.1×