[Lingbot World] Bound image condition storage and extend temporal RoPE

[wtz2333]

Purpose

Condition. Realtime and stepwise generation previously stopped after ten blocks because the image condition covered only 117 pixel frames (30 latent frames). This PR advances a session-owned causal Wan VAE encoder to produce each three-latent-frame condition block from the initial image followed by zero pixel frames. Carrying the encoder history forward preserves the evolving causal context, including the initial image's contribution to later blocks. The implementation retains one current condition block and bounded encoder histories, removes the request/example caps, and accounts for committed and in-flight state in session admission.

RoPE. The transformer also rejected temporal positions beyond its precomputed RoPE table (1024 latent frames by default). Cached positions keep using the existing table; blocks crossing or exceeding it compute only their own temporal cos/sin values using absolute positions and the original frequencies. The table stays fixed in size, and spatial limits remain unchanged.

These model-side changes follow RFC #7074. Condition encoding requires the unpatched, non-tiled Wan encoder. Shared paged-KV length handling and metadata compaction remain a separate follow-up: at 480×832, the default 1 << 20 token limit is reached during the 225th generation block. This PR does not claim unbounded end-to-end realtime generation.

What Changes

File	Change
vllm_omni/diffusion/models/lingbot_world/pipeline.py	Advance causal condition encoding in realtime and stepwise paths; commit encoder history after successful blocks, release it on close/reset, account for encoder caches, and remove the frame cap.
vllm_omni/diffusion/models/lingbot_world/transformer.py	Compute temporal RoPE beyond the cached horizon without growing the table.
examples/offline_inference/diffusion/lingbot_world_v2.py	Remove the 117-frame example limit.
examples/offline_inference/diffusion/lingbot_world_v2_realtime.py	Remove the ten-event limit and describe stateful condition encoding.
tests/diffusion/models/lingbot_world/test_pipeline_lingbot_world.py	Cover causal condition progression, session isolation and cleanup, failure handling, memory accounting, and tick/stepwise agreement.
tests/diffusion/models/lingbot_world/test_lingbot_world_transformer.py	Cover RoPE boundary crossings and fixed cache size.
docs/design/feature/realtime_ar_diffusion.md	Document condition encoder histories, their lifetime, memory reservation, and supported VAE path.

Test Plan

vLLM Version: 0.29.0

vLLM-Omni Commit: main db40ed3a3179d9125408dce9c89f1c97b5a5d03a vs current 142ff5dcac382cb9d65909120be8846c7ac7e22f, based on the same main commit. The separate KV fix is excluded.

Environment: one H200, PyTorch 2.13.0+cu130, diffusers 0.38.0, transformers 5.14.1; 832×480, BF16, TP1, eager, FLASH_ATTN, batch/concurrency 1, four DMD steps, flow shift 5, seed 42.

Check	Workload	Status
Chunk latency and latent equality	10 chunks / 117 equivalent video frames; fresh main → current → main runs on the same GPU, no separate warmup	Completed.
Side-by-side video	Decode the completed main/current latent outputs with the same full BF16 Wan VAE and MP4 settings, 117 frames at 16 FPS	Completed: decoded RGB and MP4 bytes identical.
Temporal RoPE boundary	Current encoder, 344 chunks / 1032 latent frames / 4125 video frames at 16 FPS; continuous session and streaming decode	Completed with the diagnostic KV override below; all 344 chunks and 4125 decoded frames validated.

All checks use the same lakeside image/prompt. Each camera event has an increasing event_id and three action lists, e.g. {"event_id": 0, "frames": [["d"], ["d"], ["d"]]}. Short runs use ten d events; the long run alternates 20 d events with 20 a events.

At this resolution, each latent frame consumes 1560 tokens. The default shared KV limit (1 << 20) is reached in block 225, before the RoPE crossing in block 342 (zero-based index 341, temporal positions 1023–1025). The long test therefore uses an isolated copy of the current commit with only max_model_len=1 << 21 added to LingBot's ARDiffusionKVCacheSpec. This experiment-only override is not part of the PR; 344 blocks require 1,609,920 tokens. It isolates RoPE behavior and cannot establish that the default shared-KV limit has been removed.

The existing entrypoint below exercises the short configuration. $INPUT_IMAGE, $PROMPT, $EVENTS_JSONL, and $OUTPUT_DIR are caller-supplied. The original image/prompt are not bundled in this PR, so replacement inputs reproduce the procedure rather than the exact numerical results. The example saves latent chunks; video decoding/export is a separate step. The long diagnostic additionally requires the stated KV override and 344 events.

python examples/offline_inference/diffusion/lingbot_world_v2_realtime.py \
  --model robbyant/lingbot-world-v2-14b-causal-fast-diffusers \
  --image "$INPUT_IMAGE" --prompt "$PROMPT" \
  --events "$EVENTS_JSONL" --output-dir "$OUTPUT_DIR" \
  --height 480 --width 832 --seed 42 \
  --gpu-memory-fraction 0.1 --tensor-parallel-size 1 --enforce-eager

python -m pytest tests/diffusion/models/lingbot_world \
  tests/diffusion/ar_diffusion/test_capability_runner.py \
  tests/diffusion/ar_diffusion/test_worker_lifecycle.py \
  tests/diffusion/ar_diffusion/test_streaming_decode.py \
  -q -m 'core_model and cpu' -o 'addopts=--strict-markers --strict-config'

Test Result

480p numerical equality

All ten latent chunks are exactly equal across main → current → main: maximum absolute difference and RMSE are both 0. All 30 outputs are finite and have the expected shape. This covers one image/prompt and control sequence. The 117-frame main/current videos also have exactly matching float RGB values and MP4 bytes; the side-by-side video contains 117 frames at 16 FPS (1664×480).

comparison.mp4

480p long rollout / RoPE boundary

• 480p long video — 344 chunks, 4125 frames, 16 FPS; experimental KV cap 1 << 21

The current stateful encoder completed 344 continuous chunks (1032 latent frames), producing a validated 4125-frame video at 832×480 and 16 FPS (257.8125 seconds). Every latent chunk was finite with the expected shape and consecutive event IDs. Chunk index 341 covers temporal positions 1023–1025 and crosses the original 1024-position RoPE table; chunks 342 and 343 also completed, ending at position 1031. The existing streaming Wan decoder maintained one continuous state across the full video.

This verifies the tested RoPE crossing with the experiment-only shared KV cap raised to 1 << 21. The default KV cap remains unchanged in the PR; this result does not establish unbounded default-configuration generation or long-video perceptual quality.

480p chunk latency

Times are milliseconds. Main is the mean of its two runs; current has one run. Request latency is the sum of ten session.next_chunk() wall-clock measurements, excluding model loading, between-chunk saving, VAE decoding, and MP4 encoding. Subsequent chunks have changing context lengths and are not independent steady-state repeats.

Metric	Main mean	Current	Change
First chunk	5,120.18	3,635.03	−29.01%
Subsequent 9 chunks, total	22,326.03	23,937.27	+7.22%
All 10 chunks, total	27,446.21	27,572.30	+0.46%

All three runs succeeded. Main's full-request totals were 27,836.25 and 27,056.17 ms (2.84% spread), so the +0.46% total difference is within observed baseline variation. Its subsequent-nine-chunk totals differed by only 0.124%; current adds 171–185 ms per subsequent chunk, averaging 179 ms.

Why the latency shifts. Main prepares all 117 condition pixel frames in the first chunk: 30 Wan encoder forwards, followed by condition slicing in later chunks. Current performs three encoder forwards per chunk: nine opening pixel frames, then twelve fresh pixel frames per subsequent chunk, carrying causal history across calls. The first chunk therefore avoids 27 encoder forwards, while each later chunk performs work that main already prepaid. Both paths perform 30 encoder forwards over this ten-chunk workload. The observed first-chunk reduction and later increase are consistent with this redistribution; exact per-stage attribution has not been measured.

This is not an additional condition/KV writeback pass. Committing encoder_cache replaces a Python reference, and preparing the pending cache shallow-copies a list rather than cloning all tensors. Wan's internal feature-cache clone/cat operations still run as part of encoding; main performed those operations during its initial full encode. The existing four denoising forwards plus the fifth clean-latent KV-commit forward are unchanged. Additional per-chunk projection, normalization, allocation, and cache lifetime may contribute, but their individual costs are unmeasured. The short test never reaches the RoPE extension path.

Process wall times including startup/shutdown were 101,410 / 78,422 / 75,004 ms in A/B/A order; model loading alone was 19,714 / 17,588 / 16,865 ms. Startup variation is not treated as an inference speedup. All three reported 77.18 GiB of process GPU memory after loading, not peak memory. Both revisions admitted one session after automatic KV-budget adjustment. This is an exploratory comparison, not a steady-state or tail-latency benchmark.

CPU checks: 286 passed, 3 deselected. Ruff lint/format checks and git diff --check passed.

**BEFORE SUBMITTING:**read CONTRIBUTING.md and run the precheck-pr skill with the code agent for a self-check against project conventions.
(anything written below this line will be removed by GitHub Actions)

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[vllm-omni-review-bot]

This PR appears to belong to: docs/design/module/diffusion/diffusion_model_integration.md.

Module owners: @fhfuih @Bounty-hunter @wtomin @RuixiangMa

@wtz2333, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[vllm-omni-review-bot]

Omni ReviewBot triage note

Resolved as of 8adcb9449c29: the high-priority or low-quality signal noted on an earlier commit no longer applies.

[wtz2333]

Two remaining entrypoint limits found at 142ff5dcac382cb9d65909120be8846c7ac7e22f:

1. [P2] The camera loader still truncates trajectories to 117 frames. The PR removes the request and CLI frame caps, but camera.py:210–212 still slices to _MAX_ACTION_FRAMES = 117. Loading 129-frame poses.npy / intrinsics.npy through action_path therefore returns only 117 frames, and the downstream camera-length check rejects the newly accepted 129-frame request. Reproduced with real NPY files: source=129, returned=117, request=129. The new 129-frame test supplies _lingbot_camera_trajectory directly, bypassing this loader. Please update the shared loader to retain the required trajectory while preserving its source-size and dtype checks, and add a regression through the file-loading path.

2. [P2] The streaming-pixel example still rejects the eleventh block. ar_diffusion_streaming_decode.py:157–162 still derives max_chunks = 10 from the old 117-frame horizon. A valid eleven-block action script is rejected before reaching the extended stepwise pipeline. Calling the actual validation helper accepts ten blocks and rejects eleven with 117-frame image-condition horizon fits at most 10. Please remove this stale horizon check, retain per-block action-shape validation, and cover eleven blocks in the example tests.

Both entrypoint issues were reproduced on CPU without model weights.

[wtz2333]

Two remaining entrypoint limits found at 142ff5dcac382cb9d65909120be8846c7ac7e22f:

1. [P2] The camera loader still truncates trajectories to 117 frames. The PR removes the request and CLI frame caps, but camera.py:210–212 still slices to _MAX_ACTION_FRAMES = 117. Loading 129-frame poses.npy / intrinsics.npy through action_path therefore returns only 117 frames, and the downstream camera-length check rejects the newly accepted 129-frame request. Reproduced with real NPY files: source=129, returned=117, request=129. The new 129-frame test supplies _lingbot_camera_trajectory directly, bypassing this loader. Please update the shared loader to retain the required trajectory while preserving its source-size and dtype checks, and add a regression through the file-loading path.
2. [P2] The streaming-pixel example still rejects the eleventh block. ar_diffusion_streaming_decode.py:157–162 still derives max_chunks = 10 from the old 117-frame horizon. A valid eleven-block action script is rejected before reaching the extended stepwise pipeline. Calling the actual validation helper accepts ten blocks and rejects eleven with 117-frame image-condition horizon fits at most 10. Please remove this stale horizon check, retain per-block action-shape validation, and cover eleven blocks in the example tests.

Both entrypoint issues were reproduced on CPU without model weights.

Fixed it

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-15T06:50:33.016988Z	8adcb94	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8adcb9449c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Update the shipped recipe after lifting the frame cap

When requests above 117 frames are accepted here, the canonical recipes/Robbyant/LingBot-World-2.0.md:50-53 instructions become incorrect: they still state that raw frame counts are capped at 117. Users following the model recipe will therefore avoid the longer offline rollouts this change explicitly enables, so the documented limit should be removed or replaced with the actual remaining constraints.

Useful? React with 👍 / 👎.

[tzhouam]

LGTM