Mergedphys - #447
Open
3a1b2c3 wants to merge 20 commits into
Open
Conversation
…che) Rebuild the per-block cross-attention text KV in place at a chunk boundary (storage addresses survive, so captured CUDA graphs stay valid) while the self-attention history carries the scene forward under the new prompt. Optional two-prompt edit guidance runs the cond branch under old and new text against the same history and extrapolates flow_old + s*(flow_new - flow_old) for N chunks after a swap; the KV commit always runs single-branch under the new prompt. ReCache (LongLive / Hunyuan-GameCraft-2) re-commits the previous chunk's KV under the new text via a same-index cache-bracket re-open. GPU-verified: swaps are RNG-clean (zero pre-swap divergence), and weather/lighting edits land convincingly at s=3 with training-caption-style phrasing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Route datachannel event messages to a free-text prompt swap (event_id carries the prompt; clear states restore the scene prompt) and to /spawn <preset> [dist] [speed] [lateral] / /clear-actors commands. Spawned actors follow a constant-velocity world trajectory and enter the conditioning through the same Ludus bbox path as gRPC dynamic actors, so the model materializes grounded vehicles/pedestrians the game shell can track. The web client gains a scene-prompt panel and spawn buttons; WASD typed into text fields no longer drives the car. GPU-verified: a spawned car materializes photorealistically within one chunk and vanishes within one chunk of /clear-actors. Guidance defaults (s=3, 6 chunks) follow the calibration sweep. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
smoke_text_edit rolls RNG-matched control/swap/guided/recache variants and reports per-chunk divergence; sweep_text_edit calibrates an edit prompt bank (incl. the scene bundle's native weather phrasings) against one control; smoke_spawn_actor drives the WebRTC runtime headless and exercises /spawn and /clear-actors. guidance_distill/PLAN.md specifies the Tier-2a LoRA recipe that bakes two-prompt edit guidance into the student (post-swap-gated, premerge-deployed). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The ego pose is the rig origin (~1.5 m above the road), so spawned boxes floated at eye level — off-distribution for the bbox conditioning, and the model under-rendered them (a moving truck box was ignored entirely). Offset the bbox center by the rig height; verified against the scene's own actor boxes in the rendered conditioning. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
/spawn <preset> [dist] [speed] [lateral] [yaw_deg] — box heading relative to the ego (0 = same direction, 180 = oncoming). The rendered box's front/back face colors encode travel direction. Probing found the model paints static boxes in place (parked-vehicle prior) but renders constant-gap moving boxes as a plausible oncoming pass regardless of yaw; the argument stays for scene priors where lead vehicles exist. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Distill the two-prompt text-edit guidance into a LoRA so a plain prompt swap responds at guided strength: on-policy rollouts with a mid-stream swap, teacher = the frozen base running the guidance combine on the same states, student = the LoRA'd single branch under the new prompt (per-term immediate backward under functional attention — the teacher's in-place KV loads would otherwise invalidate the student's checkpoint recompute). Prompt bank reuses the calibration-sweep phrasings plus no-op swaps as a drift regularizer; embeddings precomputed so the 14 GB text encoder is not resident during training. r64 / 1600 steps passes the eval gate on held-out clips: the LoRA'd plain swap reaches 0.854 of guided divergence (bar 0.8; base 0.376), visually clean, generalizing across scene types. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
omnidreams/_edit_lora.py caches base and base-plus-delta weight sets at load and toggles them by in-place copy_ at edit-window boundaries, so weight storage addresses survive and captured CUDA graphs stay valid (the drift corrector's pointer-rebinding swap is not graph-safe). With the hook attached (text_edit_lora_path on the wrapper / WebRTC runtime config, EDIT_LORA on the probe script), replace_text_embeddings builds a use_lora window: single forward per denoise step at guided strength, KV commits included, base weights restored on expiry and on new rollouts. GPU-validated on the rain benchmark: plain swaps stay bit-identical to the hookless run, and the LoRA window's divergence curve tracks the two-branch guided reference (same endpoint) with zero extra forwards — replacing the +84 ms/chunk guidance cost. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review follow-up (PR NVIDIA#431): the ReCache context forward drew its noise from the model RNG, so enabling ReCache shifted every subsequent noise draw relative to a plain-swap rollout. Any noise rendition of the same clean latent is in-distribution for the context forward (each chunk's original commit already uses an independent draw), but drawing from a per-AR-index seeded generator makes the re-commit deterministic and leaves the rollout's noise stream untouched with or without ReCache. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Main moved the runner video helpers to flashdreams.infra.runner_io (load_video_tensor / load_first_frame_tensor / write_video_tensor) — port the three GPU probe scripts to the new API. The guidance_distill trainer/eval/precompute scripts import the Clean Forcing training infra (drift_correction/), which is not on main yet — keep PLAN.md here and land the scripts with that stack (NVIDIA#398); the deploy hook (_edit_lora.py) is self-contained and stays. Also satisfy ty: cast the torch.compile unwrap, require the test checkpoint path, annotate the two intentional test monkeypatches; ruff-format the touched files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
Greptile SummaryThe PR expands OmniDreams interactive-drive support, particularly Windows startup behavior, prompt editing, actor conditioning, text-edit LoRA support, diagnostics, and setup tooling.
Confidence Score: 2/5The PR does not appear safe to merge because disk-space failures can permanently stall interactive generation and remote clients can drive unbounded actor-conditioning work. A dropped render command leaves Files Needing Attention: integrations/omnidreams/omnidreams/interactive_drive/video_model/chunk_pipeline.py, integrations/omnidreams/omnidreams/interactive_drive/runtime/loop.py, integrations/omnidreams/omnidreams/webrtc/session.py Important Files Changed
Sequence DiagramsequenceDiagram
participant Client
participant Session as OmniDreams Session
participant Runtime
participant Pipeline
participant Model
Client->>Session: prompt, controls, or actor command
Session->>Runtime: update session conditioning
Runtime->>Pipeline: request next world-model chunk
Pipeline->>Model: generate with prompt, pose, and actors
Model-->>Pipeline: rendered frames
Pipeline-->>Session: completed chunk
Session-->>Client: display or WebRTC video
Reviews (3): Last reviewed commit: "make prompt" | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
+86
to
+90
| import sys as _sys | ||
| # Skip warmup on Windows (torch.compile hangs with CUDA graphs) | ||
| if _sys.platform == "win32": | ||
| logger.info("[WARMUP] Skipping warmup on Windows (torch.compile disabled)") | ||
| return |
Contributor
There was a problem hiding this comment.
When the default interactive-drive backend starts on Windows, this branch returns before self._session.warmup_model() constructs the pipeline. Scene preparation does not build it later, so the first render raises RuntimeError("warmup() must be called before rendering world-model chunks") instead of generating video.
Suggested change
| import sys as _sys | |
| # Skip warmup on Windows (torch.compile hangs with CUDA graphs) | |
| if _sys.platform == "win32": | |
| logger.info("[WARMUP] Skipping warmup on Windows (torch.compile disabled)") | |
| return |
Knowledge Base Used:
Comment on lines
+339
to
+345
| return | ||
| except DiskSpaceError as exc: | ||
| logger.error( | ||
| f"[chunk-pipeline] DISK SPACE ERROR: {exc}\n" | ||
| "Free up space or set HF_HOME to another drive and retry." | ||
| ) | ||
| continue |
Contributor
There was a problem hiding this comment.
When a queued render raises DiskSpaceError, the worker consumes the command and continues without producing frames or reporting command failure. The runtime therefore never decrements chunks_outstanding, permanently stops requesting chunks, and keeps displaying the last frame even after disk space is freed.
Knowledge Base Used: Core Engine (flashdreams/flashdreams/core/)
| lateral_m=lateral_m, | ||
| yaw_offset_deg=yaw_offset_deg, | ||
| ) | ||
| self._spawned_actors.append(actor) |
Contributor
There was a problem hiding this comment.
Each /spawn event appends another persistent actor, and every later chunk rebuilds conditioning data for the complete list with work and allocations proportional to actors times frame timestamps. Add a per-session actor cap or eviction policy to prevent sustained client input from causing excessive generation latency or exhausting GPU memory.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.