[crazy-robotaxi] Live-edit abilities: skin swap, coins, spawned obstacles, weather (stacked on #463) - #494
Conversation
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
…edit Signed-off-by: aidanfnv <aidanf@nvidia.com>
… off and guidance-window knobs Fixes the three perf problems reported against PR NVIDIA#494 + NVIDIA#496: - --live-edit-corrector-mode off disables every drift corrector even when checkpoints are configured: no DriftCorrectorDispatch is built, no weight sets are snapshotted, and the scheduler/finalize gate drivers are never installed (weights verified bit-identical in tests). - The native-DIT guard is now precise and actionable: it fires only for the prompt-swap abilities that genuinely need the Python transformer forward (replace_text_embeddings raises NotImplementedError under the native executor, and TextEditLoRA's copy_ toggles never reach the native fp8 weight snapshot), names the exact flags to drop, and only mentions corrector flags when a corrector is actually enabled. Coins and other pixel-only abilities never construct the ability and run under native_dit_acceleration untouched. - New knobs --live-edit-skin-guidance-chunks (default 6) and --live-edit-weather-guidance-chunks (default 20). The skin window is LoRA-realized single-branch (length is not a per-chunk cost); a GPU A/B on a cyberpunk swap showed 6 lands the style as fast and as strong as 20. Weather has no LoRA: its guided window costs ~2x per chunk while open, and the 8-chunk re-swap refresh re-opens it - now documented in the README with the knob to shorten it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…sident The presenter used to materialize the model frame to host numpy for PIL compositing, which on the native Vulkan HUD fast path (lazy CUDA uint8 HWC frames) forced a GPU->CPU->GPU round trip per frame (~10 fps observed). LiveEditFrameCompositor keeps the frame on device: coin sprites, contact shadows, and HUD chips are pre-rendered once (PIL) and cached as premultiplied per-size textures; the per-frame work is one float32 conversion, a fused lerp ROI blend or two per coin, and a single round/clamp/uint8 cast (plus an optional separable-Gaussian unsharp for skins). Measured on GB300 at 1280x704: 0.09 ms/frame with 0 coins, 0.59 ms with 9 coins (~4.7 ms per 8-frame chunk), issued from the presenter thread so it overlaps model compute. The composited result is re-wrapped as a LazyCudaFrame whose CUDA event is recorded after the blends, so both consumers stay ordered (Vulkan interop copy stream, MJPEG host prefetch) and the single-materialization numpy-XOR-cuda contract holds. Host/numpy sources keep the PIL path, as does the obstacle box-outline annotation (debug aid). All compositor math is device-agnostic and unit-tested on CPU tensors. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
obstacle_ability/test_runner import-block ordering (repo ruff 0.12.7 --select I), C408 dict literals, PLR0402/RUF059/RUF046/UP037 from the current ruff default set on the live-edit files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nv/flashdreams-wq into robotaxi-463-live-edit
Every HTTP endpoint (page, /stream, /bev_stream, /control, /state, /scenes, /thumbnail, /scene/select, /taxi/name) now requires the shared token via ?token= query param or X-Stream-Token header when the flag is set; requests without a valid token get 403. The served page reads the token from its own URL and appends it to all of its requests. Omitting the flag keeps the historical open behavior. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Escalates storm along the cues that materialize (2026-08-21 A/B at guidance 3.5/4.0 vs the shipped storm frames): visibility collapse, walls of spray/mist, static debris lying on the flooded road, and a black-green emergency-gloom sky. Dynamic wind wording (bending trees, flying debris) is deliberately absent — it never materializes in the history-anchored model. Shipped at weather guidance 4.0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
aidanfnv
left a comment
There was a problem hiding this comment.
Awesome!
The style transfers probably need a follow-up to clear up some issues with how they look in motion (see my message in Slack), but the weather effects look incredible, and the coins and obstacles work well. My new map format will break the obstacle ability, but that will of course be on me to fix.
Thank you for this!
Aidan reported coins still hurting native-path perf after the CUDA compositor fix. Profiling at real course scale (4395 coins on the shipped suburb scene) found two costs the 9-coin measurement missed: - visible_sprites/advance_frames scanned all coins every frame (0.60 ms/frame CPU at 4400 coins). A static 32 m spatial hash with per-cell cached windows plus a vectorized projection filter makes both O(nearby): 0.20 ms/frame, pickup 0.10 -> 0.015 ms. - The compositor is CPU-launch-bound, not bandwidth-bound (GPU drain adds ~0 in every measurement), and dense areas put 211 sprites inside the 120 m render radius -> 11-14 ms/frame enqueue. Fixes: one blend per coin (shadow+coin+quantized fade pre-composited into one cached texture, 2.6x fewer kernels: 1.14 -> 0.44 ms at 9 sprites, 5.20 -> 1.84 ms at 48) and a nearest-N sprite cap (--live-edit-coin-max-visible, default 64) so dense courses cannot blow the budget. Also adds the ROI-only uint8 blend path (LIVE_EDIT_COMPOSITOR=roi) that never converts the full frame to float32 - exact parity with the float path (max diff 0) and minimal GPU traffic, but slower wall-clock on launch-bound machines, so the float canvas stays the default - and a perf self-report (--live-edit-perf-log N / LIVE_EDIT_PERF_LOG) that logs p50/p95 coin CPU ms, compositor enqueue CPU ms, and compositor GPU ms (lazy CUDA event pairs, no added sync) every N composited frames so remote users can send numbers instead of guesses. Validated headless MJPEG at course scale: chunk cadence unchanged (219-242 ms, matching the no-coins baseline), coin rendering parity vs the previous compositor within 1.5 LSB mean on sprite pixels (fade quantization + <=1 px shadow anchor). New tests cover the windowed culling, the sprite cap, ROI/float parity, and the perf log. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One O press now spawns --live-edit-obstacle-count clones (default 1 = unchanged single obstacle): distinct crossing/oncoming template tracks (never pace-matched lead cars, which render at ghost strength), staggered across a 16-40 m band ahead of the ego (spacing_m per slot) and --live-edit-obstacle-stagger-chunks apart in time, each retimed+rigidly shifted with ground-z correction and despawning after its own pass. Guidance already strips every OBSTACLE_ENTITY_PREFIX actor from the shadow branch, so the box-axis path handles simultaneous boxes unchanged. Mask-anchored validation (RNG-matched baseline, per-clone camera-projected boxes, 26-chunk headless runs): s=2.0 lands 2/3 clones at N=3 (52/61/18% of real-car in-box benchmark) and 4/5 at N=5 (52/60/15/69/83%), out-box clean (median 5.7-7.5); s=2.5 is stronger still at N=3 (63/79/22%) with no breakup, so the traffic demo ships at s=2.5. Simultaneous spawns (stagger 0) are stable; the 1-chunk default is for on-screen pacing. The one weak clone is the fastest template (~11 m/s) which holds distance until late. The presenter annotates every active clone and shows a TRAFFIC xN chip. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The always-guided weather deployment (guidance_chunks 20, re-opened by the 8-chunk skin re-swap) paid a second forward per denoise step for the whole hold — and, worse, every re-swap snapshotted its old KV from buffers that already held the weather text, so the re-opened windows guided along a ZERO direction (2x cost, no edit pressure). RNG-matched A/B on the real model (scratch weather_hold probe, rain, 27-chunk hold): a 6-chunk guided landing followed by a plain unguided hold matches the always-guided policy both in divergence-vs-clear (peak 35.1 vs 38.4 uint8, identical late-run decay) and eyes-on frames; a rebased 2-chunk maintenance pulse every 16 adds ~1 uint8 (noise), so the default hold has no maintenance. - weather guidance_chunks default 20 -> 6 (landing window only) - weather no longer rides the skin re-swap; optional maintenance knobs --live-edit-weather-maintain-interval / -maintain-chunks (default 0/2), and a maintenance pulse rebases (plain base swap first) so its guidance direction is weather-minus-base, not zero Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every skin/weather swap paid a text-encoder forward inside replace_text at the chunk boundary (450-930 ms model_ms on the boundary chunk). All swap prompts are known up front, so StyleAbility.attach() now encodes the configured skin + weather prompts once (and start() encodes the scene's base prompt once), and _replace_text injects the cached [1,1,L,D] embeddings through the pipeline's existing replace_text_from_embeddings — no main-repo change needed. Uncached prompts (or pipelines without the embedding API / a resident text encoder, e.g. the offload path) fall back to the encode-per-swap replace_text. Each swap logs cached_embeddings=<bool> swap_ms=<ms> for measurement. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rebuild The guidance previously forced use_cuda_graph=False session-wide (~545 ms per chunk for the WHOLE session, events or not). It never needed to: the transformer's CUDAGraphWrapper stages every top-level tensor kwarg — including hdmap_condition — into static buffers per call, so the two forwards of a guided step are two REPLAYS of the same captured cond graph with box/no-box conditioning staged in (the mechanism two-prompt text-edit guidance already rides), and the predict_flow dispatch runs eagerly outside any capture. The one genuinely graph-unsafe seam was the shadow ENCODER: the Wan VAE's graph wrapper passes its streaming cache dict through verbatim, binding captured kernels to one cache's buffer addresses — feeding it the shadow cache would silently read/write the real cache (and alternating real/shadow calls through one wrapper would capture against whichever cache was live). Shadow encodes now run eagerly via _eager_vae_scope, which also keeps the wrapper's warmup/capture stream fed by the real cache only. install_obstacle_guidance_on_backend keeps the accelerated session; the transformer guard now rejects only the native optimized-DiT executor. GPU validation (graphs ON, s=2.5, 3-clone burst, RNG-matched baseline, scratch obstacle_graphs probe): one graph captured at chunk 4 and never re-captured through spawn/despawn; pre-spawn |diff| exactly 0.0; event chunks ~340 ms model vs ~230 non-event (non-event unchanged vs baseline); per-clone in-box |diff| 44/60/44% of the real-car benchmark with out-box clean (6.1 median) — materialization matches the graph-free reference. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
--live-edit-skin-duration-chunks N auto-reverts an activated skin to the base world after N generated chunks through the existing plain-swap revert path (guidance 1.0/0; corrector dispatch returns to base). 0 keeps the hold-until-cycled behavior. K during an active timed skin keeps its cycle semantics: next skin with a fresh timer (same key meaning in both modes; base reachable by cycling; a user press at the expiry boundary wins over the auto-revert). Duty-cycle re-swaps are skipped when the duration fits inside the re-swap interval; re-swaps never reset the timer. HUD skin chip shows the remaining seconds (chunk-granular, manifest-derived chunk length). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Mirrors --live-edit-weather-first: rotate the K cycle so a named skin comes first for direct one-press selection — timed power-up demos would otherwise burn transitional chunks cycling through the skins ahead of it. README documents the power-up mode and its K semantics. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Slow clients (VPN / SSH tunnel) were watching seconds-old frames: the LatestFrameBus already drops to the newest frame, but the default kernel SO_SNDBUF (several MB) let TCP queue seconds of encoded JPEGs anyway. Cap the per-connection send buffer to ~2 frames so writes block as soon as the link saturates and the next bus read skips to the newest frame. Per-connection sent/dropped/bandwidth stats are logged every 10 s and on disconnect. New knobs: --stream-jpeg-quality (default 85, unchanged) and --stream-scale (e.g. 0.5 = 640x352) downscale before JPEG encode, off the render thread as before. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Effect items: sparse pickups along the lanes (rain/snow icons -> that weather; mystery box -> seeded-random timed skin burst) reusing the coin course/projection/culling/compositing machinery via per-sprite bank keys. Pickups dispatch through the existing ability state machines at the next chunk boundary — the same path as the K/V keys, which stay fully live. Weather stays base-world-only: a pickup during a skin is ignored with a HUD hint (not queued). Item sprites are local-only config paths; defaults render procedural placeholder icons. Timed weather: --live-edit-weather-duration-chunks (default 90 ~ 24 s) auto-reverts any weather activation to clear via a GUIDED clear landing (clear is itself a weather transition; plain swaps leave precipitation running on KV momentum). HUD weather chip gains a countdown. Re-picking the active weather refreshes the timer without a zero-direction re-swap. Weather-state corrector stays default-off per policy (clean-forcing corrector only for game-skin states); knob kept for A/B. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Real maps chop lanes into segments shorter than the item spacing (the shipped suburb map laid out 4 items instead of ~300): walk candidates every ~spacing/4 and accept one only when no accepted item lies within spacing_m (spacing-sized spatial hash), so rarity is a property of the whole network rather than each polyline. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Global min-distance suppression let whichever directed lane was walked first claim every spacing-disc along a road, so drivers of the opposite lane never passed within pickup radius of an item (0 pickups over a 30 s GPU drive that collected 60 coins). Suppress a candidate only against accepted items with a similar heading (dot > 0.5): each direction of a road carries its own items while overlapping/duplicated lanes still dedup. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
--live-edit-obstacle-static-count N lays out N parked-track clones midroad ahead of the spawn pose, in the conditioning from the session's first chunk, alternating laterals so the ego weaves between them; they persist until a rollout reset re-anchors them, and O-key bursts keep working alongside. Probed 2026-08-23 (RNG-matched, mask-anchored): static clones stay at ghost strength unguided even from chunk 0 (the initial camera frame anchors an empty road), materialize solid stopped cars in the 5-25 m band with box-axis guidance s=2.0, and break up when spawned mid-stream at s=2.5 - so the mode pairs with guide_scale 2.0 and nearer slots are expected to lag. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
--live-edit-obstacle-static-lateral-m widens the alternating slots so the ego has a slalom line with clearance beyond the 3 m collision radius; the first capture at the 2.8 m default clipped the clone boxes, and a camera-through-box overlap sends the box-axis guidance into frame-wide breakup. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A nitro item (GPU icon) joins the effect-item course mix at the same rarity as the other kinds. Picking one up applies a temporary boost inside the app-authoritative taxi physics tick — the integrate_fn seam in RolloutSpec — multiplying max speed AND max acceleration by --live-edit-nitro-boost (1.6) for --live-edit-nitro-duration-s (4 s game time). Unlike the weather/skin items there is no chunk-boundary handshake: nitro is physics-only, so the effect lands on the very next sampled physics tick and composes with every skin/weather/obstacle state. A second pickup while boosted resets the timer (no stacking). The boosted max speed is hard-capped at --live-edit-nitro-max-speed (16 m/s default) so the conditioned ego never outruns the world model's manifold on the suburb map. The HUD shows the NITRO! pickup flash plus a boost chip with a game-time countdown. --live-edit-item-types restricts the course mix (single-effect capture courses); --live-edit-item-nitro-sprite supplies the local-only icon, with a procedural placeholder as the shipped default. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…istry Merge origin/dev/aidanf/game/crazy-robotaxi (b3b9581), which adds the FlashdreamsWorldModelSession.replace_prompt passthrough and an extra_key_handlers registry on both presenters (NVIDIA#496). Conflict resolutions favor the upstream API plus our live-edit package: - hud_presenter: drop our k/c/v/o keysym branches and _request_live_edit in favor of the upstream _dispatch_extra_key registry; the explicit k/c/v/o entries in _build_key_codes go too (the registry loop adds handler keysyms). - streaming_presenter: keep our stream_token kwarg alongside the upstream extra_key_handlers kwarg; replace our k/c/v/o live-edit branch in _apply_control with the upstream case-folded handler lookup. Our drop-to-latest / SO_SNDBUF / quality / scale streaming changes stay. - tests: keep both our stream-scale/stats tests and the upstream extra-key-handler tests. The composition root (cli.py) now passes extra_key_handlers={k,c,v,o -> LiveEditRequests channels} to both presenter constructors, resolving the requests object at press time via the app's long-lived keyboard (the presenters are built first). k/c/v/o do not collide with either presenter's reserved-key set. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
StyleAbility._replace_text's encode-per-swap fallback now calls the public FlashdreamsWorldModelSession.replace_prompt instead of reaching into session.pipeline/_cache, dropping the TODO(upstream) note. Since replace_prompt flushes the adapter's deferred chunk finalize itself, the manual flush in the continue_generation wrapper is gone; the embeddings fast path (no upstream equivalent yet) keeps an explicit _flush_pending_finalize so finalize still runs under the old text. The fake sessions in the live-edit tests grow a matching replace_prompt that mirrors the real flush-then-swap semantics. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
@aidanfnv — #494 is now rebased/merged onto your post-#496 base and consumes the new API: 🤖 Generated with Claude Code |
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
039d4b4
into
NVIDIA:dev/aidanf/game/crazy-robotaxi
Stacked on #463 (Crazy Robotaxi standalone) — only the top 2 commits are new; implements the plan from #489 using the mechanisms from #431/#458. cc @aidanfnv
Adds an optional, flag-gated
live_edit/ability package to Crazy Robotaxi (vanilla behavior unchanged with flags off; all keys work in both the native HUD and the browser/MJPEG path):Validation: end-to-end headless MJPEG runs (65 chunks) with all abilities; 511 CPU tests pass (+62 new in
tests/test_live_edit*.py) against the synthetic fixtures; ruff clean. Known limits documented in code/tests: corrector currently requires CUDA graphs + compile off (graph-safe corrector is the follow-up), snow strength is honest-partial at the pinned guidance, revert-to-clear can't undo accumulated scene change (history-carrying model).Demo videos attached below.
robotaxi_liveedit_demo3_rain.mp4
robotaxi_liveedit_demo3_annotated.mp4
robotaxi_liveedit_demo3_cyberpunk.mp4
Update: real-time performance + weather refinements (head
a13d4c74)torch._foreach_copy_before each CUDA-graph replay — so the corrector, the style-LoRA delta, and the per-timestep gate all run with CUDA graphs on. This also makes the dispatch the sole writer of the self-attn projections, resolving the LoRA/corrector composition cleanly.--live-edit-corrector-mode unfusedkeeps the legacy path.--live-edit-base-corrector), reducing long-session drift with no features active.--live-edit-coin-sprite <path>accepts a custom RGBA sprite.Test suites: 87 live_edit CPU tests green; ruff clean.
Final feature set (head
8ae7277d)Controls: WASD drive · K skin power-up · C coins toggle · O obstacle/traffic burst · V weather cycle · plus in-world pickups (weather icons trigger their weather; a mystery box grants a random timed skin burst) — pickups and keys drive the same ability state machines.
--live-edit-skin-duration-chunks, e.g. 11 ≈ 3 s) with HUD countdown and residue-free auto-revert — style bursts end before long-hold drift begins. Swap boundary ~48 ms (prompts pre-encoded at startup).--live-edit-obstacle-count). Static roadblock mode exists behind default-off flags (requires continuous guidance from chunk 0; quality is limited — documented, not recommended).--stream-jpeg-quality/--stream-scalebandwidth knobs.Performance (GB300, end-to-end): ~34 fps steady with features on; ~31 fps with obstacle guidance armed; transients only at swap boundaries (~48 ms) and during weather landings / obstacle events (~1.5–2× for a few seconds). Native-DIT: coins/keys compatible; skins/weather require it off (the executor lacks
replace_textwiring and doesn't re-read LoRA weight toggles — error messages name the exact flags; upstream item).Tests: 203 live-edit CPU tests (664 across the app suite) against the synthetic fixtures; ruff clean. Demo videos below/attached.
all.mp4
Update: pickup items + nitro (head
1f6727be)The abilities are now also reachable as in-world pickups (same state machines as the keys; keys unchanged):
--live-edit-items,--live-edit-item-spacing): weather icons trigger their weather (rain/snow; storm/hurricane stay key-only), a mystery box grants a random timed skin burst (seeded RNG, per-activation duration override), and a nitro item gives an instant 1.6× speed+acceleration boost for 4 s (pure game physics, zero model cost; re-pickup resets the timer; boosted max speed capped at 16 m/s to stay on the world model's manifold — frames verified coherent at 13–15 m/s).--live-edit-weather-duration-chunksvia a guided clear-landing; accumulated effects fade realistically (roads dry, snow melts). Weather-during-skin pickups are ignored with a HUD hint (matches the key semantics).--live-edit-item-{rain,snow,mystery,nitro}-sprite); defaults render procedural placeholder icons — no bundled assets.Corrector policy per in-game A/B evidence: active only during skin states (decisive win — an unaided held skin collapses by +60 chunks); base-state corrector measured as a wash in game driving and defaults off; weather corrector gain defaults 0.
Tests: 687 passing across the app CPU suite (56 new for items/nitro); ruff clean.