Migrate FlashVSR demos to runtime API - #442
Conversation
Define the initial flashdreams.runtime package with minimal T1 boundaries for runtime config, user/model inputs, schemas, input mapping, model adapters, runtime/session protocols, output targets, and metrics. Add focused CPU tests for the new API surface without migrating existing runners.
* WIP implementation of T2, T3, partial T4 * Fix issues found by Claude * Rewrite based on discussion, port after merge * doc update * doc updates * Update based on new diagrams * Align closer to diagrams
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Introduce the experimental runtime/session/input envelopes and a shared demo-level API for replay and WebRTC flows. Add the shared runner, output target plumbing, fake-model coverage, and benchmark hooks. Port OmniDreams replay and WebRTC onto the shared demo path via a thin model-owned adapter, add local/remote validation docs, and update the migration plan to track remaining output/stat work and legacy demo cleanup.
* initial commit * more porting * Claude reviewed fixes * Add keyboard parity and event-driven GPU tests for the Lingbot port Pin the runtime-API camera path against the WebRTC path it will replace. KeyboardResampler + CameraPoseIntegrator and the canonicalizer + mapping path are compared over 11 key streams, single- and multi-chunk, including edges landing exactly on a chunk boundary where KeyboardResampler's inclusive drain meets TimeWindow's half-open slice. They agree. Also cover event-driven camera control on CUDA, and repair the existing CUDA test, which monkeypatched trace-loading helpers that moved out of the session and called step() with an empty InferenceInput. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Update integrations/lingbot/lingbot/runtime.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Signed-off-by: aidanfnv <aidanf@nvidia.com> --------- Signed-off-by: aidanfnv <aidanf@nvidia.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Signed-off-by: Gangzheng Tong <gtong@nvidia.com>
Signed-off-by: Gangzheng Tong <gtong@nvidia.com>
* Use inferencesession * Remove legacy lingbot, finish port * Address greptile comments * Address greptile comments * Address greptile comments * Address greptile comments
* Canonicalize generated video step results * Simplify video output stream consumption * Pass step results through WebRTC delivery * Run serving runtimes on thread-affine workers * Unify Lingbot model session execution * Unify OmniDreams model session execution * Make WebRTC manager capabilities explicit * Define explicit WebRTC app adapter contracts * Route outputs through integration capabilities * Drive WebRTC generation with step requests * Fix serving type-check regressions * Apply repository-wide lint fixes * Skip unavailable Transformer Engine in CPU tests * Record serving architecture validation * Unify WebRTC session manager implementations * Consolidate WebRTC runtime lifecycle * refactor(omnidreams): use shared WebRTC demo APIs * refactor(omnidreams): remove legacy WebRTC implementation * refactor(omnidreams): remove WebRTC postprocessing * Consolidate WebRTC demo integrations * Simplify WebRTC demo launch path * Share demo application lifecycle
* Restore shared WebRTC manager input helpers * fix(webrtc): restore session completion and lingbot adapter Treat completed inference sessions as terminal in the shared WebRTC manager, restore LingBot's session-branch adapter against the current model-session core, and update affected WebRTC tests for the current result and manager APIs.
unify demo runtime through OmniDreams migration Add the shared demo runtime stack for replay, WebRTC, input providers, output sinks, timing, warmup, metrics, and error handling. Migrate OmniDreams null, precomputed MP4, Ludus MP4, and default WebRTC paths onto the shared runtime, add GPU CI coverage for those paths, and clean up the OmniDreams demo layout while keeping legacy fallback compatibility.
Migrate LingBot demos onto unified runtime Routes LingBot replay and WebRTC paths through the shared demo runtime stack, adds provider/shared WebRTC parity coverage, and documents the new demo commands.
Migrate LingBot demo onto unified runtime Port LingBot replay, null, MP4, and WebRTC demo paths to the shared demo runtime, add focused CI coverage for null and MP4 outputs, and update demo docs with the validated commands. Clean up shared keyboard/input ownership, preserve legacy compatibility where still needed, and improve shared WebRTC responsiveness for interactive demos.
Signed-off-by: Gangzheng Tong <gtong@nvidia.com>
Greptile SummaryThe PR migrates FlashVSR replay and WebRTC demos to the shared runtime API and adds browser MP4 upload support. The upload handoff currently uses an unowned, insufficiently synchronized process-global slot.
Confidence Score: 2/5The PR is not safe to merge until staged uploads are isolated per client and upload writes are serialized with offer consumption. The externally reachable upload flow permits one browser to inspect or consume another browser's video, while a concurrent accepted upload can also be silently cleared by an in-progress offer. Files Needing Attention: integrations/flashvsr/flashvsr/demo/server.py and flashdreams/flashdreams/serving/webrtc/manager.py
|
| Filename | Overview |
|---|---|
| integrations/flashvsr/flashvsr/demo/server.py | Adds upload and status endpoints, but stages uploads globally without client ownership or synchronization with offer consumption. |
| integrations/flashvsr/flashvsr/demo/webrtc.py | Connects FlashVSR to the shared WebRTC runtime and dynamically derives session dimensions from the pending upload. |
| integrations/flashvsr/flashvsr/runtime.py | Implements the native runtime/session lifecycle, cold-versus-steady chunking, cache reuse, and shape-specific pipeline reconstruction. |
| integrations/flashvsr/flashvsr/demo/providers.py | Supplies exact requested input chunks for finite and looping sources with consistent tail handling. |
| flashdreams/flashdreams/serving/webrtc/web/request_session.js | Adds adapter connection hooks, status helpers, and dynamic visibility for model controls. |
| integrations/flashvsr/pyproject.toml | Registers the new demo command, serving dependencies, and packaged browser assets. |
Sequence Diagram
sequenceDiagram
participant A as Browser A
participant API as FlashVSR upload API
participant Slot as pending_session_input
participant B as Browser B
participant RTC as WebRTC offer handler
A->>API: POST /api/session/input (video A)
API->>Slot: Store video A
B->>API: GET /api/session/input
API-->>B: Filename and video metadata
B->>RTC: POST /api/webrtc/offer
RTC->>Slot: Read and clear video A
RTC-->>B: Session processing video A
Reviews (1): Last reviewed commit: "Migrate FlashVSR demos to runtime API" | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| session_input = FlashVSRWebRTCSessionInput( | ||
| prepared_video=prepared, | ||
| original_name=original_name, | ||
| ) | ||
| self.manager.set_pending_session_input(session_input) |
There was a problem hiding this comment.
If multiple users can access the externally bound server, every upload is stored in one unowned pending_session_input slot, while the public status route exposes its filename and dimensions and the next offer consumes it. This lets another client inspect, replace, or process the first client's uploaded video. How this was verified: The upload and status routes share one manager slot, and the next offer consumes it without an identity or capability check.
Knowledge Base Used: WebRTC Serving Flow
| prepared_video=prepared, | ||
| original_name=original_name, | ||
| ) | ||
| self.manager.set_pending_session_input(session_input) |
There was a problem hiding this comment.
If an upload finishes while create_answer is negotiating a previously staged input, this unlocked setter can replace the slot after the offer reads it, and the offer then unconditionally clears the newer value. The upload request reports success, but the uploaded video is silently lost and must be submitted again.
Knowledge Base Used: WebRTC Serving Flow
1ba6d84 to
4711570
Compare
3 participants
No description provided.