Simplify t2v app by moving common logic to core lib - #446
Open
gtong-nv wants to merge 25 commits into
Open
Conversation
Define the initial flashdreams.runtime package with minimal T1 boundaries for runtime config, user/model inputs, schemas, input mapping, model adapters, runtime/session protocols, output targets, and metrics. Add focused CPU tests for the new API surface without migrating existing runners.
* WIP implementation of T2, T3, partial T4 * Fix issues found by Claude * Rewrite based on discussion, port after merge * doc update * doc updates * Update based on new diagrams * Align closer to diagrams
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Introduce the experimental runtime/session/input envelopes and a shared demo-level API for replay and WebRTC flows. Add the shared runner, output target plumbing, fake-model coverage, and benchmark hooks. Port OmniDreams replay and WebRTC onto the shared demo path via a thin model-owned adapter, add local/remote validation docs, and update the migration plan to track remaining output/stat work and legacy demo cleanup.
* initial commit * more porting * Claude reviewed fixes * Add keyboard parity and event-driven GPU tests for the Lingbot port Pin the runtime-API camera path against the WebRTC path it will replace. KeyboardResampler + CameraPoseIntegrator and the canonicalizer + mapping path are compared over 11 key streams, single- and multi-chunk, including edges landing exactly on a chunk boundary where KeyboardResampler's inclusive drain meets TimeWindow's half-open slice. They agree. Also cover event-driven camera control on CUDA, and repair the existing CUDA test, which monkeypatched trace-loading helpers that moved out of the session and called step() with an empty InferenceInput. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Update integrations/lingbot/lingbot/runtime.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Signed-off-by: aidanfnv <aidanf@nvidia.com> --------- Signed-off-by: aidanfnv <aidanf@nvidia.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Signed-off-by: Gangzheng Tong <gtong@nvidia.com>
Signed-off-by: Gangzheng Tong <gtong@nvidia.com>
* Use inferencesession * Remove legacy lingbot, finish port * Address greptile comments * Address greptile comments * Address greptile comments * Address greptile comments
* Canonicalize generated video step results * Simplify video output stream consumption * Pass step results through WebRTC delivery * Run serving runtimes on thread-affine workers * Unify Lingbot model session execution * Unify OmniDreams model session execution * Make WebRTC manager capabilities explicit * Define explicit WebRTC app adapter contracts * Route outputs through integration capabilities * Drive WebRTC generation with step requests * Fix serving type-check regressions * Apply repository-wide lint fixes * Skip unavailable Transformer Engine in CPU tests * Record serving architecture validation * Unify WebRTC session manager implementations * Consolidate WebRTC runtime lifecycle * refactor(omnidreams): use shared WebRTC demo APIs * refactor(omnidreams): remove legacy WebRTC implementation * refactor(omnidreams): remove WebRTC postprocessing * Consolidate WebRTC demo integrations * Simplify WebRTC demo launch path * Share demo application lifecycle
* Restore shared WebRTC manager input helpers * fix(webrtc): restore session completion and lingbot adapter Treat completed inference sessions as terminal in the shared WebRTC manager, restore LingBot's session-branch adapter against the current model-session core, and update affected WebRTC tests for the current result and manager APIs.
unify demo runtime through OmniDreams migration Add the shared demo runtime stack for replay, WebRTC, input providers, output sinks, timing, warmup, metrics, and error handling. Migrate OmniDreams null, precomputed MP4, Ludus MP4, and default WebRTC paths onto the shared runtime, add GPU CI coverage for those paths, and clean up the OmniDreams demo layout while keeping legacy fallback compatibility.
Migrate LingBot demos onto unified runtime Routes LingBot replay and WebRTC paths through the shared demo runtime stack, adds provider/shared WebRTC parity coverage, and documents the new demo commands.
Migrate LingBot demo onto unified runtime Port LingBot replay, null, MP4, and WebRTC demo paths to the shared demo runtime, add focused CI coverage for null and MP4 outputs, and update demo docs with the validated commands. Clean up shared keyboard/input ownership, preserve legacy compatibility where still needed, and improve shared WebRTC responsiveness for interactive demos.
* Preserve setup-time driver invariants Keep setup-time DriverInvariantError exceptions out of normal setup error policy handling in both batch and realtime demo drivers. Add regression coverage for setup invariant propagation and document MP4 generation handling across realtime resets. * Move LingBot shared provider off input mappings Make the migrated LingBot demo provider convert UserInputWindow values directly into model InferenceInput, including trace slicing, live camera integration, skipped WebRTC input catch-up, and text-event prompt updates. Allow shared replay runs to use mapping-free model providers while keeping legacy compatibility runners mapping-gated, and update LingBot parity coverage against the old mapping path. * Decouple LingBot WebRTC startup from legacy input mapping Update the LingBot WebRTC inference-session startup path so shared demo sessions only require an initialized rollout, not the legacy input mapping. Clarify the remaining direct-session compatibility path and add assertions that the migrated LingBot WebRTC demo uses a mapping-free shared scenario. * Fix ruff errors * Update LingBot input mapping test for shared provider path Adjust the event-driven LingBot scenario test to assert the migrated mapping-free shared provider behavior. The test now drives the scenario through LingbotInputProvider instead of expecting PreparedScenario.mapping to contain a legacy LingbotInputMapping.
* Unify demo launch commands Signed-off-by: Gangzheng Tong <gtong@nvidia.com> * Move launch manifests to configs Signed-off-by: Gangzheng Tong <gtong@nvidia.com> * Add OmniDreams launch aliases Signed-off-by: Gangzheng Tong <gtong@nvidia.com> * Fail loudly for missing launch manifests Signed-off-by: Gangzheng Tong <gtong@nvidia.com> --------- Signed-off-by: Gangzheng Tong <gtong@nvidia.com>
* Add app-owned T2V replay demo Signed-off-by: Gangzheng Tong <gtong@nvidia.com> * Add reusable WebRTC prompt generation controls Signed-off-by: Gangzheng Tong <gtong@nvidia.com> * Improve persistent T2V WebRTC playback Signed-off-by: Gangzheng Tong <gtong@nvidia.com> * Add SPDX header to T2V package Signed-off-by: Gangzheng Tong <gtong@nvidia.com> * Add T2V flashdreams-run launcher Signed-off-by: Gangzheng Tong <gtong@nvidia.com> * fix precommit --------- Signed-off-by: Gangzheng Tong <gtong@nvidia.com>
Signed-off-by: Gangzheng Tong <gtong@nvidia.com>
Signed-off-by: Gangzheng Tong <gtong@nvidia.com>
Signed-off-by: Gangzheng Tong <gtong@nvidia.com>
Signed-off-by: Gangzheng Tong <gtong@nvidia.com>
Contributor
|
Too many files changed for review (231 files, 100 file limit). Bypass the limit by tagging |
gtong-nv
changed the base branch from
main
to
dev/jarcher/inference-runtime-api
August 12, 2026 00:52
jarcherNV
force-pushed
the
dev/jarcher/inference-runtime-api
branch
2 times, most recently
from
August 12, 2026 01:43
6f494af to
fc3d2a5
Compare
jarcherNV
force-pushed
the
dev/jarcher/inference-runtime-api
branch
3 times, most recently
from
August 12, 2026 03:12
1ba6d84 to
4711570
Compare
gtong-nv
changed the base branch from
dev/jarcher/inference-runtime-api
to
main
August 12, 2026 04:12
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.