feat(data): add camera selector to annotation workspace and fix AV1 frame extraction#591
Merged
Conversation
…rame extraction - expose cameras list and setCameraName from media-sources hook with selection preserved across episodes - render CameraSelector alongside ViewerDisplayControls in playback card - prefer imageio-ffmpeg static binary so frame thumbnails work for AV1 videos 🎥 - Generated by Copilot
Contributor
Dependency ReviewThe following issues were found:
Snapshot WarningsEnsure that dependencies are being submitted on PR branches and consider enabling retry-on-snapshot-warnings. See the documentation for more information and troubleshooting advice. OpenSSF ScorecardScorecard details
Scanned Files
|
- Add libdav to cspell dictionary (libdav1d reference in lerobot_handler docstring) - Sort imports in AnnotationWorkspacePlaybackCard for simple-import-sort - Update ffmpeg extraction tests to mock _resolve_ffmpeg directly 🤖 - Generated by Copilot
- Format ffmpeg test monkeypatch calls onto single lines for ruff - Add cameras/selectedCamera/onSelectCamera to test fixtures for new AnnotationWorkspacePlaybackCard required props 🤖 - Generated by Copilot
- Apply prettier to AnnotationWorkspacePlaybackCard and useAnnotationWorkspaceShell - Add CameraSelector to episode-viewer mock so AnnotationWorkspace tests render - Derive cameraName synchronously via useMemo with separate override state so initial render no longer produces a transient null videoSrc that triggers the frame-only autoplay path twice 🤖 - Generated by Copilot
- format paths with both v2.x (episode_chunk/episode_index) and v3 placeholders - derive chunk index from episode_index // chunks_size for v2.x layouts - read episode lengths from meta/episodes.jsonl when present - add tests covering v2.x path resolution and episode metadata 🛠️ - Generated by Copilot
…dataviewer - add LanguageInstructionAnnotation model, store actions, and panel widget - load task descriptions from meta/tasks.jsonl and tasks.parquet - preserve dataset feature keys when transforming snake_case API responses - support v2.x LeRobot layout in loader and read episodes.jsonl lengths - cover v2.x layout and jsonl metadata with new loader tests 🏷️ - Generated by Copilot
- move trajectory plot and subtask timeline above the subtasks pane - shrink trajectory plot to a fixed 180px height - collapse the labels panel back to a single full-height column 🎨 - Generated by Copilot
- compute finite-difference velocities in build_trajectory when not provided - guards against zero or non-monotonic timestamps with a small dt floor 📈 - Generated by Copilot
…y layout tests - Track the LanguageInstructionWidget component that was referenced from the annotation-panel barrel but never committed - Mock LanguageInstructionWidget in the workspace test support module - Update trajectory tab layout tests to match the compact single-column labels panel and graph-inside-playback-group structure 🤖 - Generated by Copilot
- rename HMI_STORAGE_BACKEND to STORAGE_BACKEND across backend, tests, docker-compose, Dockerfile, README, and dataviewer Terraform module - rename HMI_DATA_PATH to DATA_DIR across config, routers, services, tests, .env examples, and Dockerfile - add --data-dir/--data-dir=<path> argument to data-management/viewer/start.sh with sensible default of <repo>/datasets and missing-path warning - default dev:backend DATA_DIR to ../../../datasets in viewer package.json so launching from the VS Code task picks up the repo-root datasets folder 🤖 - Generated by Copilot
… tab 🤖 - Generated by Copilot
katriendg
reviewed
Apr 29, 2026
Collaborator
katriendg
left a comment
katriendg
left a comment
There was a problem hiding this comment.
Thanks so much @akzaidi, this is great work and adding a big new update for dataviewer!
Reviewed with Copilot and there are a number of things we can enhance before merging - also happy to pick up any of these in your branch directly before we merge. Let's discuss!
Multi-camera selection, AV1 thumbnails, LeRobot v2.x compatibility, and the language-instruction widget are all valuable additions, and I confirmed locally that the test suite (414 backend, 658 frontend) is green on the branch. A few items to resolve before merge:
📚 Missing user-facing documentation for the new features
This PR ships two net-new user-visible capabilities, and neither is documented anywhere a user (or an agent driving the dataviewer) would discover them:
- Camera selector — no mention in data-management/viewer/README.md, .github/skills/dataviewer/SKILL.md, or the docs site. A user with a multi-camera dataset has no way to know switching is now possible, where the control lives, or how the fallback (
cameras[0]when override is stale) behaves. - Language instruction / VLA annotation — same gap. The 317-line
LanguageInstructionWidgetintroduces new annotation semantics (source: human / template / llm-generated / retroactive, paraphrases for augmentation, subtask decomposition for hierarchical conditioning) that downstream training consumers will need to understand. Today this is only documented in TS/Py docstrings.
Please add a short "Annotation features" or "What you can annotate" subsection to the viewer README covering both, and update the dataviewer SKILL.md so the agent knows these surfaces exist.
📝 Documentation drift from the env var rename
The HMI_DATA_PATH → DATA_DIR / HMI_STORAGE_BACKEND → STORAGE_BACKEND rename was not propagated to:
- .github/skills/dataviewer/SKILL.md — 12 references (lines 33, 102, 109, 116, 118, 123, 129, 135, 142, 151, 306, 378)
- .github/agents/dataviewer-developer.agent.md — 3 references (lines 34, 37, 162)
- .github/prompts/start-dataviewer.prompt.md — line 17
Also rename HMI_LOCAL_DATA_PATH at the source. It's still used as the docker-compose host-mount selector in data-management/viewer/docker-compose.yml and in the README example at data-management/viewer/README.md. Keeping the HMI_ prefix only here will read as an incomplete rename and confuse future readers. Suggest renaming to something like DATAVIEWER_HOST_DATA_DIR (or LOCAL_DATA_PATH) in the compose file, README, and any CI workflow that references it (e.g., .github/workflows/dast-zap-scan.yml).
🧪 Test coverage gaps for the new code paths
| Surface | Why it matters |
|---|---|
LanguageInstructionWidget.tsx (new, 317 LOC) |
Currently mocked as <div>Language Instructions</div> in test support; zero direct tests for empty state, "Use as Instruction" template seeding, paraphrase/subtask add+remove, Enter-key submit, or clear. |
LanguageInstructionAnnotation round-trip |
No backend test posts/reads an EpisodeAnnotation containing a language instruction. |
LeRobotLoader.get_tasks() |
New method, neither the jsonl nor the parquet branch is exercised. The new TestV2EpisodeLayout fixture already writes task rows — extending it is one assertion. |
build_trajectory velocity estimation |
Brand-new finite-difference path with no unit test, including the length<=1 and zero-dt edge cases. |
_resolve_ffmpeg |
The test refactor monkeypatches the resolver itself, which removes coverage of the actual imageio_ffmpeg → shutil.which fallback chain that this PR introduces. Add a focused test for the resolver. |
Camera override in useAnnotationWorkspaceMediaSources |
Author calls this "indirect" — please add direct assertions for stale-override fallback and stable first-render cameraName. |
🔒 Security / robustness
LanguageInstructionAnnotation.paraphrasesandsubtask_instructionsare unbounded lists of unbounded strings.instructionitself ismax_length=1000, but the lists are not, so a client can post multi-MB payloads that get persisted to the on-disk JSON store. Please add amax_lengthon each list and a per-itemmax_lengthto mirrorinstruction.
🧠 Code soundness (medium)
LeRobotLoader._is_v2_layout: substring check"{episode_index"indata_pathworks today, but the comment promisescodebase_version-independent detection. A future v3 template that uses{episode_index}for naming would silently flip to v2 mode. Tighten by also requiring{episode_chunkor matching theepisode_*.parquetfilename pattern.build_trajectory: the velocity-estimation guard checks thelengthparameter, notjoint_positions.shape[0]. If a caller ever passeslength>1against a single-row positions array,np.diffreturns empty and the assignment raises. Addjoint_positions.shape[0] > 1to the guard.LeRobotLoader.get_tasks()parquet branch uses bareexcept Exceptionwhile the jsonl branch narrows to(json.JSONDecodeError, OSError). Make them consistent.
✅ What's solid
- Camera-override design (override state +
useMemo) avoids the autoplay double-fire cleanly. - AV1 fix via
imageio_ffmpegis the right call. - v2.x layout regression test (
TestV2EpisodeLayout) is well-built. start.sh --data-dirparsing handles both spellings with empty-value guarding.- Terraform
STORAGE_BACKENDrename is consistent.
…rename - bound paraphrases/subtask_instructions, tighten v2 layout detection, fix velocity guard - rename HMI_DATA_PATH to DATA_DIR and HMI_LOCAL_DATA_PATH to DATAVIEWER_HOST_DATA_DIR - add backend + frontend tests for new annotation surfaces - document camera selector and language instruction widget 🛠️ - Generated by Copilot
- reflow build_trajectory velocity guard onto a single line per ruff - reformat new LanguageInstructionWidget and media-sources tests with prettier 🎨 - Generated by Copilot
Contributor
Author
|
Thank you @katriendg for the review! Tried to tackle most of those misses and gaps. I am doing a refactor of the agentic harness in a separate branch, but gave the dataviewer a pass with a couple datasets to make sure it works with the new features now. But much more to come in a future PR! |
…aviewer-multi-camera - Cover playback card controls, range slider, and media-source branches - Add trajectory tab and content guard tests - Extend LanguageInstructionWidget, api-client, and annotation type tests - PR-scoped frontend coverage: 96.5% lines, 97.0% functions, 79.0% branches - Track remaining branch gaps under #210 sub-issues (#214, #215, #217) 🤖 - Generated by Copilot
…me interpolation 🔧 - Generated by Copilot
katriendg
approved these changes
Apr 30, 2026
Collaborator
katriendg
left a comment
katriendg
left a comment
There was a problem hiding this comment.
Thanks @akzaidi! I added a few front-end tests, though we need to cover more tests which is already tracked by front-end test issue backlog. Just focused on some of the changes.
I believe this is good to merge, and then @WilliamBerryiii we can pick-up and work on #590
WilliamBerryiii
pushed a commit
that referenced
this pull request
May 8, 2026
🤖 I have created a release *beep* *boop* --- ## [0.8.0](v0.7.4...v0.8.0) (2026-05-08) ### ⚠ BREAKING CHANGES * **dataviewer:** bump frontend stack to React 19, Vite 8, Tailwind v4, MSAL 5, ESLint 10 ([#524](#524)) ### ✨ Features * **agents:** add automated validation for high-risk Dependabot bumps ([#574](#574)) ([8c3686a](8c3686a)), closes [#573](#573) * **data:** add camera selector to annotation workspace and fix AV1 frame extraction ([#591](#591)) ([c809d2f](c809d2f)) * **data:** seed dataviewer frontend test foundation and per-section codecov flags ([#594](#594)) ([c06c4e3](c06c4e3)) * **dataviewer:** add OWASP security middleware stack ([#439](#439)) ([239edb9](239edb9)) * **infrastructure:** add conversion pipeline Terraform module ([#542](#542)) ([244531e](244531e)) * **infrastructure:** upgrade OSMO to chart 1.2.1 / image 6.2 with secure auth and skrl 2.0.0 compatibility ([#492](#492)) ([edfd7a5](edfd7a5)) * **pipeline:** add ACSA setup for ROS2 bag sync to Blob ([#451](#451)) ([c271a54](c271a54)) * **workflows:** add advisory Dependabot PR reviewer agentic workflow ([#498](#498)) ([d4bb140](d4bb140)) * **workflows:** trigger AW Dependabot PR reviewer after PR Validation ([#580](#580)) ([7ab3d16](7ab3d16)) ### 🐛 Bug Fixes * **ci:** correct stale version comment for actions/create-github-app-token ([#506](#506)) ([b2e9a54](b2e9a54)) * **ci:** restore data-pipeline and training broken tests by domain folder restructure ([#547](#547)) ([06d8472](06d8472)) * **docs:** update remaining stale 'Coming soon' labels in docs/README.md ([#507](#507)) ([02439d6](02439d6)) * **docs:** update stale coming soon label for Training section ([#472](#472)) ([46db49b](46db49b)) * **evaluation:** scope SIL AzureML validation code path and script reference ([#387](#387)) ([9f138a9](9f138a9)) * **infrastructure:** OSMO workflow execution, PostgreSQL public access, and quickstart corrections ([#477](#477)) ([9ed2da6](9ed2da6)) * **scripts:** exclude CHANGELOG.md from changed-files msdate check ([#644](#644)) ([8133bdc](8133bdc)) * **workflows:** allow dependabot[bot] to activate AW Dependabot PR Review ([#586](#586)) ([39dc022](39dc022)) * **workflows:** correct branches filter on AW Dependabot PR Review workflow_run trigger ([#584](#584)) ([fe06b52](fe06b52)) * **workflows:** normalize validate.yaml placeholder env/compute values ([#510](#510)) ([340ff44](340ff44)) * **workflows:** recompile aw-dependabot-pr-review lock file ([#576](#576)) ([d77c167](d77c167)) * **workflows:** switch AW Dependabot PR Review to pull_request_target ([#589](#589)) ([3f1edd1](3f1edd1)) ### 📚 Documentation * **docs:** Fix deployment guide links ([#614](#614)) ([0070b04](0070b04)) * document dependency-pinning-artifacts directory purpose ([#508](#508)) ([50e0010](50e0010)) ### 📦 Build System * **training:** standardize on Python 3.12 across manifests, containers, and runtime scripts ([#541](#541)) ([7ad014a](7ad014a)) ### 🔧 Operations * **build:** add Copilot cloud agent setup-steps workflow ([#593](#593)) ([c912668](c912668)) ### 🔧 Miscellaneous * **build:** exclude auto-generated CHANGELOG.md from cspell and seed dictionary ([#582](#582)) ([de1dd57](de1dd57)) * **build:** redesign codecov flags and split pytest CI per component ([#520](#520)) ([357e745](357e745)) * **dataviewer:** bump frontend stack to React 19, Vite 8, Tailwind v4, MSAL 5, ESLint 10 ([#524](#524)) ([50f8ad4](50f8ad4)) * **dataviewer:** repoint stale src/dataviewer references to data-management/viewer ([#504](#504)) ([88fa1b4](88fa1b4)), closes [#503](#503) * **deps-dev:** bump basic-ftp from 5.3.0 to 5.3.1 ([#618](#618)) ([ca10f2a](ca10f2a)) * **deps-dev:** bump globals from 15.15.0 to 17.5.0 in /data-management/viewer/frontend ([#527](#527)) ([0e0b2ae](0e0b2ae)) * **deps-dev:** bump ip-address from 10.1.0 to 10.2.0 ([#616](#616)) ([816c9cf](816c9cf)) * **deps-dev:** bump lint-staged from 16.4.0 to 17.0.2 in the root-npm-dependencies group across 1 directory ([#626](#626)) ([0e2f293](0e2f293)) * **deps-dev:** bump pydantic from 2.13.3 to 2.13.4 in the python-dependencies group across 1 directory ([#629](#629)) ([c24f1c1](c24f1c1)) * **deps-dev:** bump the python-dependencies group across 1 directory with 2 updates ([#514](#514)) ([8410f4b](8410f4b)) * **deps:** bump azure-core from 1.39.0 to 1.40.0 in /evaluation in the inference-dependencies group across 1 directory ([#597](#597)) ([6141db4](6141db4)) * **deps:** bump cryptography from 46.0.6 to 46.0.7 in /data-management/viewer ([#424](#424)) ([5fb6d58](5fb6d58)) * **deps:** bump cryptography from 46.0.6 to 46.0.7 in /data-management/viewer/backend ([#423](#423)) ([b516ad5](b516ad5)) * **deps:** bump lucide-react from 0.469.0 to 1.8.0 in /data-management/viewer/frontend ([#528](#528)) ([1bdfc1e](1bdfc1e)) * **deps:** bump nginx from `8aa63af` to `5616878` in /data-management/viewer/frontend ([#511](#511)) ([9e7e20e](9e7e20e)) * **deps:** bump nginx from 1.27-alpine to 1.29-alpine in /data-management/viewer/frontend ([#484](#484)) ([0e5c3dd](0e5c3dd)) * **deps:** bump node from `435f353` to `e49fd70` in /data-management/viewer/frontend ([#560](#560)) ([2884649](2884649)) * **deps:** bump react-is from 18.3.1 to 19.2.5 in /data-management/viewer/frontend ([#530](#530)) ([d51318c](d51318c)) * **deps:** bump tensordict from 0.11.0 to 0.12.1 in /evaluation in the inference-dependencies group across 1 directory ([#456](#456)) ([b24e733](b24e733)) * **deps:** bump the dataviewer-backend-dependencies group across 1 directory with 2 updates ([#531](#531)) ([171a1da](171a1da)) * **deps:** bump the dataviewer-backend-dependencies group across 1 directory with 5 updates ([#516](#516)) ([4f9a577](4f9a577)) * **deps:** bump the dataviewer-backend-dependencies group across 1 directory with 5 updates ([#602](#602)) ([6c27ab5](6c27ab5)) * **deps:** bump the dataviewer-dependencies group across 1 directory with 2 updates ([#529](#529)) ([8646971](8646971)) * **deps:** bump the dataviewer-dependencies group across 1 directory with 3 updates ([#601](#601)) ([d28fb50](d28fb50)) * **deps:** bump the dataviewer-dependencies group across 1 directory with 3 updates ([#632](#632)) ([4ca5f3e](4ca5f3e)) * **deps:** bump the dataviewer-dependencies group across 1 directory with 5 updates ([#515](#515)) ([109ee81](109ee81)) * **deps:** bump the dataviewer-frontend-patch-minor group across 1 directory with 6 updates ([#630](#630)) ([04d5dfd](04d5dfd)) * **deps:** bump the dataviewer-frontend-patch-minor group across 1 directory with 9 updates ([#563](#563)) ([c08f450](c08f450)) * **deps:** bump the docusaurus-dependencies group across 1 directory with 4 updates ([#627](#627)) ([f5825fc](f5825fc)) * **deps:** bump the docusaurus-dependencies group across 1 directory with 6 updates ([#599](#599)) ([b859344](b859344)) * **deps:** bump the github-actions group across 1 directory with 4 updates ([#459](#459)) ([2609c52](2609c52)) * **deps:** bump the github-actions group across 1 directory with 4 updates ([#517](#517)) ([f54bf5d](f54bf5d)) * **deps:** bump the inference-dependencies group across 1 directory with 11 updates ([#562](#562)) ([087f53a](087f53a)) * **deps:** bump the inference-dependencies group across 1 directory with 2 updates ([#628](#628)) ([4a3be47](4a3be47)) * **deps:** bump the pip group across 2 directories with 1 update ([#494](#494)) ([a14b6b0](a14b6b0)) * **docs:** update stale Python 3.11 references to 3.12 ([#575](#575)) ([6f85c95](6f85c95)) * **scripts:** remove redundant SC1091 disables in OSMO deploy scripts ([#509](#509)) ([ae1cb82](ae1cb82)) ### 🔒 Security * **build:** pin dependencies and hash-verify downloads ([#465](#465)) ([0289f49](0289f49)) * **build:** remediate dependency security advisories ([#479](#479)) ([7196d6d](7196d6d)) * **deps-dev:** bump basic-ftp from 5.2.1 to 5.2.2 ([#454](#454)) ([cb158f1](cb158f1)) * **deps-dev:** bump basic-ftp from 5.2.2 to 5.3.0 ([#495](#495)) ([e983b8b](e983b8b)) * **deps-dev:** bump hypothesis from 6.152.3 to 6.152.4 in the python-dependencies group ([#598](#598)) ([83384d2](83384d2)) * **deps-dev:** bump markdownlint-cli2 from 0.22.0 to 0.22.1 in the root-npm-dependencies group ([#559](#559)) ([32bde35](32bde35)) * **deps-dev:** bump picomatch from 2.3.1 to 2.3.2 in /docs/docusaurus ([#455](#455)) ([66f86ca](66f86ca)) * **deps-dev:** bump postcss from 8.5.10 to 8.5.12 in /data-management/viewer/frontend ([#569](#569)) ([a652dba](a652dba)) * **deps-dev:** bump the python-dependencies group with 2 updates ([#457](#457)) ([749d231](749d231)) * **deps-dev:** bump the python-dependencies group with 2 updates ([#485](#485)) ([71b44fd](71b44fd)) * **deps-dev:** bump the python-dependencies group with 3 updates ([#564](#564)) ([9fc52fd](9fc52fd)) * **deps-dev:** bump typescript from 6.0.2 to 6.0.3 in /docs/docusaurus in the docusaurus-dependencies group ([#513](#513)) ([5694dbc](5694dbc)) * **deps:** bump azureml/openmpi4.1.0-ubuntu22.04 from 20260303.v5 to 20260409.v4 in /evaluation/sil/docker ([#480](#480)) ([25d4df8](25d4df8)) * **deps:** bump cryptography from 46.0.6 to 46.0.7 in /evaluation in the uv group across 1 directory ([#538](#538)) ([92c5b2e](92c5b2e)) * **deps:** bump diffusers from 0.35.2 to 0.38.0 in /training/il/lerobot ([#638](#638)) ([6261d19](6261d19)) * **deps:** bump follow-redirects from 1.15.11 to 1.16.0 in /docs/docusaurus ([#469](#469)) ([0458908](0458908)) * **deps:** bump gitpython and mako for lerobot IL training ([#623](#623)) ([9f8022b](9f8022b)) * **deps:** bump node from 24.14.1-slim to 25.9.0-slim in /data-management/viewer/frontend ([#482](#482)) ([1532d09](1532d09)) * **deps:** bump packaging from 26.0 to 26.1 in /evaluation in the inference-dependencies group ([#483](#483)) ([f4afb6c](f4afb6c)) * **deps:** bump pillow from 12.1.1 to 12.2.0 ([#467](#467)) ([39fb663](39fb663)) * **deps:** bump python from 3.11-slim to 3.14-slim in /data-management/viewer/backend ([#481](#481)) ([7af9dfc](7af9dfc)) * **deps:** bump the dataviewer-backend-dependencies group across 1 directory with 15 updates ([#428](#428)) ([e4446a2](e4446a2)) * **deps:** bump the dataviewer-backend-dependencies group in /data-management/viewer/backend with 4 updates ([#487](#487)) ([0f57c5b](0f57c5b)) * **deps:** bump the dataviewer-backend-dependencies group in /data-management/viewer/backend with 8 updates ([#566](#566)) ([d6e7869](d6e7869)) * **deps:** bump the dataviewer-dependencies group across 1 directory with 5 updates ([#464](#464)) ([24c208d](24c208d)) * **deps:** bump the dataviewer-dependencies group in /data-management/viewer with 2 updates ([#486](#486)) ([90149f3](90149f3)) * **deps:** bump the dataviewer-dependencies group in /data-management/viewer with 6 updates ([#565](#565)) ([f0bb36b](f0bb36b)) * **deps:** bump the dataviewer-frontend-patch-minor group across 1 directory with 10 updates ([#613](#613)) ([e481f83](e481f83)) * **deps:** bump the github-actions group across 1 directory with 4 updates ([#534](#534)) ([5478ab6](5478ab6)) * **deps:** bump the github-actions group with 2 updates ([#488](#488)) ([4e6ce98](4e6ce98)) * **deps:** bump the github-actions group with 3 updates ([#567](#567)) ([48c38dc](48c38dc)) * **deps:** bump the github-actions group with 3 updates ([#634](#634)) ([00cfb49](00cfb49)) * **deps:** bump the github-actions group with 6 updates ([#603](#603)) ([73eb79a](73eb79a)) * **deps:** bump the training-dependencies group across 1 directory with 23 updates ([#463](#463)) ([d5a8656](d5a8656)) * **deps:** bump yaml from 2.8.2 to 2.8.3 in /data-management/viewer/frontend ([#453](#453)) ([10449df](10449df)) * pytest harness, dependabot advisories, and OSSF Scorecard remediations ([#501](#501)) ([e8756e8](e8756e8)) * **scripts:** pin and hash-verify all shell script downloads ([#468](#468)) ([0c2bb9c](0c2bb9c)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: physical-ai-toolchain-release[bot] <267194360+physical-ai-toolchain-release[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request
PR Soundtrack: Jaimie Branch - Fly or Die
Description
Multi-camera viewing, AV1 thumbnails, LeRobot v2.x compatibility, language-instruction annotations, a compact trajectory tab, and a cleaner dataviewer environment.
Multi-camera selector + AV1 thumbnails
The annotation workspace previously hard-pinned the displayed camera to
cameras[0], leaving no way to view secondary views (e.g.,wristvsfront) on the Trajectory Viewer / Object Detection tabs. Selection now lives inuseAnnotationWorkspaceMediaSources, derives synchronously viauseMemoagainst acameraOverridestate (so first render produces a stablevideoSrcand autoplay no longer double-fires), and falls back tocameras[0]when the override no longer matches the available list.The backend frame-extraction fallback called the system
ffmpegviashutil.which, but the host's ffmpeg is broken (libGL symbol error) and cv2's bundled ffmpeg does not include libdav1d. Newer LeRobot datasets ship AV1-encoded MP4s, so frame thumbnails returned 404. The handler now resolves to theimageio-ffmpegstatic binary first (ships libdav1d and libx264).LeRobot v2.x layout support
The dataset loader now handles the v2.x
meta/episodes.jsonl+meta/tasks.jsonllayout in addition to the legacy parquet metadata, and reads per-episode lengths from JSONL when available. Estimates joint velocities from positions when a dataset omits them, so the Trajectory Plot no longer renders empty for v2.x captures.Language instruction annotations
Adds a
LanguageInstructionWidgetto the annotation panel for capturing natural-language task descriptions, paraphrases for data augmentation, and subtask decompositions for hierarchical policy conditioning. Backend exposes a newLanguageInstructionAnnotationmodel, store actions, and an API endpoint; tasks are loaded frommeta/tasks.jsonl/tasks.parquetand dataset feature keys are preserved when transforming snake_case API responses.Compact trajectory tab layout
Moves the trajectory plot and subtask timeline above the subtasks pane inside a single scrollable playback group, shrinks the trajectory plot to a fixed 180px height, and collapses the labels panel to a single full-height column.
Dataviewer environment cleanup
HMI_DATA_PATH→DATA_DIRandHMI_STORAGE_BACKEND→STORAGE_BACKENDacross backend config, routers, services, tests,.envexamples, Dockerfile, docker-compose, README, and the dataviewer Terraform Container Apps module.data-management/viewer/start.shnow accepts--data-dir <path>(and--data-dir=<path>), defaultsDATA_DIRto<repo>/datasets, exports it to the backend, and warns when the path is missing.dev:backendnpm script defaultsDATA_DIRto the repo-rootdatasets/so launching from the VS Code task picks up datasets out of the box.Closes #436
Type of Change
HMI_DATA_PATH→DATA_DIR,HMI_STORAGE_BACKEND→STORAGE_BACKEND)Component(s) Affected
infrastructure/terraform/modules/dataviewer/container-apps.tfenv var renameTesting Performed
leisaac-pick-orange,so101-multi-pick-merged,so101-dual-camera-demos_*`)observation.images.*keys and switching updates both<video>source and frame thumbnails/api/datasets/<id>/episodes/<ep>/frames/<idx>?camera=...(previously 404 withFailed to read frame)Local re-validation on feat/dataviewer-multi-camera: npm run test 701/701, npm run lint 0 errors, ruff check src/ clean, pytest 432 pass (1 unrelated HDF5 video test skipped in environments without ffmpeg/cv2).
Documentation Impact
data-management/viewer/README.mdand backend.env.example/.env.azure.examplefor the env var rename.Bug Fix Checklist
tests/test_lerobot_loader.pycovers v2.x layout and JSONL metadata)use-annotation-workspace-media-controller.test.tsxand the workspace integration tests.Migration Notes
If you maintain local
.envfiles or external deployment configs for the dataviewer, rename:HMI_DATA_PATHDATA_DIRHMI_STORAGE_BACKENDSTORAGE_BACKENDdata-management/viewer/start.shandnpm run dev:backendnow defaultDATA_DIRto the repo-rootdatasets/folder, so most local setups need no override.Checklist