Project: Plotting of "signals" style data

[Wumpf]

Summary description

Allow sending/logging of multi-dimensional data and plot them without additional data transformation by the user.

Background and problem

Many datasets use state vectors to capture robot state. For example lerobot uses a 7-dimensional vector of motor positions as their state.

Today, users have to break these up into separate columns/entities is somewhat awkward compared to just being able to log and plot using a ListArray (or more direct API) instead.

Grounding use-cases & evaluation

• Hugging face loader
• IMU data
• other things our design partners may hit?
• TODO: fill in more

Outline & Vision

Previous proposals were focused around introducing new purpose-built archetypes, see #7278. A number of issues were identified with this line of thinking:

• lots of code duplication, including existing technical debt
• type explosion
• very hard to name - depending on your area this may be about "signals" or "scalars" or "vectors" etc.

Taking a step back, our actual goal is for users to be able to supply arbitrary data and then (explicit via blueprint or by automated heuristic) set up a a visualizer for their data.

In accordance with the vision of a strong archetype-visualizer correlation (see #8368), we therefore should strive for a more flexible model where archetypes describes the visualization, but do not (overly) constrain how you present your data.

Concrete this means that both SeriesLines and SeriesPoints visualizers should be able to deal with more arbitrary kinds of "scalars" - different dimensions & different types.
They source the data from an appropriately tagged component

Note that we long planned to make component tags overwriteable in the viewer (see general tagged component issue #6889). Fully implemented this should allow us to...

• present the "data"/scalar component as input of the respective visualizers == field on the archetype
• deprecate the existing Scalar archetype but (!) still pick its contents up in a SeriesLines visualizer by means of heuristically overwriting the tag such that the visualizer consumes the component array
  • open question: how far can this go, can we pick up arbitrary "untagged" data? Likely! But requires more complex heuristic. If this is instead easy to configure (from code & viewer) this should not be necessary.

Areas & first steps

Thanks to now landed eager serialization, we can easily present the viewer with much more complex data than the API specification presents as "valid". This means in turn that we can make visualizers deal with data that is arbitrarily more complex and deal with API changes completely separately.

Therefore we can split this project into three mostly separate tracks:

TODO: many of these bullet points should be sub-issues

• Visualizer enhancements
  • process multi dimensional scalars
  • process multi dimensional colors/names/etc. on SeriesLine
  • necessary styling improvements/options. TBD
    • loosely related: bool plots
• API improvements
  • [keep it Python first for now]
  • rename SeriesLine/SeriesPoint to SeriesLines/SeriesPoints
    • TODO: can we do this with a deprecation step or do we break API?
  • make it easy to log scalar
  • decide on component tag(s) which are picked up by the visualizers
  • TODO: formalize genericness of data type? How to document which types are handled by the visualizer
• Demo & usecase exploration
  • note down good hugging face demos
  • build new IMU data example
  • feed back learnings into this issue

Other open questions

• How much of tagged component features Tagged components #6889 do we need to push forward towards this goal
  • are descriptor overrides in-scope?
    * 💡 we could do descriptor overrides BEFORE introducing full tag awareness - if you're able to change the ComponentName from e.g. Translation3D to Scalar (working title) you'd automatically get plots for your translations
  • formalize generic components?
• Various UX questions
  • how does one hide individual line series (see also Sync egui_plot item visibility with entity visibility #4974)
  • what does it mean to select an individual series or point
• how are visualizer dealing with varying number of elements in the data vector. Options:
  • pad with zeros
  • give some error that bubbles up to SDK
  • gaps? which "component" of the vector though?
• TODO: add more

Non-goals and won't do

• general plot improvements are likely as part of this issue, but not the core focus nor part of completion criteria
• supporting logging of columnar scalars on a single entity
  • while it is desirable feed many columns directly to a single SeriesLine, we implicitly support several columns by creating an entity per column
  • note that having several columns on a single entity is roughly equivalent with providing several "Scalar" components on a single entity all of which are picked up then by the same visualizer. (while this is sound for our data model, there's some friction here how this interacts with our visualizers/archetypes)
• higher dimensional plots - e.g. "a line over time" (~ landscape) -> sticking to "1D vectors over time"

Maybe out of scope:

• data types other than than f64
  • very easy to convert for users
  • complexity:
    • booleans need scale to fit with the rest in current representations

Other notes

• f32->f64 conversions in SDKs other than Python