libMP: Tracking enhancement on added plugins and features

[ngphibang]

Summary

libMP (introduced in #98514) has an extensible architecture where we can extend its functionalities via adding plugins (a group of elements falls into the same scope).

This issue is to track the progress on adding plugins and features into libMP so that we can know which are already added, which will be added (in a prefered priority order) and maybe who is (interested) to work on which plugin.

Current added plugins / elements:

Built-in elements:

• mp_src
• mp_sink
• mp_transform
• mp_bin / mp_pipeline
• mp_capsfilter

Plugins:

• zvid: zvid_src, zvid_transform
• zdisp: zdisp_sink
• zaud: dmic_src, i2s_sink, (simple SW-based) zaud_gain

Describe the solution you'd like

The current priority (we see after some discussions, but not limited to this) is to leverage Zephyr (video / audio / ...) examples with libMP.

For audio, the main purpose is to keep current samples (which are small, compact and appropriate for some simple driver usecases which does not require powerful MCU, large memory footprint, etc.) but adding sample(s) to illustrate more complex pipelines, e.g. a pipeline composed of "dmic_src -> (gain or some processing) transform -> i2s_sink", the processing part of the pipeline may run on a different heterogenous core (e.g. DSP).

For video, we have 4 samples, one direction may be to keep the current non-libMP video capture sample (which is equivalent to a basic libMP pipeline "camera --> display") to illustrate the very basic usecase which uses directly Zephyr (video/display) APIs (so without needing additional overhead) then adding one more unified sample with libMP which can cover all the current four samples to illustrate the reusability and flexibility of libMP.

1. Video capture sample:

• zvid_src --> capsfilter --> (queue) --> zdisp_sink
• zvid_src --> capsfilter --> (queue) --> zvid_transform (PxP) --> (queue) --> zdisp_sink: to support usecase like the NXP's display that needs a video transformer (PxP) to rotate image 90 degree before displaying on LCD

2. Video capture-to-lvgl sample:

• zvid_src --> capsfilter --> (queue) --> "LVGL element" --> (queue) --> zdisp_sink: Currently the capture-to-lvgl Zephyr sample is the same as the capture sample, the only difference is it uses LVGL API to display frame instead of Zephyr display API. With libMP, we can extend it a bit to display an LVGL overlay like "a text showing frame rate" for example.

3. tcpserversink sample:

• zvid_src (camera) --> capsfilter --> (queue) --> jpegenc / h264enc --> (queue) -> tcpserver sink

4. uvc device sample:

• zvid_src (real camera) --> capsfilter --> (queue) --> jpegenc --> (queue) --> uvcsink

5. screen mirror demo: an additional usecase we can also support without adding much efforts.

• tcpserversrc / udpsrc --> (queue) --> jpegparse --> jpegdec --> (queue) --> zvid_transform (color conversion with PxP) --> queue --> zdisp_sink

In all above samples, the queue element is optional, added to parallellize the work between two elements so that one does not have to wait for another (which is not supported in the current Zephyr video samples).

So, based on the above pipeline analysis, the missing elements that need to be added in priority are:

• queue
• jpegenc / h264enc: Currently, the only upstream supported HW is from STMicroelectronics where we support only stateless encoder (?), so the jpegenc/h264enc element can be purely a zvid_transform element. In the future, if stateful encoder and/or seeking feature is required and/or dynamic resolution change required, we might need to add a videoencoder base element and the jpegenc/h264enc will base on it.
• tcpserversink / tcpserversrc
• uvcsink
• an LVGL (transform) element (?)
• jpegdec and jpegparse: If jpegdec is added, we will need also a jpegparse element to detect SOF / EOF
• filesrc / filesink

Alternatives

Other plugins / elements that you think need to be considered / added in priority or willing contributions are very welcome.

Additional Context

No response