Introduce libMP (Media Pipe library)

[ngphibang]

In Zephyr today, multimedia applications—such as those involving video, audio, display, vision, and graphics—are typically implemented as simple, domain-specific sample applications. While these are sufficient for basic use cases, they quickly become inadequate when dealing with:

• Complex processing chains, e.g., multiple processing components between a camera and a display.
• Cross-domain scenarios, e.g., an MPEG-DASH player handling video, audio, and subtitles streamed over a network, with dynamic resolution changes.

In such cases, application complexity increases significantly. Developers must manually manage buffer allocation, queuing and dequeuing for each component as well as synchronization between components across the pipeline. Moreover, similar functionality often needs to be reimplemented across projects, leading to duplicated effort. Applications also tend to require extensive customization for each use case and become fragile to even minor changes in requirements.

To address these challenges, this PR introduces libMP (MediaPipe library)—a lightweight multimedia framework designed specifically for Zephyr.

This PR depends on the following PRs:

• video: display: misc changes - draft #98513
• drivers: audio: Add get_caps API to DMIC, I2S, and Codec subsystems #98500
• sys: util: Add SIGN, gcd and lcm #98875

libMP aims to simplify the development of multimedia applications by:

• Abstracting buffer management and synchronization.
• Providing a modular and extensible pipeline architecture.

It also streamlines the development of multimedia components (plugins) by:
• Offering a consistent, well-defined framework for plugin developers.
• Enabling reuse across different multimedia components.

libMP reuses many concepts from GStreamer—such as elements, pads, caps negotiation, and buffer negotiation—and adopts a pipeline-based architecture that decomposes multimedia processing into discrete, interconnected elements.

Applications simply select the built-in elements suited to their purpose to construct a pipeline, and it just works. This design promotes modularity, reusability, and efficient resource management (e.g., zero-copy data flow), which are critical for resource-constrained embedded systems.

libMP features a highly modular, inheritance-based architecture inspired by GStreamer, ensuring modularity, scalability, and maintainability. For example, new custom elements can be easily added via plugins by extending existing elements—without requiring modifications to the core components. Additionally, plugins are selectively built by enabling their corresponding Kconfig options, helping to minimize memory footprint. Key design highlights include:

• Decentralized core structures such as caps and properties, allowing seamless extension without altering the core framework.
• Stable and generic public APIs, enabling application code to remain unchanged even as libMP evolves.

Currently, libMP is provided with proof-of-concept (PoC) examples for both video and audio pipelines:

• Video pipeline: A simple 4-element chain consisting of a camera source, capsfilter, video transform, and display sink.
• Audio pipeline: A 3-element chain composed of a DMIC source, gain transform, and I2S sink.

Additional plugins and example pipelines can be added in the future. Among them, the prioritized TODOs are:

• Complete the pipeline stop implementation
• Support multi-core pipeline (part of pipeline running on a different DSP / NPU core without OS)
• Support building pipelines via command line or config file so that we don’t need to add more and more examples
• Support pull mode
• Added video jpeg codec and H.264 codec plugins
• Adding useful built-in elements such as: queue, tee, appsrc, appsink, etc.

[JarmouniA]

I would start with the name (both libMP & MediaPipe): not great، ex. of some existing projects using it:
https://github.com/gpudirect/libmp
https://man.freebsd.org/cgi/man.cgi?query=libmp&sektion=3&format=html
https://github.com/google-ai-edge/mediapipe

Also, shouldn't this be hosted as an external RTOS-agnostic library, like libmpix & LVGL, it would see wider adoption that way in my opinion & would have better APIs.

[butok]

Can it be shortened to "mp" instead of "libmp"?
It's placed in the "lib" folder, so it's clear that it's a library.

[ngphibang]

yes, we can. Apart from libc which has the lib prefix that I think due to historic reason, others don't have it. Noted and will change when we are firmed on the project name.

[butok]

Is it possible to avoid the dynamic memory allocation in the library?

[ngphibang]

For this place, it's possible because sizeof(MpCaps) is fixed but caps need to be set in to MpQuery and sent across function too, so maybe we can use a static array or memslab (?) but then need to specify a max number for it. But this can be done specifically here only as we couldn't avoid dynamic alloc in the whole library. For example, the items (structure, value) in caps are dynamic and is known only at runtime when querying HW. Or when creating elements, the size of elements are not known beforehand because plugined elements sizes are variable.

[josuah]

Some possible way to dodge k_alloc() is k_heap_alloc() with a heap local to libMP configured with Kconfig.

With a possible K_FOREVER instead of K_NO_WAIT in functions that need to return void.

[ngphibang]

I would start with the name (both libMP & MediaPipe): not great، ex. of some existing projects using it:
https://github.com/gpudirect/libmp
https://man.freebsd.org/cgi/man.cgi?query=libmp&sektion=3&format=html
https://github.com/google-ai-edge/mediapipe

Indeed, these are existing projects which have the same name with libMP. In fact, we tried to change the name several times, it's difficult to have an intuitive name which does not overlap with the existing ... What about libMPL, I do not see this elsewhere ? Do you know if we need to change also the prefix (mp_) in the code when changing the project name ?

Also, shouldn't this be hosted as an external RTOS-agnostic library, like libmpix & LVGL, it would see wider adoption that way in my opinion & would have better APIs.

I think it's a bit different compared to libmpix & LVGL. AFAIK, libmpix is mainly about math, algorithms (color / format conversions, etc.) which is rather OS-agnostic. LVGL exists nearly at the same time with Zephyr and has it own life-cycle outside Zephyr. Moreover, LVGL has its own eco-system and does not interact much with the OS, except when it comes to touch (input) and HW accelerators (such as PxP, VGLite, but for this, LVGL calls directly to the low level drivers and bypasses Zephyr subsystems and drivers. About this, I don't know how it works when we want to use LVGL and Zephyr stuffs on the PxP in the same application, does it lead to conflicts because Zephyr stuffs will pass by the video subsystem through the PxP Zephyr driver to the PxP low level driver while LVGL stuffs will pass directly to the PxP low level driver ? ...). So, to port LVGL to Zephyr, we need mainly an OSA and some glue codes.

BTW, libMP used (and will use) heavily Zephyr mechanismes such as devicetree, iterable section, work queue, rtio, etc. to optimize the implementation. If we make an OS-agnostic version, we need to create something equivalent or the implementation cannot be optimized. The media components (plugins, elements) in libMP interact deeply with the OS, they calls directly to the (Zephyr) video, audio, display, vision, ... subsystems. So even in a generic libMP version, these components need to be created separately for each OS. And to support FreeRTOS, as an example, where there are no such subsystems we need to create all of them (kind of a HAL layer and need to reproduce the APIs a bit like in Zephyr).

Another reason is, as an external module, libMP is required to have its own life-cycle outside the Zephyr Project, that is, reside in its own repository, and have its own contribution, testing and maintenance workflow and release process. We need to do integration into Zephyr regularly (like LVGL) and review all code from contributors (that may come from many different domains : video, audio, display, vision, NPU, etc.) where we don't have enough resource to do that.

Looking that such a unified multimedia framework does not exist yet in Zephyr (there are some frameworks for audio such as Maestro, but when integrated into Zephyr it bypasses all Zephyr subsystems, so not a real integration), making it inside Zephyr, we expect much more contribution and helps from the Zephyr community and benefit Zephyr infrastructure (the current code base is just an initiative).

So, IMHO, if we support FreeRTOS in the future, we could port it or maintain two versions where the generic version may not be optimized and the Zephyr version may grow much faster and has its own development cycle.

[butok]

Also we need to understand if this is a Zephyr Subsystem or a Zephyr Library.

[ngphibang]

Also we need to understand if this is a Zephyr Subsystem or a Zephyr Library.

It seems to me that it's a Zephyr library (?)

[josuah]

For this message, I only look at the content of lib/mp/src/core: the framework source itelf.

It seems like there is some RTOS abstraction layer, which needs to stay if this is not meant as Zephyr-first/only implementation:

• mp_bus.c -> Zbus
• mp_event.c -> Zbus / Events
• mp_messages.c -> Zbus
• mp_task.c -> k_thread_...() (maybe no wrapper needed if Zephyr only target)
• mp_pixel_format.h -> four character codes
• mp_plugin.c -> SYS_INIT() in plugins (does not work if plugins need to support non-Zephyr)
• mp_buffer.c -> net_buf / RTIO buffers / other
• mp_utils.c. -> sys/utils

And then a very small core on top of it, bringing the bulk of what a media subsystem would need to do in an RTOS.
If I got it right:

• mp_value.c, mp_structure.c: A generic configuration library
• mp_bin.c, mp_element.c, mp_element_factory.c: A pipeline configuration API
• mp_pipeline.c, mp_pad.c, mp_src.c, mp_sink.c: A pipeline runtime API
• mp_caps.c, mp_property.h, mp_object.c, mp_query.c, : A pipeline control API

So +1 to try to reduce the number of elements to integrate and abstraction layers:

• Vendor-specific solution: Vendor SDK > Vendor library > Application
• External library on Zephyr: Vendor HAL > Zephyr > RTOS abstraction layer > library > application
• Internal library on Zephyr: Vendor HAL > Zephyr > Application

[josuah]

Some "complex" or "multi-component" camera/video hardware is arriving to Zephyr:

• i.MXRT1170: MIPI + scaler + display (maybe more evolved hardware coming)
• STM32N6: MIPI + debayer + ISP + scaler + encoder + display
• ESP32-P4: MIPI + debayer + ISP + scaler + encoder + display
• Renesas RA8D2: MIPI + ISP + scaler + display
• MPUs converted from Linux to Zephyr (like STM32MP1 or SG2000)
• ...

Depending on the hardware, a different application has to be written (currently managed with a growing number of #ifdef), unless there exists a framework to turn this variability into configuration.

In that sense, libMP can also be seen as an essential part of video hardware integration as it enables writing the basic video samples without hundreads of lines of copy-pasta boilerplate.

For instance, here is Zephyr implementation of libMP's element->srcpads locally inside the UVC sample:

zephyr/samples/subsys/usb/uvc/src/main.c

Lines 46 to 53 in a8bf08b

	static struct video_caps *app_uvc_source_caps(void)
	{
	if (app_has_videoenc()) {
	return &videoenc_out_caps;
	} else {
	return &video_caps;
	}
	}

This encourages adding a dependency from Zephyr video samples to libMP, whichever way it is integrated.

[ngphibang]

Thanks for the comment.

It seems like there is some RTOS abstraction layer, which needs to stay if this is not meant as Zephyr-first/only implementation:

In fact, these are not RTOS abstraction layer but the "components" that we built to use in libMP. But you are right, there are things that we could (change /)move to other places to lighten the library.

mp_bus.c -> Zbus
mp_messages.c -> Zbus

The "bus" and "message" concept in libMP are much lighter than Zbus. Basically it's just a FIFO containing messages from the pipeline sent to the application (one way) so I think using Zbus is a bit overkill.

mp_event.c -> Zbus / Events

Event in libMP is different from the generic event mechanism and event in Zephyr. As seen in the code, it's simply a structure that contains a pointer to a data structure. There is no mechanism for "listening" or "broadcasting" the event. Element sends an event to downstream or upstream by simply putting it in the function parameters, and the element can handle the event or propagate it but this is implemented inside the element itself.

mp_event and mp_query are nearly the same and should be refactored (will do).

mp_task.c -> k_thread_...() (maybe no wrapper needed if Zephyr only target)

Actually we use k_thread and just refactored into functions to not to duplicate code. Task will be extended more in the future.

mp_pixel_format.h -> four character codes

mp_pixel_format are just enums to unify formats from different domains (video, display, vision. etc.) so that they can understand each other. So an enum is sufficient, I don't see why we need a FOURCC ... and there are some formats (in display) that don't have FOURCC.

mp_plugin.c -> SYS_INIT() in plugins (does not work if plugins need to support non-Zephyr)

Yes, that's right. Instead of calling mp_init() in each application. libMP can be initialized with SYS_INIT(). I will do that. So, by this, it turns out that libMP should be a subsys than a lib.

mp_buffer.c -> net_buf / RTIO buffers / other

Currently libMP buffer pool is just an array of buffer structures to map to the real data buffers comming from each subsystem, no FIFO, no handling mechanism required (it's already done differently in each subsystem, e.g. video subsystem already used RTIO - ongoing work), element push buffer to downstream one by one after processing it. So, I am not sure to be able to use RTIO for this but I thought of that. Will rethink about this when we finished switching to RTIO for video subsystem.

mp_utils.c. -> sys/utils

That's right. This can be taken out and upstream to sys/utils.

mp_value.c, mp_structure.c: A generic configuration library

That's right too. These can be taken out from libMP. But currently I don't know where to put it in Zephyr.

• mp_value are wrappers to support handling value of primitive and non-primitive (range, list) types, doing comparison and intersection operators on them.
• mp_structure is a generic abstracion for a dynamic data structure built on top of mp_value, kind of {field, value} pair which can be appended into a caps structure.

Both are used for caps and query. Basically they are generic and can be used outside libMP but it's hard to find another usage than this one.

[josuah]

Thank you for walking through these points, this helps estimating the overlap with Zephyr features and figure out how to reuse existing Zephyr code to lighten libMP, and where it is not useful/possible to do so.

[josuah]

This could act as integration layer to all of these?

• Image/Audio input drivers (MIPI, DVP, PDM, I2S...)
• Image/Audio output drivers (displays, speakers, I2S...)
• Bluetooth Audio (LEA/auracast?)
• USB (Audio UAC2, Video UVC, both host/device)
• Networking (simple TCP capture, libsrtp support is coming, HTTP-based streaming)
• Storage (recording, playback)
• NPU
• Container formats (mkv, lc3, mpeg-ts, mp3, ogg, opus...)
• Processing (i.e. echo cancellation library, color tuning)
• SOF (?)

Maybe even sensors: combine temperature data with an audio recording of engine noise and send both to an NPU.

[ngphibang]

Updates:

• Dropped License file, dropped tests (will re-add a more complete version in a separate commit)
• Some compliance check and coding style fixups
• Some refactors in mp_value. Also add support for uint type. (Still need to find a way to move mp_value and mp_structure to a separate lib / util)

[ngphibang]

Updates:

• Add capsfilter element
• Some fixes in mp_value
• Implement default get/set/transform_caps in mp_transform
• Fix compliance failures

[josuah]

@butok:

Also we need to understand if this is a Zephyr Subsystem or a Zephyr Library.

Connecting this question with your own message from yesterday:

• Define the boundaries between driver class, library and subsystem #99250

[Copilot]

Pull Request Overview

This PR introduces libMP (MediaPipe library), a lightweight multimedia framework for Zephyr that simplifies development of multimedia applications. The framework provides a modular pipeline architecture inspired by GStreamer, with support for video and audio processing chains.

Key Changes

• New libMP core framework with element-based pipeline architecture
• Video plugin (zvid) supporting camera sources and video transforms
• Display plugin (zdisp) for rendering video output
• Audio plugin (zaud) with DMIC source, gain transform, and I2S codec sink
• Three sample applications demonstrating video and audio pipelines

Reviewed Changes

Copilot reviewed 93 out of 93 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
samples/subsys/libmp/video_examples/camera_transform_display/*	Video pipeline sample with camera, transform, and display
samples/subsys/libmp/video_examples/camera_display/*	Simpler video pipeline sample without transform
samples/subsys/libmp/audio_example/*	Audio pipeline sample with DMIC, gain, and speaker
lib/libmp/src/plugins/zvid/*	Video plugin implementation
lib/libmp/src/plugins/zdisp/*	Display plugin implementation
lib/libmp/src/plugins/zaud/*	Audio plugin implementation
lib/libmp/src/core/*	Core framework components (elements, buffers, values, etc.)

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[ngphibang]

Updates:

• Shortcut the capsfilter element after finishing caps negotiation
• Change from camel case to snake case
• Drop typedef uses
• Fix compliance
• Add libMP introduction in the commit message

[kartben]

Given this seems to have no user in Zephyr, it looks like you might want to contribute this as an external module instead, similar to libmpix

[JarmouniA]

Indeed, these are existing projects which have the same name with libMP. In fact, we tried to change the name several times, it's difficult to have an intuitive name which does not overlap with the existing ... What about libMPL, I do not see this elsewhere ?

ZMP :) (imitating ZBUS and ZMS)

Do you know if we need to change also the prefix (mp_) in the code when changing the project name ?

Yes

[butok]

Given this seems to have no user in Zephyr, it looks like you might want to contribute this as an external module instead, similar to libmpix

Should it be an "external" or "optional" module?

[ngphibang]

Given this seems to have no user in Zephyr, it looks like you might want to contribute this as an external module instead, similar to libmpix

It can be injected to current Zephyr video, display, audio samples, what I am doing.

[ngphibang]

Updates:

• zvid: Add PROP_DEVICE for src and DEFAULT_PROP_DEVICE for transform, refactor set/get_property() into zvid_object.
• Enable EOS message in video examples by using PROP_NUM_BUFS of mp_src to demonstrate message sending between pipeline and application
• Use enum for element and pipeline name / registry instead of string name for type safety and performance. The trade-off is we lost some level of modularity, i.e. when adding new plugin / element, we have to modify some (1 or 2) files in the core framework as well. Using enum instead of string for caps keys e.g. "format", "framerate", "width", "height" is planning as well
• Move bus field from pipeline to bin, rework mp_element_get_bus()
• Some fixes and refactors in mp_value
• Some minor fixes

[sonarqubecloud]

Quality Gate passed

Issues
35 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.8% Duplication on New Code

See analysis details on SonarQube Cloud