Video 4 Zephyr - Video API, Drivers and Samples #17194

loicpoulain · 2019-07-01T12:55:22Z

DEPENDS on HAL_NXP change: zephyrproject-rtos/hal_nxp#6

PRESENTATION: https://docs.google.com/presentation/d/1j44YHUqynN-Vw67NgHHx741yzLIbjbc76tcObi8J3Zc/edit?usp=sharing

Generic Video API
MCUX CSI Video driver (CMOS Sensor Interface) => Video Capture driver
MT9M114 Video camera driver => Video control driver
SW Video pattern generator => Video Capture driver
A basic capture sample application, counting received frames
A tpcserversink sample pushing video to a TCP client

zephyrbot · 2019-07-01T13:12:10Z

All checks are passing now.

checkpatch (informational only, not a failure)

-:276: WARNING:LEADING_SPACE: please, no spaces at the start of a line
#276: FILE: drivers/video/Kconfig.mt9m114:10:
+        bool "MT9M114 Aptina CMOS digital image sensor"$

-:277: WARNING:LEADING_SPACE: please, no spaces at the start of a line
#277: FILE: drivers/video/Kconfig.mt9m114:11:
+        depends on I2C$

-:278: WARNING:LEADING_SPACE: please, no spaces at the start of a line
#278: FILE: drivers/video/Kconfig.mt9m114:12:
+        help$

-:279: WARNING:LEADING_SPACE: please, no spaces at the start of a line
#279: FILE: drivers/video/Kconfig.mt9m114:13:
+          Enable driver for MT9M114 CMOS digital image sensor device.$

-:295: WARNING:LEADING_SPACE: please, no spaces at the start of a line
#295: FILE: drivers/video/Kconfig.sw_generator:10:
+        bool "Video Software Generator"$

-:296: WARNING:LEADING_SPACE: please, no spaces at the start of a line
#296: FILE: drivers/video/Kconfig.sw_generator:11:
+        help$

-:297: WARNING:LEADING_SPACE: please, no spaces at the start of a line
#297: FILE: drivers/video/Kconfig.sw_generator:12:
+          Enable video pattern generator (for testing purpose).$

-:890: WARNING:LONG_LINE: line over 80 characters
#890: FILE: drivers/video/video_mcux_csi.c:116:
+	data->csi_config.polarityFlags = kCSI_HsyncActiveHigh | kCSI_DataLatchOnRisingEdge;

-:891: WARNING:LONG_LINE_COMMENT: line over 80 characters
#891: FILE: drivers/video/video_mcux_csi.c:117:
+	data->csi_config.workMode = kCSI_GatedClockMode; /* use VSYNC, HSYNC, and PIXCLK */

-:989: WARNING:TYPO_SPELLING: 'ouput' may be misspelled - perhaps 'output'?
#989: FILE: drivers/video/video_mcux_csi.c:215:
+		/* Flush driver ouput queue */

-:1275: WARNING:LINE_SPACING: Missing a blank line after declarations
#1275: FILE: drivers/video/video_sw_generator.c:84:
+			int color_idx =  data->ctrl_vflip ? 7 - w / bw : w / bw;
+			if (data->fmt.pixelformat == VIDEO_PIX_FMT_RGB565) {

-:1575: WARNING:LONG_LINE_COMMENT: line over 80 characters
#1575: FILE: include/drivers/video-controls.h:33:
+#define VIDEO_CTRL_CLASS_GENERIC	0x00000000	/* Generic class controls */

-:1576: WARNING:LONG_LINE_COMMENT: line over 80 characters
#1576: FILE: include/drivers/video-controls.h:34:
+#define VIDEO_CTRL_CLASS_CAMERA		0x00010000	/* Camera class controls */

-:1577: WARNING:LONG_LINE_COMMENT: line over 80 characters
#1577: FILE: include/drivers/video-controls.h:35:
+#define VIDEO_CTRL_CLASS_MPEG		0x00020000	/* MPEG-compression controls */

-:1578: WARNING:LONG_LINE_COMMENT: line over 80 characters
#1578: FILE: include/drivers/video-controls.h:36:
+#define VIDEO_CTRL_CLASS_JPEG		0x00030000	/* JPEG-compression controls */

-:1579: WARNING:LONG_LINE_COMMENT: line over 80 characters
#1579: FILE: include/drivers/video-controls.h:37:
+#define VIDEO_CTRL_CLASS_VENDOR		0xFFFF0000	/* Vendor-specific class controls */

-:1582: WARNING:LONG_LINE_COMMENT: line over 80 characters
#1582: FILE: include/drivers/video-controls.h:40:
+#define VIDEO_CID_HFLIP			(VIDEO_CTRL_CLASS_GENERIC + 0) /* Mirror the picture horizontally */

-:1583: WARNING:LONG_LINE_COMMENT: line over 80 characters
#1583: FILE: include/drivers/video-controls.h:41:
+#define VIDEO_CID_VFLIP			(VIDEO_CTRL_CLASS_GENERIC + 1) /* Mirror the picture vertically */

-:1666: WARNING:TYPO_SPELLING: 'suported' may be misspelled - perhaps 'supported'?
#1666: FILE: include/drivers/video.h:59:
+ * @param pixelformat is a list of suported pixel formats (0 terminated).

-:1903: WARNING:LONG_LINE_COMMENT: line over 80 characters
#1903: FILE: include/drivers/video.h:296:
+ * Enqueue an empty (capturing) or filled (output) video buffer in the driver’s

-:1987: WARNING:LONG_LINE_COMMENT: line over 80 characters
#1987: FILE: include/drivers/video.h:380:
+ * video_stream_start is called to enter ‘streaming’ state (capture, output...).

-:2154: WARNING:LONG_LINE: line over 80 characters
#2154: FILE: include/drivers/video.h:547:
+	((u32_t)(a) | ((u32_t)(b) << 8) | ((u32_t)(c) << 16) | ((u32_t)(d) << 24))

-:2157: WARNING:LONG_LINE_COMMENT: line over 80 characters
#2157: FILE: include/drivers/video.h:550:
+#define VIDEO_PIX_FMT_BGGR8  video_fourcc('B', 'G', 'G', 'R') /*  8  BGBG.. GRGR.. */

-:2158: WARNING:LONG_LINE_COMMENT: line over 80 characters
#2158: FILE: include/drivers/video.h:551:
+#define VIDEO_PIX_FMT_GBRG8  video_fourcc('G', 'B', 'R', 'G') /*  8  GBGB.. RGRG.. */

-:2159: WARNING:LONG_LINE_COMMENT: line over 80 characters
#2159: FILE: include/drivers/video.h:552:
+#define VIDEO_PIX_FMT_GRBG8  video_fourcc('G', 'R', 'B', 'G') /*  8  GRGR.. BGBG.. */

-:2160: WARNING:LONG_LINE_COMMENT: line over 80 characters
#2160: FILE: include/drivers/video.h:553:
+#define VIDEO_PIX_FMT_RGGB8  video_fourcc('R', 'G', 'G', 'B') /*  8  RGRG.. GBGB.. */

-:2163: WARNING:LONG_LINE_COMMENT: line over 80 characters
#2163: FILE: include/drivers/video.h:556:
+#define VIDEO_PIX_FMT_RGB565 video_fourcc('R', 'G', 'B', 'P') /* 16  RGB-5-6-5 */

-:2851: WARNING:LONG_LINE: line over 80 characters
#2851: FILE: soc/arm/nxp_imx/rt/dts_fixup.h:236:
+#define DT_VIDEO_MCUX_CSI_BASE_ADDRESS		DT_NXP_IMX_CSI_402BC000_BASE_ADDRESS

-:2853: WARNING:LONG_LINE: line over 80 characters
#2853: FILE: soc/arm/nxp_imx/rt/dts_fixup.h:238:
+#define DT_VIDEO_MCUX_CSI_IRQ_PRI		DT_NXP_IMX_CSI_402BC000_IRQ_0_PRIORITY

-:2855: WARNING:LONG_LINE: line over 80 characters
#2855: FILE: soc/arm/nxp_imx/rt/dts_fixup.h:240:
+#define DT_VIDEO_MCUX_CSI_SENSOR_NAME		DT_NXP_IMX_CSI_402BC000_SENSOR_LABEL

- total: 0 errors, 30 warnings, 2636 lines checked

NOTE: For some of the reported defects, checkpatch may be able to
      mechanically convert to the typical style using --fix or --fix-inplace.

Your patch has style problems, please review.

NOTE: Ignored message types: AVOID_EXTERNS BRACES CONFIG_EXPERIMENTAL CONST_STRUCT DATE_TIME FILE_PATH_CHANGES MINMAX NETWORKING_BLOCK_COMMENT_STYLE PRINTK_WITHOUT_KERN_LEVEL SPLIT_STRING VOLATILE

NOTE: If any of the errors are false positives, please report
      them to the maintainers.

Tip: The bot edits this comment instead of posting a new one, so you can check the comment's history to see earlier messages.

loicpoulain · 2019-07-05T16:26:02Z

Hi all, any comments on this PR?

microbuilder · 2019-07-05T18:22:54Z

Hi all, any comments on this PR?

I think with the 4th of July weekend and summer holidays a lot of people are out. I'll try to have a look at this Monday myself, though.

include/drivers/video.h

gon1332 · 2019-07-08T08:57:02Z

That's the first time I see this API, so I don't think I can comment more, until I actually try to port my camera module to it.
So, it looks good to me for now.

pizi-nordic

For me there are few blank areas in this API definition. I will try to highlight them using he following example:

[ Camera ] => [ Software scaler/filter/color space converter ] => [ HW Encoder ] => Storage

Assumptions:

All components in the brackets could (and should) be represented using this API.

Questions:

Each of the device provides its own video frame allocator. Am I supposed to reallocate frame and copy its contents manually between each processing step or I can pass the frame directly?
I assume that I enqueue uncompressed frame to the HW encoder and dequeue the compressed one. Since most of the video compression algorithms do not support compression in place, the HW encoder will allocate the output frame. Who is responsible for freeing the input frame? If user of the API, how he/she could know when the frame is no longer used. If the HW encoder driver, how it could know which driver is owner of the frame and which version of video_api_release_frame_t should be called?
Some of the video encoders need some kind of out of band data describing properties of the compressed stream (the H264 is good example). How we are supposed to retrieve this data from the encoder and pass it to the decoder? Will we have concept of the stream in the API?

include/drivers/video.h

pizi-nordic · 2019-07-09T13:36:50Z

include/drivers/video.h

Is this API valid for YUV data with color subsampling?

carlescufi · 2019-10-15T16:11:10Z

From the API meeting:

@jfischer-phytec-iot can take a look post-ELCE (November timeframe)
@MaureenHelm will take another look
An option is to merge this by marking it as experimental in Kconfig and Doxygen
Deadline of 22nd of October to merge as experimental if there is no further activity

MaureenHelm

Have you tried sending video to the LCD display?

MaureenHelm · 2019-10-16T18:56:23Z

dts/bindings/video/aptina,mt9m114.yaml

This needs to be updated

thanks done

MaureenHelm · 2019-10-16T18:56:43Z

dts/arm/nxp/nxp_rt.dtsi

Stray change?

yes, fixed now

This generic video API can be used to capture/output video frames. Once a video buffer is enqueued to a video device endpoint, device owns the buffer and can process to capture (camera), output (disk, display), convert (hw encoder)... User can then call dequeue to retrieve the processed buffer (video driver ensure cache coherency). Once dequeued, video buffer is owned by user (e.g. for frame processing, display buffer update, write to media, etc...). For each video-buffer, user needs allocate the associated frame buffer via video_buffer_alloc. Buffer format is defined by video device endpoint configuration. Video device can be controlled (e.g. contrast, brightness, flip...) via controls. Signed-off-by: Loic Poulain <[email protected]>

Add support for CMOS Sensor Interface video driver. Signed-off-by: Loic Poulain <[email protected]>

Add CSI node to generic nxp rt dtsi. Add corresponding dts binding. Add CSI capability for rt MCUs. Signed-off-by: Loic Poulain <[email protected]>

This enables CSI node, and configures pinmux when driver is enabled. Signed-off-by: Loic Poulain <[email protected]>

Add myself to owner of drivers/video. Signed-off-by: Loic Poulain <[email protected]>

MT9M114 is a CMOS digital image sensor. Implement video interface. Only VGA (640x480) supported for now. Signed-off-by: Loic Poulain <[email protected]>

Sensor can be connected to the 24pin camera connector. Signed-off-by: Loic Poulain <[email protected]>

This is a virtual device generating video pattern for testing purpose. It supports colobar pattern for now. Signed-off-by: Loic Poulain <[email protected]>

Simple video sample getting frames from video capture device. Tested with mimxrt1064_evk and MT9M114 sensor. Signed-off-by: Loic Poulain <[email protected]>

This samples captures frames from video capture device (in any format), and sends them to its TCP client. Tested with: - mimxrt1064 + MT9M114 video sensor. - Gstreamer 1.8.3 running on host Signed-off-by: Loic Poulain <[email protected]>

Keep flat video driver directory for now. Signed-off-by: Loic Poulain <[email protected]>

loicpoulain · 2019-10-17T06:20:57Z

Have you tried sending video to the LCD display?

No, I do not have LCD for the iMX RT, but AFAIR, @JunYangNXP knows quite well how to do this, not sure if he has time, but it would be a good end-to-end sample to add.

MaureenHelm · 2019-10-17T18:37:25Z

Have you tried sending video to the LCD display?

No, I do not have LCD for the iMX RT, but AFAIR, @JunYangNXP knows quite well how to do this, not sure if he has time, but it would be a good end-to-end sample to add.

I'm wondering about the interaction between this video api and the existing display api in include/drivers/display.h. Will they work together? Would it make sense to build a video output driver on top of the display api?

There is a display driver for iMX RT in drivers/display/display_mcux_elcdif.c

loicpoulain · 2019-10-18T12:37:29Z

Have you tried sending video to the LCD display?

No, I do not have LCD for the iMX RT, but AFAIR, @JunYangNXP knows quite well how to do this, not sure if he has time, but it would be a good end-to-end sample to add.

I'm wondering about the interaction between this video api and the existing display api in include/drivers/display.h. Will they work together? Would it make sense to build a video output driver on top of the display api?

It's certainly do-able yes, the APIs are quite similar and a glue driver is possible. I'll also investigate if it would be worth to convert display API to video one. But it's a long term task. There is steps in between like making both API including same pixformat, fourcc definitions, etc...

There is a display driver for iMX RT in drivers/display/display_mcux_elcdif.c

Ok, I'm going to try getting one.

carlescufi · 2019-10-22T16:06:03Z

Have you tried sending video to the LCD display?

No, I do not have LCD for the iMX RT, but AFAIR, @JunYangNXP knows quite well how to do this, not sure if he has time, but it would be a good end-to-end sample to add.

I'm wondering about the interaction between this video api and the existing display api in include/drivers/display.h. Will they work together? Would it make sense to build a video output driver on top of the display api?

There is a display driver for iMX RT in drivers/display/display_mcux_elcdif.c

@vanwinkeljan could you please take a look at the interaction between this new API and the existing display API?

vanwinkeljan · 2019-10-22T18:11:54Z

@vanwinkeljan could you please take a look at the interaction between this new API and the existing display API?

@carlescufi / @MaureenHelm It looks pretty straightforward to build a glue logic video endpoint on top of the existing display API. But I think there is still a use case to support both APIs; to me it looks like the video API is intended to operate on full screen size and update the complete display at once whereas the display API allows to do partial updates, this feature could be of importance to memory constrained devices. Of Course the video API could be extended to support this but I would not block this PR for that.

include/drivers/video.h

loicpoulain · 2019-10-23T08:09:17Z

.

@vanwinkeljan could you please take a look at the interaction between this new API and the existing display API?

@carlescufi / @MaureenHelm It looks pretty straightforward to build a glue logic video endpoint on top of the existing display API. But I think there is still a use case to support both APIs; to me it looks like the video API is intended to operate on full screen size and update the complete display at once whereas the display API allows to do partial updates, this feature could be of importance to memory constrained devices. Of Course the video API could be extended to support this but I would not block this PR for that.

Thanks for your review, indeed it's easy to feed the display API with video API buffer content. Let's see later if there is a need and a way to extend video API to include display/graphics capabilities. For now we keep both and I'll submit a sample using Video API for capture + colorspace conversion and Display API for rendering.

carlescufi · 2019-10-23T12:04:04Z

@vanwinkeljan could you please take a look at the interaction between this new API and the existing display API?

@carlescufi / @MaureenHelm It looks pretty straightforward to build a glue logic video endpoint on top of the existing display API. But I think there is still a use case to support both APIs; to me it looks like the video API is intended to operate on full screen size and update the complete display at once whereas the display API allows to do partial updates, this feature could be of importance to memory constrained devices. Of Course the video API could be extended to support this but I would not block this PR for that.

Thank you @vanwinkeljan for the review.
@MaureenHelm would you mind approving the PR or removing your NACK?

stale, not involved in the project anymore

loicpoulain requested review from MaureenHelm, carlescufi, galak and nashif as code owners July 1, 2019 12:55

loicpoulain mentioned this pull request Jul 1, 2019

[WIP] Introduce Video 4 Zephyr API #14666

Closed

loicpoulain added area: API Changes to public APIs area: Video Video subsystem platform: NXP NXP area: camera area: Samples Samples labels Jul 1, 2019

loicpoulain changed the title ~~Video~~ Video API and driver Jul 1, 2019

loicpoulain requested review from microbuilder and pizi-nordic July 1, 2019 13:01

zephyrbot added area: Devicetree area: Boards EXT Has change or related to ext/ (obsolete) labels Jul 1, 2019

loicpoulain force-pushed the video branch 2 times, most recently from db3ab12 to 3720757 Compare July 1, 2019 15:34

loicpoulain changed the title ~~Video API and driver~~ Video API, Driver and Sample Jul 1, 2019

loicpoulain force-pushed the video branch 3 times, most recently from 10f0fdd to b991ac7 Compare July 5, 2019 16:22

loicpoulain changed the title ~~Video API, Driver and Sample~~ Video API, Drivers and Sample Jul 5, 2019

gon1332 reviewed Jul 8, 2019

View reviewed changes

include/drivers/video.h Outdated Show resolved Hide resolved

pizi-nordic reviewed Jul 9, 2019

View reviewed changes

loicpoulain requested a review from dbkinder July 10, 2019 18:32

MaureenHelm requested changes Oct 16, 2019

View reviewed changes

loicpoulain force-pushed the video branch from b8e531f to adf2e66 Compare October 17, 2019 06:12

Loic Poulain added 11 commits October 17, 2019 08:12

drivers: video: Add MCUX CSI video driver

dd617cd

Add support for CMOS Sensor Interface video driver. Signed-off-by: Loic Poulain <[email protected]>

soc: nxp: rt: Add CSI video support

aa4fcb5

Add CSI node to generic nxp rt dtsi. Add corresponding dts binding. Add CSI capability for rt MCUs. Signed-off-by: Loic Poulain <[email protected]>

boards: arm: nxp: mimxrt1064_evk: Enable CSI video support

4f52f3d

This enables CSI node, and configures pinmux when driver is enabled. Signed-off-by: Loic Poulain <[email protected]>

CODEOWNERS: Add drivers/video path

5fdf0ac

Add myself to owner of drivers/video. Signed-off-by: Loic Poulain <[email protected]>

drivers: video: Add Aptina MT9M114 driver

30a3e23

MT9M114 is a CMOS digital image sensor. Implement video interface. Only VGA (640x480) supported for now. Signed-off-by: Loic Poulain <[email protected]>

boards: mimxrt1064_evk: Add MT9M114 digital image sensor node

541dc70

Sensor can be connected to the 24pin camera connector. Signed-off-by: Loic Poulain <[email protected]>

drivers: video: Add software video pattern generator

981d56f

This is a virtual device generating video pattern for testing purpose. It supports colobar pattern for now. Signed-off-by: Loic Poulain <[email protected]>

samples: video: Add video capture sample

2b79c42

Simple video sample getting frames from video capture device. Tested with mimxrt1064_evk and MT9M114 sensor. Signed-off-by: Loic Poulain <[email protected]>

samples: video: Add TCP server sink sample

391829e

This samples captures frames from video capture device (in any format), and sends them to its TCP client. Tested with: - mimxrt1064 + MT9M114 video sensor. - Gstreamer 1.8.3 running on host Signed-off-by: Loic Poulain <[email protected]>

drivers: video: Remove dedicated sensor dir

adf2e66

Keep flat video driver directory for now. Signed-off-by: Loic Poulain <[email protected]>

wentongwu self-requested a review October 17, 2019 06:24

MaureenHelm requested a review from vanwinkeljan October 17, 2019 18:37

vanwinkeljan reviewed Oct 22, 2019

View reviewed changes

include/drivers/video.h Show resolved Hide resolved

MaureenHelm approved these changes Oct 23, 2019

View reviewed changes

MaureenHelm merged commit 14a0def into zephyrproject-rtos:master Oct 25, 2019

nashif added the Release Notes To be mentioned in the release notes label Nov 6, 2019

josuah mentioned this pull request May 17, 2024

drivers: video: API enhancement #72959

Open

josuah mentioned this pull request Jul 27, 2024

drivers: video: image sensor list (from Linux, others) #76378

Open

Video 4 Zephyr - Video API, Drivers and Samples #17194

Video 4 Zephyr - Video API, Drivers and Samples #17194

Uh oh!

Conversation

loicpoulain commented Jul 1, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zephyrbot commented Jul 1, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

checkpatch (informational only, not a failure)

Uh oh!

loicpoulain commented Jul 5, 2019

Uh oh!

microbuilder commented Jul 5, 2019

Uh oh!

Uh oh!

gon1332 commented Jul 8, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pizi-nordic left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pizi-nordic Jul 9, 2019

Choose a reason for hiding this comment

Uh oh!

carlescufi commented Oct 15, 2019

Uh oh!

MaureenHelm left a comment

Choose a reason for hiding this comment

Uh oh!

MaureenHelm Oct 16, 2019

Choose a reason for hiding this comment

Uh oh!

loicpoulain Oct 17, 2019

Choose a reason for hiding this comment

Uh oh!

MaureenHelm Oct 16, 2019

Choose a reason for hiding this comment

Uh oh!

loicpoulain Oct 17, 2019

Choose a reason for hiding this comment

Uh oh!

loicpoulain commented Oct 17, 2019

Uh oh!

MaureenHelm commented Oct 17, 2019

Uh oh!

loicpoulain commented Oct 18, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

carlescufi commented Oct 22, 2019

Uh oh!

vanwinkeljan commented Oct 22, 2019

Uh oh!

Uh oh!

loicpoulain commented Oct 23, 2019

Uh oh!

carlescufi commented Oct 23, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

13 participants

loicpoulain commented Jul 1, 2019 •

edited

Loading

zephyrbot commented Jul 1, 2019 •

edited

Loading

gon1332 commented Jul 8, 2019 •

edited

Loading

pizi-nordic left a comment •

edited

Loading

loicpoulain commented Oct 18, 2019 •

edited

Loading