staging: vc04_services: isp: Set the YUV420/YVU420 format stride to 64 bytes #4419
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
naushir
changed the title
|
@davidplowman and @6by9 This seems to fix the issue at raspberrypi/rpicam-apps#42 |
|
Wording of the commit text only The bcm2835 ISP requires the base address of all input/output planes to have 32 byte alignment.
That's already enforced, so "as the luma height will always be an even number of lines". |
…4 bytes The bcm2835 ISP requires the base address of all input/output planes to have 32 byte alignment. Using a Y stride of 32 bytes would not guarantee that the V plane would fulfil this, e.g. a height of 650 lines would mean the V plane buffer is not 32 byte aligned for YUV420 formats. Having a Y stride of 64 bytes would ensure both U and V planes have a 32 byte alignment, as the luma height will always be an even number of lines. Signed-off-by: Naushir Patuck <[email protected]>
|
Done! |
popcornmix
added a commit
to raspberrypi/firmware
that referenced
this pull request
Jul 2, 2021
…mplete See: raspberrypi/linux#4421 kernel: Fix CPU hang when using CEC while HDMI output is disabled See: raspberrypi/linux#4418 kernel: drm/vc4: hdmi: Add missing clk_disable_unprepare on error path See: raspberrypi/linux#4426 kernel: staging: vc04_services: isp: Set the YUV420/YVU420 format stride to 64 bytes See: raspberrypi/linux#4419 kernel: media: i2c: imx477: Add support for imx378 as a compatible sensor See: raspberrypi/linux#4420
popcornmix
added a commit
to Hexxeh/rpi-firmware
that referenced
this pull request
Jul 2, 2021
…mplete See: raspberrypi/linux#4421 kernel: Fix CPU hang when using CEC while HDMI output is disabled See: raspberrypi/linux#4418 kernel: drm/vc4: hdmi: Add missing clk_disable_unprepare on error path See: raspberrypi/linux#4426 kernel: staging: vc04_services: isp: Set the YUV420/YVU420 format stride to 64 bytes See: raspberrypi/linux#4419 kernel: media: i2c: imx477: Add support for imx378 as a compatible sensor See: raspberrypi/linux#4420
6by9
added a commit
to 6by9/linux
that referenced
this pull request
Nov 29, 2021
Matching raspberrypi#4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]>
pelwell
pushed a commit
that referenced
this pull request
Nov 29, 2021
Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]>
pelwell
pushed a commit
that referenced
this pull request
Nov 30, 2021
Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]>
pelwell
pushed a commit
that referenced
this pull request
Dec 2, 2021
Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Dec 7, 2021
Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Dec 7, 2021
Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Dec 16, 2021
Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Dec 16, 2021
Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Dec 17, 2021
Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Dec 22, 2021
Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Dec 22, 2021
Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Dec 31, 2021
Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Dec 31, 2021
Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Jan 5, 2022
Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Jan 5, 2022
Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Jan 10, 2022
Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Jan 11, 2022
Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Jan 11, 2022
Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Sep 16, 2024
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <[email protected]> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <[email protected]> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <[email protected]> Signed-off-by: Dom Cobley <[email protected]> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <[email protected]> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <[email protected]> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <[email protected]> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <[email protected]> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <[email protected]> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <[email protected]> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <[email protected]> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Oct 2, 2024
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <[email protected]> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <[email protected]> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <[email protected]> Signed-off-by: Dom Cobley <[email protected]> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <[email protected]> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <[email protected]> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <[email protected]> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <[email protected]> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <[email protected]> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <[email protected]> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <[email protected]> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Oct 2, 2024
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <[email protected]> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <[email protected]> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <[email protected]> Signed-off-by: Dom Cobley <[email protected]> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <[email protected]> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <[email protected]> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <[email protected]> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <[email protected]> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <[email protected]> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <[email protected]> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <[email protected]> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Oct 7, 2024
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <[email protected]> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <[email protected]> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <[email protected]> Signed-off-by: Dom Cobley <[email protected]> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <[email protected]> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <[email protected]> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <[email protected]> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <[email protected]> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <[email protected]> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <[email protected]> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <[email protected]> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <[email protected]> Signed-off-by: Kieran Bingham <[email protected]> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <[email protected]> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <[email protected]> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Oct 10, 2024
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <[email protected]> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <[email protected]> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <[email protected]> Signed-off-by: Dom Cobley <[email protected]> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <[email protected]> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <[email protected]> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <[email protected]> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <[email protected]> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <[email protected]> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <[email protected]> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <[email protected]> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Oct 10, 2024
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <[email protected]> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <[email protected]> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <[email protected]> Signed-off-by: Dom Cobley <[email protected]> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <[email protected]> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <[email protected]> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <[email protected]> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <[email protected]> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <[email protected]> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <[email protected]> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <[email protected]> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Oct 14, 2024
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <[email protected]> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <[email protected]> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <[email protected]> Signed-off-by: Dom Cobley <[email protected]> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <[email protected]> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <[email protected]> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <[email protected]> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <[email protected]> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <[email protected]> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <[email protected]> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <[email protected]> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <[email protected]> Signed-off-by: Kieran Bingham <[email protected]> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <[email protected]> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <[email protected]> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Oct 14, 2024
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <[email protected]> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <[email protected]> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <[email protected]> Signed-off-by: Dom Cobley <[email protected]> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <[email protected]> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <[email protected]> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <[email protected]> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <[email protected]> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <[email protected]> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <[email protected]> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <[email protected]> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <[email protected]> Signed-off-by: Kieran Bingham <[email protected]> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <[email protected]> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <[email protected]> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Oct 17, 2024
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <[email protected]> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <[email protected]> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <[email protected]> Signed-off-by: Dom Cobley <[email protected]> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <[email protected]> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <[email protected]> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <[email protected]> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <[email protected]> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <[email protected]> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <[email protected]> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <[email protected]> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Oct 21, 2024
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <[email protected]> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <[email protected]> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <[email protected]> Signed-off-by: Dom Cobley <[email protected]> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <[email protected]> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <[email protected]> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <[email protected]> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <[email protected]> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <[email protected]> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <[email protected]> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <[email protected]> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <[email protected]> Signed-off-by: Kieran Bingham <[email protected]> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <[email protected]> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <[email protected]> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Oct 23, 2024
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <[email protected]> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <[email protected]> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <[email protected]> Signed-off-by: Dom Cobley <[email protected]> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <[email protected]> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <[email protected]> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <[email protected]> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <[email protected]> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <[email protected]> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <[email protected]> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <[email protected]> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Oct 28, 2024
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <[email protected]> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <[email protected]> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <[email protected]> Signed-off-by: Dom Cobley <[email protected]> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <[email protected]> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <[email protected]> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <[email protected]> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <[email protected]> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <[email protected]> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <[email protected]> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <[email protected]> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <[email protected]> Signed-off-by: Kieran Bingham <[email protected]> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <[email protected]> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <[email protected]> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Nov 1, 2024
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <[email protected]> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <[email protected]> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <[email protected]> Signed-off-by: Dom Cobley <[email protected]> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <[email protected]> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <[email protected]> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <[email protected]> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <[email protected]> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <[email protected]> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <[email protected]> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <[email protected]> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Nov 5, 2024
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <[email protected]> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <[email protected]> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <[email protected]> Signed-off-by: Dom Cobley <[email protected]> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <[email protected]> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <[email protected]> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <[email protected]> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <[email protected]> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <[email protected]> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <[email protected]> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <[email protected]> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <[email protected]> Signed-off-by: Kieran Bingham <[email protected]> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <[email protected]> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <[email protected]> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Nov 6, 2024
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <[email protected]> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <[email protected]> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <[email protected]> Signed-off-by: Dom Cobley <[email protected]> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <[email protected]> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <[email protected]> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <[email protected]> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <[email protected]> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <[email protected]> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <[email protected]> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <[email protected]> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <[email protected]> Signed-off-by: Kieran Bingham <[email protected]> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <[email protected]> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <[email protected]> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Nov 8, 2024
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <[email protected]> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <[email protected]> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <[email protected]> Signed-off-by: Dom Cobley <[email protected]> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <[email protected]> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <[email protected]> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <[email protected]> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <[email protected]> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <[email protected]> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <[email protected]> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <[email protected]> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Nov 11, 2024
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <[email protected]> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <[email protected]> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <[email protected]> Signed-off-by: Dom Cobley <[email protected]> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <[email protected]> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <[email protected]> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <[email protected]> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <[email protected]> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <[email protected]> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <[email protected]> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <[email protected]> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <[email protected]> Signed-off-by: Kieran Bingham <[email protected]> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <[email protected]> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <[email protected]> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Nov 18, 2024
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <[email protected]> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <[email protected]> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <[email protected]> Signed-off-by: Dom Cobley <[email protected]> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <[email protected]> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <[email protected]> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <[email protected]> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <[email protected]> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <[email protected]> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <[email protected]> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <[email protected]> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <[email protected]> Signed-off-by: Kieran Bingham <[email protected]> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <[email protected]> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <[email protected]> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Nov 18, 2024
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <[email protected]> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <[email protected]> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <[email protected]> Signed-off-by: Dom Cobley <[email protected]> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <[email protected]> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <[email protected]> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <[email protected]> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <[email protected]> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <[email protected]> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <[email protected]> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <[email protected]> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <[email protected]> Signed-off-by: Kieran Bingham <[email protected]> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <[email protected]> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <[email protected]> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Nov 18, 2024
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <[email protected]> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <[email protected]> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <[email protected]> Signed-off-by: Dom Cobley <[email protected]> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <[email protected]> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <[email protected]> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <[email protected]> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <[email protected]> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <[email protected]> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <[email protected]> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <[email protected]> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Nov 25, 2024
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <[email protected]> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <[email protected]> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <[email protected]> Signed-off-by: Dom Cobley <[email protected]> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <[email protected]> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <[email protected]> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <[email protected]> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <[email protected]> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <[email protected]> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <[email protected]> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <[email protected]> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <[email protected]> Signed-off-by: Kieran Bingham <[email protected]> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <[email protected]> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <[email protected]> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Nov 25, 2024
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <[email protected]> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <[email protected]> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <[email protected]> Signed-off-by: Dom Cobley <[email protected]> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <[email protected]> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <[email protected]> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <[email protected]> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <[email protected]> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <[email protected]> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <[email protected]> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <[email protected]> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Dec 7, 2024
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <[email protected]> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <[email protected]> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <[email protected]> Signed-off-by: Dom Cobley <[email protected]> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <[email protected]> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <[email protected]> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <[email protected]> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <[email protected]> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <[email protected]> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <[email protected]> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <[email protected]> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <[email protected]> Signed-off-by: Kieran Bingham <[email protected]> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <[email protected]> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <[email protected]> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Dec 9, 2024
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <[email protected]> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <[email protected]> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <[email protected]> Signed-off-by: Dom Cobley <[email protected]> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <[email protected]> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <[email protected]> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <[email protected]> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <[email protected]> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <[email protected]> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <[email protected]> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <[email protected]> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <[email protected]> Signed-off-by: Kieran Bingham <[email protected]> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <[email protected]> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <[email protected]> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Dec 10, 2024
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <[email protected]> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <[email protected]> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <[email protected]> Signed-off-by: Dom Cobley <[email protected]> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <[email protected]> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <[email protected]> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <[email protected]> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <[email protected]> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <[email protected]> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <[email protected]> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <[email protected]> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <[email protected]> Signed-off-by: Kieran Bingham <[email protected]> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <[email protected]> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <[email protected]> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Dec 16, 2024
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <[email protected]> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <[email protected]> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <[email protected]> Signed-off-by: Dom Cobley <[email protected]> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <[email protected]> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <[email protected]> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <[email protected]> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <[email protected]> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <[email protected]> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <[email protected]> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <[email protected]> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <[email protected]> Signed-off-by: Kieran Bingham <[email protected]> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <[email protected]> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <[email protected]> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Dec 16, 2024
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <[email protected]> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <[email protected]> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <[email protected]> Signed-off-by: Dom Cobley <[email protected]> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <[email protected]> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <[email protected]> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <[email protected]> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <[email protected]> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <[email protected]> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <[email protected]> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <[email protected]> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <[email protected]> Signed-off-by: Kieran Bingham <[email protected]> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <[email protected]> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <[email protected]> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Dec 20, 2024
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <[email protected]> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <[email protected]> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <[email protected]> Signed-off-by: Dom Cobley <[email protected]> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <[email protected]> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <[email protected]> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <[email protected]> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <[email protected]> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <[email protected]> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <[email protected]> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <[email protected]> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <[email protected]> Signed-off-by: Kieran Bingham <[email protected]> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <[email protected]> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <[email protected]> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Jan 2, 2025
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <[email protected]> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <[email protected]> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <[email protected]> Signed-off-by: Dom Cobley <[email protected]> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <[email protected]> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <[email protected]> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <[email protected]> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <[email protected]> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <[email protected]> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <[email protected]> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <[email protected]> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <[email protected]> Signed-off-by: Kieran Bingham <[email protected]> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <[email protected]> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <[email protected]> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this pull request
Jan 2, 2025
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <[email protected]> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <[email protected]> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <[email protected]> Signed-off-by: Dom Cobley <[email protected]> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <[email protected]> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <[email protected]> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <[email protected]> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <[email protected]> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <[email protected]> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <[email protected]> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <[email protected]> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <[email protected]> Signed-off-by: Kieran Bingham <[email protected]> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <[email protected]> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <[email protected]> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The bcm2835 ISP requires all input/output planes to have 32 byte alignment.
Using a Y stride of 32 bytes would not guarantee that the V plane would fulfil
this, e.g. a height of 650 lines would mean the V plane buffer is not 32 byte
aligned for YUV420 formats.
Having a Y stride of 64 bytes would ensure both U and V planes have a 32 byte
alignment, provided the height is an even number of lines.
Signed-off-by: Naushir Patuck [email protected]