Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
staging: vc04_services: Add a V4L2 M2M codec driver
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <[email protected]> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <[email protected]> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <[email protected]> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <[email protected]> Signed-off-by: Dom Cobley <[email protected]> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <[email protected]> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <[email protected]> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <[email protected]> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <[email protected]> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <[email protected]> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <[email protected]> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <[email protected]> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <[email protected]> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <[email protected]> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <[email protected]> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <[email protected]> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <[email protected]> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <[email protected]>
- Loading branch information
1 parent
0cb2d23
commit 3e84ce1