can't record full-frame in mjpeg on pi zero v1.3 with camera v2.1 #315
Comments
|
It's https://github.com/waveform80/picamera/blob/master/picamera/mmalobj.py#L760 that is going to be the problem. The default buffer size off video_encode is 64kB. The MJPEG codec is pretty shonky and will stall badly if it can't get rid of the output data. Disable whilst it still has output data queued and it is game over :-( Ideally for 1080P we want 768kB or so to be able to take an entire frame in one buffer (the same optimisation would be useful on H264), and preferably a few more buffers to keep data flowing cleanly. |
|
(Just for the record, 1080p h.264 actually works fine for me on the pi zero, as does the 2x2 binning at full frame (mode 4). At least... so far... It's just the 3280x2464 mjpeg at 15fps that I can't make happen. Maybe that's all obvious but just wanted to be clear. I'm editing the issue to clarify as well.) |
|
H264 doesn't stall as the codec architecture is different to work more efficiently with the hardware. |
|
Well, the project is obviously in capable hands and I will await whatever developments with interest and appreciation. :-) |
|
Hmm, picamera should have a workaround for the MJPEG buffer size issue ... which I thought was in Ah, here it is, down in
Hmm, at the moment the PiVideoEncoder class does disable MMAL_PARAMETER_CAPTURE on the camera video port, but I don't think it waits for EOS. The callback does check for EOS and terminates if it sees it, but at the moment the main thread disables capture then immediately disables the output port. I should try tweaking that to see what happens if we just wait for EOS (I vaguely recall trying this at some point in the past and having some issue with it, but I can't recall what it was - oh well, worth having a play!) |
|
Thanks @waveform80 - I knew we'd had the discussion over buffer size, but couldn't see it. |
|
Hey -- was just curious if there was any movement or interest on this? No pressure, just curious. :-) It'd be cool to have full-res MJPEG. |
|
Admittedly I'm on a Pi2, but using That would imply that gpu_mem needs to be set to at least 176MB, and I'd suggest playing safe at 192MB as PiCamera allocates memory slightly differently to raspivid. Anyone fancy giving that a try with PiCamera? |
|
Thanks! -- just for the record, I tried a variety of memory sizes and via Python and raspivid, and nothing worked for me (on the pi zero) at the high resolution. Thread here: https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=156458&p=1021146 ...but I'll make sure i'm on the latest version and will try your exact raspivid command to see if it works. |
|
Hmmm, doesn't seem to be working for me on the pi zero (your raspivid llne.) It seems to work at first, but the file seemed to only have one actual frame in it (in VLC anyway), as reported above. And my python test code (see first post in this issue) still hangs. After I ran my python test, it hangs, I hit ctrl-Z and kill %1, and I did "vcdbg reloc". I'll try to attach the result, in case it's useful (github is giving me trouble). Let me know if there's anything I can do to help. |
|
I can't attach a .txt file for some reason, so here's the debug output: |
|
VLC is broken when it comes to MJPEG - raspberrypi/userland#345. It sees the JFIF header in our bitstream and incorrectly identifies it as a single JPEG rather than MJPEG. Just checking with the file I captured yesterday and there is something off. It looks like if the bitrate is too low then the encode needs to try and drop the Q factor below the minimum and things fail. NB |
|
Thanks 6x9 -- that vlc fix works great. Some interesting test results here... Test 1: The second raspivid command works to record the full-res clips. (But i'm curious why you specified 3264x2448 instead of 3280x2464 which I understand to be mode 2?) But the posterization and blocking is so rough on the full-res files that if I simply scale a full-res h.264 video (1640x1232 with 2x2 binning, and at only 15mbps) to the same size, the image quality is roughly equivalent. Test 2: the python code still hangs -- 'top' shows no activity, and I have to ctrl-Z / kill %1 to get out of it (and the camera is borked for further use). A partial mjpg file is recorded with only a few frames in it, but they are full resolution and look great. This is what I'm currently doing: Here are some sample images extracted from the videos:
It's tantalizingly close! :-) |
|
Resolution is just a number - the system will crop/scale almost anything to anything. 3264x2448 is the normal "standard" for 4:3 8MPix. Just doing the maths, even at 25Mbit/s 15fps, you're looking at 203kB per frame for an 8MPix (12MB) image. That's a pretty high compression factor hence compression artefacts. MJPEG is a hopelessly inefficient codec in comparison to H264. |
|
Thanks -- that makes sense, but the part I don't understand is why the python code results above look so much better than the raspivid results (i.e. little/no compression artifacts, no posterization) despite being assigned the same bitrate, and that the python code also hangs and borks the camera... ? I had given up on the mjpg, assuming the bitrate was the cause, but then I saw the (partial) results of the python code video and it looked so beautiful and made me think that it was possible to make it work after all... is it perhaps recording at too-high a bitrate (ignoring the specified 25mbps) and that's causing it to hang/fail? |
|
The codec always starts at quality 7 and then nudges the quality up/down by 1 per frame if the resultant image was more than 12% outside the target range, therefore the first frames may well end up well beyond bitrate budget but better quality. You can turn on some GPU side logging if you're that interested. Decoding it: So pretty close to the target. There is a known lockup problem with the MJPEG codec which I think I've just hit and got logging lines of |
|
Thanks -- crazy theory: perhaps the bitrate setting with mjpeg is being interpreted (through the python API) as a "per-frame" bit target instead of mbps? Here's the debug output (from running my python code above, which sets fps 15 and bitrate 25000000): It looks like it is indeed hitting the lockup problem you describe, after five frames. But the target is listed as "3125000"... at 25mbps and 15fps (set via python code) I would have expected 208333 as your debug output has, but it's ~exactly 15X that value, which happens to be the fps... so I ran another test: I used my python code to set the bitrate to 1666666 (25000000 / 15). This time the log messages show 208333 as the target and the python script works just fine -- no hang: |
|
Er, ouch! I'm not sure what is going on there. One for @waveform80 I think, if only for guidance. Technically the "mjpg_enc_frame3 blocked" message isn't actually the lockup, but is indicative that the codec is having to stall due to thinking that it has insufficient space in the output FIFO. |
|
Here's the DEBUG=True output in case it's useful (python code otherwise as shown above, and I ran it in python 2.7.9 instead of my normal 3.4.2 because the other bug said that debugging didn't work in python 3): As an aside: is the 25mbps limit a non-negotiable hardware limitation? |
|
Nothing terribly useful in that debug output, although I am a little surprised to see "1x65536" on the output of video_encode - https://github.com/waveform80/picamera/blob/master/picamera/mmalobj.py#L832 should bump it up to 512kB, but perhaps that happens later on as it's part of the enable call. 25MBit/s isn't a real hardware limit, although it may be a practical one from shovelling data around the place. The JPEG block is expected to achieve about 160MPix/s, but I don't recall a strict limit on the quality setting on the output side. |
|
Well, let me hereby register my humble request that we clueless users be allowed to push the mbps as high as hardware/software/codec standards will permit. :-) Meaning, if a 25Mbps limit for mjpg isn't meaningful, it'd be cool to have that limit lifted. For security cams, it is, of course, all about the clarity of the image, so if I can push a little more clarity out of the camera with a higher bitrate and mjpg (once it gets sorted out), that'd be awesome! |
Hmmm, that's interesting. Bitrate is set for H.264 and MJPEG by the same chunk of code, i.e. there's no special casing for H.264 or MJPEG (the bitrate setter is down in mmalobj and is quite straight-forward). I'll add bitrate to the debug output and see what happens...
Yup - in the absence of any particular knowledge about the JPEG block (or the H264) block I just copied the limit for H264 to MJPEG. If no limit exists I can certainly remove that (although it looks like excessively high values may have something to do with the lockup from the observations above). |
Ah, I've just moved where the debug print happens to fix that - it's pretty misleading otherwise. So, here's the output from a quick session with the new debug stuff printing bitrate too: >>> from picamera import *
>>> camera = PiCamera(resolution='1280x720', framerate=30)
>>> PiEncoder.DEBUG = True
>>> camera.start_recording('foo.mjpg', bitrate=1000000)
vc.ril.camera [1] [0] vc.ril.video_splitter [1] [0] vc.ril.video_encode [0]
encoding OPQV-dual OPQV-single encoding OPQV-single OPQV-dual encoding MJPG
buf 10x128 --> 10x128 buf 10x128 --> 10x128 buf 1x524288
bitrate 0bps 0bps bitrate 0bps 0bps bitrate 1000000bps
frame 1280x720@30fps 1280x720@0fps frame 1280x720@0fps 1280x720@0fps frame 1280x720@0fps
>>> camera.stop_recording()So, the buffer size looks okay, and the bitrate has indeed been set to 1,000,000 bps. Hence we'd expect to see 1,000,000 / (8 * 30) ~= 4166 as the target byte size for a frame. However, the target size reported by the MJPEG debug output is 125,000 (or 1,000,000/8): So I'd guess MJPEG isn't taking into account the framerate when calculating the target size? I could certainly tweak this on the picamera side of things (fudge the bitrate setting so it divides by the framerate when setting bitrate for MJPEG), but I don't know if it's feasible to tweak this on the firmware side (which is probably preferable as it fixes it for all applications). I should definitely make picamera's buffer size fudge a bit more intelligent though (it should at least base it on the bitrate + some fudge factor which might alleviate the MJPEG lockup). |
The H264 limits come from H264 level spec. Actually the current limit is incorrect but only on a technicality (from https://en.wikipedia.org/wiki/H.264/MPEG-4_AVC#Levels): I will try to sort the MJPEG lockup at some point. It can probably be worked around more effectively by ensuring state transitions in an appropriate order, but fixing the firmware would be the better option. |
OK, I know where to look now, but it can't be right now (this annoying work thing). Racking my memory, H264 bitrate control can run based solely on the timestamps of the frames that come in should the framerate not be provided. The MJPEG encoder doesn't support that (although I guess it could), but that probably explains what we're seeing. It's probably firing a div0 exception on the GPU (bytes_per_frame = bitrate/framerate), but that probably doesn't get logged anywhere accessible. |
Ah, that's good to know! I'll expand the checks in picamera with the relevant bits from that table.
Hmm, just had a look at the debug printing code and I can't see how it'd be reporting anything other than what's actually set so I think that's actually the state of the pipeline. I'll see if I can figure out where the framerate info's getting lost (I don't think I set it to zero anywhere, but I could be wrong - or it might be happening implicitly somewhere).
Given the lack of framerate on the output port that certainly sounds likely. I'll try hacking that manually in a mo and see what happens... |
A little more on this - it seems committing the port format wipes the framerate info: >>> from picamera import PiCamera
>>> camera = PiCamera()
>>> camera._splitter.inputs[0]
<MMALVideoPort "vc.ril.video_splitter:in:0(OPQV)": format=MMAL_FOURCC('OPQV') buffers=10x128 frames=1920x1080@0fps>
>>> camera._camera.outputs[1]
<MMALVideoPort "vc.ril.camera:out:1(OPQV)": format=MMAL_FOURCC('OPQV') buffers=10x128 frames=1920x1080@30fps>
>>> camera._splitter.enabled = False
>>> camera._splitter.inputs[0].copy_from(camera._camera.outputs[1])
>>> camera._splitter.inputs[0]
<MMALVideoPort "vc.ril.video_splitter:in:0(OPQV)": format=MMAL_FOURCC('OPQV') buffers=10x128 frames=1920x1080@30fps>
>>> camera._splitter.inputs[0].commit()
>>> camera._splitter.inputs[0]
<MMALVideoPort "vc.ril.video_splitter:in:0(OPQV)": format=MMAL_FOURCC('OPQV') buffers=3x128 frames=1920x1080@0fps>
>>> camera._splitter.enabled = TrueI can't see anything in picamera's mmalobj layer which'd be causing that so I'll have a dig into userland and see if I can figure out what's going on! |
|
Fair cop. |
|
I have a patch that waveform80 has tested for me (thank you) and works. I'm pushing it to Pi Towers, so it'll be released in a firmware update relatively soon and should fix MJPEG getting the bitrate totally wrong due to having no frame rate information. Any updates to PiCamera to handle more H264 levels or altering MJPEG max bitrate are totally up to waveform80 to do if/when he can. |
|
Awesome, thanks! |
|
Firmware update so that video_splitter forwards on the framerate has now been released via rpi-update (Hexxeh/rpi-firmware@3ffa1c6) Normal warnings over rpi-update releases potentially being unstable. I don't know when the next firmware bump will be made to the Raspbian repo to make a fully official release. |
|
Have the same issue with full-size mjpeg on RPi1 with v1.3 camera. Made this as a workaround: class MyEncoder(picamera.PiVideoEncoder):
def start(self, output, motion_output=None):
self.frame = picamera.PiVideoFrame(
index=0,
frame_type=None,
frame_size=0,
video_size=0,
split_size=0,
timestamp=0,
complete=False,
)
self.output_port._port[0].buffer_size_recommended = 1024*1024*2
super(picamera.PiVideoEncoder, self).start(output)
class MyCamera(picamera.PiCamera):
def _get_video_encoder(self, *args, **kwargs):
return MyEncoder(self, *args, **kwargs) |
Hello -- My understanding is that full-frame (3280x2464) H.264 is not possible (without 2x2 binning). I thus tried mjpeg, but I can only record a few frames before the python module appears to lock up, requiring ctrl-Z and kill %1 and leaving the camera inaccessible until reboot.
At the pi forums, 6by9 suggested that this was a known thing, and suggested that I check for an existing issue, which I did not find: "I remember this now from discussion with waveform80 ... The buffer size is too low and ends up stalling the codec and then bad things happen. ... it'll need a rebuild of picamera to get it working vaguely reliably."
The code I'm testing with:
This results in an mjpeg being created of 2-3MB in size that does not play in vlc (first frame only -- not sure if vlc handles mjpeg anyhow), and when transcoded reveals itself to contain a few frames only. CPU continues to be used and the process never exits. I've tried with various gpu_mem settings to no avail.
Happy to test/tweak anything else that may be useful. It'd be great for security applications if the pi zero could record full frames at a reasonable rate (which to me is >=15 fps). Thanks!
The text was updated successfully, but these errors were encountered: