Use both cores?

[ericfont]

A quick skim of the code suggests it only uses one core, though many ESP32 modules have two cores. Frame rate could be improved if used two cores for encoding, such as by breaking each frame into two halves and letting the extra core if available encode one half.

[ericfont]

I'm going to take a stab at this. The discrete cosine transform stage of JPEG compression is easily parallelized because the discrete cosine transform portion is done at the macroblock granularity (which is a square pixel region of 16×16 luma (Y) samples and 8×8 chroma samples).

[easytarget]

Thankyou for this; it's appreciated, and certainly something that is in my roadmap.

My plan (such as it is) is to combine multi-core use with multi-client streaming,
See #51 (comment) for instance

If working on this; a couple of caveats: 😉
Basically; whatever we do needs to be a RTOS based solution, core switching hacks are not appropriate for this.
Also, we need to consider the ESP32-C3 (eg a single cored device) in how it is implemented, I want to support devices based on this core too since I'm sure they will appear over time. Using RTOS and providing a new # define NUMCORES in the pin config should allow for this.

I'm heavily involved in other work at present and poor at task switching, so It will be winter time before I can work on this project with any urgency again. I'm going to close this issue as a duplicate of #51 for now; if you want to get started on RTOS (or maybe even multiclient streaming) please work against that issue 😃 and have fun, learn cool stuff, etc..

[ericfont]

turns out the OV2640 already has a built-in comporession engine with hardware for discrete cosine transform (https://www.uctronics.com/download/OV2640_DS.pdf):

And it has JPEG output capability:

And that built-in encoder is apparently already being used when set PIXFORMAT to JPEG here https://github.com/easytarget/esp32-cam-webserver/blob/master/esp32-cam-webserver.ino#L493

And according to some microsecond-accurate timing measurements breakdown of the stream_handler main loop (excluding the Serial.printf debug calls ericfont@a504027
SVGA_usecs_breakdown.txt
), the software converter function frame2jpg is never called because the camera driver returns a framebuffer in PIXFORMAT_JPEG, so it seems the hardware JPEG compression is used already. And I notice the esp_camera_fb_get may sometimes just takes a few microseconds sometimes but other times may take milliseconds and sometimes takes over 10 milliseconds...(I think that is because sometimes the framebuffer data is already pulled out of the camera module and ready to go waiting in the PSRAM). But regardless what takes the most time by far of every frame is the httpd_resp_send_chunk calls, particularly the thrid call which is what contains the jpg data.

So at this point I am almost certain that network communication is the bottleneck, particularly if want to do multi-client streaming. It might even be necessary to go for hardware wired Ethernet to do multi-client reliably.

[easytarget]

So at this point I am almost certain that network communication is the bottleneck, particularly if want to do multi-client streaming. It might even be necessary to go for hardware wired Ethernet to do multi-client reliably.

I concur.

If you are using lower resolution modes then multiple streams will be more possible. But I've learned not to expect too much from ESP32 series 1 Wifi, it works well enough for it's price-point, but has distinct limits.

Note that the esp's internal transcoder mechanism was (I believe, I's some time since I actually look at this code) used when in Face Recognition mode to provide the bitmaps used by the detector code, and the surface drawn on for the feedback boxes and text. Since I have stripped the FR stuff out for V4 ]we now only stream the mjpeg as created by the camera module itself.