Issue with using detections within a loop for drawing detection results on canvas (flicker)

[mantzaris]

The detections work as expected with await faceapi.detectAllFaces(photo) in a loop that uses requestAnimationFrame(Draw_Descriptors) . This code here has commented out the part that requests landmarks and face descriptors. The rectangles drawn with the canvas context also draw fine when only the face detection bounding boxes are requested

async function Draw_Descriptors() {
        if( canvas.width > 0 && canvas.height > 0 ) {
            Take_Picture()
            detections = await faceapi.detectAllFaces(photo)//.withFaceLandmarks()//.withFaceDescriptors()
            faceapi.draw.drawDetections(canvas, detections, { withScore: true })
            ctx.rect(100,100,100,100) 
            ctx.strokeStyle = 'red'
            ctx.stroke()  
            for( const face of detections ) {
                const {x,y,width,height} = face.detection.box
                 ctx.rect(100,100,100,100)  //( x , y , width*1.4 , height*1.4 ) //shift more to the top left in proportion to its size
                ctx.strokeStyle = 'red'
                ctx.lineWidth = '8'
                ctx.stroke()  
            }            
        } 
    }

If the landmarks and face descriptors are requested as well via await faceapi.detectAllFaces(photo).withFaceLandmarks().withFaceDescriptors() The context drawing on the canvas produces a 'flicker' (interrupted drawing) and I am not sure why this occurs. It occurs with the draw detections or even the ctx.rect. Has this been encountered before? Is there something which I am missing on how to uses the functionality within the animation loop? Or is there a better more 'controlled' approach?

[vladmandic]

you've provided no details on your system, browser and backend - and those heavily matter.

but in general, studder is expected since tfjs model execution is a blocking call.
what you can do is have dual loops:

• first that stores results in a global object and calls itself with `setTimeout(xxx, 0)
• second that simply draws whatever is available in the global object and runs with requestAnimationFrame

unfortunately, faceapi does not support running in web workers since it heavily relies on dom.
if you want, you can try out my far newer library (faceapi internal architecture is quite old) which does support webworkers
https://github.com/vladmandic/human

[mantzaris]

(Chrome with NodeJS, within ElectronJS)

It works now without flicker using this refactor.

The approach you suggested on having dual loops with one of them controlled with a manual setTimeout( detectFaces, 1000/fps ) that places the results in a global object every time it refreshes so it can be available to the other loop which draws the canvas content updated with requestAnimationFrame, worked perfectly. I found a post which also says that this is the best way as well, https://stackoverflow.com/a/48412686/410975 .

I also have it so that it does only the face-box detection 3 times before face-box detection with landmarks and facedescriptors as an effort to reduce processing times. Do you think that the speed up will be substantial or not? I ran an experiment and saw that the
detection only faceapi.detectAllFaces(photo) from faceapi.detectAllFaces(photo).withFaceLandmarks().withFaceDescriptors() was not more faster by much on average. Each time around 130ms to 220ms. Are your findings similar? Is it that detectAllFaces(photo) is the most computationally intensive?

Does the human package offer the production of face descriptor arrays and bounding box detection? Are the neural network weights better?

[vladmandic]

manual setTimeout( detectFaces, 1000/fps )

This would be correct if processing took 0ms (loop performance is processing time + timeout time), but since your processing time is likely much larger you can decrease the delay. Just keep it non-zero so main thread has time to update other stuff.

I found a post which also says that this is the best way as well, https://stackoverflow.com/a/48412686/410975

Wasn't aware of it, but yes, its nicely written,

I also have it so that it does only the face-box detection 3 times before face-box detection with landmarks and facedescriptors as an effort to reduce processing times

Where does this idea come from?
To me, it would make sense to do it like that on startup as it would allow for quicker warmup, but not really during actual processing loop.

Is it that detectAllFaces(photo) is the most computationally intensive?

Yes. Its not total majority, but it does count for more than 50% of processing time. Primary reason is that processing must happen on a much larger sample to detect faces before they can be cropped and send to a second model for analysis.

Does the human package offer the production of face descriptor arrays and bounding box detection? Are the neural network weights better?

Yes and much more.

For example couple of big improvements would be:

• Frame change detection algorithm which allows for caching of detection results so if input has not changed much, it can re-use previous detection results (slightly adjusted) instead of re-running entire pipeline. Result is that for video input, you're skipping detection 90% of time.
• Results interpolation. When running in scenarios such as double-loop, it can provide algorithmic time-based interpolation so results that you use in draw loop is smoothened.
• Compatibility with web workers so you could run detection loop in a separate worker thus completely freeing up main thread.