-
Notifications
You must be signed in to change notification settings - Fork 124
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Virtual background or background blur #166
Comments
It's not planned for the immediate future, but it would be a nice feature to have. It should be implemented in the client, so that the background is blurred before the video is even sent to the server. I can see two ways to implement it:
Please note that for privacy reasons Galene bundles all of the libraries it uses, so any library needs to be freely licensed and sufficiently small to be bundled with Galene. |
Jitsi uses TFLite for this purpose: with the MediaPipe Meet Segmentation model: and paired with WebAssembly SIMD instructions: |
Perhaps something useful can be gleaned from this repository: https://github.com/minhkhue3214/new_virtualbackground |
I tried using the @mediapipe/selfie_segmentation library to solve this issue
I added a blur filter:
I figured out how to apply filters to canvas, but I don't know where to send JavaScript Solution API: Can you help me? |
If I understand correctly, then: But the image freezes after 8 seconds, so I guess I'm wrong somewhere. |
I'm unable to find the docs of the library you're using, so I'm not sure, but I suspect that a number of the functions you are calling are async, so you'd need to synchronise things (and deal with dropping frames when you're getting overtaken by the video). Also, you're creating a new instance of the library each time — I suggest you create it just once, in an |
I rewrote the code, but still haven't figured out where I should use asynchronous functions? I now get a static picture immediately when the video conference starts. The background blur effect is applied successfully, but the picture is static. Please help to understand, I have provided all the information about the library above in google github repository. The model description is also in the links above. I need your help very much.
|
If I understand correctly, selfieSegmentation.send() will be called asynchronously as part of the setInterval operation. As a result, after clearing the browser cache, the code I gave yesterday works, but problems are observed at any interaction with the video stream, for example, when the blackboardMode setting is enabled, then the video frame freezes and errors are thrown into the console:
Can you give any recommendations? Maybe you should look at the library, it's from Google Inc, not unknown developers? The model works well, the same model is used in Jitsi. I made a demo, everything works fine on it, but I can't integrate it with your code, I'm stuck, I need help. For now I'm loading the library externally by disabling Content-Security-Policy in the webserver.go module, but in the final version I see no problem including the library in the package with the rest of the code. Body Segmentation with MediaPipe and TensorFlow.js: |
The reason why Galene is awesome is because the author has done an excellent job at avoiding feature creep. To be honest, how much is this feature worth to you? Would you be willing to fund this feature with your cash or time? |
The policy in Galene is to minimise the amount of server-side features: the server needs to be simple, economical and rock solid. On the other hand, I have no objection to putting functionality in the client. I've had some feedback that indicated that background blur is important for some people, and I actually know people who have promised to switch from Zoom to Galene when it implements background blur. The reason I'm still waiting is that the Chrome people have been working on native background blur (https://developer.chrome.com/blog/background-blur), and it would be much preferable to have the feature implemented natively rather than in user code. I suggest waiting some more to see if something comes out of the Chrome effort; if nothing does, I'll definitely consider implementing it in user code. |
Any plans to add a virtual background? Or at least a background blur?
The text was updated successfully, but these errors were encountered: