New demos from abhilash26

[abhilash26]

First of all thank you Jeeliz for this amazing piece of code.

I have created a minified reusable version of this repository Face Primer for my use. Since I want use your technology in various applications. I hope this will help others who just want to use your AI mode for face detection.
I have previously contacted you with my work account and got great feedbacks from you on many issues. I wanted to repay you by sending you the demo of what I was building but it was a non-disclosed client project. This is my home account. I will sure make something wonderful and share a demo with you.

I have few questions.

1. Did you use tensorflow to make the model?
2. How can we improve the model with more sample images?

Some suggestions for improving animation which I have used while working on the client project.

1. I have noticed that the influences (expression) values are having very high precision, this works against animation because the model seems to be moving more than we desire. I solved the issue by using .toFixed() function.
2. There since the eyes generally close and open at the same time (except for winking) we can average out and use the same value for both the eyes, the same goes for eyebrows. So a function that can calculate a mean value for them would be great.
3. I had created a mean-displacement function to calculate the change in last-frame and the current frame using the change in rotation and influences values. If the mean-displacement is below a certain threshold we do not update the weboji. This reduce stutter.

[xavierjs]

Hi @abhilash26

Thank you for this great feedback and for your interest about Jeeliz Weboji.
We would be glad to share and promote your work and demos using Jeeliz libs.

Here are the answers:

1. We don't use Tensorflow for the training. We do the training in browser so that we can monitor it.
   And it enables us to re-use a large part of the code used for running neural network. Since everything is WebGL, there is no computing overload because of the browser environment. The GPU are still running at 100% during training. We also don't have any bridge between our framework and tensorflow.
2. We don't use images at all, but we dynamically generate each training sample from a randomly generated 3D model (random mesh, random texture, random orientation...). So we avoid many overfitting problems (and dataset acquistion cost...).

About the improvements, we are currently focusing on other use of our technology, especially around object detection and tracking. But I have ideas to implement since a while... I hope I will find some time in the next months.

This library is at a quite low level and it gives many morphs... It is a good idea to group them to get a higher precision.
The detection of eyes opening is not good because there is a bias: the neural network learns wrinkles around the eyes. For some people, depending on the wrinkles, their lighting and the shape of the eyes it can be really bad.

The stabilization needs to be improved. I was also thinking abour using median/kalman filters. But there is always the problem of the balance between too much stabilization => low reactivity and high reactivity but slutter.

Best,
Xavier

[abhilash26]

Hello @xavierjs,

Thanks for the quick reply...

I have created a drowsiness detection system here https://dont-drive-drowsy.glitch.me/
you can use it as a demo.

I found its inspiration here. https://www.youtube.com/watch?v=Q23K7G1gJgY

I will also provide the code which I used for the improvements in the weboji animation.

It is better than the video since it can run on mobile.

Also, I have added half-sec delay to avoid alarm goes in while the person is blinking.

Regards,
Dinodroid

[xavierjs]

The demo works nicely, thanks a lot for sharing!
I have added a link from the README.md.
I will film a short video about it to share it on Jeeliz Youtube and Linkedin.

Best regards,
Xavier

[abhilash26]

Thank you so much for your appreciation.
I felt indebted to you and therefore I made this app to give you as a tribute.

Currently I am working on emotion-detector.
It might not be accurate but I will convert the influences values to emotions.
Any tips are appreciated.

Regards,
Dinodroid

[xavierjs]

Hi,

The simplest way would be to do a linear system, to just weight high level emotions to low level face morphs coefficients given by Weboji ( see jeeliz/jeelizFaceFilter#104 ).

But I really don't know what would be the accuracy of such system.
People can show their emotions in really different way...

Best,
Xavier

[abhilash26]

Hello @xavierjs,

I have created a working demo.
https://emotion-reader.glitch.me

Source code is here....
https://glitch.com/edit/#!/emotion-reader

Regards,
Dinodroid

[xavierjs]

Thanx for sharing, it works pretty good!
I will add it to the readme. For your name/link to your profile or website, do you want I put Dinodroid or Abilash26? I put the link from your name to https://ko-fi.com/dinodroid or to your github profile?

Did you tune the coefficients by hand or did you use a solver? (I am referring to https://glitch.com/edit/#!/emotion-reader?path=assets/js/emotion.js:1:0 )

Thanx,
Xavier

[xavierjs]

I also have 2 suggestions for the "Don't drive drowsy app":

• you should not use a .WAV file, it is really heavy. MP3 is really better,
• you should use webAudio API instead of <audio> element. It is more robust.
  Sometimes I have the bug of the alarm not playing.
  If you try to play multiple time a <audio> element it could do some weird stuffs, whereas webAudio API offers a better control.

[abhilash26]

Hi @xavierjs,

For my name and link use " abhilash26 aka dinodroid ".

Did you tune the coefficients by hand or did you use a solver?

No Sir, I tune it by hand. So this needs some work. Any suggestions on how to create a solver?

You should not use a .WAV file, it is really heavy. MP3 is really better. you should use webAudio API instead of <audio> element. It is more robust. Sometimes I have the bug of the alarm not playing.

Well this was my idea at first but I previously had some problems with web audio api specially with AudioContext . Also I wanted to release the app as quickly as possible. No worries, i will work on your improvements and give you the solution in less than 2 days, i hope so.

Regards,
Dinodroid

[xavierjs]

Hi,

I think it should be better to create a solver.

Currently you could organize EXPRESSION2EMOTION as a 5x11 matrix. Each row is a 11 vector which match currently a value of EXPRESSION2EMOTION.
Let call A this matrix. You are looking for the coefficients of A. There are 11*5 = 55 values to find.

You can order the scores of your 5 expressions into a vector having 5 dimensions. let call it e. The output of the neural network is a 11 vector, called x.

You have: e = A.x
Now just save several samples where you record yourself and other people doing the expression.

To solve it, you need at least 11 records because each equation e_n = A.x_n can be written as 5 scalar equations (n goes from 0 to the number of records) and you have 55 unknows (so you need at least 55 scalar equation to solve it).

For example for the first sample you get the anger face, so e_0 = [1,0,0,0,0] and you save the output of JEETRANSFERTAPI as x_0, and so on with other expressions.

you register 11 expression, then you use a linear solver to solve this (with Python there are good maths packages http://kitchingroup.cheme.cmu.edu/blog/2013/02/27/Solving-linear-equations/).
Or you can register more and use more advanced optimizations stuffs.

Best,
Xavier

[abhilash26]

Thanks I will try that...but i have never used or created a solver before.
Please can you hint me what tools to use? Any tutorial specific for making solver?

Meanwhile, check
https://github.com/abhilash26/dont-drive-drowsy

I have updated the changes you requested for webaudio api.
Check if everything is ok as per your suggestion and i will make it live on the same demo url.

Also, i have imported the emotion code to https://github.com/abhilash26/emotion-reader

Regards,
Dinodroid

[xavierjs]

Hi,

Great, I have updated the README.md to include this new demo and links to source code.
It is far better with AudioContext.
If I may, I think you should not put JavaScript file (or code) in assets path, it may be confusing.
Usually assets are only static stuffs (images, video, 3D models). You should put them into src or lib path.

For the solver python is better than JavaScript for these kind of tasks. People use it in research.
I don't know good tutorials. First you have to learn linear algebra (vector, matrix operations). It is very useful in machine learning, 3D, optimization, ... It is the most difficult part. Then Python libs or Matlab implement linear algebra operations and algorithm in a quite straightforward way.

Best,
Xavier

[abhilash26]

Hello,
For the assets path, yes, you are right about that and thanks for pointing it out. I have updated the path in both repositories.
One question... May I use your video my repository as a link?

Ok, I will try to understand python core first and then import Matlab libs. However meanwhile if you come up with something please create a pull-request to at least give me something basic to work with.

Do you have any cool ideas for the application of jeelizWeboji?

Thanks and regards,
Abhilash aka Dinodroid

[abhilash26]

Hello @xavierjs,

I have created another demo for you to use. This is sit-straight
Source code is here https://github.com/abhilash26/sit-straight

It is basically a posture corrector which tells if you have incorrect posture while sitting. If yes, then it alarms you about that.

Thanks and regards,
Abhilash aka Dinodroid

[xavierjs]

Hi @abhilash26

Sure you can use imags and videos from the Jeeliz Youtube channel.
Currently we don't focus a lot on Jeeliz Weboji library. We still maintain it and will keep maintenance but we focus more on augmented reality use cases with some major update coming for this lib: https://github.com/jeeliz/jeelizAR

We also got threatened by Apple lawyers ( we took off our demo app, https://webojis.com ) and we don't want to get into trouble.

I think a good application of weboji would be to run it on a Nvidia Jetson TX2 with a small touchscreen to build a device like this: https://ae01.alicdn.com/kf/HTB17l4pIXXXXXX5XXXXq6xXFXXXg/customer-satisfaction-investigation-terminal-for-customer-feedback-and-evaluation-rating-device.jpg but without the buttons. It should be sellable in retail.

Nice demo for sit-straight :).

Best,
Xavier