Adding emotions the tacotron2

[astricks]

I'm interested in adding emotion_id in addition to speaker_id so that while inference, I can choose which speaker as well as which emotion. The dataset would have to include an additional parameter (emotion_id) as well. This will enable me to do inference based on a specific speaker and specific emotion (angry, sad, happy, etc.)

I'm wondering how to go about doing this? Has something similar been done? Is this useful to anyone else?

[erogol]

You either need a dataset with emotion annotations and use these annotations as extra inputs to the model or if the dataset has no annotation but the speech is expressive then we can use sentiment classification to create these pseudo labels and train on them.

Curious to hear that others say. So maybe there are more alternatives.

[Sadam1195]

This might be not specifically related but could be answer to your question. You can improve things until they reach to point of diminishing return after that no matter how much you train the model mostly likely the chances are you won't get things improve more significantly. Your bottle neck could be the data itself (quality or quantity) or the model you have choose to train. @astricks

[astricks]

I'm going to gather more data and re-try. Will share back results in a month.

[astricks]

Update - Instead of waiting for more data, I gave it another sincere try. This time, it worked.

This time,

1. I used the speaker encoder to calculate speaker embeddings and trained using the resulting external_speaker_encoder_file
2. I diligently reduced r and the learning rate

Here are some samples
Angry
Scared
Surprised

This was an interesting exercise. Thank you all for your helpful comments.

[erogol]

@astricks good results and interesting use of the speaker encoder.

Maybe @thorstenMueller can try the same approach. 💪

[ridhisoni7]

@astricks do you have the code from this experiment up anywhere? would be very helpful for a project im working on atm! thanks :)