About multispeaker multilingual model

[Marioando]

Hi, I'm training the model for a multilingual model but speaker and language are deeply entanglated. I just add a language embedding by following the path of the speaker embedding ( in text encoder, and cfm decoder). Can you give some suggestion to achieve code switching capabilities please. Thank you!

[shivammehta25]

Hello,
Disentanglement is a research problem in itself. Are you adding language tag per token or per sentence? I would suggest add it per token. If you can get alignments, interleave some audio so that the model sees codeswitching while training. But you'll have to look more into code switching literature.