Is it possible to limit the languages within the language detection

[kormoczi]

Hi, I am tying to use the Language Classifier 95 model, but the accuracy is not so good.
I have tried to increase the top_n value, but did not help too much.
I thought I can neglect most of the languages (which I do not care about) with specifying a reduced set of languages in the lang_dict and the lang_group_dict parameters in the following line:
languages, language_groups = get_language_and_group(wav, model, lang_dict, lang_group_dict, top_n=2)
but it does not work.
Is it possible somehow to specify a subset of the languages for this model?
Thanks!

[snakers4]

Hi,

Hi, I am tying to use the Language Classifier 95 model, but the accuracy is not so good.

Can you describe your validation dataset, its size and composition, maybe share some metrics (accuracy, precision, recall)?

Is it possible somehow to specify a subset of the languages for this model?

You can just rewrite this function - https://github.com/snakers4/silero-vad/blob/master/utils_vad.py#L339-L362

[kormoczi]

Hi,
I do not have a validation dataset yet, we are just speaking short sentences into the microphone...
But I will try to setup some repeatable tests and share some metrics with you soon.
On the other hand, I will check this function in the utils_vad.py.
Thanks!

[kormoczi]

I have modified the utils_vad.py in a way, that the response should contain the selected languages only (English, Hungarian, Chinese) together with their probabilities (independently from the other languages and the order of the probabilities).
And for small sentences (like "Good morning", "Nice to see you here", etc.) the probability of the English language sometimes is still 0.0... Is there any presumption or requirement for the average volume of the speech? Or anything else other than the sample rate? Right now the wav files are with sample rate 16000, and 16 bits per sample...

[kormoczi]

Sure... Here are four short samples.

audio_samples.zip
.

[snakers4]

Tested the model with your files and with some random files on random langagues.

Most likely the problem with the English language is that the model cannot learn the words, it likely learns pronunciation (most blatantly it is obvious for example with Russian and Ukrainian - they sometimes have literally the same words).

And if you speak English with an accent, it picks up this accent over the content of the speech.

Some people from chat tested the model, and it mostly worked well with language groups, but worked poorly with exact languages.

This is reflected even in the metrics for English on held out validation:

Afrikaans: {'Afrikaans': 0.896, 'Luxembourgish, Letzeburgesch': 0.014, 'Yiddish': 0.012}
Albanian: {'Albanian': 0.854, 'Macedonian': 0.014, 'Belarusian': 0.012}
Amharic: {'Amharic': 0.908, 'Hausa': 0.01, 'Telugu': 0.006}
Arabic: {'Arabic': 0.71, 'Maltese': 0.044, 'Somali': 0.022}
Armenian: {'Armenian': 0.944, 'Bulgarian': 0.008, 'Tajik': 0.008}
Assamese: {'Assamese': 0.862, 'Bengali': 0.054, 'Nepali': 0.012}
Azerbaijani: {'Azerbaijani': 0.86, 'Turkish': 0.04, 'Turkmen': 0.01}
Bashkir: {'Bashkir': 0.826, 'Tatar': 0.074, 'Kazakh': 0.032}
Belarusian: {'Belarusian': 0.924, 'Slovenian': 0.006, 'Ukrainian': 0.006}
Bengali: {'Bengali': 0.864, 'Assamese': 0.02, 'Marathi': 0.008}
Bosnian: {'Bosnian': 0.37, 'Serbian': 0.268, 'Croatian': 0.194}
Bulgarian: {'Bulgarian': 0.858, 'Macedonian': 0.044, 'Albanian': 0.018}
Burmese: {'Burmese': 0.838, 'Tibetan': 0.022, 'Central Khmer': 0.018}
Catalan, Valencian: {'Catalan, Valencian': 0.82, 'Spanish, Castilian': 0.032, 'Italian': 0.03}
Central Khmer: {'Central Khmer': 0.91, 'Vietnamese': 0.026, 'Lao': 0.012}
Chinese: {'Chinese': 0.93, 'Thai': 0.0115, 'Vietnamese': 0.011}
Croatian: {'Croatian': 0.41, 'Bosnian': 0.194, 'Serbian': 0.18}
Czech: {'Czech': 0.692, 'Slovak': 0.122, 'Polish': 0.024}
Danish: {'Danish': 0.82, 'Norwegian Nynorsk': 0.026, 'Swedish': 0.026}
Dutch, Flemish: {'Dutch, Flemish': 0.778, 'Western Frisian': 0.03, 'English': 0.03}
English: {'English': 0.7535070140280561, 'German': 0.03006012024048096, 'Portuguese': 0.02404809619238477}
Estonian: {'Estonian': 0.876, 'Finnish': 0.042, 'Kinyarwanda': 0.012}
Finnish: {'Finnish': 0.892, 'Estonian': 0.026, 'Swedish': 0.014}
French: {'French': 0.87, 'Occitan': 0.02, 'Kinyarwanda': 0.016}
Galician: {'Galician': 0.836, 'Occitan': 0.048, 'Spanish, Castilian': 0.02}
German: {'German': 0.824, 'English': 0.036, 'Dutch, Flemish': 0.02}
Greek, Modern (1453–): {'Greek, Modern (1453–)': 0.872, 'Portuguese': 0.01, 'Macedonian': 0.01}
Gujarati: {'Gujarati': 0.652, 'Marathi': 0.086, 'Hindi': 0.038}
Haitian, Haitian Creole: {'Haitian, Haitian Creole': 0.884, 'Lingala': 0.024, 'Shona': 0.014}
Hausa: {'Hausa': 0.898, 'Shona': 0.016, 'Haitian, Haitian Creole': 0.008}
Hindi: {'Hindi': 0.632, 'Urdu': 0.182, 'Punjabi, Panjabi': 0.044}
Hungarian: {'Hungarian': 0.864, 'Luxembourgish, Letzeburgesch': 0.014, 'Swedish': 0.008}
Indonesian: {'Indonesian': 0.742, 'Malay': 0.054, 'Sundanese': 0.04}
Italian: {'Italian': 0.77, 'Portuguese': 0.042, 'Spanish, Castilian': 0.022}
Japanese: {'Japanese': 0.836, 'Korean': 0.056, 'Chinese': 0.014}
Kannada: {'Kannada': 0.772, 'Telugu': 0.062, 'Tamil': 0.024}
Kazakh: {'Kazakh': 0.938, 'Belarusian': 0.008, 'Mongolian': 0.008}
Kinyarwanda: {'Kinyarwanda': 0.8937875751503006, 'Indonesian': 0.03006012024048096, 'Estonian': 0.01002004008016032}
Kirghiz, Kyrgyz: {'Kirghiz, Kyrgyz': 0.876, 'Turkish': 0.038, 'Kinyarwanda': 0.02}
Korean: {'Korean': 0.954, 'Chinese': 0.014, 'Vietnamese': 0.004}
Lao: {'Lao': 0.686, 'Thai': 0.182, 'Central Khmer': 0.034}
Latin: {'Latin': 0.87, 'Occitan': 0.038, 'Norwegian Nynorsk': 0.016}
Lingala: {'Lingala': 0.848, 'Haitian, Haitian Creole': 0.026, 'Occitan': 0.026}
Luxembourgish, Letzeburgesch: {'Luxembourgish, Letzeburgesch': 0.958, 'Yiddish': 0.01, 'Norwegian': 0.006}
Macedonian: {'Macedonian': 0.802, 'Bulgarian': 0.044, 'Serbian': 0.03}
Malagasy: {'Malagasy': 0.928, 'Haitian, Haitian Creole': 0.008, 'Amharic': 0.006}
Malay: {'Malay': 0.754, 'Sundanese': 0.046, 'Indonesian': 0.036}
Malayalam: {'Malayalam': 0.832, 'Telugu': 0.038, 'Tamil': 0.032}
Maltese: {'Maltese': 0.874, 'Italian': 0.01, 'Yiddish': 0.01}
Marathi: {'Marathi': 0.8, 'Hindi': 0.034, 'Bengali': 0.03}
Mongolian: {'Mongolian': 0.904, 'Tibetan': 0.008, 'Kazakh': 0.008}
Nepali: {'Nepali': 0.85, 'Bengali': 0.016, 'Assamese': 0.016}
Norwegian: {'Norwegian': 0.78, 'Swedish': 0.074, 'Norwegian Nynorsk': 0.072}
Norwegian Nynorsk: {'Norwegian Nynorsk': 0.374, 'Norwegian': 0.352, 'Swedish': 0.112}
Occitan: {'Occitan': 0.856, 'Galician': 0.042, 'Latin': 0.012}
Oriya: {'Oriya': 0.97, 'Indonesian': 0.01, 'Tamil': 0.006}
Pashto, Pushto: {'Pashto, Pushto': 0.858, 'Somali': 0.012, 'Uzbek': 0.01}
Persian: {'Persian': 0.716, 'Turkish': 0.046, 'English': 0.022}
Polish: {'Polish': 0.756, 'Ukrainian': 0.044, 'Chinese': 0.016}
Portuguese: {'Portuguese': 0.802, 'Italian': 0.018, 'French': 0.016}
Punjabi, Panjabi: {'Punjabi, Panjabi': 0.879, 'Hindi': 0.019, 'Sindhi': 0.016}
Romanian, Moldavian, Moldovan: {'Romanian, Moldavian, Moldovan': 0.77, 'Albanian': 0.022, 'Bulgarian': 0.016}
Russian: {'Russian': 0.7525354969574036, 'Belarusian': 0.12981744421906694, 'Ukrainian': 0.04056795131845842}
Serbian: {'Serbian': 0.642, 'Bosnian': 0.118, 'Croatian': 0.086}
Shona: {'Shona': 0.846, 'Haitian, Haitian Creole': 0.02, 'Central Khmer': 0.016}
Sindhi: {'Sindhi': 0.708, 'Punjabi, Panjabi': 0.044, 'Urdu': 0.036}
Sinhala, Sinhalese: {'Sinhala, Sinhalese': 0.904, 'Telugu': 0.008, 'Sindhi': 0.008}
Slovak: {'Slovak': 0.85, 'Czech': 0.028, 'Macedonian': 0.022}
Slovenian: {'Slovenian': 0.706, 'Croatian': 0.046, 'Macedonian': 0.04}
Somali: {'Somali': 0.88, 'Hausa': 0.022, 'Sindhi': 0.01}
Spanish, Castilian: {'Spanish, Castilian': 0.6417004048582996, 'Catalan, Valencian': 0.06680161943319839, 'Galician': 0.058704453441295545}
Sundanese: {'Sundanese': 0.822, 'Indonesian': 0.032, 'Central Khmer': 0.03}
Swahili: {'Swahili': 0.88, 'Shona': 0.018, 'Lingala': 0.018}
Swedish: {'Swedish': 0.757, 'Norwegian': 0.035, 'English': 0.027}
Tagalog: {'Tagalog': 0.898, 'Sundanese': 0.014, 'Malay': 0.014}
Tajik: {'Tajik': 0.906, 'Uzbek': 0.026, 'Pashto, Pushto': 0.008}
Tamil: {'Tamil': 0.836, 'Malayalam': 0.026, 'Telugu': 0.026}
Tatar: {'Tatar': 0.738, 'Bashkir': 0.126, 'Kirghiz, Kyrgyz': 0.03}
Telugu: {'Telugu': 0.806, 'Kannada': 0.04, 'Tamil': 0.016}
Thai: {'Thai': 0.772, 'Lao': 0.082, 'Vietnamese': 0.058}
Tibetan: {'Tibetan': 0.912, 'Chinese': 0.008, 'Norwegian Nynorsk': 0.008}
Turkish: {'Turkish': 0.776, 'Azerbaijani': 0.048, 'Kinyarwanda': 0.016}
Turkmen: {'Turkmen': 0.958, 'Uzbek': 0.008, 'Turkish': 0.006}
Ukrainian: {'Ukrainian': 0.678, 'Belarusian': 0.154, 'Polish': 0.024}
Urdu: {'Urdu': 0.694, 'Hindi': 0.122, 'Punjabi, Panjabi': 0.066}
Uzbek: {'Uzbek': 0.836, 'Kazakh': 0.034, 'Tajik': 0.01}
Vietnamese: {'Vietnamese': 0.908, 'Sundanese': 0.016, 'Central Khmer': 0.016}
Western Frisian: {'Western Frisian': 0.774, 'Dutch, Flemish': 0.124, 'Portuguese': 0.014}
Yiddish: {'Yiddish': 0.886, 'Luxembourgish, Letzeburgesch': 0.036, 'Latin': 0.018}
Yoruba: {'Yoruba': 0.868, 'Haitian, Haitian Creole': 0.024, 'Shona': 0.022}

Probably this can be fixed by actually using a metric learning approach or some larger pre-training scheme, but this is out of scope for us for now.

[kormoczi]

On one hand, I can understand that an accent will make detection harder.
But my native language (language group) is not appearing in the probability list either...
And if my accent is bad, why the detection looks accurate (or at least much better) with longer sentences?
With short sentences I usually receive English: 0.0 (maybe 0.1 - 0.2, but that is rare), but with longer sentences (more than 10 word) I generally can receive English: 0.5 - 0.6 (and a lot of times even 0.7 - 0.9).

[snakers4]

With short sentences I usually receive English: 0.0 (maybe 0.1 - 0.2, but that is rare), but with longer sentences (more than 10 word) I generally can receive English: 0.5 - 0.6 (and a lot of times even 0.7 - 0.9).

To be honest we did not research the dependence of metrics on utterance length
Average utterance length in the training data was around 7-10s (some subsets have a mean around 5s, some around 10s)
This probably can explain the discrepancy
It may be a good starting place in future should we dig further

[kormoczi]

Ok, understand. Thanks for the explanation.