offline vad vs online vad

[garymmi]

offline vad output
{'start': 106000, 'end': 137000}

online vad output
[{114500: 'start'}]
[{126000: 'end'}]
[{129000: 'start'}]
[{140000: 'end'}]
[{196000: 'start'}]
[{201000: 'end'}]
[{237500: 'start'}]
[{241500: 'end'}]

offline vad can correctly got start and end boundary
but online vad got many segments
ex : segment1 from offline vad : start = 106000, end=137000 split into two segments
114500 ~ 126000
129000 ~ 140000
other segments are not speech

how to let online vad output close to offline vad output ?

[snakers4]

Hi, were you using the adaptive post-processing - https://github.com/snakers4/silero-vad#adaptive-way ?
Can you please post your audio and code?

[snakers4]

@adamnsandle

[garymmi]

thanks for your quick reply

offline vad code

import glob
import onnxruntime
import torch
from pprint import pprint

from IPython.display import Audio

_, utils = torch.hub.load(repo_or_dir='snakers4/silero-vad',
                              model='silero_vad_mini',
                              force_reload=False)

(get_speech_ts,
 get_speech_ts_adaptive,
 save_audio,
 read_audio,
 state_generator,
 single_audio_stream,
 collect_speeches) = utils

def init_onnx_model(model_path: str):
    return onnxruntime.InferenceSession(model_path)

def validate_onnx(model, inputs):
    with torch.no_grad():
        ort_inputs = {'input': inputs.cpu().numpy()}
        outs = model.run(None, ort_inputs)
        outs = [torch.Tensor(x) for x in outs]
    return outs[0]

model = init_onnx_model(f'{files_dir}/model_mini.onnx')
wav = read_audio(f'test_16k_0.wav')

# get speech timestamps from full audio file
speech_timestamps = get_speech_ts(wav, model, num_steps=4, run_function=validate_onnx, visualize_probs=True)
pprint(speech_timestamps)

online vad code

for batch in single_audio_stream(model, 'test_16k_0.wav', run_function=validate_onnx):
    if batch:
       print(batch)

test wav file : https://drive.google.com/file/d/1nn714AkeF4VGnWbaKQgVFSkWAkrJ1qOd/view?usp=sharing

I use onnx mini model, because I want to run on android device

[snakers4]

Surprisingly, despite the nature of the audio (it is quite adversarial), the chart for probabilities looks good for ordinary VAD:

It has kind of high default threshold and it does not capture the second speaker.

I listened to the audio carefully, and there are 2 voices - loud male voice and silent female voice in the background. In my opinion, the adaptive vad does a slighly better job, because it captures the female speaker as well:

Look, the peaks are kind of normalized, that is why this is "adaptive" because we keep a running mean to adapt to sudden changes of speaker volume / quality.

So I believe this behaviour is kind of expected and intentional.
In any case feel free to tweak the code of the examples for more optimal result. Also make sure to use visualize_probs=True flag when debugging, it usually helps.

[garymmi]

i also output the graph , but no use adaptive

but i curious why online vad split first male into two segments
i didn't see low prob. at first segment in below graph

[snakers4]

@adamnsandle
can you please take a look why the streaming breaks with the adaptive interface

[garymmi]

My question :
but i curious why online vad split first male into two segments
i didn't see low prob. at first segment in below graph

Answer:
because offline vad algorithm has two criteria : min_silence_samples, min_speech_samples,
online vad has no these two . so offline vad output is different from online vad

[snakers4]

this probably concerns online + adaptive, right?

[garymmi]

speech_timestamps = get_speech_ts(wav, model, num_steps=4, run_function=validate_onnx, visualize_probs=True) 
for e in speech_timestamps:
   print(e)

output : {'start': 106000, 'end': 134000}

get_speech_ts_adaptive(wav, model, run_function=validate_onnx, visualize_probs=True) 
for e in speech_timestamps:
   print(e)

output : {'start': 106000, 'end': 134000}

for this test, no adaptive and adaptive have the same output result