Questions using Silero with Microsoft.ML.OnnxRuntime

[kkppstudios]

Hi. I am playing around with silero-vad and I think I've had some success, but I have some questions about the input and output.

Background:
I am developing an Android app with Xamarin Forms. I am using the nuget package Microsoft.ML.OnnxRuntime. I am streaming audio from the device's microphone and trying to process it in real-time. The audio is 16kHz, PCM-16bit, one channel. I am converting the audio bytes into a normalized (-1,1) float array, then batching it in batch sizes of 1536 frames/samples. So far so good. I can create a OnnxRuntime InferenceSession using the .onnx model. I can send the audio data in as input and I am seeing some output.

Questions:
After enough searching and playing around, I found that the inputs are "input", "h0", and "c0". What is h0 and c0? To satisfy the model, I am sending in an empty float[] with dimensions 2,1,64 for each of these inputs. I have no idea what they do and if I am setting them up correctly.
As for the output, I am seeing two floats. My understanding was that this model would return one float that represented the probability of speech. So why are there two floats and what do each of them mean? There is also a "hn" and "cn" being output, which I am again ignoring for now.

Any insight into this would be really helpful. I think I am getting close, I just need to understanding these inputs/outputs.

[snakers4]

Hi,

After enough searching and playing around, I found that the inputs are "input", "h0", and "c0". What is h0 and c0?

You can just follow this wrapper logic in your code:

silero-vad/utils_vad.py

Lines 10 to 52 in 7c671a7

	class OnnxWrapper():
	def __init__(self, path):
	import numpy as np
	global np
	import onnxruntime
	self.session = onnxruntime.InferenceSession(path)
	self.session.intra_op_num_threads = 1
	self.session.inter_op_num_threads = 1
	self.reset_states()
	def reset_states(self):
	self._h = np.zeros((2, 1, 64)).astype('float32')
	self._c = np.zeros((2, 1, 64)).astype('float32')
	def __call__(self, x, sr: int):
	if x.dim() == 1:
	x = x.unsqueeze(0)
	if x.dim() > 2:
	raise ValueError(f"Too many dimensions for input audio chunk {x.dim()}")
	if sr != 16000 and (sr % 16000 == 0):
	step = sr // 16000
	x = x[::step]
	sr = 16000
	if x.shape[0] > 1:
	raise ValueError("Onnx model does not support batching")
	if sr not in [16000]:
	raise ValueError(f"Supported sample rates: {[16000]}")
	if sr / x.shape[1] > 31.25:
	raise ValueError("Input audio chunk is too short")
	ort_inputs = {'input': x.numpy(), 'h0': self._h, 'c0': self._c}
	ort_outs = self.session.run(None, ort_inputs)
	out, self._h, self._c = ort_outs
	out = torch.tensor(out).squeeze(2)[:, 1] # make output type match JIT analog
	return out

input is just input samples, h0 and c0 are just internal model states that are kept between iterations. We did not find an easier way to do so with ONNX.

My understanding was that this model would return one float that represented the probability of speech.

This line should help:

silero-vad/utils_vad.py

Lines 47 to 50 in 7c671a7

	ort_outs = self.session.run(None, ort_inputs)
	out, self._h, self._c = ort_outs
	out = torch.tensor(out).squeeze(2)[:, 1] # make output type match JIT analog

[shrestha-pranita]

@kkppstudios Hi!! I have been trying to create an empty float[] with dimensions 2,1,64 to pass as h0 and c0, however, I keep getting byteLength error. Could you post how you created these?

[kkppstudios]

In my class I have the following:

private readonly float[] h0Value = new float[128];
private readonly float[] c0Value = new float[128];

private readonly NamedOnnxValue h0NamedOnnxValue = null;
private readonly NamedOnnxValue c0NamedOnnxValue = null;

In the class constructor:

var h0Input = new DenseTensor<float>(this.h0Value, new[]
{
    2, 1, 64
});

var c0Input = new DenseTensor<float>(this.c0Value, new[]
{
    2, 1, 64
});

this.h0NamedOnnxValue = NamedOnnxValue.CreateFromTensor("h0", h0Input);
this.c0NamedOnnxValue = NamedOnnxValue.CreateFromTensor("c0", c0Input);

Then in a function within the class, I call session.Run:

using var results = this.session.Run(new List<NamedOnnxValue>
{
    NamedOnnxValue.CreateFromTensor("input", input),
    this.h0NamedOnnxValue,
    this.c0NamedOnnxValue
});

Note, my use case is streaming audio.

[shrestha-pranita]

Thanks for the help. For the feeds's input are you passing the input in the shape of (1, .....)?? I keep getting the following error:

E:onnxruntime:, sequential_executor.cc:368 Execute] Non-zero status code returned while running Conv node. Name:'Conv_81' Status Message: Invalid input shape: {1}