Latency on embedded platforms - Experiences

[becocabana]

Hi everyone,

I was wondering if anyone is having success with running low latency models on e.g. jetson nano or raspberry.

All my attempts so far end up with latency of around 300-400 ms wich makes most live applications rather unappealing.

I've tried it with various configurations. The Latest one was spherical with:
CAPACITY = 16
LATENT_SIZE = 128
N_BAND = 16
PHASE_1_DURATION = 1000000
RATIOS = [4, 4, 4, 2]
SAMPLING_RATE = 44100

Is that still far too big overall?

I would also be happy to share models with the community, however my current ones all seem rather bad to me.

Thanks

(Here are the specifics of my jetson nano setup, if anyone cares: https://bennoheisel.de/1-2/ I just noted my steps for my own use, so take it with a grain of salt.)

[hermes2507]

Raspberry Pi is probably not the best option for real-time machine learning inference. I have been trying to run my model on a Bela + Beaglebone, but it seems there is an issue with the compatibility of pytorch during the compilation of ~nn. See my forum entry here: https://forum.bela.io/d/2965-installing-nn-external-for-rave-models/2

For a hardware comparison you can check out: https://www.nime.org/proceedings/2016/nime2016_paper0005.pdf

[becocabana]

Thank you. Interesting approach with the bela. Maybe I will check it out.
But neither the Bela nor any raspberry pi can compete with a Jetson Nano in terms of performance for this kind of application, or am I missing something? In In the end, somehow Mr. Caillon must have reached the 20ms on his nano showcase. That would be absolutely workable for me. I managed to get it to about 100ms at the moment. Maybe I will try further reductions, but I doubt it changes much. At the moment I am stumped.
Thank you for responding! And food for thought for sure.