iOS crash due to recording sampleRate != device's supported sampleRate

[oleksandrtaran]

At first, thank you so much for this nice plugin. It works great on Android and is well documented.
There is a corner case that makes SR fail on some older iOS devices due to not correctly set sampleRate.

Reproducible on:

• iPhone 5s, 12.4.4

Not reproducible on:

• latest iPhones with the latest OS updates

Issue

• SR initializes properly and works several times for listen-cancel cycles.
• at another "listen" method call my app fails and I see following logs reported to Firebase Crashlytics

Fatal Exception: com.apple.coreaudio.avfaudio required condition is false: format.sampleRate == hwFormat.sampleRate
Fatal Exception: com.apple.coreaudio.avfaudio
0 CoreFoundation 0x1c1243180 __exceptionPreprocess
1 libobjc.A.dylib 0x1c041b9f8 objc_exception_throw
2 CoreFoundation 0x1c115c88c +[_CFXNotificationTokenRegistration keyCallbacks]
3 AVFAudio 0x1c70b8244 AVAE_RaiseException(NSString*, ...)
4 AVFAudio 0x1c70b787c _AVAE_Check(char const*, int, char const*, char const*, bool)
5 AVFAudio 0x1c7158224 AVAudioIONodeImpl::SetOutputFormat(unsigned long, AVAudioFormat*)
6 AVFAudio 0x1c710e39c AUGraphNodeBaseV3::CreateRecordingTap(unsigned long, unsigned int, AVAudioFormat*, void (AVAudioPCMBuffer*, AVAudioTime*) block_pointer)
7 AVFAudio 0x1c70eae70 AVAudioEngineGraph::InstallTapOnNode(AVAudioNode*, unsigned long, unsigned int, AVAudioFormat*, void (AVAudioPCMBuffer*, AVAudioTime*) block_pointer)
8 AVFAudio 0x1c716391c AVAudioEngineImpl::InstallTapOnNode(AVAudioNode*, unsigned long, unsigned int, AVAudioFormat*, void (AVAudioPCMBuffer*, AVAudioTime*) block_pointer)
9 AVFAudio 0x1c7152be8 -[AVAudioNode installTapOnBus:bufferSize:format:block:]
10 speech_to_text 0x101f0f1d4 (Missing)
11 speech_to_text 0x101f0d7b4 (Missing)
12 speech_to_text 0x101f0db74 (Missing)
13 Flutter 0x100d6ef40 (Missing)
14 Flutter 0x100d08db0 (Missing)
15 Flutter 0x100d5f4c0 (Missing)
16 Flutter 0x100d18c90 (Missing)
17 Flutter 0x100d1a908 (Missing)
18 CoreFoundation 0x1c11d5554 CFRUNLOOP_IS_CALLING_OUT_TO_A_TIMER_CALLBACK_FUNCTION
19 CoreFoundation 0x1c11d5284 __CFRunLoopDoTimer
20 CoreFoundation 0x1c11d4ab8 __CFRunLoopDoTimers
21 CoreFoundation 0x1c11cfa08 __CFRunLoopRun
22 CoreFoundation 0x1c11cefb4 CFRunLoopRunSpecific
23 GraphicsServices 0x1c33d079c GSEventRunModal
24 UIKitCore 0x1ed876c38 UIApplicationMain
25 Runner 0x1007bc290 main + 22 (AppDelegate.swift:22)
26 libdyld.dylib 0x1c0c928e0 start

My analysis result

1. iOS devices have different recording sample rates and this value changed at some point in the iOS history
2. The issue happens when recording sampleRate that must be defined in the plugin's iOS implementation is not the same as device's supported one.

Possible solutions

• provide a parameter in the plugin's "initialize()" method to set the sample rate
• set sample rate based on the supported one by the device like suggested here 1st source or 2nd source

Source of info

• crash format. SampleRate == hwFormat. SampleRate AudioKit/AudioKit#1851
• AKMicrophone fails to initialize AudioKit/AudioKit#1556

PS: I'd very much appreciate a fix in the upcoming 2.3.0 version!

[sowens-csd]

Thank you for the detailed report very helpful. I'll check out those links and see what I can do. I have seen posts about the sample rate problem and it seemed likely that the plugin was susceptible to it but I hadn't yet had any reports in the wild so I was being hopeful.

[oleksandrtaran]

@sowens-csd if this helps in any way I did the implementation of Google Streaming Speech API in both Kotlin and Swift and it worked very well (but the issue was Google Speech API is too expensive for my project).
In that implementation, I used a sample rate of 16000 and it worked without any crashes on both Android and iOS (all OS and device versions).
I was recording an audio stream with the Mic and streaming it to Google API.

func prepare(specifiedSampleRate: Int) -> OSStatus {

    var status = noErr
    let session = AVAudioSession.sharedInstance()
    do {
      try session.setCategory(AVAudioSessionCategoryRecord)
      try session.setPreferredIOBufferDuration(10)
    } catch {
      return -1
    }
    var sampleRate = session.sampleRate
    print("hardware sample rate = \(sampleRate), using specified rate = \(specifiedSampleRate)")
    sampleRate = Double(specifiedSampleRate)
...
    // Set format for mic input (bus 1) on RemoteIO's output scope
    var asbd = AudioStreamBasicDescription()
    asbd.mSampleRate = sampleRate
...

[sowens-csd]

This didn't make it into 2.3.0 but I'll try to get it into a patch release.

[oleksandrtaran]

@sowens-csd I tried a few fixes to set audioSession.setPreferredSampleRate and both of them stopped the app crashing with the mentioned error but there were other problems like not returning any result after the 1st successful recognition or returning "error_listen_failed" from the catch block.

1. try self.audioSession.setPreferredSampleRate(inputNode.inputFormat(forBus: 0).sampleRate)
2. try self.audioSession.setPreferredSampleRate(recordingFormat.sampleRate)

Any results on your side?

[oleksandrtaran]

tried this solution too and it didn't make the things OK. As a result, the 1st listening is successful and any subsequent doesn't return any result.

private var hwSRate: Double = 48000.0 // guess of device hardware sample rate
private var sampleRate: Double = 44100.0 // default audio sample rate
...
// choose 44100 or 48000 based on hardware rate
// sampleRate = 44100.0
var preferredIOBufferDuration = 0.0058 // 5.8 milliseconds = 256 samples
hwSRate = audioSession.sampleRate // get native hardware rate
if hwSRate == 48000.0 { sampleRate = 48000.0 } // set session to hardware rate
if hwSRate == 48000.0 { preferredIOBufferDuration = 0.0053 }
let desiredSampleRate = sampleRate
try self.audioSession.setPreferredSampleRate(desiredSampleRate)
try self.audioSession.setPreferredIOBufferDuration(preferredIOBufferDuration)

[sowens-csd]

If you have time could you try this experimental fix? The version on main right now has some new code to uninitialize the audio unit which, according to some threads, may address this issue. If this doesn't work I'll add a sample rate parameter to the initialization but it would be nice if we didn't need more parameterization.

[oleksandrtaran]

I tried out the AudioUnit fix from the main branch and it didn't work at all on iPhone 5s, 12.4.4 and iPhone 6, 13x. I get back
Exception: SpeechRecognitionError msg: error_listen_failed, permanent: true right away after listening start.
For now, I use a workaround by setting always a sample rate to 44100. It works 90% of the time on all devices I tested, i.e. iOS 12 and 13, iPhone 5s, 6, X, XS, iPad, etc.
See here https://github.com/oleksandrtaran/speech_to_text/blob/main/ios/Classes/SwiftSpeechToTextPlugin.swift#L360.

[sowens-csd]

Thanks for trying! I'll add the sample rate as an option to the initialize for now.

[sowens-csd]

The latest version on main now supports a sampleRate property on the listen method. It is optional, if you don't provide one the behaviour is the same as the current behaviour. I decided to use listen because theoretically the correct sample rate can change based on the input device, at least according to some articles I read. Since it is set for each start listen it made sense to pass it in on that method. Let me know if this works for you.

I'll also continue to try to find the right way to reliably avoid the sample rate crash. This still seems like a workaround that shouldn't be required.

[sowens-csd]

2.4.0 is now available with this change. I'm going to leave this issue open while I look for a more permanent solution.

[sowens-csd]

I believe this issue was fixed as of 4.2.0 with a change to the way the sample rate is established for new sessions. Closing this, please open new issues if anyone sees it again.