Memory and Latency Regression Tests

[Abhinay1997]

• Log memory during regression test
• Keep track of latency during regression test
• Summarize measurements and save for every N(=100) samples
• Dump measurements to a json for easy visualization.

Checklist:

• code clean up
• add more stats for latency tests
  - [ ] add WER calculations in swift Moved out of this PR. Plan to include it as part of EN Normalization PR
• get processor info and add it to test output. See Utils.swift
• use a longer audio file. > 1 hr for tests. Using earnings22
• test dynamically on all models similar to testOutputAll

See colab for plots

[ZachNagengast]

We also need a scheme to detect regressions for these tests on a chip-by-chip basis. We can store the data inside the test resources directly, or host them in a huggingface repo.

Files should be named something like this:
Resources/Fixtures/RegressionTests/\(Process.processor)/\(modelName)/\(date).json

All of these should be at or below 1.05x of the baseline data. I.e. a given regression test can be up to 5% slower than the baseline to account for various hardware fluctuations.

• Word error rate
• Peak memory
• Tokens per second
• Full pipeline (for model loading times)

We don't yet have code to calculate WER from swift, but from python we use this metric: https://huggingface.co/spaces/evaluate-metric/wer

Should also have a test that runs all available models from the whisper-coreml repo, similar to testOutputAll, however I wouldn't expect this to run on the github runners due to the limited resources they have, it will just be run manually before releases.

[Abhinay1997]

• WER calculations are detailed here, we'll have to implement this in Swift.
• Chip/Processor info is possible to obtain. Shouldn't be an issue. I'll add that.
• Understood on the file name ! Let me do that.

[atiorh]

Should resolve #61

[ZachNagengast]

These are all subject to change over time unfortunately, so these tests should actually search the full HF repo for all models available in there in order to run these tests. You can use this method to find all available models:

WhisperKit/Sources/WhisperKit/Core/WhisperKit.swift

Lines 131 to 136 in 8564ce2

	public static func fetchAvailableModels(from repo: String = "argmaxinc/whisperkit-coreml", matching: [String] = ["openai_*", "distil-whisper_*"]) async throws -> [String] {
	let hubApi = HubApi()
	let modelFiles = try await hubApi.getFilenames(from: repo, matching: matching)
	return formatModelFiles(modelFiles)
	}

[Abhinay1997]

Updated the code to use this function, Zack

[Abhinay1997]

You'll need to install xcparse via brew install chargepoint/xcparse/xcparse

xcodebuild clean build-for-testing -scheme whisperkit-Package -destination generic/platform=macOS | xcpretty
xcodebuild test -only-testing WhisperKitTests/RegressionTests -scheme whisperkit-Package -destination "platform=macOS,arch=arm64" -resultBundlePath ~/Downloads
xcparse attachments ~/Downloads/<latest_xc_result_file>.xcresult

Note: xcparse command above will output attachments as files in current directory

[ZachNagengast]

Looks great! And low risk because its all new code. Needs just a bit of linting which I can run for you.

Can you give a bit of guidance on how to test this? I.e. what commands to run and what the expected output should be?

[Abhinay1997]

Do let me know how I can improve on the linting !

As for running it, the commands above should work, if not, you can manually run it from XCode and see the test attachments in the Xcode test result.

[ZachNagengast]

Do let me know how I can improve on the lining !

Will have linting rules setup soon, until then this is good to merge 👍