Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement memory and latency regression tests #61

Closed
atiorh opened this issue Mar 7, 2024 · 3 comments · Fixed by #99
Closed

Implement memory and latency regression tests #61

atiorh opened this issue Mar 7, 2024 · 3 comments · Fixed by #99
Assignees
Labels
enhancement Improves existing code help wanted Extra attention is needed

Comments

@atiorh
Copy link
Contributor

atiorh commented Mar 7, 2024

Implement tests to transcribe long audio files (at least several minutes worth) and measure the memory and latency over time. This is to guard against memory leaks or slowdowns potentially being introduced by new PRs (e.g. #40 fixed by #56 thanks to @finnvoor!)

@ZachNagengast ZachNagengast added enhancement Improves existing code help wanted Extra attention is needed labels Mar 8, 2024
@Abhinay1997
Copy link
Contributor

Any pointers on how to start with this. I'd like to take it up.

I assume the tests would be functionaltests and we'd be measuring the metrics but not sure on the details.

@ZachNagengast
Copy link
Contributor

@Abhinay1997 Check inside FunctionalTests.swift for the measure calls. We'll want to save these results somewhere so they can be compared to in subsequent runs, but we'd need to makes sure it is hardware specific so we don't think there's a regression on a slower computer vs a faster one.

@ZachNagengast ZachNagengast moved this to TODO: Features in WhisperKit Mar 28, 2024
@ZachNagengast ZachNagengast moved this from TODO: Features to TODO in WhisperKit Mar 28, 2024
@Abhinay1997
Copy link
Contributor

PR: #99

@ZachNagengast ZachNagengast moved this from TODO to In Progress in WhisperKit Apr 2, 2024
@ZachNagengast ZachNagengast linked a pull request Apr 12, 2024 that will close this issue
5 tasks
@github-project-automation github-project-automation bot moved this from In Progress to Done in WhisperKit Apr 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Improves existing code help wanted Extra attention is needed
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

3 participants