Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optional quadratic duration correction for dynamic bucketing sampler #950

Merged
merged 3 commits into from
Jan 27, 2023

Conversation

pzelasko
Copy link
Collaborator

I've also removed some deprecated options like specifying max frames/samples and related tests.

@pzelasko
Copy link
Collaborator Author

pzelasko commented Jan 27, 2023

I gave it some runs, it seems to work alright, even though I still haven't calibrated it very well.

As a piece anegdotic evidence, I was able to adjust a training which had max_duration=250 into max_duration=400 with quadratic_duration=30 without OOM. So I almost doubled the duration, but the average duration in my dataset is about 5-6 seconds, which is far below the penalty that effectively doubles a cut duration. It's hard to give effective increase in throughput without measuring it carefully, but I think throwing around numbers like 20-30% is not unreasonable.

I'll merge it. If somebody uses it, it's important to remember that you need to adjust max_duration after setting quadratic_duration (and likely learning rate as well).

@pzelasko pzelasko merged commit 0fca592 into master Jan 27, 2023
@pzelasko pzelasko deleted the feature/quadratic-duration branch January 27, 2023 21:25
@danpovey
Copy link
Collaborator

That's great!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants