New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parallel processing of training docs in MLLM #483

Closed

osma opened this issue Apr 20, 2021 · 1 comment · Fixed by #511

Labels

Milestone

Member

osma commented Apr 20, 2021

Training the MLLM backend can be a bit slow. Most of the time is spent generating candidates from the training documents. This could probably be done faster by using parallel processing. It's noted as a TODO item in the code: https://github.com/NatLibFi/Annif/blob/master/annif/backend/mllm.py#L23

osma added the enhancement label

osma added this to the Short term milestone

Member Author

osma commented Aug 10, 2021

Actually the TODO item mentioned above is only relevant for the hyperparameter optimization functionality of the MLLM backend.

The training docs are processed in this loop and it could probably be parallelized.

osma mentioned this issue

Process training docs in parallel in MLLM backend #511

Merged

osma closed this as completed in #511

juhoinkinen modified the milestones: Short term, 0.54

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment