[Proposal] Early stopping while doing hyperparameter tuning with Ray integration

[ozhanozen]

Proposal

I propose adding an early stopping mechanism for Ray integration.

Motivation

When doing hyperparameter tuning with Ray integration, the trials are continued until whatever limit is set within the training config. Even if a trial is going badly and hopelessly, Ray waits until it is finished before proceeding to the next one. This is not always the most efficient thing to do, especially when it is possible to predict how the end performance will be based on the current performance.

If we can add an optional early stopping mechanism, e.g., using the Stoppers, this would increase the hyperparameter tuning efficiency by giving up on bad trials early.

Additional context

I have actually managed to integrate the optional provision of a custom Stopper that correctly raises the stop flags based on the performance metrics. However, I did not manage to stop the ongoing trials for some reason; potentially because they are on a different subprocess that runs the simulation, or the simulation being unresponsive to the stop signals from Ray.

Hi @garylvov, if you have any feedback regarding how to manage this, I would be happy to try. I could create a PR draft to show you what I did and for us to work on together if you wish.

Checklist

• I have checked that there is no similar issue in the repo (required)

Acceptance Criteria

• Eary stopping for ray tuning is added.

[garylvov]

Hi @ozhanozen thank you for looking into this. If you could make a PR with your work for us to collaborate on that would be great! I'd be happy to help troubleshoot the rogue process issue!

[RandomOakForest]

Thank you for posting this proposal. The team will review it.