feat: 💥 Add _pseudo_ genetic search #777
Conversation
The proposed implementation of a genetic algorithm for hyper optimization. Even if genetic optimization might be costly for CNN, the applications in numeric analysis or Design of Experiment (DoE) make it still interesting. Fixes: keras-team#47 Further Reading: 1. [Vishwakarma G, et al Towards Autonomous Machine Learning in Chemistry via Evolutionary Algorithms. **ChemRxiv.**](https://chemrxiv.org/engage/api-gateway/chemrxiv/assets/orp/resource/item/60c7445a337d6c2849e26d98/original/towards-autonomous-machine-learning-in-chemistry-via-evolutionary-algorithms.pdf) 2. [Rosanna Nichols et al 2019 _Quantum Sci. Technol._ **4** 045012](https://iopscience.iop.org/article/10.1088/2058-9565/ab4d89/meta?casa_token=db7uZRqRMEAAAAAA:fRO9qB25dAkeoskS6MMyzpZw2jSiMkpsN4zA_k6lheWUXaSUU8fPS-JPMoNFcIl9tka4OPCG5AtDtiM)
Codecov ReportPatch coverage:
Additional details and impacted files@@ Coverage Diff @@
## master #777 +/- ##
==========================================
+ Coverage 95.41% 99.86% +4.45%
==========================================
Files 50 45 -5
Lines 3247 3028 -219
==========================================
- Hits 3098 3024 -74
+ Misses 149 4 -145
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report at Codecov. |
|
@Anselmoo Thanks for the contribution! We will need to have an internal review. I will get back to you afterwards. |
@haifeng-jin thx for the quick response and to the team. I know introducing a new algorithm is not straightforward, so see what's next 😊 |
|
@Anselmoo However, this would serve as a great example of implementing custom algorithms for KerasTuner in the KerasTuner guides on keras.io. https://keras.io/guides/keras_tuner/ Would you like to contribute it there? |
|
@Anselmoo I should have closed the issue in the first place as we are very conservative on accepting new algorithms. Please mention me in the thread before making any big contributions to ensure the PR can be accepted in the future. |
This sound promising, let's do it. How, does this match with your @haifeng-jin latest comment? |
There was a problem hiding this comment.
Thanks for the PR! I have left some comments. You can either change it here in the PR. Or create a new PR in https://github.com/keras-team/keras-io, where the final contribution would be. You need to create a new file here: https://github.com/keras-team/keras-io/tree/master/guides/keras_tuner. Please use the other files in the directory as examples.
| self.ranges = self._make_ranges | ||
| self.population = {"hyperparameters": [], "scores": []} | ||
| self.new_population = {"hyperparameters": [], "scores": []} | ||
| self.values = {hp.name: hp.default for hp in self.get_space().space} |
There was a problem hiding this comment.
We don't need this. It can be a local variable in self.populate_space() and an argument in self._check_score(..., values).
There was a problem hiding this comment.
The problem is if you define in self.populate_space(), it actually starts to re-initialize the hp.default again and again for every trial. In general, this implementation is kind of nested because I was focusing on using the oracle trial_id for making use of the parallelisation so far.
Make it sense a little?
| self.population = {"hyperparameters": [], "scores": []} | ||
| self.new_population = {"hyperparameters": [], "scores": []} |
There was a problem hiding this comment.
These 2 should be lists of trial_ids.
The score information and hyperparameters information can be retrieved from self.trials[trial_id].
There was a problem hiding this comment.
Does this mean that self.trials[trial_id] should get a list of dict?
There was a problem hiding this comment.
@haifeng-jin can you briefly explain to me, what a possible implementation should look like, please. Pseudo code is fine.
|
@haifeng-jin can you take a brief look, please? With regards to https://keras.io/guides/keras_tuner/ it is more like a tutorial on how to build your individual solver solution? |
The proposed implementation of a genetic algorithm for hyper optimization as discussed in #47
Even if genetic optimization might be costly for CNN, the applications in numeric analysis or Design of Experiment (DoE) make it still interesting.
Fixes: #47
Further Reading: