TPE discrete-categorical-loguniform space support #389

donglinjy · 2020-05-08T13:44:17Z

Add discrete and categorical dimension into TPE support.

Now TPE will support uniform, uniform discrete and choices as prior. As for choices prior, the probabilities if any given will be ignored.

codecov-io · 2020-05-08T13:51:24Z

Codecov Report

Merging #389 into develop will increase coverage by 0.62%.
The diff coverage is 100.00%.

@@             Coverage Diff             @@
##           develop     #389      +/-   ##
===========================================
+ Coverage    46.61%   47.23%   +0.62%     
===========================================
  Files           67       67              
  Lines        12754    12905     +151     
  Branches       317      321       +4     
===========================================
+ Hits          5945     6096     +151     
  Misses        6784     6784              
  Partials        25       25

Impacted Files	Coverage Δ
tests/functional/algos/test_algos.py	`100.00% <100.00%> (ø)`
tests/unittests/algo/test_tpe.py	`100.00% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d1057ea...7bc6808. Read the comment docs.

bouthilx · 2020-05-11T20:05:08Z

src/orion/algo/tpe.py

+        if list(candidate_points):
+            sampler_above = CategoricalSampler(self, above_points, choices)
+
+            lik_blow = sampler_below.get_loglikelis(candidate_points)


Suggested change

lik_blow = sampler_below.get_loglikelis(candidate_points)

lik_below = sampler_below.get_loglikelis(candidate_points)

bouthilx · 2020-05-11T20:05:25Z

src/orion/algo/tpe.py

+            lik_blow = sampler_below.get_loglikelis(candidate_points)
+            lik_above = sampler_above.get_loglikelis(candidate_points)
+
+            new_point = compute_max_ei_point(candidate_points, lik_blow, lik_above)


Suggested change

new_point = compute_max_ei_point(candidate_points, lik_blow, lik_above)

new_point = compute_max_ei_point(candidate_points, lik_below, lik_above)

bouthilx · 2020-05-11T20:05:52Z

src/orion/algo/tpe.py

+        if list(candidate_points):
+            sampler_above = CategoricalSampler(self, above_points, choices)
+
+            lik_blow = sampler_below.get_loglikelis(candidate_points)


Suggested change

lik_blow = sampler_below.get_loglikelis(candidate_points)

lik_below = sampler_below.get_loglikelis(candidate_points)

bouthilx · 2020-05-11T20:06:02Z

src/orion/algo/tpe.py

+            lik_blow = sampler_below.get_loglikelis(candidate_points)
+            lik_above = sampler_above.get_loglikelis(candidate_points)
+
+            new_point_index = compute_max_ei_point(candidate_points, lik_blow, lik_above)


Suggested change

new_point_index = compute_max_ei_point(candidate_points, lik_blow, lik_above)

new_point_index = compute_max_ei_point(candidate_points, lik_below, lik_above)

bouthilx · 2020-05-11T20:35:43Z

tests/unittests/algo/test_tpe.py

@@ -262,13 +370,13 @@ def test_set_state(self, tpe):
    def test_unsupported_space(self):
        """Test tpe only work for supported search space"""
        space = Space()
-        dim = Integer('yolo1', 'uniform', -2, 4)
+        dim = Fidelity('epoch', 1, 9, 3)


It could support easily Fidelity by just returning the max like random search does. Integer('yolo1', 'normal', 0, 1) for testing unsupported priors.

It is pretty straight forward for random as all suggestions simply follow the same pattern, for tpe, we need to add separate paths to handle fidelity dimension and mention such support in doc. The only value for random or tpe to support fidelity I can think is to make it easier for user to switch from different search algorithms, or user can simply specify fixed fidelity value for random and tpe. And it is actually a little bit confusing as fidelity prior means differently between tpe/random and hyperband/asha,

Yes the advantage of supporting fidelity dimension in algos like random search and TPE is really just to make it easier to switch from one algo to another using the same search space definition.

bouthilx · 2020-05-11T20:45:10Z

tests/unittests/algo/test_tpe.py

+        above_points = [obs_points[25:]]
+        points = tpe.sample_one_dimension(dim1, 1,
+                                          below_points, above_points, tpe._sample_int_point)
+        assert len(points) == 1


It would be good to test the values as well. That they are integers in the correct interval and correspond roughly to the best region given below_points and above_points.

bouthilx · 2020-05-11T20:49:42Z

tests/unittests/algo/test_tpe.py

        assert len(points) == 2

        tpe.n_ei_candidates = 0
-        points = tpe.sample_real_dimension(dim2, 2, below_points, above_points)
+        points = tpe.sample_one_dimension(dim2, 2,
+                                          below_points, above_points, tpe._sample_real_point)


I see that we didn't test for values for test_sample_real_dimensions. We could sample separately below and above with different intervals to artificially create the best regions and verify the sampling of the TPE.

It was same idea for int, categorical and real, as there are separate cases for gmm and categorical sampler. But sure, we can add some specific verification at these places.

codecov-commenter · 2020-05-23T14:51:29Z

Codecov Report

Merging #389 into develop will increase coverage by 0.67%.
The diff coverage is 99.48%.

@@             Coverage Diff             @@
##           develop     #389      +/-   ##
===========================================
+ Coverage    46.62%   47.30%   +0.67%     
===========================================
  Files           67       67              
  Lines        12768    12932     +164     
  Branches       317      321       +4     
===========================================
+ Hits          5953     6117     +164     
  Misses        6790     6790              
  Partials        25       25

Impacted Files	Coverage Δ
tests/unittests/core/test_transformer.py	`21.66% <0.00%> (ø)`
tests/functional/algos/test_algos.py	`100.00% <100.00%> (ø)`
tests/unittests/algo/test_space.py	`99.27% <100.00%> (-0.01%)`	⬇️
tests/unittests/algo/test_tpe.py	`100.00% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2b956ec...30db153. Read the comment docs.

src/orion/algo/tpe.py

bouthilx · 2020-06-08T17:25:36Z

src/orion/algo/tpe.py

@@ -283,7 +280,8 @@ def suggest(self, num=1):
                                                       above_points[idx: idx + shape[0]],
                                                       self._sample_categorical_point)
                else:
-                    raise NotImplementedError()
+                    # fidelity dimension


maybe elif dimension.type == 'fidelity' to be sure to catch unsupported dimensions that would have been missed in __init__

…-discrete-categorical

…jy/orion into tpe-discrete-categorical

tpe discrete categorical space support

dce2c9d

bouthilx reviewed May 11, 2020

View reviewed changes

donglinjy added 2 commits May 12, 2020 17:29

refine some code

7bc6808

add loguniform

5536624

donglinjy changed the title ~~TPE discrete categorical space support~~ TPE discrete-categorical-loguniform space support May 23, 2020

sync categorical change

1c51f5f

bouthilx reviewed May 25, 2020

View reviewed changes

src/orion/algo/tpe.py Outdated Show resolved Hide resolved

add fidelity support

055d900

bouthilx reviewed Jun 8, 2020

View reviewed changes

bouthilx and others added 5 commits June 8, 2020 13:26

Merge branch 'develop' into tpe-discrete-categorical

5e4a27e

Merge branch 'develop' of https://github.com/Epistimio/orion into tpe…

7f5d570

…-discrete-categorical

fix issues

4971825

Merge branch 'tpe-discrete-categorical' of https://github.com/donglin…

dd729b4

…jy/orion into tpe-discrete-categorical

Merge branch 'develop' into tpe-discrete-categorical

dbaa11b

bouthilx approved these changes Jun 12, 2020

View reviewed changes

Merge branch 'develop' into tpe-discrete-categorical

30db153

Thomsch merged commit 047aee5 into Epistimio:develop Jun 12, 2020

bouthilx added the v0.1.8 label Jun 22, 2020

This was referenced Jun 23, 2020

Release 0.1.8rc #412

Closed

Release 0.1.8rc2 #414

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TPE discrete-categorical-loguniform space support #389

TPE discrete-categorical-loguniform space support #389

donglinjy commented May 8, 2020

codecov-io commented May 8, 2020 •

edited

Loading

bouthilx May 11, 2020

bouthilx May 11, 2020

bouthilx May 11, 2020

bouthilx May 11, 2020

bouthilx May 11, 2020

donglinjy May 12, 2020 •

edited

Loading

bouthilx May 25, 2020

bouthilx May 11, 2020

bouthilx May 11, 2020

donglinjy May 12, 2020

codecov-commenter commented May 23, 2020 •

edited

Loading

bouthilx Jun 8, 2020

	lik_blow = sampler_below.get_loglikelis(candidate_points)
	lik_below = sampler_below.get_loglikelis(candidate_points)

	new_point = compute_max_ei_point(candidate_points, lik_blow, lik_above)
	new_point = compute_max_ei_point(candidate_points, lik_below, lik_above)

	new_point_index = compute_max_ei_point(candidate_points, lik_blow, lik_above)
	new_point_index = compute_max_ei_point(candidate_points, lik_below, lik_above)

TPE discrete-categorical-loguniform space support #389

TPE discrete-categorical-loguniform space support #389

Conversation

donglinjy commented May 8, 2020

codecov-io commented May 8, 2020 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

donglinjy May 12, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-commenter commented May 23, 2020 • edited Loading

Codecov Report

Choose a reason for hiding this comment

codecov-io commented May 8, 2020 •

edited

Loading

donglinjy May 12, 2020 •

edited

Loading

codecov-commenter commented May 23, 2020 •

edited

Loading