-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathbbob2016.qmd
224 lines (180 loc) · 10.6 KB
/
bbob2016.qmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
# GECCO Workshop on Real-Parameter Black-Box Optimization Benchmarking (BBOB 2016) - focus on multi-objective problems
Welcome to the web page of the 6th GECCO Workshop on Real-Parameter
Black-Box Optimization Benchmarking (BBOB 2016) with focus on
multi-objective problems with two objective functions which took place
during GECCO 2016.
> **WORKSHOP ON REAL-PARAMETER BLACK-BOX OPTIMIZATION BENCHMARKING (BBOB
> 2016) - with a new focus on multi-objective problems**
>
> | hold as part of the
> |
> | **2016 Genetic and Evolutionary Computation Conference
> (GECCO-2016)**
> | July 20-24, Denver, CO, USA
> | <http://gecco-2016.sigevo.org/>
| Submission Deadline: extended to Sunday, April 17, 2016 (from
Saturday, April 3, 2016)
|
------------------------------------------------------- ------------------------------------------------------------------------ -----------------------------------------------------------------
[register for news](http://numbbo.github.io/register) [Coco quick start (scroll down a bit)](https://github.com/numbbo/coco) [latest Coco release](https://github.com/numbbo/coco/releases/)
------------------------------------------------------- ------------------------------------------------------------------------ -----------------------------------------------------------------
<br />
Quantifying and comparing the performance of optimization algorithms is
a difficult and tedious task to achieve. Previously, the Coco platform
has provided tools to ease this process for *single-objective problems*
by: (1) an implemented, well-motivated benchmark function testbed, (2) a
simple and sound experimental set-up, (3) the generation of output data
and (4) the post-processing and presentation of the results in graphs
and tables. For the first time, this year, we provide(d) an extension of
the Coco platform towards a *multi-objective testbed* (with two
objective functions) and with nearly the same procedure as in previous
BBOB workshops.
The remaining tasks for participants were therefore: run your favorite
multi-objective black-box optimizer (old or new) by using the wrappers
provided and run the post-processing procedure (provided as well) that
will generate automatically all the material for a workshop paper (LaTeX
templates are provided). A description of the algorithm and the
discussion of the results completes the paper writing.
We encourage(d) particularly submissions related to multi-objective
algorithms for *expensive optimization* (with a limited budget) and also
*algorithms from outside the evolutionary computation community*. Please
note that submissions related to the existing *single-objective BBOB
testbeds* (noiseless and noisy) were still welcome although the focus
was on the new bi-objective testbed.
During the workshop, algorithms and results were presented by the
participants. An overall analysis and comparison was accomplished by the
organizers and the overall process was critically reviewed.
## Updates and News
Get updated about the latest news regarding the workshop and releases
and bugfixes of the supporting NumBBO/Coco plateform, by registering at
<http://numbbo.github.io/register>.
## Supporting material
Most likely, you want to read the [Coco quick
start](https://github.com/numbbo/coco) (scroll down a bit). This page
also provides the code for the benchmark functions, for running the
experiments in C, Java, Matlab, Octave, and Python, and for
postprocessing the experiment data into plots, tables, html pages, and
publisher-conform PDFs via provided LaTeX templates. Please refer to
<http://numbbo.github.io/coco-doc/experimental-setup/> for more details
on the general experimental set-up for black-box optimization
benchmarking.
The latest (hopefully) stable release of the Coco software can be
downloaded as a whole [here](https://github.com/numbbo/coco/releases/).
Please use at least version v1.0 for running your benchmarking
experiments.
Documentation of the functions used in the [bbob-biobj]{.title-ref}
suite for BBOB 2016 are provided at
<http://numbbo.github.io/coco-doc/bbob-biobj/functions/> .
Note that the current release of the new Coco platform does not contain
the original noisy BBOB testbed, such that you must use the old code at
<https://numbbo.github.io/coco/oldcode/bboball15.03.tar.gz> for the time
being if you want to compare your algorithm on the noisy testbed.
## Submissions
Submissions of benchmarking results of new or existing numerical
optimization algorithms in terms of bi-objective or single-objective
optimization were welcome and should have been done through the
following form at <http://numbbo.github.io/submit>.
Eventually, the following papers have been accepted:
- Ilya Loshchilov and Tobias Glasmachers: **Anytime Bi-Objective
Optimization with a Hybrid Multi-Objective CMA-ES (HMO-CMA-ES)**
- Oswin Krause, Tobias Glasmachers, Nikolaus Hansen, and Christian
Igel: **Unbounded Population MO-CMA-ES for the Bi-Objective BBOB
Test Suite**
- Kouhei Nishida and Youhei Akimoto: **Evaluating the Population Size
Adaptation Mechanism for CMA-ES on the BBOB Noiseless Testbed**
- Kouhei Nishida and Youhei Akimoto: **Evaluating the Population Size
Adaptation Mechanism for CMA-ES on the BBOB Noisy Testbed**
- Cheryl Wong, Abdullah Al-Dujaili, and Suresh Sundaram:
**Hypervolume-based DIRECT for Multi-Objective Optimisation**
- Abdullah Al-Dujaili and Suresh Sundaram: **A MATLAB Toolbox for
Surrogate-Assisted Multi-Objective Optimization: A Preliminary
Study**
- Anne Auger, Dimo Brockhoff, Nikolaus Hansen, Dejan Tušar, Tea Tušar,
and Tobias Wagner: **Benchmarking the Pure Random Search on the
Bi-objective BBOB-2016 Testbed**
- Anne Auger, Dimo Brockhoff, Nikolaus Hansen, Dejan Tušar, Tea Tušar,
and Tobias Wagner: **The Impact of Variation Operators on the
Performance of SMS-EMOA on the Bi-objective BBOB-2016 Test Suite**
- Anne Auger, Dimo Brockhoff, Nikolaus Hansen, Dejan Tušar, Tea Tušar,
and Tobias Wagner: **Benchmarking MATLAB\'s gamultiobj (NSGA-II) on
the Bi-objective BBOB-2016 Test Suite**
- Anne Auger, Dimo Brockhoff, Nikolaus Hansen, Dejan Tušar, Tea Tušar,
and Tobias Wagner: **Benchmarking RM-MEDA on the Bi-objective
BBOB-2016 Test Suite**
- Tea Tušar and Bogdan Filipič: **Performance of the DEMO algorithm on
the bi-objective BBOB test suite**
- Anne Auger, Dimo Brockhoff, Nikolaus Hansen, Dejan Tušar, Tea Tušar,
and Tobias Wagner: **The Impact of Search Volume on the Performance
of RANDOMSEARCH on the Bi-objective BBOB-2016 Test Suite**
## Data
The data sets of all submitted algorithms can be found at
<https://numbbo.github.io/data-archive/bbob-biobj/> for the `bbob-biobj`
test suite and at <https://numbbo.github.io/data-archive/bbob> and
<https://numbbo.github.io/data-archive/bbob-noisy/> for the `bbob` and
`bbob-noisy` test suites respectively.
## Workshop Schedule
All BBOB-2016 sessions took place on the first day of GECCO (July 20,
2016) in the Wind Star B room. Speakers are highlighted with a star
behind the name. Please click on the provided links to download the
slides.
----------- --------------------------------------------------------------------------------------------------------------
**Session
I**
08:30 - The BBOBies: Introduction to Blackbox Optimization Benchmarking
09:30 ([slides](./presentations/2016-GECCO/01_Dimo_bbob-2016-intro.pdf))
09:30 - Tea Tušar\*, Bogdan Filipič: Performance of the DEMO algorithm on the bi-objective BBOB test suite
09:55 ([slides](./presentations/2016-GECCO/02_Tea_DEMO_handouts.pdf))
09:55 - Ilya Loshchilov, Tobias Glasmachers*: Anytime Bi-Objective Optimization with a Hybrid Multi-Objective CMA-ES (HMO-CMA-ES)
10:20 ([slides](./presentations/2016-GECCO/03_Tobias_hmocmaes.pdf))
**Session
II**
10:40 - The BBOBies: Session Introduction
10:55 ([slides](./presentations/2016-GECCO/04_Dimo_bbob-2016-turbointro.pdf))
10:55 - Cheryl Wong\*, Abdullah Al-Dujaili, and Suresh Sundaram: Hypervolume-based DIRECT for Multi-Objective
11:20 Optimisation
([slides](./presentations/2016-GECCO/05_Cheryl_MO-DIRECT.pdf))
11:20 - Abdullah Al-Dujaili and Suresh Sundaram (speaker: Cheryl Wong): A MATLAB Toolbox for Surrogate-Assisted
11:45 Multi-Objective Optimization: A Preliminary Study
([slides](./presentations/2016-GECCO/06_Cheryl_MO-MATSuMoTo.pdf))
11:45 - Oswin Krause\*, Tobias Glasmachers, Nikolaus Hansen, and Christian Igel: Unbounded Population MO-CMA-ES for
12:10 the Bi-Objective BBOB Test Suite
([slides](./presentations/2016-GECCO/07_Oswin_UP-MO-CMA-ES.pdf))
12:10 - The BBOBies: Session Wrap-up
12:30 ([slides](./presentations/2016-GECCO/08_Dimo_session2-wrapup.pdf))
**Session
III**
14:00 - The BBOBies: Session Introduction
14:15 ([slides](./presentations/2016-GECCO/09_Anne_bbob-2016-turbointro.pdf))
14:15 - Kouhei Nishida\* and Youhei Akimoto: Evaluating the Population Size Adaptation Mechanism for CMA-ES
14:40 ([slides](./presentations/2016-GECCO/10_Kouhei_PSA.pdf))
14:40 - The BBOBies: Wrap-up of all BBOB-2016 Results
15:05 ([slides](./presentations/2016-GECCO/11_Anne_bbob-2016-wrap-up.pdf))
15:05 - Thomas Weise\*: optimizationBenchmarking.org: An Introduction
15:30
15:30 - Open Discussion
15:50
----------- --------------------------------------------------------------------------------------------------------------
## Important Dates
- **01/20/2016** first version of the new Coco platform released as
[0.5-beta](https://github.com/numbbo/coco/releases/)
- **01/30/2016** (planned: 01/29/2016) release
[0.7-beta](https://github.com/numbbo/coco/releases/) of the Coco
software with the main functionality to run experiments
- (planned: 02/12/2016, replaced by 7 intermediate releases) first
complete release [0.9](https://github.com/numbbo/coco/releases/) of
the software
- **03/29/2016** (planned: 03/18/2016) final release
[1.0](https://github.com/numbbo/coco/releases/) for producing the
papers
- **04/17/2016** new *paper and data submission deadline* (extended
from 04/02/2016)
- **04/20/2016** decision notification
- **05/04/2016** deadline camera-ready papers
- **07/20/2016** workshop
## Organizers
- Anne Auger, Inria Saclay - Ile-de-France
- Dimo Brockhoff, Inria Lille - Nord Europe
- Nikolaus Hansen, Inria Saclay - Ile-de-France
- Dejan Tušar, Inria Lille - Nord Europe
- Tea Tušar, Inria Lille - Nord Europe
- Tobias Wagner, TU Dortmund University