merge development into master for release 2.3

.circleci/config.yml

@@ -93,6 +93,10 @@ jobs:
           name: Run coco C tests
           command: python do.py test-c
           no_output_timeout: 1800
+      - run:
+          name: Run regression test
+          command: python do.py test-suites
+          no_output_timeout: 1800
 
   test_ubuntu_latest_python2:
     docker:
@@ -180,6 +184,11 @@ jobs:
           name: Run coco C tests
           command: python do.py test-c
           no_output_timeout: 1800
+      - run:
+          name: Run regression test
+          command: python do.py test-suites
+          no_output_timeout: 1800
+
 
 workflows:
 

README.md

@@ -2,6 +2,7 @@ numbbo/coco: Comparing Continuous Optimizers
 ============================================
 
 [![CircleCI](https://circleci.com/gh/numbbo/coco/tree/master.svg?style=shield)](https://circleci.com/gh/numbbo/coco/tree/master)
+[![Appveyor](https://ci.appveyor.com/api/projects/status/4dawpqr7aq2ioici/branch/master?svg=true)](https://ci.appveyor.com/project/nikohansen/coco-j53aywshl8udzvb/branch/master)
 
 [This code](https://github.com/numbbo/coco) reimplements the original Comparing
 Continous Optimizer platform, now rewritten fully in `ANSI C` and `Python` with
@@ -140,9 +141,10 @@ Getting Started <a name="Getting-Started"></a>
   Another entry point for your own experiments can be the `code-experiments/examples`
   folder. 
 
-5. Now you can **run** your favorite algorithm on the `bbob` suite (for 
-  single-objective algorithms) or on the `bbob-biobj` and `bbob-biobj-ext`
-  suites (for multi-objective algorithms). Output is automatically generated in the 
+5. Now you can **run** your favorite algorithm on the `bbob` and `bbob-largescale` suites
+  (for single-objective algorithms), on the `bbob-biobj` suite (for multi-objective 
+  algorithms), or on the mixed-integer suites (`bbob-mixint` and `bbob-biobj-mixint` 
+  respectively). Output is automatically generated in the 
   specified data `result_folder`. By now, more suites might be available, see below. 
 
   <a name="Getting-Started-pp"></a>
@@ -164,16 +166,17 @@ Getting Started <a name="Getting-Started"></a>
   ```
 
   processes the referenced archived BFGS data set. The given substring must
-  have a unique match in the archive. Otherwise, all matches are listed but none
-  is processed with this call. For more information in how to obtain and
-  display specific archived data, see
+  have a unique match in the archive or must end with `!` or `*` or must be a
+  [regular expression](https://docs.python.org/3/library/re.html#regular-expression-syntax)
+  containing a `*` and not ending with `!` or `*`. Otherwise, all matches are listed
+  but none is processed with this call. For more information in how to obtain
+  and display specific archived data, see
   [`help(cocopp)`](http://coco.gforge.inria.fr/apidocs-cocopp/cocopp.html) or
-  [`help(cocopp.archives`)](http://coco.gforge.inria.fr/apidocs-cocopp/cocopp.
-  archiving.KnownArchives.html) or the class
-  [`COCODataArchive`](http://coco.gforge.inria.fr/apidocs-cocopp/cocopp.
-  archiving.COCODataArchive.html).
+  [`help(cocopp.archives)`](http://coco.gforge.inria.fr/apidocs-cocopp/cocopp.archiving.KnownArchives.html)
+  or the class
+  [`COCODataArchive`](http://coco.gforge.inria.fr/apidocs-cocopp/cocopp.archiving.COCODataArchive.html).
 
-  For the `bbob` test suite, the data descriptions can be found at
+  Data descriptions can be found for the `bbob` test suite at
   http://coco.gforge.inria.fr/doku.php?id=algorithms and for the `bbob-biobj`
   test suite at http://coco.gforge.inria.fr/doku.php?id=algorithms-biobj.
 
@@ -257,15 +260,19 @@ our issue tracker at https://github.com/numbbo/coco/issues.
     problem inheriting thereby all properties of a coco problem)
   - most other files implement more or less what they say, e.g. the actual
     benchmark functions, transformations, benchmark suites, etc.
-  - currently, four benchmark suites and corresponding logging facilities are
-    implemented:
+  - currently, the following benchmark suites and corresponding logging facilities are
+    supported:
     * `bbob`: standard single-objective BBOB benchmark suite with 24 noiseless,
       scalable test functions
     * `bbob-biobj`: a bi-objective benchmark suite, combining 10 selected
       functions from the bbob suite, resulting in 55 noiseless functions
-    * `bbob-biobj-ext`: an extended version of the bi-objective `bbob-biobj`
-       benchmark suite, adding `bbob` function combinations from the same
-       `bbob` function groups, resulting in 92 noiseless functions overall
+    * `bbob-largescale`: a version of the `bbob` benchmark suite with dimensions
+      20 to 640, employing permuted block-diagonal matrices to reduce the 
+      execution time for function evaluations in higher dimension.
+    * `bbob-mixint`: a mixed-integer version of the original `bbob` and
+      `bbob-largescale` suites in which 80% of the variables have been discretized
+    * `bbob-biobj-mixint`: a version of the (so far not supported) `bbob-biobj-ext`
+      test suite with 92 functions with 80% discretized variables
     * `toy`: a simple, probably easier-to-understand example for reading and testing
 
 * code-experiments/tools are a few meta-tools, mainly the amalgamate.py to merge all
@@ -595,10 +602,15 @@ Links and Documentation <a name="Links"></a>
 * For details on the experimental set-up to carry out the benchmarking
   please refer to http://numbbo.github.io/coco-doc/experimental-setup/.
 * More detailed documentation of the existing benchmark suites can be found here:
-  - for the **`bbob`** problem suite at http://coco.lri.fr/downloads/download15.03/bbobdocfunctions.pdf 
+  - for the **`bbob`** problem suite at http://coco.gforge.inria.fr/downloads/download16.00/bbobdocfunctions.pdf 
     with the experimental setup at http://coco.lri.fr/downloads/download15.03/bbobdocexperiment.pdf
   - for the **`bbob-biobj`** and **`bbob-biobj-ext`** problem suites
     at http://numbbo.github.io/coco-doc/bbob-biobj/functions
+  - for the **`bbob-largescale`** problem suite
+    at http://numbbo.github.io/coco-doc/bbob-largescale/functions
+  - for the **`bbob-mixint`** and **`bbob-biobj-mixint`** problem suites, we refer to 
+    https://hal.inria.fr/hal-02067932/document and to
+    http://coco.gforge.inria.fr/preliminary-bbob-mixint-documentation/bbob-mixint-doc.pdf
 * Online documentation of the NumBBO/Coco API (i.e. for the ANSI C code) is available at 
   http://numbbo.github.io/coco-doc/C
 * More detailed documentation on how the performance assessement is done can 
@@ -613,12 +625,15 @@ Comprehensive List of Documentations <a name="Documentations"></a>
 * General introduction: http://numbbo.github.io/coco-doc
 * Experimental setup: http://numbbo.github.io/coco-doc/experimental-setup/
 * Testbeds
-  - bbob: http://coco.lri.fr/downloads/download15.03/bbobdocfunctions.pdf ^1
+  - bbob: http://coco.gforge.inria.fr/downloads/download16.00/bbobdocfunctions.pdf
   - bbob-biobj: http://numbbo.github.io/coco-doc/bbob-biobj/functions/
   - bbob-biobj-ext: http://numbbo.github.io/coco-doc/bbob-biobj/functions/
   - bbob-noisy (only in old code basis): http://coco.lri.fr/downloads/download15.03/bbobdocnoisyfunctions.pdf
+  - bbob-largescale: http://numbbo.github.io/coco-doc/bbob-largescale/functions/
+  - bbob-mixint and bbob-biobj-mixint: https://hal.inria.fr/hal-02067932/document and
+    http://coco.gforge.inria.fr/preliminary-bbob-mixint-documentation/bbob-mixint-doc.pdf
   - bbob-constrained (in progress): http://numbbo.github.io/coco-doc/bbob-constrained/functions/
-  - bbob-largescale (in progress): http://numbbo.github.io/coco-doc/bbob-largescale/functions/
+
 
 * Performance assessment: http://numbbo.github.io/coco-doc/perf-assessment/
 * Performance assessment for biobjective testbeds: http://numbbo.github.io/coco-doc/bbob-biobj/perf-assessment/
@@ -633,9 +648,7 @@ Comprehensive List of Documentations <a name="Documentations"></a>
 * Somewhat outdated documents:
   - Full description of the platform: http://coco.lri.fr/COCOdoc/
   - Experimental setup before 2016: http://coco.lri.fr/downloads/download15.03/bbobdocexperiment.pdf
- - Old framework software documentation: http://coco.lri.fr/downloads/download15.03/bbobdocsoftware.pdf
+  - Old framework software documentation: http://coco.lri.fr/downloads/download15.03/bbobdocsoftware.pdf
 
  * Some examples of [results](https://github.com/numbbo/coco/wiki/COCO-Wiki-Home).
 
-^1: see [#837](https://github.com/numbbo/coco/issues/837) and [#1266](https://github.com/numbbo/coco/issues/1266) for two pending errata
-

appveyor.yml

@@ -33,4 +33,5 @@ test_script:
   - cmd: python do.py test-java
   - cmd: python do.py test-preprocessing
   - cmd: python do.py test-postprocessing
+  - cmd: python do.py test-suites
   # - cmd: python do.py test-postprocessing-all

code-experiments/build/c/Makefile.in

@@ -33,4 +33,4 @@ example_experiment: example_experiment.o coco.o
 coco.o: coco.h coco.c
 	${CC} -c ${CCFLAGS} -o coco.o coco.c
 example_experiment.o: coco.h coco.c example_experiment.c
-	${CC} -c ${CCFLAGS} -o example_experiment.o example_experiment.c
+	${CC} -c ${CCFLAGS} -o example_experiment.o example_experiment.c