Missing lines

[nikohansen]

Looking at this output

https://numbbo.github.io/ppdata-archive/bbob/2019/pprldflex.html

I see GNN-CMA and IPOP-CMA in <= 10-D but not in 20-D. This is a little scary, how can this happen? I didn't check whether everything else is in order.

[brockho]

These are two entries which are incomplete, according to https://numbbo.it/testsuites/bbob/data-archive.html#inofficialincomplete-data-sets. Everything else should be fine. Shall I remove these two data sets from the ppdata output and even from the official archives? If we remove them, where do we put them?

[nikohansen]

We could move these to an incomplete sub archive? Generally, we should try to keep the number of incomplete sets in the official archive as low as possible.

[brockho]

Do you prefer to have the incomplete folder at the same level than 2009 etc.? Or is it okay to have them in a 2019-incomplete folder like the ones in 2014-others [which we might want to rename to 2014-incomplete then]?

[nikohansen]

I suggest to have one bbob/incomplete archive. It may have per year subfolders.

I also think we should consider renaming 2014-others. In the current list we only see the year but not the subfolder where the data are in, which is however part of its name in the archive and can be used for searching. As of now bbob/2014/bfgs* would ignore entries in 2014-others for a pretty nontransparent reason.

Similarly, I'd prefer, accordingly, to move 2015-GECCO/ to 2015/ etc. We may mention this in a list (we see it from the reference), but I don't see a good reason why the venue should be part of the data set name.

[brockho]

Should the entries in 2014-others/ then go to bbob/incomplete/?

[nikohansen]

Should the entries in 2014-others/ then go to bbob/incomplete/?

yes, given they are incomplete.

[nikohansen]

The naming in the mixint archive is also neither consistent with any of the above, nor is gecco-benchmark a particularly helpful qualifier.

>>> cocopp.archives.bbob_mixint()

['2019-gecco-benchmark/CMA-ES-pycma.tgz',
 '2019-gecco-benchmark/DE-scipy.tgz',
 '2019-gecco-benchmark/RANDOMSEARCH.tgz',
 '2019-gecco-benchmark/TPE-hyperopt.tgz',
 '2022/CMA-ESwM_Hamano.zip',
 '2024/DE-CoBi-r1-L_Tanabe.zip',
 '2024/DE-NO-r1-B_Tanabe.zip',
 '2024/DE-j-r2-B_Tanabe.zip']

[brockho]

Agreed. All this comes from a time where we did not have continuous submissions of data sets. A year without anything always indicates a BBOB workshop, the other years have specific suffixes (like -gecco-benchmark for our GECCO paper or -others or -CEC or ...). If we want, I can remove those suffixes in all data-archive paths.