Add performance benchmarking scripts

[rouault]

And run them by Travis-CI

Note: I've some doubt about the reproductibility and trustability in those tests on Travis infrastructure. Initially I had tried them on OSX workers too but this seemed far too unreliable : https://travis-ci.org/rouault/openjpeg/jobs/228744947

This relates to #911

[detonin]

Seems fine ! A few comments:

• move performance-related files (scripts and filenames list to a "performance" folder in the "tests" directory)
• instead of having the coding options hard-coded in the script, it would be useful to read an external file containing the options used in different environments (see https://basecamp.com/1888053/projects/9781426/messages/62230968). The test_perf_filelist.csv would then contain the image, the number of threads, the action (compress or decompress) and an optional reference to this list of options. As we are more interested in the decoding speed, this is maybe more a nice-to-have than a mandatory feature but I think it would globally ease the benchmarking of OpenJPEG.
• I agree with your reserve on benchmarking on virtual platforms so we'll see if it gives relevant results or not (apparently the answer is no for macos). In the meantime, I suggest we evaluate the improvements based on local runs.

[detonin]

strangely enough, the correct prefix or suffix for OpenJPEG is opj and not ojp ... As everything in the code follows opj, I suggest we keep this rule.

[rouault]

Done

[detonin]

do we need a newline here ?

[rouault]

Done (wasn't strictly needed, but better indeed)

[rouault]

move performance-related files (scripts and filenames list to a "performance" folder in the "tests" directory)

Done

[rouault]

instead of having the coding options hard-coded in the script, it would be useful to read an external file containing the options used in different environments

Agreed. I imagined that the COMPRESS string could be later extended to accept options like COMPRESS[reversible=yes,...] depending on what options we want to test.

[rouault]

I've disabled making the job fail in case of performance regression, because even on Linux this is too unreliable. See https://travis-ci.org/uclouvain/openjpeg/jobs/229085228 that failed whereas previous builds (with same code) succeeded

[detonin]

Agreed. I imagined that the COMPRESS string could be later extended to accept options like COMPRESS[reversible=yes,...] depending on what options we want to test.

ok. This way is possible indeed but would require a "translation" from this generic syntax to the actual kdu or opj options I guess.

I've disabled making the job fail in case of performance regression, because even on Linux this is too unreliable. See https://travis-ci.org/uclouvain/openjpeg/jobs/229085228 that failed whereas previous builds (with same code) succeeded

ok. I think you're right : benchmark results on Travis will never be trustful. I wonder if we should not even disable the test but provide an easy way to run the script locally, as you suggested. To be discussed.
Do you already have results or graphs showing the current performances of OPJ vs KDU ?

[rouault]

I wonder if we should not even disable the test but provide an easy way to run the script locally, as you suggested

The test can be run locally with
OPJ_SKIP_REBUILD=1 OPJ_CI_PERF_TESTS=1 OPJ_NONCOMMERCIAL=1 tools/travis-ci/run.sh

I think we can keep it on Travis in non-failing mode as I did in the latest version of the patch, so as a hint of possible regressions or improvements, to be confirmed locally.

Do you already have results or graphs showing the current performances of OPJ vs KDU ?

Yes, extract from https://travis-ci.org/uclouvain/openjpeg/jobs/229431228

Comparing current version with Kakadu
../../data/input/nonregression/kodak_2layers_lrcp.j2c, 3 iterations, 1 threads, DECOMPRESS: ref_time 1830 ms, new_time 3196 ms, (regression) 74.6 %, ERROR_THRESHOLD
../../data/input/nonregression/kodak_2layers_lrcp.j2c, 5 iterations, 2 threads, DECOMPRESS: ref_time 1708 ms, new_time 4584 ms, (regression) 168.4 %, ERROR_THRESHOLD
../../data/input/nonregression/kodak_2layers_lrcp.j2c, 10 iterations, 4 threads, DECOMPRESS: ref_time 2138 ms, new_time 7831 ms, (regression) 266.3 %, ERROR_THRESHOLD
../../data/input/conformance/p0_07.j2k, 3 iterations, 1 threads, DECOMPRESS: ref_time 1948 ms, new_time 7418 ms, (regression) 280.8 %, ERROR_THRESHOLD
../../data/input/conformance/p0_04.j2k, 10 iterations, 1 threads, DECOMPRESS: ref_time 1096 ms, new_time 1270 ms, (regression) 15.9 %, ERROR_THRESHOLD
../../data/input/nonregression/X_4_2K_24_185_CBR_WB_000.tif, 3 iterations, 1 threads, COMPRESS: ref_time 4960 ms, new_time 9890 ms, (regression) 99.4 %, ERROR_THRESHOLD
TOTAL: ref_time 13683 ms, new_time 34192 ms, (regression) 149.9 %, ERROR_THRESHOLD