Update tests to permit any order of output lines by sorting the editorconfig output

[cxw42]

@xuhdev @ppalaga Here we go again! :D

• Fixes Test suite: Permit editorconfig programs to provide correct output lines in any order editorconfig#375, which has the discussion leading up to this PR.

• Replaces Update tests to permit any order of output lines #22 per Update tests to permit any order of output lines #22 (comment)

• Adds new new_ec_test_multiline function that adds a test accepting multiple
  output lines in any order. The matching regex for this function must be specified in CMake's sort order - see discussion at https://gitlab.kitware.com/cmake/cmake/issues/18551

• Rewrites the pertinent tests to use new_ec_test_multiline.

• Bumps the required CMake version up to 3.5.0 per @xuhdev at Update tests to permit any order of output lines #22 (comment)

editorconfig-core-c and my editorconfig-core-vimscript both pass 100% of tests on Cygwin (edit) and Windows with this PR in place (except for utf-8 filenames).

Note: also includes one commit for the benefit of Cygwin (my development environment) to not run tests that can't possibly succeed on Cygwin.

ppalaga: Would you please test your Java CI?

Edit I have now run these tests on mingw-w64 x86 with Windows CMake 3.6.1 in cmd. All pass except for utf_8_char, which I didn't touch :) . I see per this (linked from editorconfig/editorconfig-core-c#31 (comment)) that this is a known issue on Windows.

[cxw42]

@ppalaga Have you had a chance to try out the Java core? Unless this change causes problems for you, I think it's ready to merge.

[xuhdev]

Thanks for your hard work, @cxw42 ! Except for the minor thing I pointed out, can you also make the coding style consistent? Sometimes you have spaces inside braces but sometimes you don't. Other than those, I think it's ready to merge.

[cxw42]

@xuhdev Fixed, plus I found and fixed a few EQUAL comparisons that should have been STREQUAL. I re-ran {C core, Vimscript core} x {Cygwin, MinGW} and it looks good. Thanks!

I updated the copyright dates across the board so no one else would have to do them for the rest of the year :) .

I removed the spaces inside the parens for consistency with the existing code.

[ppalaga]

There are 20 failing tests with ec4j:

• Windows: https://ci.appveyor.com/project/ppalaga/ec4j/builds/20452747?fullLog=true#L826
• Linux: https://travis-ci.org/ec4j/ec4j/jobs/457809294#L3021

Let me check what is the problem.

[ppalaga]

Here is the detailed output for the first failure meta_multiline:

2/189 Testing: meta_multiline
2/189 Test: meta_multiline
Command: "C:/Program Files (x86)/CMake/bin/cmake.exe" "-D" "EDITORCONFIG_CMD=C:/Program Files/Java/jdk1.7.0/bin/java.exe;-cp;C:/projects/ec4j/core/target/ec4j-core.jar;org.ec4j.core.cli.Cli" "-D" "ECARGS:LIST=-f;meta.in;C:/projects/ec4j/core/editorconfig-core-test/meta/meta.c" "-P" "C:/projects/ec4j/core/editorconfig-core-test/cmake/ec_sort.cmake"
Directory: C:/projects/ec4j/core/editorconfig-core-test/meta
"meta_multiline" start time: Nov 21 07:01 Coordinated Universal Time
Output:
----------------------------------------------------------
CMake Error at C:/projects/ec4j/core/editorconfig-core-test/cmake/ec_sort.cmake:55 (message):
  C:/Program
  Files/Java/jdk1.7.0/bin/java.exe;-cp;C:/projects/ec4j/core/target/ec4j-core.jar;org.ec4j.core.cli.Cli
  -f;meta.in;C:/projects/ec4j/core/editorconfig-core-test/meta/meta.c
  returned a nonzero exit code
<end of output>

source: https://ci.appveyor.com/project/ppalaga/ec4j/builds/20452747?fullLog=true#L894

Maybe the reason is that my EDITORCONFIG_CMD consists of java executable and some space separated arguments? Here is its definition:
https://github.com/ec4j/ec4j/blob/8b8fa5578e293bf2e209432f2a1a399ff23fe36f/core/CMakeLists.txt#L20

Should I change something there?

[ppalaga]

When I cd to the proper directory, replace semicolons with spaces in the command that returned a nonzero exit code above, and run it, it works properly. It returns answer=42 as expected.

I am not sure where the semicolons come from. Maybe they should not be there?

[cxw42]

@ppalaga Thanks! Would you please try the new commit? Before doing so, in your Travis/Appveyor config, would you please add --output-on-failure to the ctest command line? That will give us a bit more information on failures.

The semicolons in .../java.exe;-cp;C:/projects/... are added by CMake. This is because the definition

set(EDITORCONFIG_CMD ${Java_JAVA_EXECUTABLE} -cp ${PROJECT_SOURCE_DIR}/target/ec4j-core.jar org.ec4j.core.cli.Cli)

creates a list-valued variable, and lists in CMake are just strings with semicolons between list items (reference). By contrast, set(CMD "${Java} -cp ${...}") (with "") would create a string-valued variable.

I have updated cmake/ec_sort.cmake to accept list-valued variables or single strings. It should now pass the tests using your current EDITORCONFIG_CMD. Please let me know. Much appreciated!

[ppalaga]

Yes, with eccf9d4 I see ec4j passing locally. Let's see if both CI jobs will pass too https://ci.appveyor.com/project/ppalaga/ec4j/builds/20462289 https://travis-ci.org/ec4j/ec4j/jobs/457960159

[ppalaga]

ec4j is passing with eccf9d4. This PR is good to merge.

[xuhdev]

Great work! Thanks!