Support cases where CRS is not supported

[allanfrankwork]

Hi

As mentioned here #185 the tests fail if the service tested does not support CRS. As CRS support is not mandatory for a OGC API Feature service, then I think the CRS tests should be skipped. This way only if a service supports CRS the CRS tests will run.

Fixes #185.

Thanks.

[dstenger]

Thank you for providing the pull request.

This questing targets more in the direction how the test result is organized.
If you take a look at the HTML report, Core is listed under Core conformance classes which means that just this conformance class must be passed to get following result in the HTML report Passed Core conformance classes (Implementations passing these classes can be certified): Yes.
Coordinate Reference Systems by Reference, on the other hand, is not listed under Core conformance classes and thus not mandatory to get certified.

If you take the XML or EARL report, it is also possible to just consider the Core tests:

• XML: Just consider content of <test name="Core">.
• EARL: Check /rdf:RDF/cite:TestRun/cite:areCoreConformanceClassesPassed

I am more in favor of keeping the mechanism that the CRS tests fail as skipped tests could also mean pass/ no error.

I will discuss this in the CITE team and come back to you.

[allanfrankwork]

Hi

I'm not exactly sure which html report you are referring to but the emailable-report.html report also shows the problem I'm facing:

Without change my service reports as:

With this change my service reports as:

Its the # failed I'm trying to avoid. In my opinion failed > 0 should mean that there is something wrong with the service.

I know that I can just ignore the CRS part and as stated in #185 a workaround could be to change the testng.xml.

[dstenger]

This is the report generated by TestNG. There is also a custom HTML report generated by TEAM Engine.
Still, the TestNG report states that 0 tests are failing for Core. Shouldn't that solve you use case?

Can you please describe which file you use in your integration test?

[allanfrankwork]

We execute

java -cp '/usr/local/tomcat/webapps/teamengine/WEB-INF/lib/:/usr/local/tomcat/lib/' -DTE_BASE=config/ org.opengis.cite.ogcapifeatures10.TestNGController config/test-run-props.xml --generateHtmlReport true

and then do

grep -q "<testng-results.*failed="0"" testng-results.xml

Which of course fails if failed is not 0 and since the service is compliant but does not implement CRS I do not think this is correct.

Usually we do not look at the html report - only if the above grep makes the pipeline fail then we look at the html report for what to correct (its usually pretty clear from the testng tests already).

I cannot see that I can find results from just the Core part - that does not seem to be in the testng-results.xml file.

[dstenger]

Discussion with CITE team (2022-06-27):
The CITE team decided to accept the pull request as this mechanism is also fine for compliance certification.

These changes will be part of the next release of ets-ogcapi-features10. There is no date scheduled yet.

[allanfrankwork]

Thanks, that sounds good.

Looking forward to its release.