Running tests against a non CRS service results in test failure.

[allanfrankwork]

Describe the bug
We are using these excellent tests to test our OGC API Features service and use this as part of our build pipeline to ensure that our API conforms to the OGC API Features standard. Since the CRS tests have been added to this suite this is failing since we are not conforming to the CRS requirements class (https://docs.ogc.org/is/18-058/18-058.html). We have no plans to ever support CRS beyond the default https://www.opengis.net/def/crs/OGC/1.3/CRS84.

There already is functionality in the tests to enable/disable the tests depending on if the CRS requirements class is in the conformsTo section, however this test fails (so subsequent CRS tests are not executed) and it therefore looks like we are not passing the tests.

Is there a good way of disabling CRS tests - it looks like this is done automatically using the CrsRequirementClassPrecondition test, but that does result in one failed test.

Workaround
We could use a docker image that was built before CRS tests were applied, but that would limit us from any other improvements.

We could also just accept one failing test, but that sounds like that could hide a real bug in future.

To Reproduce
Run tests against a service that does not list CRS as a requirements class.

Expected behavior
CRS tests are skipped and no failurs are listed.

Screenshots
This is the resulting output of us running our tests:

Additional context
Thanks for this test suite, this is most excellent.

[allanfrankwork]

I tried just now to add the if(!requirementClassIsImplemented) statement part and this causes expected behaviour for me:

public void verifyRequirementClass() {
    boolean requirementClassIsImplemented = this.requirementClasses != null
                                            && this.requirementClasses.contains( CRS );

    if (!requirementClassIsImplemented) {
        throw new SkipException("Skipping the test case");
    }

    assertTrue( requirementClassIsImplemented, "Requirement class " + CRS.name()
                                               + " is not supported by the test instance. Tests will be skipped." );
}

Would this be acceptable? If so I can supply a PR on this.

[dstenger]

Thank you for reporting.

May I ask what interface you are using to execute the tests (REST, Web Browser, CLI)?

There are several solutions solving your problem. You could also edit the testng.xml and remove the CRS tests there.
We will discuss in the CITE team which solution we prefer.

[allanfrankwork]

CLI.

Its part of our build pipeline and using the docker image we execute:

java -cp '/usr/local/tomcat/webapps/teamengine/WEB-INF/lib/:/usr/local/tomcat/lib/' -DTE_BASE=config/ org.opengis.cite.ogcapifeatures10.TestNGController config/test-run-props.xml --generateHtmlReport

where config/test-run-props.xml has been generated to have the builds endpoint.

We then check the testng-results.xml file for errors.

Agree that editing testng.xml would also be an option but that still has a potential of us not executing new tests being added in coming releases of this test suite.

[tomkralidis]

I'm having the same issue testing pygeoapi. The ETS should skip CRS testing if it is not specified as a conformance class by the server.

[allanfrankwork]

We are currently pinning the docker image to ogccite/ets-ogcapi-features10:1.1-teamengine-5.4, while we wait for what the preferred solution is. For us the proposed SkipException solution would work fine - programatically editing the testng.xml file to exclude CRS tests seems like hard to get right assuming that there could be other changes to this file.

[dstenger]

Please also see #190 (comment).