Ranges.org

Sometimes a particular configuration might be looking for a range of parameter values relative to an external tool version rather than just exact matches to the parameter. This can still be done with tasty-sugar but requires some extra handling. Here is an example, based the actual version of clang that is available to test against.

CUBE {
  validParams : [ ("llvm-ver": Just ["pre-clang12", "pre-clang13" ]) ]
}

In this example, if the tool version is clang-10, it’s desireable to only run one test for an un-parameterized good file and not one for each assumed parameter value.

For specifying upper bounds, we will consider the “supremum” to be the maximum upper bound (here, "pre-clang13").

The general configuration is that tasty-sugar will call the “mkTest” callback with various combinations. The mkTest will first filter based on the parameter matches, then it will check against the actual version of the tool available before finally performing a generate_test to create the test, or skip to skip creating a test.

tasty-sugar --> filter --> check --> generate_test
                   \         \
                    `--------------> skip

The filter will allow any Explicit match, but only an Assumed match for the supremum.

Case 1 - weaker match only

Files: test_1.good

tasty-sugar call	expfile	“llvm-ver” param	filter	check	result
1	test_1.good	Assumed “pre-clang12”	not supremum	-	skip
2	test_1.good	Assumed “pre-clang13”	supremum, OK	clang-10	generate_test
				clang-12	generate_test
				clang-13	skip
				clang-15	skip

Case 2 - explicit match only

Files: test_2.pre-clang12.good

tasty-sugar call	expfile	“llvm-ver” param	filter	check	result
1	test_2.pre-clang12.good	Explicit “pre-clang12”	always OK	clang-10	generate_test
				clang-12	skip
				clang-13	skip
				clang-15	skip

There is no more-generic (weaker) expected file to match against the constraints, so the expected number of tests is generated. Note that if there had been other parameters, then there might be a weaker match candidate, but that weaker match is still varied over the other parameter’s value, so those are reasonable distinct test candidate generations.

Case 3 - explicit and a weaker match

Files: test_3.pre-clang12.good, test_3.good

tasty-sugar call	expfile	“llvm-ver” param	filter	check	result
1	test_3.pre-clang12.good	Explicit “pre-clang12”	always OK	clang-10	generate_test
				clang-12	skip
				clang-13	skip
				clang-15	skip
2	test_3.good	Assumed “pre-clang13”	supremum, OK	clang-10	generate_test
				clang-12	skip
				clang-13	skip
				clang-15	skip

Here there is an extra invocation because an explicit match was below the supremum, and both the explicit match and the supremum allow a test to be generated. This also means that the weaker matching expected file is also checked even when there’s an overlap.

One workaround for this is the alternate technique described in Case 3a below.

A separate workaround is that the withSugarGroups callback provides the full Sweets as well as the specific Expectation currently being processed: the filter can be enhanced such that an Assumed on the supremum checks for any Explicit matches of the same parameter (all other params being the same) in the expected list of the Sweets and if one is found, it does not pass this Assumed on to the check phase.

• TODO? On Assumed calls, provide a boolean if there was a stronger match for that same parameter (i.e. provide the filter search)?

Case 3a - explicit and super-supremum instead of weaker match

In this variation, a super-supremum parameter match is used instead of a weaker match:

• The test_3.good file is renamed to a super-value test_3.pre-clang100.good.
• The valid-params now includes this super-value: [ "pre-clang12", "pre-clang13", "pre-clang100" ]
• The filter will not allow any Assumed matches through
• The check is aware of all ranges and checks more specifically.

Files: test_3.pre-clang12.good, test_3.pre-clang100.good

tasty-sugar call	expfile	“llvm-ver” param	filter	clang	check	result
1	test_3.pre-clang12.good	Explicit “pre-clang12”	always OK	10	10 < 12	generate_test
				12	! 12 < 12	skip
				13	! 13 < 12	skip
				15	! 15 < 12	skip
2	test_3.pre-clang100.good	Explicit “pre-clang100”	always OK	10	! 13 <= 10 < 100	skip
				12	! 13 <= 12 < 100	skip
				13	13 <= 13 < 100	generate_test
				15	13 <= 15 < 100	generate_test

Note that there is no call with the pre-clang13 parameter value because there is no expected file that will match that value (either explicitly or via a weaker match where no llvm-ver parameter is specified. If there was a weaker match, then there would be a call with Assumed "pre-clang13" , but this call would be ignored as described above (all Assumed matches are ignored).

The use of a super-supremum rather than a realistic supremum is to allow for future increasing values of the parameter without requiring a rename of existing matching files. The super-supremum match becomes the default match for when there are no other matches.

Case 4

Files: test_4.pre-clang12.good, test_4.pre-clang13.good, test_4.good

tasty-sugar call	expfile	“llvm-ver” param	filter	check	result
1	test_4.pre-clang12.good	Explicit “pre-clang12”	always OK	clang-10	generate_test
				clang-12	skip
				clang-13	skip
				clang-15	skip
2	test_3.pre-clang13.good	Explicit “pre-clang13”	always OK	clang-10	generate_test
				clang-12	generate_test
				clang-13	skip
				clang-15	skip

Here, there is no invocation with the weakest expectation (test_4.good) because all the parameters are explicitly matched, although there are multiple invocations for tool versions that are less than multiple upper bounds. Presumably, the tester intended to use the lowest upper bound that would satisfy, so as with Case 3a, a check that verified against all known ranges rather than just the supremum could be used to handle this.