Expand test coverage to all 100 NIST KAT values

[dstebila]

Currently our KAT tests only test against the first KAT, but most NIST submissions include (if memory serves) 100 values. We could do all of these.

[baentsch]

We could do all of these.

The environment wouldn't thank us if we'd do it in CI (mostly wasted CPU cycles). As running it at least once does make sense, though, I'd suggest adding this at most to the weekly build/regression test, possibly only to a test script to be run manually, e.g. at release.

[dstebila]

Agreed; weekly would certainly suffice.

[SWilson4]

Bumping this discussion to note that I recently encountered a subtle bug when patching HQC that only affected two of the 100 KATs. The PQClean automated tests all pass on the buggy version, since they also check only the first KAT.

[baentsch]

The PQClean automated tests all pass on the buggy version, since they also check only the first KAT.

Doesn't sound good. What about raising this issue then to one to be resolved in release "0.9.0"? If the goal is to have a high certainty that release 0.9.0 indeed has proper(ly tested) implementations for all NIST finishers and R4 candidates shouldn't this issue then be resolved before that release?

[dstebila]

It certainly would be nice. I seem to recall that when we thought about doing this several years ago the performance was rather problematic, but maybe most of the remaining algorithms it should be fine.

I don't think I'd hold the 0.9.0 release for it, but it would be good to include in our next release.

[SWilson4]

I can take a look at expanding the kat_kem and kat_sig commands to check against all 100 KAT values. e.g. by adding an optional --full flag, and do some preliminary testing to see how feasible it is to run for all (or most, or some) of our supported algorithms.