Remove test harnesses, defunct tooling

[cpu]

This PR removes the Java and Javascript test harnesses as well as associated defunct tooling and leftover Google-specific CI bits.

What remains are the bits #105 convincingly articulates as the future of the repository: JSON test vectors (testvectors/, testvectors_v1/), the associated schemas (schemas/) and supporting documentation (doc/).

Resolves #105 (and a variety of related issues that can be closed manually after this is merged).

[bleichenbacher-daniel]

I disagree with the deletion of the tests. A significant fraction of the code contains tests that are not covered by test vectors.
What should happen instead is to move the java and javascript tests into a separate component, so that the remainder of the repo does not depend on them.

[cpu]

A significant fraction of the code contains tests that are not covered by test vectors.

I think the best way to address that is to add new JSON vectors for what's missing.

What should happen instead is to move the java and javascript tests into a separate component, so that the remainder of the repo does not depend on them.

I believe that's best done in a separate repository. Is there a reason you think that approach would be unmanageable?

[bleichenbacher-daniel]

A significant fraction of the code contains tests that are not covered by test vectors.

I think the best way to address that is to add new JSON vectors for what's missing.

There are a lot of tests that cannot be translated into test vectors. Prominent test are for timing attacks,
bias in ECDSA/DSA signatures, potentially weak interfaces such as CipherInputStream and CipherOutputStream in java,
edge cases such as encrypting long messages with GCM, etc.

A main issue with the current code is that it uses a particular framework, bazel and junit, which makes it difficult to integrate
the tests into some other environment. There is already some work ongoing trying to achieve this goal.

What should happen instead is to move the java and javascript tests into a separate component, so that the remainder of the repo does not depend on them.

I believe that's best done in a separate repository. Is there a reason you think that approach would be unmanageable?

I think this is what I just proposed.

[cpu]

There are a lot of tests that cannot be translated into test vectors. Prominent test are for timing attacks,
bias in ECDSA/DSA signatures, potentially weak interfaces such as CipherInputStream and CipherOutputStream in java,
edge cases such as encrypting long messages with GCM, etc.

That makes sense, but I think those are great examples of things that end up being tightly coupled to specific implementations, requiring both significant maintenance and per-project/language expertise. My perspective is that this repository is better used as a place to house test data that isn't tied to any one implementation or API.

I believe that's best done in a separate repository. Is there a reason you think that approach would be unmanageable?

I think this is what I just proposed.

Ah ok! It sounds like we're on the same page then:

1. Remove the test harnesses/supporting code from here
2. Add it back in a separate repository

Step 2 can be done by whoever finds the removed pieces valuable and wants to own maintaining them. It doesn't need to block Step 1 since the history remains in-repo.

[FiloSottile]

I think we are all on the same page that 1) dynamic tests are valuable and do increase coverage, and 2) they are different enough from test vectors that they belong in a different repository. An important part of (2) is that the expertise and resources necessary to maintain them are different, and we don't feel like we're the right set of people to do that.

The LICENSE file is very clear, so anyone can take the tests and maintain them in a different repository.

[FiloSottile]

P.S. I would love to find portable ways to test at least some of these things with vectors. For example,

bias in ECDSA/DSA signatures

can be tested with vectors if the implementation uses draft-irtf-cfrg-det-sigs-with-noise-05 like we do in Go

edge cases such as encrypting long messages with GCM

can be tested with vectors if the input can be described like "generate X bytes from SHAKE" and the output can be hashed. We have such a test for cSHAKE in Go, and I'd love to port more tests like this to Wycheproof.

https://github.com/golang/go/blob/24b395119b4df7f16915b9f01a6aded647b79bbd/src/crypto/sha3/sha3_test.go#L374-L383

[FiloSottile]

Filed #134 to discuss strategies for large vectors.

[cpu]

I think there are a handful of issues we could close now that the harnesses are removed. I'll close those today.