Conformance testing with FIDO-alliance tool #13
Comments
|
I consider "passing" to mean all tests pass the tool check with zero exceptions thrown for the expected pass tests and expected fail tests result in a thrown exception from the library. These set of tests should all be passing: ServerAuthenticatorAttestationResponse-Resp-1 |
|
Latest run:
Note: Strange errors when running Server-ServerAuthenticatorAssertionResponse-Resp-3. |
|
I am not sure I agree with some of the test results compared to the spec. I may open an issue or two against the conformance tool. |
|
Not mine but I'm not the only one. |
|
A bunch more of these should be passing now. However, some of the failure tests are passing not because we are defending properly but because they are hitting other unexpected exceptions. It's tedious to track which ones are "good" pass and which are "bad" pass. |
|
Latest run:
We seem to have some regression, e.g. - ❌ Server-ServerAuthenticatorAssertionResponse-Resp-2 Test server processing CollectClientData. (please note that order of tests was changed between the versions of the compat tool) I agree with you about finding "holes" in the defense where we catch another error in the response before we reach what test is supposed to test for. I guess the best way to completely solve that would be to add test cases in the conformance tool. |
|
I am of the opinion that something is broken in Server-ServerAuthenticatorAssertionResponse-Resp-3. I get the same results you do and the test does not seem to be hitting any interesting code at all. |
|
I've been talking to Ackermann Yuriy and sent images and a error report, hopefully they are able to correct the test. |
|
Lots more should be passing now. Most of Server-ServerAuthenticatorAssertionResponse-Resp-3 is still broken, but it seems that is a known issue now. |
|
@aseigler I'm figuring out the issue with help form Ackerman. Error message seems to be because of the naive way we store credentials, so I'm updating the testcontroller. |
|
@aseigler this branch (https://github.com/abergs/fido2-net-lib/tree/test-fixes) contains a commit that makes the tests pass. It's still a bit ugly and the UV/UP tests seems to fail a lot. I pushed it to a branch because I too eagerly made a change from Raw.Id to Raw.RawId and want to fix that before merging to master but you can use it if you want to run tests. |
|
|
|
I will take a look at the branch when I get a chance. I am quite honestly much more useful in the crypto and server side data structure handling aspects of this project than I am in the client or web aspects, I try to defer as much web related stuff as I can to your expertise. |
|
I agree with the vice-versa @aseigler ;) But feel free to use the branch to run the tests since they work in that branch. |
|
Latest run:
💠 Pending - What does this mean? |
|
@aseigler Do you know what the pending things are? |
|
Functionally, off the top of my head, I know about these:
MDS doesn't seem to have any authenticators registered with aaguid yet, but there is an update supposed to post on 18-Sept that may change that. I am a little baffled by the MDS conformance tests, they do not seem to work like the rest of the tests which are fairly straightforward. |
|
That's good. But I was as actually refering to the tests that report status as "pending" with the blue color. (so not "not passing" or "passing". just "pending") |
|
Ah, OK. Those blue marks coincide with 1 and 2 above. The first X is 3 and 4 is not referenced yet anywhere else. |
|
Android key should pass clean after tonight's PR. I don't see a good way to implement Ed25519 support without taking a dependency on something like https://github.com/CodesInChaos/Chaos.NaCl. There doesn't seem to be native support that I can find (and I looked, hard). |
|
With the metadata handling code from #42 minus the outstanding issues fido-alliance/conformance-test-tools-resources#395 and fido-alliance/conformance-test-tools-resources#396, and without figuring out EdDSA support, we should be clean for all 4 authenticator assertion and attestation tests. |
|
After #43 we should be clean for all server tests aside from metadata service tests, for all algorithms except ED25519. Please verify independently. |
|
🎉 100% test passes with all optional algorithms (except ED25519) This calls for celebration. @aseigler an incredible well done job! |
|
I have a fix for EdDSA support I am testing here aseigler@e61e8fb and here aseigler@5743c4b. |
|
Need help with Metadata Service tests |
|
There are links to the root and issuing CRLs here: https://fidoalliance.org/metadata/ But there is no in box way to use CRL file with X509Certificate2, and I haven't gotten the cycles to write out something that will fill the gap. |
|
How can I pass the case: Already known that I have to use metadata for this case, but there is no suitable entry from metadata. |
|
You have to use the test metadata from the link in the conformance tool. See the custom metadata load stuff in the metadata code and the packed attestation code that interacts with metadata to see how that test was covered. |
|
I finally figured out how to pass ALL of the conformance tests, including the metadata service tests. You get a snazzy "submit results" button! I will document the metadata service test pass process so that it can be repeated by anyone. You do NOT need an MDS access key to pass the metadata service tests.
|
|
Really great news! 😃 |
Will update this issues with testing progress
The text was updated successfully, but these errors were encountered: