Transient CI failures for target S2nIntegrationV2SmallBatch

[torben-hansen]

Security issue notifications

If you discover a potential security issue in s2n we ask that you notify
AWS Security via our vulnerability reporting page. Please do not create a public github issue.

Problem:

I see some transient CI failures in the CI target S2nIntegrationV2SmallBatch. One example is

• Build batch
• Build

FAILED test_well_known_endpoints.py::test_well_known_endpoints[KMS-PQ-TLS-1-0-2020-07-S2N-www.apple.com-TLS1.2]
======================== 1 failed, 431 passed in 24.27s ========================
ERROR: InvocationError for command /codebuild/output/src981233478/src/github.com/aws/s2n-tls/tests/integrationv2/.tox/py39/bin/pytest -n 2 --cache-clear -rpfsq --provider-version=openssl-1.0.2 --fips-mode=0 --no-pq=0 test_well_known_endpoints.py (exited with code 1)

There is some weak consistency in failures:

• I mostly see failures for test_well_known_endpoints.py, but it can vary.
• Mostly failures for PQ cipher suites, but can vary.
• For endpoints I have seen a mix of: apple.com, att.com, microsoft.com, ebay.com

There doesn't appear to be any correlation with the libcrypto used. I have seen failures for almost all of them: openssl 1.0.2, openssl 1.1.1, aws-lc, awslc-fips.

Here are some failures for the past two weeks or so:

• libcrypto = openssl 1.0.2, endpoint = apple.com, test script = test_well_known_endpoints.py: build batch build
• libcrypto = openssl 1.0.2, endpoint = ebay.com, test script = test_well_known_endpoints.py: build batch build
• libcrypto = openssl 1.0.2, endpoint = att.com, test script = test_well_known_endpoints.py: build batch build
• libcrypto = openssl 1.1.1, endpoint = att.com, test script = test_well_known_endpoints.py: build batch build
• libcrypto = aws-lc, test script = test_cross_compatibility.py: (internal) build batch build
• libcrypto = awslc-fips, endpoint = att.com, test script = test_well_known_endpoints.py: (internal) build batch build

Solution:

A description of the possible solution in terms of S2N architecture. Highlight and explain any potentially controversial design decisions taken.

• Does this change what S2N sends over the wire? If yes, explain.
• Does this change any public APIs? If yes, explain.
• Which versions of TLS will this impact?

Requirements / Acceptance Criteria:

What must a solution address in order to solve the problem? How do we know the solution is complete?

• RFC links: Links to relevant RFC(s)
• Related Issues: Link any relevant issues
• Will the Usage Guide or other documentation need to be updated?
• Testing: How will this change be tested? Call out new integration tests, functional tests, or particularly interesting/important unit tests.
  • Will this change trigger SAW changes? Changes to the state machine, the s2n_handshake_io code that controls state transitions, the DRBG, or the corking/uncorking logic could trigger SAW failures.
  • Should this change be fuzz tested? Will it handle untrusted input? Create a separate issue to track the fuzzing work.

Out of scope:

Is there anything the solution will intentionally NOT address?

[torben-hansen]

Another error: https://us-west-2.codebuild.aws.amazon.com/project/eyJlbmNyeXB0ZWREYXRhIjoiZUJWY1BUOFREMUNLZnQ1cGdsdmVGbmdGT243Z0VlOHRCOGRGRzBweTdRWVBodUpKUE9SMWZvU1lwdTJNazBFZWsvekJIekNVS2liOWRCV0Ixa0JTTUM0cWkvdWpvZUZYVVp5TjJVZHo3ekJqRzI3RCIsIml2UGFyYW1ldGVyU3BlYyI6InVVYnllYSthVDU1T2lMUkYiLCJtYXRlcmlhbFNldFNlcmlhbCI6MX0%3D/build/e69d3f50-bf1c-4cbe-be9e-773a74c5f8b5

And here: https://us-west-2.codebuild.aws.amazon.com/project/eyJlbmNyeXB0ZWREYXRhIjoiZUJWY1BUOFREMUNLZnQ1cGdsdmVGbmdGT243Z0VlOHRCOGRGRzBweTdRWVBodUpKUE9SMWZvU1lwdTJNazBFZWsvekJIekNVS2liOWRCV0Ixa0JTTUM0cWkvdWpvZUZYVVp5TjJVZHo3ekJqRzI3RCIsIml2UGFyYW1ldGVyU3BlYyI6InVVYnllYSthVDU1T2lMUkYiLCJtYXRlcmlhbFNldFNlcmlhbCI6MX0%3D/build/2447a825-707c-4ca4-b998-baa03a2aaaf8

[torben-hansen]

And here: https://us-west-2.codebuild.aws.amazon.com/project/eyJlbmNyeXB0ZWREYXRhIjoiZUJWY1BUOFREMUNLZnQ1cGdsdmVGbmdGT243Z0VlOHRCOGRGRzBweTdRWVBodUpKUE9SMWZvU1lwdTJNazBFZWsvekJIekNVS2liOWRCV0Ixa0JTTUM0cWkvdWpvZUZYVVp5TjJVZHo3ekJqRzI3RCIsIml2UGFyYW1ldGVyU3BlYyI6InVVYnllYSthVDU1T2lMUkYiLCJtYXRlcmlhbFNldFNlcmlhbCI6MX0%3D/build/62f3095f-d244-4be1-9086-572f5d2af899

[jmayclin]

more errors, for well-known-endpoint Amazon on TLS 1.0

__________ test_well_known_endpoints[None-S2N-www.amazon.com-TLS1.0] ___________

Command '['s2nc', '--non-blocking', '-e', '-T', '-f', 
'../pems/trust-store/ca-bundle.trust.crt', '-c', 'test_all_tls12', 
'--enter-fips-mode', 'www.amazon.com', '443']' timed out after 5 seconds
 s2nc --non-blocking -e -T -f ../pems/trust-store/ca-bundle.trust.crt -c
 test_all_tls12 --enter-fips-mode www.amazon.com 443

[jmayclin]

Actually I think I'm gonna close out this issue and open a new one because it seems like failure behavior has changed significantly.

The only false negatives that I see on S2nIntegrationV2SmallBatch are the the well-known-endpoint failures, specifically amazon failures.

[jmayclin]

Closing this issue out, new issue is #3999