Validate requests are not impacted during security cache invalidation

[cwperks]

Description

Opening a PR that adds a test to ensure that requests during a cache invalidation succeed and fail if any unsuccessful status code is detected. This is related to adding an automated test for #1961

I'm currently encountering a couple of issues and am able to reproduce a separate concurrency error, but not the OptionalDataException.

This test will fail if both of these PRs are reverted:

• Use Collections.synchronizedSet and Collections.synchronizedMap for r… #1970
• Adds a check to skip serialization-deserialization if request is for same node #2765

When the above 2 PRs are reverted, the test case fails with:

Caused by: org.opensearch.OpenSearchException: Instance User [name=admin, backend_roles=[admin], requestedTenant=null] of class class org.opensearch.security.user.User is not serializable
        at org.opensearch.security.support.Base64CustomHelper.serializeObject(Base64CustomHelper.java:102) ~[main/:?]
        at org.opensearch.security.support.Base64Helper.serializeObject(Base64Helper.java:34) ~[main/:?]
        at org.opensearch.security.transport.SecurityInterceptor.ensureCorrectHeaders(SecurityInterceptor.java:312) ~[main/:?]
        at org.opensearch.security.transport.SecurityInterceptor.sendRequestDecorate(SecurityInterceptor.java:240) ~[main/:?]
        at org.opensearch.security.OpenSearchSecurityPlugin$6$2.sendRequest(OpenSearchSecurityPlugin.java:811) ~[main/:?]
        at org.opensearch.transport.TransportService.sendRequest(TransportService.java:913) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        ... 21 more
Caused by: java.lang.NullPointerException: Cannot invoke "String.length()" because "str" is null
        at org.opensearch.core.common.io.stream.StreamOutput.writeString(StreamOutput.java:443) ~[opensearch-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.core.common.io.stream.StreamOutput.writeCollection(StreamOutput.java:1187) ~[opensearch-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.core.common.io.stream.StreamOutput.writeStringCollection(StreamOutput.java:1199) ~[opensearch-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.security.user.User.writeTo(User.java:263) ~[main/:?]
        at org.opensearch.security.support.Base64CustomHelper.serializeObject(Base64CustomHelper.java:86) ~[main/:?]
        at org.opensearch.security.support.Base64Helper.serializeObject(Base64Helper.java:34) ~[main/:?]
        at org.opensearch.security.transport.SecurityInterceptor.ensureCorrectHeaders(SecurityInterceptor.java:312) ~[main/:?]
        at org.opensearch.security.transport.SecurityInterceptor.sendRequestDecorate(SecurityInterceptor.java:240) ~[main/:?]
        at org.opensearch.security.OpenSearchSecurityPlugin$6$2.sendRequest(OpenSearchSecurityPlugin.java:811) ~[main/:?]
        at org.opensearch.transport.TransportService.sendRequest(TransportService.java:913) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        ... 21 more

This test provides assurances that there are no errors related to concurrency and cache invalidation. I had to update the integrationTest framework to add backendRoles onto the User object in the integration test suite.

• Category (Enhancement, New feature, Bug fix, Test fix, Refactoring, Maintenance, Documentation)

Adds a test

Check List

• New functionality includes testing
• New functionality has been documented
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[cwperks]

To replicate the OptionalDataException you need to revert the 2 changes in the PR description and also change every instance of final boolean useJDKSerialization = connection.getVersion().before(ConfigConstants.FIRST_CUSTOM_SERIALIZATION_SUPPORTED_OS_VERSION); to final boolean useJDKSerialization = true;

• security/src/main/java/org/opensearch/security/transport/SecurityInterceptor.java

  Line 151 in 5de9686

  final boolean useJDKSerialization = connection.getVersion().before(ConfigConstants.FIRST_CUSTOM_SERIALIZATION_SUPPORTED_OS_VERSION);

• security/src/main/java/org/opensearch/security/ssl/transport/SecuritySSLRequestHandler.java

  Lines 89 to 92 in 5de9686

  	threadContext.putTransient(
  	ConfigConstants.USE_JDK_SERIALIZATION,
  	channel.getVersion().before(ConfigConstants.FIRST_CUSTOM_SERIALIZATION_SUPPORTED_OS_VERSION)
  	);

• security/src/test/java/org/opensearch/security/transport/SecurityInterceptorTests.java

  Line 111 in 5de9686

  boolean useJDKSerialization = remoteNodeVersion.before(ConfigConstants.FIRST_CUSTOM_SERIALIZATION_SUPPORTED_OS_VERSION);

[peternied]

Looks great, thanks!