Make test-suite runnable under FIPS JVM #18491

beanuwave · 2025-06-11T08:36:56Z

Description

Makes the required changes to build and test under FIPS-140-3 compliance support. FIPS mode can be activated by adding the -Pcrypto.standard=FIPS-140-3 Gradle parameter.

NOTE:

This PR includes all remaining changes from the original PR#14912
Due to the netty FIPS reflection bug in v4.1.125.Final and v4.1.126.Final, the :plugins:transport-reactor-netty4:test task will fail. To run properly, netty needs to be updated to a newer version, such as v4.1.127.Final. Not relevant after update to netty 4.2.7

Related Issues

Resolves RFC

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2025-06-11T09:41:17Z

❌ Gradle check result for 9b5da5c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-07-07T12:36:21Z

❌ Gradle check result for 986dce7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

terryquigleysas · 2025-07-14T10:36:42Z

I'd like to raise a general point here to keep in mind with this development, especially as instructions will be required for Java setups other than the bundled version. The Red Hat JDK 21, for example, has a default of fips.keystore.type: PKCS12 - see https://docs.redhat.com/en/documentation/red_hat_build_of_openjdk/21/html/configuring_red_hat_build_of_openjdk_21_on_rhel_with_fips/fips_settings#fips_settings .

We'd like to ensure that code checks aren't so stringent as to prevent this setup from working.

github-actions · 2025-07-14T14:42:29Z

❌ Gradle check result for 939e6b5: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-07-15T11:21:52Z

❌ Gradle check result for 11da667: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-07-15T11:31:38Z

❌ Gradle check result for 9a387a4: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-07-15T12:12:16Z

❌ Gradle check result for 4e0af75: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-07-15T12:43:23Z

❌ Gradle check result for 9efd838: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-07-15T13:49:30Z

❌ Gradle check result for 4fc6b40: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-07-15T15:10:51Z

❌ Gradle check result for 0139eaa: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-07-15T17:54:34Z

❌ Gradle check result for f52e720: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

beanuwave · 2025-07-16T12:21:25Z

The Red Hat JDK 21, for example, has a default of fips.keystore.type: PKCS12

@terryquigleysas Thank you for pointing out those limitations.

We'd like to ensure that code checks aren't so stringent as to prevent this setup from working.

We rely on SunPKCS12 provider to load the JVM's default truststore. In case of OpenJKD the default type is the same as RHEL's - so nothing changes for us.

terryquigleysas · 2025-07-16T13:28:01Z

The Red Hat JDK 21, for example, has a default of fips.keystore.type: PKCS12

@terryquigleysas Thank you for pointing out those limitations.

We'd like to ensure that code checks aren't so stringent as to prevent this setup from working.

We rely on SunPKCS12 provider to load the JVM's default truststore. In case of OpenJKD the default type is the same as RHEL's - so nothing changes for us.

Good news. Thank you for the reply. Much appreciated!

github-actions · 2025-08-01T16:25:22Z

❌ Gradle check result for cb7949d: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-08-04T12:12:41Z

❌ Gradle check result for 0680de8: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-08-04T13:20:27Z

❌ Gradle check result for 732e412: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

reta · 2025-11-07T15:42:31Z

...terTest/java/org/opensearch/discovery/azure/classic/AzureDiscoveryClusterFormationTests.java

-                (PrivilegedAction<Version>) () -> Version.parse(System.getProperty("java.version"))
-            );
-            if (full.compareTo(Version.parse("12.0.1")) < 0) {
+        if (!inFipsJvm()) {


Not related, but our baseline is JDK-21, we could clean up this code

I've reverted my changes to this plugin. Azure classic is generally not worth trying to make it FIPS-compliant. We should keep it as is - unsupported.

reta · 2025-11-07T15:45:38Z

.../repository-gcs/src/main/java/org/opensearch/repositories/gcs/GoogleCloudStorageService.java

+            if (Security.getProvider("BCFIPS") != null) {
+                certTrustStore = KeyStore.getInstance("BCFKS");
+                InputStream keyStoreStream = getClass().getResourceAsStream("/google.bcfks");
+                SecurityUtils.loadKeyStore(certTrustStore, keyStoreStream, "notasecret");


I believe we definitely should not hardcode password here, please have a setting for it

Let's make a setting for SSL trust as well as it's passphrase w/o a fallback and use this google.bcfks store only for tests.

reta · 2025-11-07T15:56:55Z

@cwperks @reta I believe this PR is ready for review - can you take a look please?

Thanks @beanuwave , sorry haven't looked yet but will do this week, thank you

Thanks @beanuwave , did a first pass, a few minor comments but I have two major concerns:

Incredible amount of "if" logic related to FIPS mode checks (and its variations like assumeFalse). I would like to suggest to split test cases in XxxBaseTest and XxxNoFipsTest extends XxxBaseTest (the naming is not very important). With this setup, we could easily exclude *NoFipsTest* from test runs that use FIPS mode (and XxxBaseTest would be run in any mode)
We now have to run all test suites in 2 modes - w/o FIPS. AFAICT this is not done yet, what is your plan here?

Thank you.

beanuwave · 2025-11-07T17:13:30Z

We now have to run all test suites in 2 modes - w/o FIPS. AFAICT this is not done yet, what is your plan here?

I believe @cwperks suggested creating a new GitHub workflow that supports both scheduled runs and PR comments.

…or TYPE_TO_EXTENSION_MAP; revert changes to SecureSettingsHelpers.getSecureSettingsProvider; revert changes to AzureClassic Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]>

Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]>

reta · 2025-11-14T15:30:54Z

I believe @cwperks suggested creating a new GitHub workflow that supports both scheduled runs and PR comments.

Sorry, a bit late here, I think this is in general viable option, but:

we need to make sure out build scripts are adapted (we build on Jenkins, not GA)
we need to run the these checks in parallel

Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]>

… google.p12 in non-FIPS mode and an exception in FIPS mode when no truststore is set up. Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]>

github-actions · 2025-11-17T12:21:56Z

❌ Gradle check result for 07c6358: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

cwperks · 2025-11-17T14:55:14Z

We now have to run all test suites in 2 modes - w/o FIPS. AFAICT this is not done yet, what is your plan here?

I believe @cwperks suggested creating a new GitHub workflow that supports both scheduled runs and PR comments.

What I was thinking was something similar to benchmark-pull-request where the benchmarks are run ad hoc

To trigger the run requires leaving a comment on a PR. Example: Allow collectors take advantage of preaggregated data using collectRange API #20009 (comment)
This would then open an issue where a maintainer can approve to run the workflow. Example: Request to approve/deny benchmark run for PR #19573 #19969

@peterzhuamazon @rishabh6788 Would you be able to help us with setup for such a workflow?

Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]>

github-actions · 2025-11-17T16:25:19Z

✅ Gradle check result for 826d92f: SUCCESS

Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]>

…oreUtils Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]>

github-actions · 2025-11-19T15:13:00Z

✅ Gradle check result for f3f0880: SUCCESS

beanuwave · 2025-11-19T15:14:47Z

@reta @cwperks The earlier comments have been resolved. Whenever it’s convenient, a second review would be appreciated.

cwperks

Thank you for the diligence on this PR @beanuwave! With these changes it greatly expands the test coverage when running in FIPS approved mode.

@peterzhuamazon what would be required to add another gradle check to run with the FIPS approved setting? i.e. ./gradlew check -Pcrypto.standard=FIPS-140-3

reta · 2025-11-22T17:53:43Z

...etty4/src/test/java/org/opensearch/http/netty4/ssl/SecureNetty4HttpServerTransportTests.java

+            final KeyStore keyStore = KeyStore.getInstance(keyStoreType, jcaProvider);
+
+            keyStore.load(
+                SecureNetty4HttpServerTransportTests.class.getResourceAsStream("/netty4-server-keystore" + fileExtension),


try (InputStream in = SecureNetty4HttpServerTransportTests.class.getResourceAsStream("/netty4-server-keystore" + fileExtension) { keyStore.load(in, PASSWORD); }

reta · 2025-11-22T17:54:58Z

...etty4/src/test/java/org/opensearch/http/netty4/ssl/SecureNetty4HttpServerTransportTests.java

+
+    // Cached factory to avoid repeated keystore loading (especially slow in FIPS mode with BCFKS).
+    // TODO: Replace with @BeforeAll when migrating to JUnit 5.
+    private static volatile KeyManagerFactory cachedKeyManagerFactory;


Why we cannot use @BeforeClass (JUnit 4) here?

reta · 2025-11-22T17:56:55Z

...tty4/src/test/java/org/opensearch/transport/netty4/ssl/SimpleSecureNetty4TransportTests.java

+
+    // Cached factories to avoid repeated keystore loading (especially slow in FIPS mode)
+    // TODO: Replace with @BeforeAll when migrating to JUnit 5.
+    private static volatile KeyManagerFactory cachedServerKeyManagerFactory;


Same question please, @BeforeClass should be sufficient

reta · 2025-11-22T17:58:32Z

...tty4/src/test/java/org/opensearch/transport/netty4/ssl/SimpleSecureNetty4TransportTests.java

-
-                    SSLEngine engine = SslContextBuilder.forServer(keyManagerFactory)
+                return Optional.of(
+                    SslContextBuilder.forServer(getServerKeyManagerFactory())


Don't we need to provide trustManager here as well?

reta · 2025-11-22T18:02:56Z

...st/java/org/opensearch/repositories/gcs/GoogleCloudStorageBlobContainerRetriesFipsTests.java

+import static org.opensearch.repositories.gcs.GoogleCloudStorageClientSettings.TRUSTSTORE_PATH_SETTING;
+import static org.opensearch.repositories.gcs.GoogleCloudStorageClientSettings.TRUSTSTORE_TYPE_SETTING;
+
+@SuppressForbidden(reason = "use a http server")


Sorry, why do we need @SuppressForbidden here?

reta · 2025-11-22T18:07:07Z

test/framework/src/main/java/org/opensearch/test/KeyStoreUtils.java


+    // Cached keystore to avoid expensive cert generation on every test (especially slow in FIPS mode)
+    // TODO: Replace with @BeforeAll when migrating to JUnit 5.
+    private static volatile KeyStore cachedServerKeyStore;


Please use @BeforeClass

reta · 2025-11-22T18:09:32Z

gradle/fips.gradle

        testTask.systemProperty 'javax.net.ssl.trustStoreProvider', 'BCFIPS'
        testTask.systemProperty 'javax.net.ssl.trustStorePassword', 'changeit'
+
+        // Exclude base test classes that have corresponding FipsTests or FipsIT


Should we just rename these tests to:

XxxNoFipsTests.java XxxNoFipsIT.java

?

reta · 2025-11-22T18:21:39Z

libs/ssl-config/src/main/java/org/opensearch/common/ssl/KeyStoreUtil.java

 */
 final class KeyStoreUtil {

+    public static final Supplier<Boolean> inFipsMode = () -> {


Why do need Supplier here? The value of the FIPS mode will not change, we need to capture it only once:

public static final boolean IN_FIPS_MODE = isInFipsMode(); private static boolean isInFipsMode() { try { // Equivalent to: boolean approvedOnly = CryptoServicesRegistrar.isInApprovedOnlyMode() var registrarClass = Class.forName("org.bouncycastle.crypto.CryptoServicesRegistrar"); var isApprovedOnlyMethod = registrarClass.getMethod("isInApprovedOnlyMode"); return (Boolean) isApprovedOnlyMethod.invoke(null); } catch (ReflectiveOperationException e) { return false; } }

reta · 2025-11-22T18:24:27Z

libs/ssl-config/src/main/java/org/opensearch/common/ssl/KeyStoreUtil.java

        }
        try {
-            KeyStore keyStore = KeyStore.getInstance(type);
+            if (CryptoServicesRegistrar.isInApprovedOnlyMode() && !FIPS_COMPLIANT_KEYSTORE_TYPES.contains(type)) {


So why we are calling CryptoServicesRegistrar directly here but using reflection above (inFipsMode)? If we do not require CryptoServicesRegistrar to be present on the classpath, we should use IN_FIPS_MODE , otherwise we should just use CryptoServicesRegistrar.isInApprovedOnlyMode()

reta · 2025-11-22T18:25:43Z

@reta @cwperks The earlier comments have been resolved. Whenever it’s convenient, a second review would be appreciated.

Thanks a lot @beanuwave , it looks great, few quite minor comments but awesome work by all means! Thank you!

opensearch-ci-bot mentioned this pull request Jun 11, 2025

[AUTOCUT] Gradle Check Flaky Test Report for RecoveryWhileUnderLoadIT #14509

Open

beanuwave force-pushed the fips_compliance3 branch from 939e6b5 to 11da667 Compare July 15, 2025 11:17

beanuwave force-pushed the fips_compliance3 branch from 11da667 to 9a387a4 Compare July 15, 2025 11:26

beanuwave force-pushed the fips_compliance3 branch from 9a387a4 to 4e0af75 Compare July 15, 2025 12:07

beanuwave force-pushed the fips_compliance3 branch from 4e0af75 to 9efd838 Compare July 15, 2025 12:20

beanuwave force-pushed the fips_compliance3 branch from 9efd838 to 4fc6b40 Compare July 15, 2025 13:22

beanuwave force-pushed the fips_compliance3 branch from 4fc6b40 to 0139eaa Compare July 15, 2025 14:58

beanuwave force-pushed the fips_compliance3 branch from 0139eaa to f52e720 Compare July 15, 2025 15:51

beanuwave force-pushed the fips_compliance3 branch from f52e720 to cb7949d Compare August 1, 2025 14:22

opensearch-ci-bot mentioned this pull request Aug 1, 2025

[AUTOCUT] Gradle Check Flaky Test Report for FullRollingRestartIT #18490

Closed

beanuwave force-pushed the fips_compliance3 branch from cb7949d to 0680de8 Compare August 4, 2025 11:47

beanuwave changed the title ~~remaining changes from 14912~~ Make test-suite runnable under FIPS JVM Aug 4, 2025

beanuwave force-pushed the fips_compliance3 branch from 0680de8 to 732e412 Compare August 4, 2025 13:15

beanuwave force-pushed the fips_compliance3 branch from 732e412 to 6164088 Compare August 4, 2025 13:36

reta reviewed Nov 7, 2025

View reviewed changes

iigonin and others added 2 commits November 13, 2025 10:45

Separate FIPS-specific tests into dedicated FipsTests classes

10bec59

Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]>

iigonin and others added 3 commits November 14, 2025 17:32

Merge branch 'main' into fips_compliance3

5a83e64

fix spotlessCheck

f8cd57b

Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]>

Add the @after annotation; regenerate all test certs with SANs included

826d92f

Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]>

iigonin and others added 3 commits November 19, 2025 14:49

Optimize FIPS test performance by caching keystores and SSL factories

e788f13

Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]>

add required FIPS deps

b27a9d3

Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]>

fix forbiddenApisTest & use RSA 2048 alg to generate certs with KeySt…

f3f0880

…oreUtils Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]>

cwperks approved these changes Nov 20, 2025

View reviewed changes

reta reviewed Nov 22, 2025

View reviewed changes

Make test-suite runnable under FIPS JVM #18491

Are you sure you want to change the base?

Make test-suite runnable under FIPS JVM #18491

Conversation

beanuwave commented Jun 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

github-actions bot commented Jun 11, 2025

Uh oh!

github-actions bot commented Jul 7, 2025

Uh oh!

terryquigleysas commented Jul 14, 2025

Uh oh!

github-actions bot commented Jul 14, 2025

Uh oh!

github-actions bot commented Jul 15, 2025

Uh oh!

github-actions bot commented Jul 15, 2025

Uh oh!

github-actions bot commented Jul 15, 2025

Uh oh!

github-actions bot commented Jul 15, 2025

Uh oh!

github-actions bot commented Jul 15, 2025

Uh oh!

github-actions bot commented Jul 15, 2025

Uh oh!

github-actions bot commented Jul 15, 2025

Uh oh!

beanuwave commented Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

terryquigleysas commented Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Aug 1, 2025

Uh oh!

github-actions bot commented Aug 4, 2025

Uh oh!

github-actions bot commented Aug 4, 2025

Uh oh!

reta Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

reta commented Nov 7, 2025

Uh oh!

beanuwave commented Nov 7, 2025

Uh oh!

reta commented Nov 14, 2025

Uh oh!

github-actions bot commented Nov 17, 2025

Uh oh!

cwperks commented Nov 17, 2025

Uh oh!

github-actions bot commented Nov 17, 2025

Uh oh!

github-actions bot commented Nov 19, 2025

Uh oh!

beanuwave commented Nov 19, 2025

Uh oh!

cwperks left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

beanuwave commented Jun 11, 2025 •

edited

Loading

beanuwave commented Jul 16, 2025 •

edited

Loading

terryquigleysas commented Jul 16, 2025 •

edited

Loading

reta Nov 7, 2025 •

edited

Loading

reta Nov 22, 2025 •

edited

Loading

reta commented Nov 22, 2025 •

edited

Loading