FIPS - Failing to parse PEM key in full-cluster-restart QA test

[tvernum]

In 7.3 running BWC against 6.8.3

• https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.3+matrix-java-periodic/ES_BUILD_JAVA=openjdk12,ES_RUNTIME_JAVA=java8fips,nodes=general-purpose/80/

The test times out waiting for the ports file, but the node has actually failed with:

java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:614) ~[elasticsearch-6.8.3-SNAPSHOT.jar:6.8.3-SNAPSHOT]
..
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
..
Caused by: java.lang.IllegalStateException: Error parsing Private Key from: /var/lib/jenkins/workspace/elastic+elasticsearch+7.3+matrix-java-periodic/ES_BUILD_JAVA/openjdk12/ES_RUNTIME_JAVA/java8fips/nodes/general-purpose/x-pack/qa/full-cluster-restart/build/cluster/v6.8.3#oldClusterTestCluster node0/elasticsearch-6.8.3-SNAPSHOT/config/testnode.pem
        at org.elasticsearch.xpack.core.ssl.PemUtils.readPrivateKey(PemUtils.java:107) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.PEMKeyConfig.readPrivateKey(PEMKeyConfig.java:97) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.PEMKeyConfig.createKeyManager(PEMKeyConfig.java:58) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:395) ~[?:?]
        at java.util.HashMap.computeIfAbsent(HashMap.java:1127) ~[?:1.8.0_221]
        at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:461) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:103) ~[?:?]
..
Caused by: javax.crypto.BadPaddingException: Error finalising cipher data: pad block corrupted
        at org.bouncycastle.jcajce.provider.BaseCipher.engineDoFinal(Unknown Source) ~[bc-fips-1.0.1.jar:?]
        at javax.crypto.Cipher.doFinal(Cipher.java:2164) ~[?:1.8.0_191]
        at org.elasticsearch.xpack.core.ssl.PemUtils.possiblyDecryptPKCS1Key(PemUtils.java:366) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.PemUtils.parsePKCS1Rsa(PemUtils.java:257) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.PemUtils.readPrivateKey(PemUtils.java:93) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.PEMKeyConfig.readPrivateKey(PEMKeyConfig.java:97) ~[?:?]
..

Reproduces for me

RUNTIME_JAVA_HOME=$JAVA8FIPS_HOME ./gradlew :x-pack:qa:full-cluster-restart:v6.8.3#bwcTest

[elasticmachine]

Pinging @elastic/es-security

[jkakavas]

The error shown means that the password for the private key is wrong. To be more precise, it looks like the keystore setting with the key password is not set in the keystore ( node's keystore doesn't contain the entry after this test reproducibly fails locally). This manifests only in FIPS, as in non-FIPS JVMS we use a JKS keystore and set it's password in the yaml settings with setting instead of setting secure_passphrase with keystoreSetting

[jkakavas]

This was caused by the changes introduced in #41392, I have raised #45960 to resolve it.