Configure operator to use TLS v.13 #9998

st96d045 · 2024-04-18T18:15:39Z

st96d045
Apr 18, 2024

I have SSL problems talking to the k8s API and I have to specify the TLS version for the operator to use TLSv1.3
How does one go about doing that for the operator (not the broker) in the operator helm chart? Any input is appreciated.
Thanks

{
   "instant":{
      "epochSecond":1713463722,
      "nanoOfSecond":363343485
   },
   "thread":"-502386384-pool-3-thread-1",
   "level":"ERROR",
   "loggerName":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector",
   "message":"Exception occurred while acquiring lock 'LeaseLock: default - strimzi-cluster-operator (strimzi-cluster-operator-74f46fc897-9dl65)'",
   "thrown":{
      "commonElementCount":0,
      "localizedMessage":"Operation: [get]  for kind: [Lease]  with name: [strimzi-cluster-operator]  in namespace: [default]  failed.",
      "message":"Operation: [get]  for kind: [Lease]  with name: [strimzi-cluster-operator]  in namespace: [default]  failed.",
      "name":"io.fabric8.kubernetes.client.KubernetesClientException",
      "cause":{
         "commonElementCount":10,
         "localizedMessage":"Received fatal alert: protocol_version",
         "message":"Received fatal alert: protocol_version",
         "name":"java.io.IOException",
         "cause":{
            "commonElementCount":3,
            "localizedMessage":"Received fatal alert: protocol_version",
            "message":"Received fatal alert: protocol_version",
            "name":"javax.net.ssl.SSLHandshakeException",
            "extendedStackTrace":[
               {
                  "class":"sun.security.ssl.Alert",
                  "method":"createSSLException",
                  "line":-1,
                  "exact":false,
                  "location":"?",
                  "version":"?"
               },
               {
                  "class":"sun.security.ssl.Alert",
                  "method":"createSSLException",
                  "line":-1,
                  "exact":false,
                  "location":"?",
                  "version":"?"
               },
               {
                  "class":"sun.security.ssl.TransportContext",
                  "method":"fatal",
                  "line":-1,
                  "exact":false,
                  "location":"?",
                  "version":"?"
               },
               {
                  "class":"sun.security.ssl.Alert$AlertConsumer",
                  "method":"consume",
                  "line":-1,
                  "exact":false,
                  "location":"?",
                  "version":"?"
               },
               {
                  "class":"sun.security.ssl.TransportContext",
                  "method":"dispatch",
                  "line":-1,
                  "exact":false,
                  "location":"?",
                  "version":"?"
               },
               {
                  "class":"sun.security.ssl.SSLTransport",
                  "method":"decode",
                  "line":-1,
                  "exact":false,
                  "location":"?",
                  "version":"?"
               },
               {
                  "class":"sun.security.ssl.SSLEngineImpl",
                  "method":"decode",
                  "line":-1,
                  "exact":false,
                  "location":"?",
                  "version":"?"
               },
               {
                  "class":"sun.security.ssl.SSLEngineImpl",
                  "method":"readRecord",
                  "line":-1,
                  "exact":false,
                  "location":"?",
                  "version":"?"
               },
               {
                  "class":"sun.security.ssl.SSLEngineImpl",
                  "method":"unwrap",
                  "line":-1,
                  "exact":false,
                  "location":"?",
                  "version":"?"
               },
               {
                  "class":"sun.security.ssl.SSLEngineImpl",
                  "method":"unwrap",
                  "line":-1,
                  "exact":false,
                  "location":"?",
                  "version":"?"
               },
               {
                  "class":"javax.net.ssl.SSLEngine",
                  "method":"unwrap",
                  "line":-1,
                  "exact":false,
                  "location":"?",
                  "version":"?"
               },
               {
                  "class":"jdk.internal.net.http.common.SSLFlowDelegate$Reader",
                  "method":"unwrapBuffer",
                  "line":-1,
                  "exact":false,
                  "location":"java.net.http",
                  "version":"?"
               },
               {
                  "class":"jdk.internal.net.http.common.SSLFlowDelegate$Reader",
                  "method":"processData",
                  "line":-1,
                  "exact":false,
                  "location":"java.net.http",
                  "version":"?"
               },
               {
                  "class":"jdk.internal.net.http.common.SSLFlowDelegate$Reader$ReaderDownstreamPusher",
                  "method":"run",
                  "line":-1,
                  "exact":false,
                  "location":"java.net.http",
                  "version":"?"
               },
               {
                  "class":"jdk.internal.net.http.common.SequentialScheduler$LockingRestartableTask",
                  "method":"run",
                  "line":-1,
                  "exact":false,
                  "location":"java.net.http",
                  "version":"?"
               },
               {
                  "class":"jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask",
                  "method":"run",
                  "line":-1,
                  "exact":false,
                  "location":"java.net.http",
                  "version":"?"
               },
               {
                  "class":"jdk.internal.net.http.common.SequentialScheduler$SchedulableTask",
                  "method":"run",
                  "line":-1,
                  "exact":false,
                  "location":"java.net.http",
                  "version":"?"
               }
            ]
         },
         "extendedStackTrace":[
            {
               "class":"io.fabric8.kubernetes.client.dsl.internal.OperationSupport",
               "method":"waitForResult",
               "file":"OperationSupport.java",
               "line":533,
               "exact":false,
               "location":"io.fabric8.kubernetes-client-6.3.1.jar",
               "version":"?"
            },
            {
               "class":"io.fabric8.kubernetes.client.dsl.internal.OperationSupport",
               "method":"handleResponse",
               "file":"OperationSupport.java",
               "line":570,
               "exact":false,
               "location":"io.fabric8.kubernetes-client-6.3.1.jar",
               "version":"?"
            },
            {
               "class":"io.fabric8.kubernetes.client.dsl.internal.OperationSupport",
               "method":"handleGet",
               "file":"OperationSupport.java",
               "line":482,
               "exact":false,
               "location":"io.fabric8.kubernetes-client-6.3.1.jar",
               "version":"?"
            },
            {
               "class":"io.fabric8.kubernetes.client.dsl.internal.BaseOperation",
               "method":"handleGet",
               "file":"BaseOperation.java",
               "line":742,
               "exact":false,
               "location":"io.fabric8.kubernetes-client-6.3.1.jar",
               "version":"?"
            },
            {
               "class":"io.fabric8.kubernetes.client.dsl.internal.BaseOperation",
               "method":"getMandatory",
               "file":"BaseOperation.java",
               "line":177,
               "exact":false,
               "location":"io.fabric8.kubernetes-client-6.3.1.jar",
               "version":"?"
            }
         ]
      },
      "extendedStackTrace":[
         {
            "class":"io.fabric8.kubernetes.client.KubernetesClientException",
            "method":"launderThrowable",
            "file":"KubernetesClientException.java",
            "line":159,
            "exact":false,
            "location":"io.fabric8.kubernetes-client-api-6.3.1.jar",
            "version":"?"
         },
         {
            "class":"io.fabric8.kubernetes.client.dsl.internal.BaseOperation",
            "method":"getMandatory",
            "file":"BaseOperation.java",
            "line":182,
            "exact":false,
            "location":"io.fabric8.kubernetes-client-6.3.1.jar",
            "version":"?"
         },
         {
            "class":"io.fabric8.kubernetes.client.dsl.internal.BaseOperation",
            "method":"get",
            "file":"BaseOperation.java",
            "line":144,
            "exact":false,
            "location":"io.fabric8.kubernetes-client-6.3.1.jar",
            "version":"?"
         },
         {
            "class":"io.fabric8.kubernetes.client.dsl.internal.BaseOperation",
            "method":"get",
            "file":"BaseOperation.java",
            "line":93,
            "exact":false,
            "location":"io.fabric8.kubernetes-client-6.3.1.jar",
            "version":"?"
         },
         {
            "class":"io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.LeaseLock",
            "method":"get",
            "file":"LeaseLock.java",
            "line":44,
            "exact":false,
            "location":"io.fabric8.kubernetes-client-api-6.3.1.jar",
            "version":"?"
         },
         {
            "class":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector",
            "method":"tryAcquireOrRenew",
            "file":"LeaderElector.java",
            "line":193,
            "exact":false,
            "location":"io.fabric8.kubernetes-client-api-6.3.1.jar",
            "version":"?"
         },
         {
            "class":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector",
            "method":"lambda$acquire$4",
            "file":"LeaderElector.java",
            "line":153,
            "exact":false,
            "location":"io.fabric8.kubernetes-client-api-6.3.1.jar",
            "version":"?"
         },
         {
            "class":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector",
            "method":"lambda$loop$8",
            "file":"LeaderElector.java",
            "line":258,
            "exact":false,
            "location":"io.fabric8.kubernetes-client-api-6.3.1.jar",
            "version":"?"
         },
         {
            "class":"java.util.concurrent.CompletableFuture$AsyncRun",
            "method":"run",
            "line":-1,
            "exact":false,
            "location":"?",
            "version":"?"
         },
         {
            "class":"java.util.concurrent.ThreadPoolExecutor",
            "method":"runWorker",
            "line":-1,
            "exact":false,
            "location":"?",
            "version":"?"
         },
         {
            "class":"java.util.concurrent.ThreadPoolExecutor$Worker",
            "method":"run",
            "line":-1,
            "exact":false,
            "location":"?",
            "version":"?"
         },
         {
            "class":"java.lang.Thread",
            "method":"run",
            "line":-1,
            "exact":false,
            "location":"?",
            "version":"?"
         }
      ]
   },
   "endOfBatch":false,
   "loggerFqcn":"org.apache.logging.slf4j.Log4jLogger",
   "threadId":24,
   "threadPriority":5
}

And seeing similar logs from the debug logging for the JVM

javax.net.ssl|DEBUG|A1|JdkHttpClientFactory-1389984438-pool-2-thread-2|2024-04-17 21:52:20.155 UTC|null:-1|WRITE: TLSv1.2 handshake, length = 320
javax.net.ssl|DEBUG|A1|JdkHttpClientFactory-1389984438-pool-2-thread-2|2024-04-17 21:52:20.156 UTC|null:-1|Raw write (
### omitted raw output
javax.net.ssl|DEBUG|A1|JdkHttpClientFactory-1389984438-pool-2-thread-2|2024-04-17 21:52:20.156 UTC|null:-1|Raw read (
  0000: 15 03 01 00 02 02 46                               ......F
)
javax.net.ssl|DEBUG|A1|JdkHttpClientFactory-1389984438-pool-2-thread-2|2024-04-17 21:52:20.156 UTC|null:-1|READ: TLSv1 alert, length = 2
javax.net.ssl|DEBUG|A1|JdkHttpClientFactory-1389984438-pool-2-thread-2|2024-04-17 21:52:20.156 UTC|null:-1|Received alert message (
"Alert": {
  "level"      : "fatal",
  "description": "protocol_version"
}
)
javax.net.ssl|ERROR|A1|JdkHttpClientFactory-1389984438-pool-2-thread-2|2024-04-17 21:52:20.156 UTC|null:-1|Fatal (PROTOCOL_VERSION): Received fatal alert: protocol_version (
"throwable" : {
  javax.net.ssl.SSLHandshakeException: Received fatal alert: protocol_version
        at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source)
        at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source)
        at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source)
        at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Unknown Source)
        at java.base/sun.security.ssl.TransportContext.dispatch(Unknown Source)
        at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source)
        at java.base/sun.security.ssl.SSLEngineImpl.decode(Unknown Source)
        at java.base/sun.security.ssl.SSLEngineImpl.readRecord(Unknown Source)
        at java.base/sun.security.ssl.SSLEngineImpl.unwrap(Unknown Source)
        at java.base/sun.security.ssl.SSLEngineImpl.unwrap(Unknown Source)
        at java.base/javax.net.ssl.SSLEngine.unwrap(Unknown Source)
        at java.net.http/jdk.internal.net.http.common.SSLFlowDelegate$Reader.unwrapBuffer(Unknown Source)
        at java.net.http/jdk.internal.net.http.common.SSLFlowDelegate$Reader.processData(Unknown Source)
        at java.net.http/jdk.internal.net.http.common.SSLFlowDelegate$Reader$ReaderDownstreamPusher.run(Unknown Source)

scholzj · 2024-04-18T18:17:53Z

scholzj
Apr 18, 2024
Maintainer

Maybe you should start by sharing a full log ideally in a readable format and additional information such as software versions etc.

4 replies

st96d045 Apr 18, 2024
Author

I'm running strimzi v0.34.0
Not sure the logs would shed much more information than what's already shown. If there is a way to configure the TLS version used in the operator, maybe there is a section in the documentation that describes that? I was following the information shared here, but seems this is only applicable to the broker config.

st96d045 Apr 18, 2024
Author

And the issue I think it boils down to the kubernetes API server being configured with the tls-min-version flag requiring TLSv1.3 (as per the /etc/kubernetes/manifests/kube-apiserver.yaml config available in control plane node)

    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_
ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
    - --tls-min-version=VersionTLS13
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key

scholzj Apr 18, 2024
Maintainer

The operator is just a Java application. I think that should use TLS 1.3 by default. So does Kafka as far as I know. So not exactly sure what exactly is the problem in your case. It also might not be a TLS version issue as much as a cipher suites issue if you require some specific cipher suites. Another obvious thing to try would be to upgrade as 0.34 is quite old. But I'm of course not 100% sure it helps.

st96d045 Apr 18, 2024
Author

strimzi-operator-ssl-log-base.log

I think that should use TLS 1.3 by default

Thanks for the info. That's what the documentation says, however I see the SSL handshake being done with TLSv1.2, so maybe the newer versions of strimzi use 1.3 by default but not the 0.34.0 version?

Here's the log in case it helps.

st96d045 · 2024-04-22T20:46:23Z

st96d045
Apr 22, 2024
Author

I have verified that relaxing the minimum required version of SSL to v1.2 on my target k8s cluster allows the operator to deploy properly without any of the SSL errors/exception shown earlier. I have not been successful in updating the operator to use my preferred version of TLS (with version 0.34.0).
The question here is: what is the suggested approach to configure the TLS version used in the operator? Is there a section in the documentation that describes that?
Thanks

2 replies

scholzj Apr 22, 2024
Maintainer

The question here is: what is the suggested approach to configure the TLS version used in the operator? Is there a section in the documentation that describes that?

i don't think anyone raised this before. So there is not documentation, you might need to figure it out.

st96d045 Apr 23, 2024
Author

Found the issue where this was addressed: fabric8io/kubernetes-client#5125
Version 0.34.0 uses version of k8s client library that has hard-coded version SSLv1.2 for SSL context init (fabric8io v6.3.1).
Only starting with strimzi version 0.36.0 does operator use k8s client which supports SSLv1.3 (using fabric8io v6.7.2)
So I think the answer to my question is this is not possible with strimzi 0.34.0, and starting with 0.36.0 this is based on what is supported by the jdk version:

  public static SSLContext sslContext(KeyManager[] keyManagers, TrustManager[] trustManagers) {
    SSLContext sslContext = null;
    NoSuchAlgorithmException noSuch = null;
    // v1.3 is not supported on all vms, and of course there may be later versions added.
    // so try to find one starting with the latest
    for (TlsVersion version : TlsVersion.values()) {
      try {
        sslContext = SSLContext.getInstance(version.javaName());
        break;
      } catch (NoSuchAlgorithmException e) {
        if (noSuch == null) {
          noSuch = e;
        }
        continue;
      }
    }

st96d045 · 2024-05-01T20:34:40Z

st96d045
May 1, 2024
Author

Closing.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strimzi

Configure operator to use TLS v.13 #9998

{{title}}

Replies: 3 comments 6 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Strimzi

Configure operator to use TLS v.13 #9998

st96d045 Apr 18, 2024

Replies: 3 comments · 6 replies

scholzj Apr 18, 2024 Maintainer

st96d045 Apr 18, 2024 Author

st96d045 Apr 18, 2024 Author

scholzj Apr 18, 2024 Maintainer

st96d045 Apr 18, 2024 Author

st96d045 Apr 22, 2024 Author

scholzj Apr 22, 2024 Maintainer

st96d045 Apr 23, 2024 Author

st96d045 May 1, 2024 Author

st96d045
Apr 18, 2024

Replies: 3 comments 6 replies

scholzj
Apr 18, 2024
Maintainer

st96d045 Apr 18, 2024
Author

st96d045 Apr 18, 2024
Author

scholzj Apr 18, 2024
Maintainer

st96d045 Apr 18, 2024
Author

st96d045
Apr 22, 2024
Author

scholzj Apr 22, 2024
Maintainer

st96d045 Apr 23, 2024
Author

st96d045
May 1, 2024
Author