OCPBUGS-35800: kubelet config controller does not always propogate tlsSecurityProfile by djoshy · Pull Request #4512 · openshift/machine-config-operator

djoshy · 2024-08-02T19:10:51Z

- What I did

All subcontrollers within the kubelet-config-controller will now apply the APIServer object settings, including their bootstrap variants. If a bootstrap APIServer manifest is provided, the controllers will use that - otherwise it will default to the intermediate TLS profile.
All the controllers will resync on updates to the APIServer object. The initial resync is done by the node config controller, which subsequently calls the kubelet-config and feature sub controllers.
Also added some unit tests to verify the above behavior.

- How to verify it

Bring up a cluster with an install time APIServer manifest named "cluster". Ensure that this manifest has the TLS profile you'd like to test.
Once the install is complete, observe the kubelet configs(located at /etc/kubernetes/kubelet.conf) on any node to ensure that it has the correct values for TLSCiphers and TLSMinVersion.
Switch between TLS profiles by updating the APIServer object named "cluster".
This will cause a new MachineConfig rollout. You can verify the TLS values on the nodes are as expected as the update rolls through the cluster.
Now, try applying a user defined KubeletConfig with TLS settings that are different from the ones in the APIServer object. This should cause another MachineConfig rollout to the pool it was targeted to. After the update, you should see the nodes in the pool with the TLS values specified by the new KubeletConfig(not the APIServer object).

Things to note

The APIServer rejects the Modern profile at the moment, according to the docs.
If no tlsSecurityProfile is provided, the MCO will default to Intermediate.

openshift-ci-robot · 2024-08-02T19:10:57Z

@djoshy: This pull request references Jira Issue OCPBUGS-35800, which is invalid:

expected the bug to target the "4.17.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

- What I did

All subcontrollers within the kubelet-config-controller will now apply the APIServer object settings, including their bootstrap variants. If a bootstrap APIServer manifest is provided, the controllers will use that - otherwise it will default to the intermediate TLS profile.

All the controllers will resync on updates to the APIServer object. The initial resync is done by the node config controller, which subsequently calls the kubelet-config and feature sub controllers.

Also added some unit tests to verify the above behavior.

- How to verify it

Bring up a cluster with an install time APIServer manifest named "cluster". Ensure that this manifest has the TLS profile you'd like to test.

Once the install is complete, observe the kubelet configs(located at /etc/kubernetes/kubelet.conf) on any node to ensure that it has the correct values for TLSCiphers and TLSMinVersion.

Switch between TLS profiles by updating the APIServer object named "cluster".

This will cause a new MachineConfig rollout. You can verify the TLS values on the nodes are as expected as the update rolls through the cluster.

Now, try applying a user defined KubeletConfig with TLS settings that are different from the ones in the APIServer object. This should cause another MachineConfig rollout to the pool it was targeted to. After the update, you should see the nodes in the pool with the TLS values specified by the new KubeletConfig(not the APIServer object).

Things to note

The APIServer rejects the Modern profile at the moment, according to the docs.

If no tlsSecurityProfile is provided, the MCO will default to Intermediate.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

djoshy · 2024-08-02T19:13:53Z

/jira refresh

openshift-ci-robot · 2024-08-02T19:14:00Z

@djoshy: This pull request references Jira Issue OCPBUGS-35800, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.17.0) matches configured target version for branch (4.17.0)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

rphillips · 2024-08-05T13:36:46Z

 	<-stopCh
 }
+func (ctrl *Controller) filterAPIServer(apiServer *configv1.APIServer) {
+	if apiServer.Name == "cluster" {


reversing the logic here would remove the nested if.

Suggested change

if apiServer.Name == "cluster" {

if apiServer.Name != "cluster" {

return

}

djoshy · 2024-08-05T16:16:01Z

/test unit

/hold for QE review

rphillips · 2024-08-05T17:20:44Z

/lgtm

This commit replaces the hardcoded TLS configuration values in the templates with rendered values from the APIServer object. This is done during bootstrap and during in cluster operation. If no tls configuration is provided by the APIServer object, the intermediate profile is used as a default.

This commit adds new APIServer based callbacks for the kubelet config controller, so that it can update the kubelet config MachineConfigs if there is an update to the cluster wide TLS settings.

djoshy · 2024-08-06T13:44:24Z

/retest

rphillips · 2024-08-06T13:52:30Z

/lgtm

openshift-ci · 2024-08-06T13:52:54Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: djoshy, rphillips

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [djoshy]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

djoshy · 2024-08-06T17:12:08Z

/test e2e-gcp-op-techpreview

/hold
holding for QE approval

djoshy · 2024-08-08T12:42:15Z

/test all

ptalgulk01 · 2024-08-08T13:55:49Z

Pre-merge verified:
For manual- installation and added the 99_apiserver.yaml file before creating cluster

$ cat > openshift/99_apiserver.yaml
apiVersion: config.openshift.io/v1
kind: APIServer
metadata:
  name: cluster
spec:
  tlsSecurityProfile:
    old: {}
    type: Old
  audit:
    profile: Default

$ oc get node
NAME                                        STATUS   ROLES                  AGE   VERSION
ip-10-0-28-203.sa-east-1.compute.internal   Ready    control-plane,master   93m   v1.30.1+9798e19
ip-10-0-28-40.sa-east-1.compute.internal    Ready    worker                 81m   v1.30.1+9798e19
ip-10-0-44-194.sa-east-1.compute.internal   Ready    worker                 82m   v1.30.1+9798e19
ip-10-0-61-35.sa-east-1.compute.internal    Ready    control-plane,master   93m   v1.30.1+9798e19
ip-10-0-76-153.sa-east-1.compute.internal   Ready    worker                 82m   v1.30.1+9798e19
ip-10-0-85-131.sa-east-1.compute.internal   Ready    control-plane,master   93m   v1.30.1+9798e19

More detail steps about verification are provided here: MCO-850

Check that applied config is configured on the node too.

 $ oc debug node/ip-10-0-76-153.sa-east-1.compute.internal -- chroot /host cat /etc/kubernetes/kubelet.conf
tlsCipherSuites:
- TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
- TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
- TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
- TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
- TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
- TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
- TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
- TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
- TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
- TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
- TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
- TLS_RSA_WITH_AES_128_GCM_SHA256
- TLS_RSA_WITH_AES_256_GCM_SHA384
- TLS_RSA_WITH_AES_128_CBC_SHA256
- TLS_RSA_WITH_AES_128_CBC_SHA
- TLS_RSA_WITH_AES_256_CBC_SHA
- TLS_RSA_WITH_3DES_EDE_CBC_SHA
tlsMinVersion: VersionTLS10

Switch the tlsSecurityProfile via apiserver

$ oc get apiservers cluster  -o yaml
apiVersion: config.openshift.io/v1
kind: APIServer
metadata:
  creationTimestamp: "2024-08-08T06:51:20Z"
  generation: 2
  name: cluster
  resourceVersion: "45244"
  uid: 4b776b26-cab8-40ed-9edc-d28f7df7dc89
spec:
  audit:
    profile: Default
  tlsSecurityProfile:
    intermediate: {}
    type: Intermediate

$ oc get po kube-rbac-proxy-crio-ip-10-0-28-203.sa-east-1.compute.internal machine-config-controller-57fb7566ff-k2thk -o yaml | grep -i tls
      - --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
      - --tls-min-version=VersionTLS12

    - --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
    - --tls-min-version=VersionTLS12

$ oc logs machine-config-server-6q6gz | grep -i tls
I0808 08:36:48.025754       1 start.go:51] Launching server with tls min version: VersionTLS12 & cipher suites [TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256]

Apply custom created kubelet config with TLS settings that are different from the the APIServer object.

apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
  name: set-kubelet-tls-security-profile
spec:
  tlsSecurityProfile:
    type: Custom 
    custom: 
      ciphers: 
      - ECDHE-ECDSA-CHACHA20-POLY1305
      - ECDHE-RSA-CHACHA20-POLY1305
      - ECDHE-RSA-AES128-GCM-SHA256
      - ECDHE-ECDSA-AES128-GCM-SHA256
      minTLSVersion: VersionTLS11
  machineConfigPoolSelector:
    matchLabels:
      pools.operator.machineconfiguration.openshift.io/master: ""

$ oc apply -f kubletconfig.yaml 
kubeletconfig.machineconfiguration.openshift.io/set-kubelet-tls-security-profile created
 
$ oc get kubeletconfigs 
NAME                               AGE
set-kubelet-tls-security-profile   13s

Wait for particular pool to be updated on which kubelet config is applied

$  oc get mcp
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master   rendered-master-ac022d4da191a5039d6d5d61a4ea9c8c   True      False      False      3              3                   3                     0                      116m
worker   rendered-worker-2362c76331294e6a2eaa105f9f5ac0db   True      False      False      3              3                   3                     0                      116m

Check the /etc/kubernetes/kubelet.conf on the node has the same tls setting applied via kubelet config and does not contain API server applied tls setting

$ oc debug node/ip-10-0-28-203.sa-east-1.compute.internal -- chroot /host cat  /etc/kubernetes/kubelet.conf | grep -i tls
Starting pod/ip-10-0-28-203sa-east-1computeinternal-debug ...
To use host binaries, run `chroot /host`

Removing debug pod ...
serverTLSBootstrap: true
tlsCipherSuites:
- TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
- TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
- TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
- TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
tlsMinVersion: VersionTLS11

The tls setting is not disturbed by kubelet config in pods we could see the same tls setting as that of API server

$ oc get po kube-rbac-proxy-crio-ip-10-0-28-203.sa-east-1.compute.internal machine-config-controller-57fb7566ff-l7h9t oyaml | grep -i tls
      - --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
      - --tls-min-version=VersionTLS12

    - --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
    - --tls-min-version=VersionTLS12

$ oc logs machine-config-server-6q6gz | grep -i tls
I0808 08:36:48.025754       1 start.go:51] Launching server with tls min version: VersionTLS12 & cipher suites [TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256]

djoshy · 2024-08-08T14:01:25Z

/unhold

ptalgulk01 · 2024-08-08T14:11:00Z

Pre-merge Verified on
GCP - private-templates/functionality-testing/aos-4_17/ipi-on-gcp/versioned-installer
SNO - private-templates/functionality-testing/aos-4_17/ipi-on-gcp/versioned-installer
Parameters - enable_spot_instance_workers: "no"
num_masters: 1
num_workers: 0
master_worker_AllInOne: "true"
Disconnected - private-templates/functionality-testing/aos-4_17/ipi-on-gcp/versioned-installer_customer_vpc-disconnected
Proxy - private-templates/functionality-testing/aos-4_17/ipi-on-gcp/versioned-installer_customer_vpc-http_proxy

Steps:

By default the tlsSecurityProfile is Intermediate , same can be seen on node in /etc/kubernetes/kubelet.conf config

$ oc debug node/ptalgulk-07dis-47szz-worker-b-czxxs -- chroot /host cat /etc/kubernetes/kubelet.conf
tlsCipherSuites:
- TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
- TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
- TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
- TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
- TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
tlsMinVersion: VersionTLS12

$ oc get po machine-config-controller-c775d6887-rmxg8 -oyaml | grep -i tls
 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
    - --tls-min-version=VersionTLS12

$ oc get po kube-rbac-proxy-crio-ptalgulk-07dis-47szz-master-0 -oyaml | grep -i tls
--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
    - --tls-min-version=VersionTLS12

$ oc logs machine-config-server-5wkzr | grep -i tls
I0807 03:28:58.175849       1 start.go:51] Launching server with tls min version: VersionTLS12 & cipher suites [TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256]

More detail steps about verification are provided here: MCO-850

Switch tlsSecurityProfile and wait till MCP is updated
Verify the tlsProfile is applied correctly

$ oc debug node/ptalgulk-07dis-47szz-worker-b-czxxs -- chroot /host cat /etc/kubernetes/kubelet.conf
tlsCipherSuites:
- TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
- TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
- TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
- TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
tlsMinVersion: VersionTLS11

$ oc get po machine-config-controller-bbd9fbd6b-7gbg9  -o yaml | grep -i tls
    - --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
    - --tls-min-version=VersionTLS11

$ oc logs machine-config-server-6rf7p | grep -i tls
I0807 05:11:03.628870       1 start.go:51] Launching server with tls min version: VersionTLS11 & cipher suites [TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256]

$ oc get po kube-rbac-proxy-crio-ptalgulk-07dis-47szz-master-0 -oyaml | grep -i tls
    - --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
    - --tls-min-version=VersionTLS11

$ oc debug node/ptalgulk-07dis-47szz-worker-b-czxxs  --image=quay.io/openshifttest/testssl@sha256:ad6fb8002cb9cfce3ddc8829fd6e7e0d997aeb1faf972650f3e5d7603f90c6ef -- testssl.sh --color 0 -E localhost:9637
TLS 1
 -
TLS 1.1
 -
TLS 1.2
 xcca9   ECDHE-ECDSA-CHACHA20-POLY1305     ECDH 253   ChaCha20    256      TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256      
 xc02b   ECDHE-ECDSA-AES128-GCM-SHA256     ECDH 253   AESGCM      128      TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256            
TLS 1.3
 x1302   TLS_AES_256_GCM_SHA384            ECDH 253   AESGCM      256      TLS_AES_256_GCM_SHA384                             
 x1303   TLS_CHACHA20_POLY1305_SHA256      ECDH 253   ChaCha20    256      TLS_CHACHA20_POLY1305_SHA256                       
 x1301   TLS_AES_128_GCM_SHA256            ECDH 253   AESGCM      128      TLS_AES_128_GCM_SHA256

Apply kubelet config

apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
  name: set-kubelet-tls-security-profile
spec:
  tlsSecurityProfile:
    old: {}
    type: Old
  machineConfigPoolSelector:
    matchLabels:
      pools.operator.machineconfiguration.openshift.io/worker: ""

$ oc apply -f kubletconfig.yaml 
kubeletconfig.machineconfiguration.openshift.io/set-kubelet-tls-security-profile created

Wait for MCP to be updated on which kubelet is applied

$ oc debug node/ptalgulk-07dis-47szz-worker-a-r9gsg -- chroot /host cat /etc/kubernetes/kubelet.conf
tlsCipherSuites:
- TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
- TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
- TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
- TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
- TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
- TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
- TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
- TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
- TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
- TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
- TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
- TLS_RSA_WITH_AES_128_GCM_SHA256
- TLS_RSA_WITH_AES_256_GCM_SHA384
- TLS_RSA_WITH_AES_128_CBC_SHA256
- TLS_RSA_WITH_AES_128_CBC_SHA
- TLS_RSA_WITH_AES_256_CBC_SHA
- TLS_RSA_WITH_3DES_EDE_CBC_SHA
tlsMinVersion: VersionTLS10

Kubelet Config does not affect the apiserver configuration applied other than on particular pool node

$ oc get po kube-rbac-proxy-crio-ptalgulk-07dis-47szz-worker-a-r9gsg -o yaml | grep -i tls
   --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
    - --tls-min-version=VersionTLS11

$ oc logs machine-config-server-6rf7p | grep -i tls
I0807 05:11:03.628870       1 start.go:51] Launching server with tls min version: VersionTLS11 & cipher suites [TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256]

$ oc debug node/ptalgulk-07dis-47szz-master-2 -- chroot /host cat /etc/kubernetes/kubelet.conf
tlsCipherSuites:
- TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
- TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
- TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
- TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
tlsMinVersion: VersionTLS11

If we apply the tls setting after the kubelet config is applied it wont override the tls setting of kubelet config

e.g in SNO cluster

$ oc get apiservers.config.openshift.io cluster -o yaml
apiVersion: config.openshift.io/v1
kind: APIServer
….
  audit:
    profile: Default

$ oc get kubeletconfigs.machineconfiguration.openshift.io set-kubelet-tls-security-profile -o yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
….
spec:
  machineConfigPoolSelector:
    matchLabels:
      pools.operator.machineconfiguration.openshift.io/master: ""
  tlsSecurityProfile:
    old: {}
    type: Old
…

$ oc debug node/ptalgulk-08sno-wj5bs-master-0   -- chroot /host cat /etc/kubernetes/kubelet.conf
tlsCipherSuites:
- TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
- TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
- TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
- TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
- TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
- TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
- TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
- TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
- TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
- TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
- TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
- TLS_RSA_WITH_AES_128_GCM_SHA256
- TLS_RSA_WITH_AES_256_GCM_SHA384
- TLS_RSA_WITH_AES_128_CBC_SHA256
- TLS_RSA_WITH_AES_128_CBC_SHA
- TLS_RSA_WITH_AES_256_CBC_SHA
- TLS_RSA_WITH_3DES_EDE_CBC_SHA
tlsMinVersion: VersionTLS10
volumeStatsAggPeriod: 0s

Switch to Custom API server

$ oc get apiservers.config.openshift.io cluster -o yaml
….
  tlsSecurityProfile:
    custom:
      ciphers:
      - ECDHE-ECDSA-CHACHA20-POLY1305
      - ECDHE-RSA-CHACHA20-POLY1305
      - ECDHE-RSA-AES128-GCM-SHA256
      - ECDHE-ECDSA-AES128-GCM-SHA256
      minTLSVersion: VersionTLS11
    type: Custom

$  oc get po kube-rbac-proxy-crio-ptalgulk-08sno-wj5bs-master-0 -o yaml | grep -i tls
    - --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
    - --tls-min-version=VersionTLS11

Delete the Kubelet Config to have same tls setting as of API server

$ oc delete kubeletconfigs.machineconfiguration.openshift.io set-kubelet-tls-security-profile 
kubeletconfig.machineconfiguration.openshift.io "set-kubelet-tls-security-profile" deleted

$  oc debug node/ptalgulk-08sno-wj5bs-master-0 -- chroot /host cat /etc/kubernetes/kubelet.conf
tlsCipherSuites:
- TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
- TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
- TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
- TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
tlsMinVersion: VersionTLS11

Note: As all the cluster installed here has API server object and if we manually try to delete the api server it shows as forbidden to remove it so hence the fail configs were not generated.

ptalgulk01 · 2024-08-08T14:11:51Z

/label qe-approved

openshift-ci-robot · 2024-08-08T14:11:58Z

@djoshy: This pull request references Jira Issue OCPBUGS-35800, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.17.0) matches configured target version for branch (4.17.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

No GitHub users were found matching the public email listed for the QA contact in Jira (ptalgulk@redhat.com), skipping review request.

Details

In response to this:

- What I did

All subcontrollers within the kubelet-config-controller will now apply the APIServer object settings, including their bootstrap variants. If a bootstrap APIServer manifest is provided, the controllers will use that - otherwise it will default to the intermediate TLS profile.

All the controllers will resync on updates to the APIServer object. The initial resync is done by the node config controller, which subsequently calls the kubelet-config and feature sub controllers.

Also added some unit tests to verify the above behavior.

- How to verify it

Bring up a cluster with an install time APIServer manifest named "cluster". Ensure that this manifest has the TLS profile you'd like to test.

Once the install is complete, observe the kubelet configs(located at /etc/kubernetes/kubelet.conf) on any node to ensure that it has the correct values for TLSCiphers and TLSMinVersion.

Switch between TLS profiles by updating the APIServer object named "cluster".

This will cause a new MachineConfig rollout. You can verify the TLS values on the nodes are as expected as the update rolls through the cluster.

Now, try applying a user defined KubeletConfig with TLS settings that are different from the ones in the APIServer object. This should cause another MachineConfig rollout to the pool it was targeted to. After the update, you should see the nodes in the pool with the TLS values specified by the new KubeletConfig(not the APIServer object).

Things to note

The APIServer rejects the Modern profile at the moment, according to the docs.

If no tlsSecurityProfile is provided, the MCO will default to Intermediate.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

djoshy · 2024-08-08T15:13:44Z

/test e2e-hypershift

djoshy · 2024-08-08T15:47:39Z

/test e2e-gcp-op

openshift-ci-robot · 2024-08-08T17:23:13Z

/retest-required

Remaining retests: 0 against base HEAD 217f607 and 2 for PR HEAD 5f0a4be in total

djoshy · 2024-08-08T21:12:06Z

/override ci/prow/e2e-hypershift

Overriding since hypershift seems to be flaking and it has passed several times prior.

openshift-ci · 2024-08-08T21:12:21Z

@djoshy: Overrode contexts on behalf of djoshy: ci/prow/e2e-hypershift

Details

In response to this:

/override ci/prow/e2e-hypershift

Overriding since hypershift seems to be flaking and it has passed several times prior.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-ci-robot · 2024-08-09T04:59:43Z

@djoshy: Jira Issue OCPBUGS-35800: All pull requests linked via external trackers have merged:

openshift/machine-config-operator#4512

Jira Issue OCPBUGS-35800 has been moved to the MODIFIED state.

Details

In response to this:

- What I did

All subcontrollers within the kubelet-config-controller will now apply the APIServer object settings, including their bootstrap variants. If a bootstrap APIServer manifest is provided, the controllers will use that - otherwise it will default to the intermediate TLS profile.

All the controllers will resync on updates to the APIServer object. The initial resync is done by the node config controller, which subsequently calls the kubelet-config and feature sub controllers.

Also added some unit tests to verify the above behavior.

- How to verify it

Bring up a cluster with an install time APIServer manifest named "cluster". Ensure that this manifest has the TLS profile you'd like to test.

Once the install is complete, observe the kubelet configs(located at /etc/kubernetes/kubelet.conf) on any node to ensure that it has the correct values for TLSCiphers and TLSMinVersion.

Switch between TLS profiles by updating the APIServer object named "cluster".

This will cause a new MachineConfig rollout. You can verify the TLS values on the nodes are as expected as the update rolls through the cluster.

Now, try applying a user defined KubeletConfig with TLS settings that are different from the ones in the APIServer object. This should cause another MachineConfig rollout to the pool it was targeted to. After the update, you should see the nodes in the pool with the TLS values specified by the new KubeletConfig(not the APIServer object).

Things to note

The APIServer rejects the Modern profile at the moment, according to the docs.

If no tlsSecurityProfile is provided, the MCO will default to Intermediate.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci · 2024-08-09T04:59:45Z

@djoshy: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-vsphere-ovn-zones	`5f0a4be`	link	false	`/test e2e-vsphere-ovn-zones`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

OCPBUGS-35800: kubelet config controller does not always propogate tlsSecurityProfile

openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Aug 2, 2024

openshift-ci Bot requested review from dkhater-redhat and yuqi-zhang August 2, 2024 19:11

openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 2, 2024

openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Aug 2, 2024

rphillips reviewed Aug 5, 2024

View reviewed changes

djoshy force-pushed the kcc-tls branch from 89879a1 to cca1ff7 Compare August 5, 2024 16:09

openshift-ci Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 5, 2024

openshift-ci Bot assigned rphillips Aug 5, 2024

openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Aug 5, 2024

djoshy added 3 commits August 6, 2024 08:56

make kubelet-config-controller react to APIServer

8b0f371

This commit adds new APIServer based callbacks for the kubelet config controller, so that it can update the kubelet config MachineConfigs if there is an update to the cluster wide TLS settings.

add unit tests for kubeletConfig TLS profiles

5f0a4be

djoshy force-pushed the kcc-tls branch from cca1ff7 to 5f0a4be Compare August 6, 2024 13:03

openshift-ci Bot removed the lgtm Indicates that a PR is ready to be merged. label Aug 6, 2024

openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Aug 6, 2024

openshift-ci Bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 8, 2024

openshift-ci Bot added the qe-approved Signifies that QE has signed off on this PR label Aug 8, 2024

openshift-merge-bot Bot merged commit edfd408 into openshift:master Aug 9, 2024

djoshy deleted the kcc-tls branch August 22, 2024 19:07

ptalgulk01 pushed a commit to ptalgulk01/machine-config-operator that referenced this pull request May 15, 2026

Merge pull request openshift#4512 from djoshy/kcc-tls

556ef4a

OCPBUGS-35800: kubelet config controller does not always propogate tlsSecurityProfile

coderabbitai Bot mentioned this pull request Aug 2, 2026

WIP: Add TLS profile compliance e2e tests openshift/ingress-node-firewall#761

Closed

Uh oh!

Conversation

djoshy commented Aug 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Aug 2, 2024

Uh oh!

djoshy commented Aug 2, 2024

Uh oh!

openshift-ci-robot commented Aug 2, 2024

Uh oh!

rphillips Aug 5, 2024

Choose a reason for hiding this comment

Uh oh!

djoshy commented Aug 5, 2024

Uh oh!

rphillips commented Aug 5, 2024

Uh oh!

djoshy commented Aug 6, 2024

Uh oh!

rphillips commented Aug 6, 2024

Uh oh!

openshift-ci Bot commented Aug 6, 2024

Uh oh!

djoshy commented Aug 6, 2024

Uh oh!

djoshy commented Aug 8, 2024

Uh oh!

ptalgulk01 commented Aug 8, 2024

Uh oh!

djoshy commented Aug 8, 2024

Uh oh!

ptalgulk01 commented Aug 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ptalgulk01 commented Aug 8, 2024

Uh oh!

openshift-ci-robot commented Aug 8, 2024

Uh oh!

djoshy commented Aug 8, 2024

Uh oh!

djoshy commented Aug 8, 2024

Uh oh!

openshift-ci-robot commented Aug 8, 2024

Uh oh!

djoshy commented Aug 8, 2024

Uh oh!

openshift-ci Bot commented Aug 8, 2024

Uh oh!

openshift-ci-robot commented Aug 9, 2024

Uh oh!

openshift-ci Bot commented Aug 9, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

djoshy commented Aug 2, 2024 •

edited

Loading

ptalgulk01 commented Aug 8, 2024 •

edited

Loading