Driver Load Testing: Set pod CPU & mem Requests for driver Container & Node components

[davidz627]

Driver Load Testing

Key

Controller Purple: Total
Node Light Blue: Total/Total Non-Evictable
Node Orange: Total Evictable (weird spike)

Configuration

• GKE cluster with 1 n1-standard-32 node
• Stable driver v0.5.1
• Disks are 6Gi

Timeline

3:35: I create 50 pods on one node
3:45: Delete 50 pods on one node
3:53 Start 50 pods again
3:59 Delete 50 pods again
4:05 Start 100 pods on one node
4:15 Delete 100 pods

Test Results

Analysis

Test Limitations

• Only used one node so probably not the max stress possible on the controller. Looks like we didn't hit the limit in terms of controller ops with the max pods per node because the CPU was still scaling linearly.
• Node backoff is probably not optimized so we probably didn't hit the max stress possible on the node (however, with the current version of the driver/csi-sidecars with its current backoff settings I believe we hit the max stress)
• 6Gi disks probably smaller than many production disks, larger disks may take longer/more cpu/mem to format

Conclusions

• Controller memory seems to stay stable at ~50MiB regardless of load
  • Setting pod request of 60MiB seems reasonable
• Node memory seems to stay stable at ~40MiB regardless of load
  • Setting pod request of 50MiB seems reasonable
  • Setting pod limit too since this runs on user node 150MiB seems reasonable

Next Steps

• Test with 100 pods on one node with "larger" PD's to stress node more (harder/longer to format?)
• Test with more pods across more nodes to stress controller more (we want to hit a point where the CPU no longer increases with larger pod startup sizes which implies some other component of the system is limiting or parallelism limit is reached)

/cc @verult @msau42

[misterikkit]

Looks like we could comfortably set the Node CPU request at 40m or 60m. Setting a CPU limit is not recommended.

[msau42]

It would be interesting to see if go has profiling tools to get a breakdown of memory usage.

[hantaowang]

cc @jingxu97

[misterikkit]

Based on discussion, it would be interesting to,

• run the Controller on a separate node from the load test
• get some CPU & memory profiles from the Controller & Node components

[davidz627]

Follow-Up No.1

Test

• Tested with same configuration as original but set disk size to "500Gi"

Analysis

• Node CPU and Memory spiked to 250m and 1024Mib+ respectively
  • This is no bueno and I will probably need to do some profiling to figure out whats going on
• Controller CPU usage increased to ~94m, mem held fairly steady but reached ~63Mib
  • I don't currently understand why these would increase with relation to disk size
  • The entirety of the CPU increase is from the gce-pd-driver container
  • The entirety of the MEM increase is from the csi-attacher & csi-provisioner

Next Steps

• Will run controller test that @misterikkit suggested to see if that resolves the controller spikes
  • Running theory is that there are more retries due to cpu/mem pressure since the controller is on the same node as all the testing pods
• Need to profile the node components to identify why cpu and mem are spiking to such incredible heights

[davidz627]

Follow Up No. 2

Test

• Set up cluster with 2 nodes each with n1-standard-32 (32vCPU 120GB MEM)
• Controller on one of the nodes
• First test around 5pm was 100 Pods each with 500Gi disk on the node without the controller
• Brought down around 5:11
• 5:34 Created 100ish Pods (50is on each node) each with 5Gi disk - configuration error, this isn't a test I was trying to run :)
• 5:52 Created 200 Pods (100 on each node) each with 5Gi disk

Analysis

• Looks like Controller CPU peaked at ~65m which is consistent with previous observations (except for the run where we had the node spikes on the same node as the controller)
  • I think we can safely set Controller CPU Request at 70m.
• Controller MEM peaked at 68MiB, which is higher than previous observations but fairly consistent when put under similar loads.
  • Looks like with 200 disks the provisioner started to take a little bit more memory (~7MiB more than 100 disks)
  • I think we can safely set Controller MEM Request at 70MiB
• Node is still wack and uses way too much CPU and MEM

Additional Notes

• At one point I created 100 Pods and discovered I had made a config error so immediately removed the 100 Pods.
  • It looks like some race condition was triggered and although all the Pods were removed successfully, it seems like ~50 PV's were cleaned up and ~50PV's and ~50 Volume Attachments were in various states of stuck (the disks were still attached on backend)
  • Common error was seeing PV is marked for deletion Attach Error, these were not cleaned up in the ~10mins I waited
  • cc @msau42

Next Steps

• Profile Node driver to see why 500Gi disk is chewing up so much CPU and MEM
• Investigate attach/detach race condition

[davidz627]

The large jumps in the node memory usage are evictable memory. The current theory is that mkfs.ext4 uses up a lot of memory for a very short burst of time then the memory becomes evictable. This should not be an issue and we should set node CPU and MEM ignoring evictable memory. CPU increases somewhat proportionally to the size of the disk which seems reasonable. I don't think theres anything we can do here to improve.

Did a lot of profiling and was unable to lock down a single perpetrator for CPU usage. StripSecrets seemed to use a lot

[davidz627]

We could also try using the Vertical Pod Autoscaler: https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler
with it

"Off": VPA does not automatically change resource requirements of the pods. The recommendations are calculated and can be inspected in the VPA object.

this would give us recommendations for cpu and mem resources for the driver

[msau42]

StripSecrets seemed to use a lot

Can we get rid of StripSecrets since it's not actually relevant to our driver and it seems to take up non-trivial cpu usage?