Envoy Proxy produce a lot of cpu load

[max-soe]

Summary

We are currently testing the route integrity feature in the version 2.34 ( cf-deploment 9.5 ). We noticed that the envoy process produces a lot of cpu load on every container, which can affect the running app subsequently.

Before we opened this issue we also discussed this topic in slack.

Expected Result

We expected that this new feature will increase the cpu usage a little bit. The running app should't be affected by the envoy and roughly running as before the feature was enabled.

Actual Result

We created a little python app and ran it on a diego cell with 8Gb Memory and 2 cpus. The app got 4GB memory quota. Then we produced load on the app by using vegeta. For this we used the following command:

echo "GET https://<ROUTE-TO-APPLICATION>" | vegeta -cpus=1 attack -insecure -duration=60s -max-body=0 -rate=<NUMBER-OF-REQUESTS>/1s -connections=20 | vegeta report | grep -v ^Get

We documented the results of cpu usage from the envoy and our app in the following table:

The last row shows a test with route-integrity disabled, so after the feature was enabled, the container could only handle 38% of the number of requests per second.

Context

• diego release 2.34
• gorouter connection properties:

disable_keep_alives: false
max_idle_conns: 49500
max_idle_conns_per_host: 100
frontend_idle_timeout: 900s
backends:
  max_conns: 500

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/167450587

The labels on this github issue will be updated when the story is started.

[sunjayBhatia]

@max-soe can you provide the python app that you used to reproduce this?

[max-soe]

@sunjayBhatia yes no problem, I attached it here. But does the python app have an impact on the cpu usage of the envoy proxy? Or do you only need this for testing?

[sunjayBhatia]

It potentially has an impact, we were reminded that in the past we have seen that the python BaseHTTPServer defaults to HTTP 1.0 and closes connections after each request and that the SimpleHTTPServer only handles one connection at a time

[max-soe]

Okay thank you for this hint! We will test this and add our results here.

[heyjcollins]

Hey @max-soe - just swinging back to see if you've completed your testing and have results to share.

Thank you!

[max-soe]

Hey @heyjcollins - We can test on our bigger scale landscape in the next week again, to confirm our results. The hint from @sunjayBhatia was good, but after all the envoy produce some load on the cpu. We will share our final results next week here...

[max-soe]

We are now finished our route integrity tests. First good news, if apps run with HTTP 1.1 or higher, the CPU load of envoy is much lower than that of the previous tests.

In the last time we did a lot of different tests with route integrity. I will give you a short summary over the importance results for diego:

In our tests we used cedar stress-app as server and the golang app hey as client.
We also used 12 GoRouter with 4cpus and 8gb memory to handle the load that we produced on the apps.

With this setup we could send 9000 request per second on one app instance without route integrity. The app used for that round about 90% CPU. The memory was under 1%.

After we enable route integrity, the same app used again round about 90% CPU. But the envoy used 60% too. So the container had a CPU usage of 140% and exceeded the cpu limit.
But of course 9000 req/sec on one instance is a very high load.

So we reduced the load and got the following results:

• 2000 req/sec produced 18% CPU usage
• 1000 req/sec produced 9% CPU usage
• 500 req/sec produced 5% CPU usage

So the CPU usage is not as bad as we thought, but afterall it can have an impact on the container and the running app.
Most of all small apps can now run into our cpu share limits more easily.

If you are interested in further details please contect me. Further hints to reduce CPU usage are also welcome ;-)

[sunjayBhatia]

Envoy has a long-standing issue with parsing HTTP 1 headers:

• Performance Issues with Envoy envoyproxy/envoy#5536
• perf: reduce the CPU usage of HTTP/1.1 header parser envoyproxy/envoy#2880

To be fair though, we are not configuring envoy as an HTTP proxy, just a TCP proxy, so I'm not sure how relevant these issues are.

[sunjayBhatia]

The only Envoy configuration we have been able to set to produce any difference is the --concurrency flag (docs here). Setting this to something lower than the hardware threads available on your diego-cell VM does produce less CPU usage but of course comes at a cost in throughput.