Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

configure maxRatePerInstance in backend #545

Closed
ukai opened this issue Nov 12, 2018 · 8 comments
Closed

configure maxRatePerInstance in backend #545

ukai opened this issue Nov 12, 2018 · 8 comments
Assignees
@rramkumar1
Labels
kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt.

Comments

@ukai
Copy link

ukai commented Nov 12, 2018

ingress configures backend with maxRatePerInstance=1 like

backends:

This comment says maxRatePerInstance value doesn't matter

ingress-gce/pkg/backends/ig_linker.go

Line 57 in 6583318

// maxRPS is the RPS setting for all Backends with BalancingMode RATE. The exact

but actually I experienced lots of 502 errors when it received too many requests per second.

I manually raised maxRatePerInstance (e.g. to 20.0), 502 errors disappeared.
can we configure maxRatePerInstance by ingress annotation or so.

we might also want to configure timeoutSec too.
@rramkumar1
Copy link
Contributor

rramkumar1 commented Nov 12, 2018 •
edited
Loading

@ukai I have another option to alleviate this which would allow us not to expose it.

We could just increase the default maxRatePerInstance we set from 1 to N (possibly even the max int as shown in the comment).

/kind cleanup

@k8s-ci-robot k8s-ci-robot added the kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. label Nov 12, 2018
@rramkumar1
Copy link
Contributor

We need to consider though how changing that value might effect existing users.

I'll give it some thought before creating a PR.

/assign

@bowei
Copy link
Member

bowei commented Nov 12, 2018

Do you have a reproable configuration? As far as we know, this should not result in 502s...

@ukai
Copy link
Author

ukai commented Nov 13, 2018

ah, changing maxRatePerInstance doesn't change rate of 502s for a day.

anyway, what caused the 502 response?
client got

HTTP/1.1 502 Bad Gateway
Content-Type: text/html; charset=UTF-8
Referrer-Policy: no-referrer
Content-Length: 332
Date: Tue, 13 Nov 2018 01:44:18 GMT
Alt-Svc: clear

<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>502 Server Error</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Server Error</h1>
<h2>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.</h2>
<h2></h2>
</body></html>

@ukai
Copy link
Author

ukai commented Nov 13, 2018

hmm, when pod is deleted while handling http request and terminated without response, ingress (load balancer) returns 502?

@ukai
Copy link
Author

ukai commented Nov 14, 2018

by making backend server robust (less eviction, OOM killed), we could reduce 502 errors, so might not need to set maxRatePerInstance.

@rramkumar1
Copy link
Contributor

@ukai Can we close this?

@ukai
Copy link
Author

ukai commented Nov 15, 2018

yeah

@ukai ukai closed this as completed Nov 15, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rramkumar1 rramkumar1

Labels
kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@bowei @ukai @k8s-ci-robot @rramkumar1