extensions: add http bandwidth limit filter by nitgoy · Pull Request #16358 · envoyproxy/envoy

nitgoy · 2021-05-06T08:01:33Z

Commit Message: adds bandwidth limit filter core functionality and UTs
Additional Description: the filter smoothens the flow of data in both direction up to the specified bandwidth limit.
Risk Level: Low, new filter
Testing: UTs added. Adding more UTs and integration tests.
Docs Changes: Added
Release Notes: Added
Fixes #13604

* Add bandwidthlimit filter skeleton * fix build * Add core logic * fix build * minor fix * Add bandwidthlimit filter skeleton * fix build * Add core logic * fix build * minor fix * minor * add generated configs * final build fixes * config and test related changes Signed-off-by: Nitin <nigoyal@microsoft.com> * fix spellings Signed-off-by: Nitin <nigoyal@microsoft.com> * update generated config Signed-off-by: Nitin <nigoyal@microsoft.com> * move configs to v3alpha * bandwidth limit filter config Signed-off-by: Nitin <nigoyal@microsoft.com> Signed-off-by: Nitin Goyal <nrk.goyal@gmail.com> * Address comments: remove enforce threshold Signed-off-by: Nitin <nigoyal@microsoft.com> Signed-off-by: Nitin Goyal <nrk.goyal@gmail.com> * move proto to alpha Signed-off-by: Nitin <nigoyal@microsoft.com> Signed-off-by: Nitin Goyal <nrk.goyal@gmail.com> * bandwidth limit filter config Signed-off-by: Nitin <nigoyal@microsoft.com> * Address comments: remove enforce threshold Signed-off-by: Nitin <nigoyal@microsoft.com> * change limit field to optional Signed-off-by: Nitin <nigoyal@microsoft.com> Signed-off-by: Nitin Goyal <nrk.goyal@gmail.com> * delete v3 file Signed-off-by: Nitin <nigoyal@microsoft.com> * generated configs Signed-off-by: Nitin <nigoyal@microsoft.com> * add spelling Signed-off-by: Nitin <nigoyal@microsoft.com> * Add v4alpha proto Signed-off-by: Nitin <nigoyal@microsoft.com> * add generated v4alpha configs Signed-off-by: Nitin <nigoyal@microsoft.com> * fix test and build Signed-off-by: Nitin <nigoyal@microsoft.com> * fix config tests Signed-off-by: Nitin <nigoyal@microsoft.com> * Add tests Signed-off-by: Nitin <nigoyal@microsoft.com> * fix tests Signed-off-by: Nitin <nigoyal@microsoft.com> * fix format Signed-off-by: Nitin <nigoyal@microsoft.com> * Add bi-directional test Signed-off-by: Nitin <nigoyal@microsoft.com> * proto format Signed-off-by: Nitin <nigoyal@microsoft.com> * remove soft limit config Signed-off-by: Nitin <nigoyal@microsoft.com> * extension: add http bandwidth limit filter (#3) Add bandwidth limit filter core functionality and UTs. The filter smoothens the flow of data in both direction up to the specified bandwidth limit. Signed-off-by: Nitin <nigoyal@microsoft.com> Signed-off-by: Nitin Goyal <nrk.goyal@gmail.com> * make token bucket conditionally thread-safe and add comment for stream rate limiter Signed-off-by: Nitin <nigoyal@microsoft.com> * fix reset Signed-off-by: Nitin <nigoyal@microsoft.com> * remove some absl guards Signed-off-by: Nitin <nigoyal@microsoft.com> * update filter Signed-off-by: Nitin Goyal <nigoyal@microsoft.com> * update proto Signed-off-by: Nitin Goyal <nigoyal@microsoft.com> * add docs Signed-off-by: Nitin Goyal <nigoyal@microsoft.com> * proto and doc fixes Signed-off-by: Nitin Goyal <nigoyal@microsoft.com> * test fix Signed-off-by: Nitin Goyal <nigoyal@microsoft.com>

repokitteh-read-only · 2021-05-06T08:01:40Z

CC @envoyproxy/api-shepherds: Your approval is needed for changes made to api/envoy/.
envoyproxy/api-shepherds assignee is @markdroth
CC @envoyproxy/api-watchers: FYI only for changes made to api/envoy/.

🐱

Caused by: #16358 was opened by nitgoy.

see: more, trace.

Signed-off-by: Nitin Goyal <nigoyal@microsoft.com>

nitgoy · 2021-05-06T19:36:44Z

Any idea about the format_pre and docs check failure? The tools/code_format/check_format.py fix and docs/build.sh commands are passing locally.

nitgoy · 2021-05-06T21:30:59Z

Actually found this warning with docs build command:
generated/rst/api-v3/extensions/filters/http/bandwidth_limit/v3alpha/bandwidth_limit.proto.rst: WARNING: document isn't included in any toctree
Not sure what am I missing since the file is auto-generated.

adisuissa · 2021-05-06T21:45:41Z

Actually found this warning with docs build command:
generated/rst/api-v3/extensions/filters/http/bandwidth_limit/v3alpha/bandwidth_limit.proto.rst: WARNING: document isn't included in any toctree
Not sure what am I missing since the file is auto-generated.

Not an expert, but I think the idea is that the entire doc will be a tree (not a forest), and so everything can be reached from the main docs page.
Maybe something similar to envoy/docs/root/api-v3/config/accesslog/accesslog.rst will work in this case.

Signed-off-by: Nitin Goyal <nigoyal@microsoft.com>

nitgoy · 2021-05-06T22:01:53Z

Actually found this warning with docs build command:
generated/rst/api-v3/extensions/filters/http/bandwidth_limit/v3alpha/bandwidth_limit.proto.rst: WARNING: document isn't included in any toctree
Not sure what am I missing since the file is auto-generated.

Not an expert, but I think the idea is that the entire doc will be a tree (not a forest), and so everything can be reached from the main docs page.
Maybe something similar to envoy/docs/root/api-v3/config/accesslog/accesslog.rst will work in this case.

I see. It didn't get picked up because http.rst only looks in the v3 folder but not v3alpha folder. Changed that one but not sure if it's acceptable. Please review.

nitgoy · 2021-05-06T23:59:33Z

@adisuissa The docs error got fixed. Any insight into the format_pre failure or how to reproduce it locally?

adisuissa · 2021-05-07T00:12:58Z

@adisuissa The docs error got fixed. Any insight into the format_pre failure or how to reproduce it locally?

According to the pipelines errors:

mixed tabs and spaces: docs/root/configuration/http/http_filters/bandwidth_limit_filter.rst

To reproduce locally, you cant try running: ci/run_envoy_docker.sh 'ci/do_ci.sh format_pre'

Signed-off-by: Nitin Goyal <nigoyal@microsoft.com>

api/envoy/extensions/filters/http/bandwidth_limit/v3alpha/bandwidth_limit.proto

docs/root/configuration/http/http_filters/bandwidth_limit_filter.rst

source/common/common/token_bucket_impl.h

Signed-off-by: Nitin Goyal <nigoyal@microsoft.com>

nitgoy · 2021-05-11T01:29:45Z

/retest

repokitteh-read-only · 2021-05-11T01:29:49Z

Retrying Azure Pipelines:
Retried failed jobs in: envoy-presubmit

🐱

Caused by: a #16358 (comment) was created by @nitgoy.

see: more, trace.

Signed-off-by: Nitin Goyal <nigoyal@microsoft.com>

nitgoy · 2021-05-13T07:51:11Z

/retest

repokitteh-read-only · 2021-05-13T07:51:15Z

Retrying Azure Pipelines:
Retried failed jobs in: envoy-presubmit

🐱

Caused by: a #16358 (comment) was created by @nitgoy.

see: more, trace.

Signed-off-by: Nitin Goyal <nigoyal@microsoft.com>

phlax

lgtm from docs pov - thanks @nitgoy

api/envoy/extensions/filters/http/bandwidth_limit/v3alpha/bandwidth_limit.proto

docs/root/configuration/http/http_filters/bandwidth_limit_filter.rst

phlax

apologies for last minute nits @nitgoy - i would really appreciate if you could fix the literal - it looks wrong with italics

see here:

https://storage.googleapis.com/envoy-pr/ce53ad6/docs/configuration/http/http_filters/bandwidth_limit_filter.html?highlight=bandwidth_limit#statistics

docs/root/configuration/http/http_filters/bandwidth_limit_filter.rst

Signed-off-by: Nitin Goyal <nigoyal@microsoft.com>

nitgoy · 2021-05-24T23:10:48Z

/retest

repokitteh-read-only · 2021-05-24T23:10:52Z

Retrying Azure Pipelines:
Retried failed jobs in: envoy-presubmit

🐱

Caused by: a #16358 (comment) was created by @nitgoy.

see: more, trace.

api/envoy/extensions/filters/http/bandwidth_limit/v3alpha/bandwidth_limit.proto

source/extensions/filters/http/bandwidth_limit/bandwidth_limit.cc

Signed-off-by: Nitin Goyal <nigoyal@microsoft.com>

phlax

lgtm for docs, thanks for cleanups @nitgoy

Signed-off-by: Nitin Goyal <nigoyal@microsoft.com>

the filter smoothens the flow of data in both direction up to the specified bandwidth limit. Risk Level: Low, new filter Testing: UTs added. Adding more UTs and integration tests. Docs Changes: Added Release Notes: Added Fixes envoyproxy#13604 Signed-off-by: Nitin Goyal <nigoyal@microsoft.com>

repokitteh-read-only bot added the api label May 6, 2021

repokitteh-read-only bot assigned markdroth May 6, 2021

nitgoy mentioned this pull request May 6, 2021

extensions: add http bandwidth limit filter #14096

Closed

docs and format fix

e2e9ff3

Signed-off-by: Nitin Goyal <nigoyal@microsoft.com>

yanavlasov self-assigned this May 6, 2021

try fix docs precheck error

42ad169

Signed-off-by: Nitin Goyal <nigoyal@microsoft.com>

nitgoy added 2 commits May 6, 2021 18:05

fix format

bcb06b3

Signed-off-by: Nitin Goyal <nigoyal@microsoft.com>

fix spelling

efc4d21

Signed-off-by: Nitin Goyal <nigoyal@microsoft.com>

yanavlasov requested changes May 7, 2021

View reviewed changes

nitgoy added 4 commits May 7, 2021 17:00

use duration type for fill_interval proto field

0e3ae50

Signed-off-by: Nitin Goyal <nigoyal@microsoft.com>

revert token bucket changes

cab2a33

Signed-off-by: Nitin Goyal <nigoyal@microsoft.com>

format fix

5059bc2

Signed-off-by: Nitin Goyal <nigoyal@microsoft.com>

try format fix again

f4cb248

Signed-off-by: Nitin Goyal <nigoyal@microsoft.com>

nitgoy requested a review from yanavlasov May 8, 2021 00:59

nitgoy added 3 commits May 9, 2021 18:32

fix format pre

e038f7d

Signed-off-by: Nitin Goyal <nigoyal@microsoft.com>

try fix clang tidy

7052748

Signed-off-by: Nitin Goyal <nigoyal@microsoft.com>

fix build errors

d128ac8

Signed-off-by: Nitin Goyal <nigoyal@microsoft.com>

Merge from upstream

e6d805b

Signed-off-by: Nitin Goyal <nigoyal@microsoft.com>

address comment

ce53ad6

Signed-off-by: Nitin Goyal <nigoyal@microsoft.com>

phlax approved these changes May 21, 2021

View reviewed changes

yanavlasov previously approved these changes May 24, 2021

View reviewed changes

lizan reviewed May 24, 2021

View reviewed changes

api/envoy/extensions/filters/http/bandwidth_limit/v3alpha/bandwidth_limit.proto Show resolved Hide resolved

phlax reviewed May 24, 2021

View reviewed changes

docs/root/configuration/http/http_filters/bandwidth_limit_filter.rst Outdated Show resolved Hide resolved

phlax requested changes May 24, 2021

View reviewed changes

docs/root/configuration/http/http_filters/bandwidth_limit_filter.rst Outdated Show resolved Hide resolved

address doc comments

f7e39d9

Signed-off-by: Nitin Goyal <nigoyal@microsoft.com>

nitgoy dismissed yanavlasov’s stale review via f7e39d9 May 24, 2021 18:46

yanavlasov previously approved these changes May 24, 2021

View reviewed changes

rename proto enum values

4fc4837

Signed-off-by: Nitin Goyal <nigoyal@microsoft.com>

nitgoy dismissed yanavlasov’s stale review via 4fc4837 May 24, 2021 20:30

nitgoy requested review from lizan, phlax and yanavlasov May 24, 2021 20:32

yanavlasov previously approved these changes May 25, 2021

View reviewed changes

lizan reviewed May 25, 2021

View reviewed changes

api/envoy/extensions/filters/http/bandwidth_limit/v3alpha/bandwidth_limit.proto Outdated Show resolved Hide resolved

source/extensions/filters/http/bandwidth_limit/bandwidth_limit.cc Outdated Show resolved Hide resolved

make getConfig return ref

54302e9

Signed-off-by: Nitin Goyal <nigoyal@microsoft.com>

nitgoy dismissed yanavlasov’s stale review via 54302e9 May 25, 2021 07:46

phlax previously approved these changes May 25, 2021

View reviewed changes

rename enum and stats values

2a70c24

Signed-off-by: Nitin Goyal <nigoyal@microsoft.com>

nitgoy dismissed phlax’s stale review via 2a70c24 May 25, 2021 10:21

nitgoy requested a review from lizan May 25, 2021 16:12

yanavlasov approved these changes May 25, 2021

View reviewed changes

lizan approved these changes May 25, 2021

View reviewed changes

repokitteh-read-only bot removed the api label May 25, 2021

lizan merged commit a467b0e into envoyproxy:main May 25, 2021

nitgoy deleted the bwlfilter branch June 3, 2021 20:20

Conversation

nitgoy commented May 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

repokitteh-read-only bot commented May 6, 2021

Uh oh!

nitgoy commented May 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nitgoy commented May 6, 2021

Uh oh!

adisuissa commented May 6, 2021

Uh oh!

nitgoy commented May 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nitgoy commented May 6, 2021

Uh oh!

adisuissa commented May 7, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nitgoy commented May 11, 2021

Uh oh!

repokitteh-read-only bot commented May 11, 2021

Uh oh!

nitgoy commented May 13, 2021

Uh oh!

repokitteh-read-only bot commented May 13, 2021

Uh oh!

phlax left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

phlax left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nitgoy commented May 24, 2021

Uh oh!

repokitteh-read-only bot commented May 24, 2021

Uh oh!

Uh oh!

Uh oh!

phlax left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

nitgoy commented May 6, 2021 •

edited

Loading

nitgoy commented May 6, 2021 •

edited

Loading

nitgoy commented May 6, 2021 •

edited

Loading

phlax left a comment •

edited

Loading