App: Limit rate of requests to http queue #18981

ethanwharris · 2023-11-09T22:49:14Z

What does this PR do?

Wraps the HTTPQueue with a RateLimitedQueue object that limits the number of requests to the inner queue (blocking the request until it is allowed). The target number of requests per second is configured with an env variable.

Before: ~10 requests per second

2023-11-09T22:05:59.039309835Z INFO: QUEUE GET 01hev1v0va0vrwrvknm8ts5qb6_DELTA_QUEUE
2023-11-09T22:05:59.144032523Z INFO: QUEUE GET 01hev1v0va0vrwrvknm8ts5qb6_ERROR_QUEUE
2023-11-09T22:05:59.246351460Z INFO: QUEUE GET 01hev1v0va0vrwrvknm8ts5qb6_DELTA_QUEUE
2023-11-09T22:05:59.351695608Z INFO: QUEUE GET 01hev1v0va0vrwrvknm8ts5qb6_ERROR_QUEUE
2023-11-09T22:05:59.454738670Z INFO: QUEUE GET 01hev1v0va0vrwrvknm8ts5qb6_DELTA_QUEUE
2023-11-09T22:05:59.559893554Z INFO: QUEUE GET 01hev1v0va0vrwrvknm8ts5qb6_ERROR_QUEUE
2023-11-09T22:05:59.566447921Z INFO: QUEUE GET 01hev1v0va0vrwrvknm8ts5qb6_ORCHESTRATOR_REQUEST_root.work
2023-11-09T22:05:59.665098012Z INFO: QUEUE GET 01hev1v0va0vrwrvknm8ts5qb6_DELTA_QUEUE
2023-11-09T22:05:59.770642883Z INFO: QUEUE GET 01hev1v0va0vrwrvknm8ts5qb6_ERROR_QUEUE
2023-11-09T22:05:59.871592700Z INFO: QUEUE GET 01hev1v0va0vrwrvknm8ts5qb6_DELTA_QUEUE
2023-11-09T22:05:59.976584576Z INFO: QUEUE GET 01hev1v0va0vrwrvknm8ts5qb6_ERROR_QUEUE
2023-11-09T22:06:00.081320867Z INFO: QUEUE GET 01hev1v0va0vrwrvknm8ts5qb6_DELTA_QUEUE
2023-11-09T22:06:00.183737145Z INFO: QUEUE GET 01hev1v0va0vrwrvknm8ts5qb6_ERROR_QUEUE
2023-11-09T22:06:00.284529924Z INFO: QUEUE GET 01hev1v0va0vrwrvknm8ts5qb6_DELTA_QUEUE
2023-11-09T22:06:00.387931522Z INFO: QUEUE GET 01hev1v0va0vrwrvknm8ts5qb6_ERROR_QUEUE
2023-11-09T22:06:00.493267088Z INFO: QUEUE GET 01hev1v0va0vrwrvknm8ts5qb6_DELTA_QUEUE
2023-11-09T22:06:00.597459733Z INFO: QUEUE GET 01hev1v0va0vrwrvknm8ts5qb6_ERROR_QUEUE
2023-11-09T22:06:00.698971475Z INFO: QUEUE GET 01hev1v0va0vrwrvknm8ts5qb6_DELTA_QUEUE
2023-11-09T22:06:00.810307525Z INFO: QUEUE GET 01hev1v0va0vrwrvknm8ts5qb6_ERROR_QUEUE
2023-11-09T22:06:00.914632895Z INFO: QUEUE GET 01hev1v0va0vrwrvknm8ts5qb6_DELTA_QUEUE

Now: 2-3 requests per second

2023-11-09T22:47:39.619434029Z INFO: QUEUE GET 01hev45myh91vfcxp37y617j53_ERROR_QUEUE
2023-11-09T22:47:39.818290465Z INFO: QUEUE GET 01hev45myh91vfcxp37y617j53_DELTA_QUEUE
2023-11-09T22:47:40.032091437Z INFO: QUEUE GET 01hev45myh91vfcxp37y617j53_ORCHESTRATOR_REQUEST_root.work
2023-11-09T22:47:40.622631077Z INFO: QUEUE GET 01hev45myh91vfcxp37y617j53_ERROR_QUEUE
2023-11-09T22:47:40.816156327Z INFO: QUEUE GET 01hev45myh91vfcxp37y617j53_DELTA_QUEUE

Before submitting

Was this discussed/agreed via a GitHub issue? (not for typos and docs)
Did you read the contributor guideline, Pull Request section?
Did you make sure your PR does only one thing, instead of bundling different changes together?
Did you make sure to update the documentation with your changes? (if necessary)
Did you write any new necessary tests? (not for typos and docs)
Did you verify new and existing tests pass locally with your changes?
Did you list all the breaking changes introduced by this pull request?
Did you update the CHANGELOG? (not for typos, docs, test updates, or minor internal changes/refactors)

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:

Reviewer checklist

Is this pull request ready for review? (if not, please submit in draft mode)
Check that all items from Before submitting are resolved
Make sure the title is self-explanatory and the description concisely explains the PR
Add labels and milestones (and optionally projects) to the PR so it can be classified

📚 Documentation preview 📚: https://pytorch-lightning--18981.org.readthedocs.build/en/18981/

cc @Borda

github-actions · 2023-11-09T22:49:44Z

⚡ Required checks status: All passing 🟢

Groups summary

🟢 lightning_app: Tests workflow

Check ID	Status
app-pytest (macOS-11, lightning, 3.8, latest)	success	✅
app-pytest (macOS-11, lightning, 3.8, oldest)	success	✅
app-pytest (macOS-11, app, 3.9, latest)	success	✅
app-pytest (macOS-12, app, 3.11, latest)	success	✅
app-pytest (ubuntu-20.04, lightning, 3.8, latest)	success	✅
app-pytest (ubuntu-20.04, lightning, 3.8, oldest)	success	✅
app-pytest (ubuntu-20.04, app, 3.9, latest)	success	✅
app-pytest (ubuntu-22.04, app, 3.11, latest)	success	✅
app-pytest (windows-2022, lightning, 3.8, latest)	success	✅
app-pytest (windows-2022, lightning, 3.8, oldest)	success	✅
app-pytest (windows-2022, app, 3.8, latest)	success	✅
app-pytest (windows-2022, app, 3.11, latest)	success	✅

These checks are required after the changes to src/lightning/app/core/constants.py, src/lightning/app/core/queues.py, tests/tests_app/core/test_queues.py.

🟢 lightning_app: Examples

Check ID	Status
app-examples (macOS-11, lightning, 3.9, latest)	success	✅
app-examples (macOS-11, lightning, 3.9, oldest)	success	✅
app-examples (macOS-11, app, 3.9, latest)	success	✅
app-examples (ubuntu-20.04, lightning, 3.9, latest)	success	✅
app-examples (ubuntu-20.04, lightning, 3.9, oldest)	success	✅
app-examples (ubuntu-20.04, app, 3.9, latest)	success	✅
app-examples (windows-2022, lightning, 3.9, latest)	success	✅
app-examples (windows-2022, lightning, 3.9, oldest)	success	✅
app-examples (windows-2022, app, 3.9, latest)	success	✅

These checks are required after the changes to src/lightning/app/core/constants.py, src/lightning/app/core/queues.py.

🟢 lightning_app: Docs

Check ID	Status
docs-make (app, doctest)	success	✅
docs-make (app, html)	success	✅

These checks are required after the changes to src/lightning/app/core/constants.py, src/lightning/app/core/queues.py.

🟢 mypy

Check ID	Status
mypy	success	✅

These checks are required after the changes to src/lightning/app/core/constants.py, src/lightning/app/core/queues.py.

🟢 install

Check ID	Status
install-pkg (ubuntu-22.04, app, 3.8)	success	✅
install-pkg (ubuntu-22.04, app, 3.11)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.8)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.11)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.8)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.11)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.8)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.11)	success	✅
install-pkg (ubuntu-22.04, notset, 3.8)	success	✅
install-pkg (ubuntu-22.04, notset, 3.11)	success	✅
install-pkg (macOS-12, app, 3.8)	success	✅
install-pkg (macOS-12, app, 3.11)	success	✅
install-pkg (macOS-12, fabric, 3.8)	success	✅
install-pkg (macOS-12, fabric, 3.11)	success	✅
install-pkg (macOS-12, pytorch, 3.8)	success	✅
install-pkg (macOS-12, pytorch, 3.11)	success	✅
install-pkg (macOS-12, lightning, 3.8)	success	✅
install-pkg (macOS-12, lightning, 3.11)	success	✅
install-pkg (macOS-12, notset, 3.8)	success	✅
install-pkg (macOS-12, notset, 3.11)	success	✅
install-pkg (windows-2022, app, 3.8)	success	✅
install-pkg (windows-2022, app, 3.11)	success	✅
install-pkg (windows-2022, fabric, 3.8)	success	✅
install-pkg (windows-2022, fabric, 3.11)	success	✅
install-pkg (windows-2022, pytorch, 3.8)	success	✅
install-pkg (windows-2022, pytorch, 3.11)	success	✅
install-pkg (windows-2022, lightning, 3.8)	success	✅
install-pkg (windows-2022, lightning, 3.11)	success	✅
install-pkg (windows-2022, notset, 3.8)	success	✅
install-pkg (windows-2022, notset, 3.11)	success	✅

These checks are required after the changes to src/lightning/app/core/constants.py, src/lightning/app/core/queues.py.

Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

for more information, see https://pre-commit.ci

lantiga

Looks great, we can probably even lower the frequency

src/lightning/app/core/queues.py

tests/tests_app/core/test_queues.py

tchaton

LGTM !

ethanwharris · 2023-11-10T09:36:39Z

@lantiga reduced the default requests per second to 0.5, so in total we will now be looking at ~1 request per second - we can try with this for a bit and then maybe reduce further if all looks good 😃

(cherry picked from commit 9085db4)

App: Limit rate of requests to http queue

cdf9f01

ethanwharris requested review from tchaton, lantiga and awaelchli as code owners November 9, 2023 22:49

github-actions bot added the app (removed) Generic label for Lightning App package label Nov 9, 2023

ethanwharris and others added 2 commits November 9, 2023 22:49

Typo

80014d8

[pre-commit.ci] auto fixes from pre-commit.com hooks

e1db4d2

for more information, see https://pre-commit.ci

lantiga approved these changes Nov 9, 2023

View reviewed changes

ethanwharris commented Nov 9, 2023

View reviewed changes

src/lightning/app/core/queues.py Outdated Show resolved Hide resolved

Update src/lightning/app/core/queues.py

2d45ece

ethanwharris commented Nov 9, 2023

View reviewed changes

src/lightning/app/core/queues.py Outdated Show resolved Hide resolved

Update src/lightning/app/core/queues.py

5382ee9

awaelchli approved these changes Nov 9, 2023

View reviewed changes

tests/tests_app/core/test_queues.py Outdated Show resolved Hide resolved

tests/tests_app/core/test_queues.py Outdated Show resolved Hide resolved

awaelchli added this to the 2.1.x milestone Nov 9, 2023

mergify bot added the ready PRs ready to be merged label Nov 9, 2023

tchaton approved these changes Nov 10, 2023

View reviewed changes

ethanwharris mentioned this pull request Nov 10, 2023

Fix(app): Reduce HTTP Queue Get Request Rates #18983

Closed

7 tasks

ethanwharris added 2 commits November 10, 2023 09:34

Suggestions

1d96857

Reduce default rps

986ce7f

ethanwharris merged commit 9085db4 into master Nov 10, 2023
79 of 81 checks passed

ethanwharris deleted the bugfix/rate-limit-queue branch November 10, 2023 10:27

Borda pushed a commit that referenced this pull request Nov 14, 2023

App: Limit rate of requests to http queue (#18981)

0c21338

(cherry picked from commit 9085db4)

lantiga pushed a commit that referenced this pull request Nov 15, 2023

App: Limit rate of requests to http queue (#18981)

4b81af6

(cherry picked from commit 9085db4)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

App: Limit rate of requests to http queue #18981

App: Limit rate of requests to http queue #18981

ethanwharris commented Nov 9, 2023 •

edited by github-actions bot

Loading

github-actions bot commented Nov 9, 2023 •

edited

Loading

lantiga left a comment

tchaton left a comment

ethanwharris commented Nov 10, 2023

App: Limit rate of requests to http queue #18981

App: Limit rate of requests to http queue #18981

Conversation

ethanwharris commented Nov 9, 2023 • edited by github-actions bot Loading

What does this PR do?

PR review

github-actions bot commented Nov 9, 2023 • edited Loading

⚡ Required checks status: All passing 🟢

Groups summary

lantiga left a comment

Choose a reason for hiding this comment

tchaton left a comment

Choose a reason for hiding this comment

ethanwharris commented Nov 10, 2023

ethanwharris commented Nov 9, 2023 •

edited by github-actions bot

Loading

github-actions bot commented Nov 9, 2023 •

edited

Loading