Repository navigation
fix(server_utils): enable TCP keepalive on global aiohttp connector #1959
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 1 commit
4ebef8a
4248314
10906a8
9d34aa7
7094d40
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -16,6 +16,7 @@ | |
| import atexit | ||
| import json | ||
| import resource | ||
| import socket | ||
| import sys | ||
| import time | ||
| from abc import abstractmethod | ||
|
|
@@ -100,14 +101,36 @@ def get_global_aiohttp_client( | |
| return set_global_aiohttp_client(cfg) | ||
|
|
||
|
|
||
| class _TCPKeepAliveConnector(TCPConnector): | ||
| _KEEPALIVE_IDLE_SECONDS: int = 60 | ||
| _KEEPALIVE_INTERVAL_SECONDS: int = 10 | ||
| _KEEPALIVE_PROBES: int = 3 | ||
|
|
||
| async def _wrap_create_connection(self, *args, **kwargs): | ||
|
ananthsub marked this conversation as resolved.
Resolved
|
||
| transport, protocol = await super()._wrap_create_connection(*args, **kwargs) | ||
| sock = transport.get_extra_info("socket") | ||
| if sock is None: | ||
| raise RuntimeError("_TCPKeepAliveConnector: transport has no underlying socket") | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. RISK — fail-hard on a keepalive optimization inverts the intended failure mode. WHAT BREAKS: BLAST RADIUS: a feature added to prevent stalled rollouts becomes a total-outage switch. Losing keepalive is a degraded state (back to today's behavior); losing all HTTP is an outage. The downside of the guard is strictly worse than the problem it guards against. FIX: degrade gracefully instead of raising —
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Switched the |
||
| sock.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1) | ||
| for opt_name, opt_value in ( | ||
| ("TCP_KEEPIDLE", self._KEEPALIVE_IDLE_SECONDS), | ||
| ("TCP_KEEPINTVL", self._KEEPALIVE_INTERVAL_SECONDS), | ||
| ("TCP_KEEPCNT", self._KEEPALIVE_PROBES), | ||
| ): | ||
| opt = getattr(socket, opt_name, None) | ||
| if opt is not None: | ||
| sock.setsockopt(socket.IPPROTO_TCP, opt, opt_value) | ||
| return transport, protocol | ||
|
|
||
|
|
||
| def set_global_aiohttp_client(cfg: GlobalAIOHTTPAsyncClientConfig) -> ClientSession: # pragma: no cover | ||
| assert not is_global_aiohttp_client_setup(), ( | ||
| "There is already a global aiohttp client setup. Please refactor your code or call `global_aiohttp_client_exit` if you want to explicitly re-make the client!" | ||
| ) | ||
|
|
||
| num_workers = get_nemo_gym_fastapi_num_workers() | ||
| client_session = ClientSession( | ||
| connector=TCPConnector( | ||
| connector=_TCPKeepAliveConnector( | ||
| limit=cfg.global_aiohttp_connector_limit // num_workers, | ||
| limit_per_host=cfg.global_aiohttp_connector_limit_per_host // num_workers, | ||
| keepalive_timeout=15.0, | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can we parameterize this via GlobalAIOHTTPAsyncClientConfig? and what are the defaults for these values if we don't specify them, can we default to those same defaults or at least display them in a comment here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good idea. I moved the three TCP keepalive values (
TCP_KEEPIDLE,TCP_KEEPINTVL,TCP_KEEPCNT) toGlobalAIOHTTPAsyncClientConfigfields so they're overridable per deployment. Linux kernel defaults are 7200s / 75s / 9 probes, so ~2h15m before TCP gives up. I think that's longer than most benchmarks need, so I used shorter times.