Repository navigation
[TransferEngine] Keep the TCP connection pool off by default - #19
Merged
Merged
Conversation
Upstream kvcache-ai#3569 turned the TCP connection pool on by default, to fix several instances sharing one IP and ephemeral-port exhaustion with the pool off (kvcache-ai#3540). With P/D disaggregation over mooncake TCP, the pool is the unsafe setting for us: measured in both directions on the same cell, enabling it made 5-41% of requests return HTTP 200 with zero completion tokens -- prefill done, the decode stream never carrying a token. Upstream's kvcache-ai#2930 (stale pooled connections to a restarted peer are never evicted; large multi-slice transfers stall silently) is still open, and none of the TCP fixes since claims to cure that failure class. Restore the pre-kvcache-ai#3569 default. MC_TCP_ENABLE_CONNECTION_POOL still decides when set: 0/false/no turns the pool off, any other value turns it on, so a workload that needs the pool opts in explicitly. Upstream: none (carry-only)
Juhyun-Kim-Memphis
added
upstream:carry-only
Juhyun-Kim-Memphis
added
upstream:carry-only
Juhyun-Kim-Memphis
marked this pull request as ready for review
September 21, 2026 10:10
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
무엇을 바꾸는가
mooncake-transfer-engine/include/transport/tcp_transport/tcp_transport.h의enable_connection_pool_기본값을true→false. 한 줄이며, upstream kvcache-ai#3569(36fdd899, 2026-08-24 병합, v0.3.13 포함) 이전의 기본값으로 되돌립니다.환경 변수
MC_TCP_ENABLE_CONNECTION_POOL은 그대로 우선합니다. 값이0/false/no면 끄고, 다른 값이면 켭니다. 그러니 pool이 필요한 워크로드는 명시적으로 켜면 됩니다.왜
engines/solar-mini4/README.md의 Constraints: P/D disaggregation과 함께 pool을 켜면 TTFT p50은 약 30% 줄지만, 요청의 5–41%가 HTTP 200에 completion token 0개로 끝납니다(prefill은 끝났는데 decode stream에 token이 한 번도 실리지 않음). 켜고 끄는 양방향으로 측정했습니다(magmaengines/solar-mini4/2026-09-10-betelgeuse.md).MC_TCP_ENABLE_CONNECTION_POOL=0을 명시하지 않은 모든 TCP pod가 조용히 pool을 쓰게 됩니다. 매니페스트마다=0을 박는 대신 fork 기본값을 되돌립니다.upstream 상태와 들고 있는 비용
if (enable_connection_pool_ || reuse_connection)). 이는 우리 base 이전부터의 동작이고 이 PR은 바꾸지 않습니다.검증
wheel 버전 PR의 커밋과 이 커밋을 쌓은 브랜치(
inf-522/tcp-pool-default-off,794ce2d8)로 확장 unit gate를 한 번 돌렸습니다: PASS, 로컬 Docker(linux/amd64, w-0529 점유로 BuildKit Job 대신), attest794ce2d800e85da6c06efe30322f61820d076637, 09:07:46Z → 09:30:23Z(22분 37초), suite 9개100% tests passed, 0 tests failed out of 9.이 gate는 store suite만 돌리지만,
file_storage_test등은 in-process master와tcptransport를 쓰므로 새 기본값으로 실행됐습니다. transfer engine의 TCP 테스트 중 pool을 쓰는 것들은MC_TCP_ENABLE_CONNECTION_POOL=1을 스스로 설정합니다.Upstream: none (carry-only)
Refs: INF-522