Repository navigation
master: bound resident memory with a watermark, not a timer - #8
Merged
Merged
Conversation
Replaces the two off-by-default levers added in #7 with one on-by-default mechanism, and removes 66 lines net while doing it. WHAT #7 GOT WRONG. It called malloc_trim(0) on a fixed 60 s interval. That reclaims eventually but cannot bound anything: how much piles up between two ticks is a function of the write rate, so the ceiling moves with the load. It also shipped off by default, which for a process whose RSS otherwise grows until the kernel kills it is the wrong default -- the flag was protecting operators from a fix, not from a risk. WHAT IS ACTUALLY HAPPENING. glibc never returns a free chunk that sits between two live ones. Read from the live master with malloc_info() (INF-363): VmRSS = VmHWM 352884 kB (never receded, for the whole run) glibc holds 465.0 MB of which free 259.2 MB = 55.7% in 210655 chunks across 56 arenas live ~206 MB => RSS is 75% overhead over live data So the excess is not one pool to sweep on a schedule; it is free memory stranded in per-arena free lists, growing with the number of operations performed rather than with what is stored: 233.7 B of permanent RSS per put at a 244k-key working set (R^2 0.988) while the live key count stayed flat to within 1%, against 8.1 B/put with the offload tier off at a matched key population. Per-resident-key fits are noise in every arm (R^2 0.002-0.020) and a stopped writer grows by nothing at all (byte-identical RSS across 22 samples / 105 s), so the axis is operations. WHAT THIS DOES INSTEAD. Trim when the allocator's free bytes exceed a fraction of what it holds, read via mallinfo2(). That bounds resident memory at roughly live x (1 + ratio) at any write rate, which is the difference between a scheduled cleanup and a bound. Verified on the live process: a single trim returned 66.3 MB (21.6% of RSS) in ~300 ms, and driving trims through a churning master held post-trim RSS at 254992 -> 239232 kB across eight cycles while the untreated slope would have added ~110 MB over the same window. Three guards, each for a measured reason. mallinfo2() walks every arena under their locks and the caller loops every 10 ms, so the deciding read gets its own 5 s cadence. The trim costs a few hundred ms, so there is a 30 s floor between trims. And a 64 MiB floor on free bytes, because a small process can sit at a high free fraction while holding almost nothing. None of the three is a knob: no prior art suggests tuning them per deployment. CONFIG SURFACE. Four flags, all default-off, become one with a working default: --malloc_trim_free_ratio (0.25) 0 disables trimming Dropped: --enable_periodic_malloc_trim, --malloc_trim_interval_ms (the timer this replaces), and --enable_metadata_rehash_on_erase / --metadata_rehash_erase_interval (the rehash lever, which was never measured -- it was speculative in #7 and shipping an unvalidated lever is a liability, not an option). Also gone: #7's jemalloc mallctl epoch+purge path, which had never been compiled. Under STORE_USE_JEMALLOC this now says once that jemalloc returns memory on its own decay schedule and that MALLOC_CONF is where to tune it, instead of calling an untested API from here. NOT IN THIS COMMIT, deliberately. Capping glibc's arena count measured a real 39% reduction in the per-op slope (233.7 -> 142.3 B/put with glibc.malloc.arena_max=4) but is not a bound -- RSS still grew monotonically and still never receded -- and it needs no code: it is one environment variable, and it belongs in the deployment manifest, where it also defends against glibc computing its arena cap from the host's core count (128 here) while ignoring the cgroup's 16. jemalloc via LD_PRELOAD remains the deeper structural answer and is likewise an environment change, to be soaked rather than rushed.
Fixes the trigger in this branch's own first draft, which the rig refuted
before it was ever built.
That draft trimmed when mallinfo2() said free bytes exceeded a fraction of held
bytes. Measured across one malloc_trim(0) on the live master:
RSS 313952 -> 246588 kB (-67.4 MB)
free/held 0.323 -> 0.315 (-0.008)
malloc_trim returns the *pages* under free chunks and leaves the *chunks* in
glibc's bins, where they are still the allocator's to hand out. So the free
ratio is very nearly invariant under the action it was supposed to trigger: the
loop would fire, reclaim 67 MB, still read 0.315 > 0.25, and fire again at every
hysteresis window -- a 30-second timer wearing a watermark's rationale, and
strictly worse than the 60-second timer #7 had, since the reasoning was wrong
too. It would also have looked fine in production, because a 30 s timer does
bound RSS at this write rate; the defect would have surfaced as a mystery the
first time someone tuned the ratio and saw nothing change.
The quantity a trim actually moves is RSS, so that is what to gate on:
trim when RSS - rss_at_last_trim > max(64 MiB, ratio x rss_at_last_trim)
This self-resets, because rss_at_last_trim is re-read after each trim and RSS
genuinely drops, and it bounds RSS at rss_at_last_trim x (1 + ratio) with that
floor sitting just above live data. On the numbers above a trim lands RSS near
247 MB, so at 0.25 the next fires near 308 MB against ~240 MB live -- and it
holds at any write rate, since a faster writer only reaches the ceiling sooner.
The read is also cheaper than what it replaces: one line of /proc/self/statm
instead of mallinfo2() walking 56 arenas under their locks. mallinfo2 stays,
but only in the log line, where held-vs-free at the moment of a trim is what
makes the trim's effect legible afterwards.
Flag renamed to match what it now means: --malloc_trim_growth_ratio.
Collaborator
Author
컴파일 게이트 통과magma 두 sentinel 모두 BuildKit 경과시간 접두사가 붙은 줄 — step header 가 RUN 전문을 echo 하므로 접두사 없는 문자열은 통과 증거가 아니라는 레시피 자체 규칙에 따른 판정.
|
This was referenced Aug 20, 2026
This was referenced Sep 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#7 을 대체한다. 같은 증상, 다른 설계 — 주기적 청소를 상한 보장으로 바꾸고, 플래그 4개(전부 기본 OFF)를 1개(기본 ON)로 줄인다.
#7 이 틀린 지점
malloc_trim(0)을 60초 고정 간격으로 불렀다. 이건 결국 회수하지만 아무것도 묶지 못한다 — 두 tick 사이에 얼마나 쌓이는지는 전적으로 쓰기 속도의 함수라서 천장이 부하에 따라 움직인다. 게다가 기본 OFF 였는데, RSS 가 커널이 죽일 때까지 자라는 프로세스에서 그 플래그는 위험이 아니라 픽스로부터 운영자를 보호하고 있었다.실제로 무슨 일이 일어나는가
glibc 는 살아있는 청크 두 개 사이에 낀 free 청크를 결코 OS 에 반납하지 않는다. 살아있는 master 에서
malloc_info()로 읽은 것:즉 일정에 맞춰 쓸어낼 단일 풀이 아니라, arena 별 free 리스트에 좌초된 메모리이고, 저장량이 아니라 수행한 연산 수에 비례해 자란다: 244k 키 워킹셋에서 put 당 영구 RSS 233.7 B(R² 0.988), 그 동안 live 키 수는 1% 내로 평탄. offload tier 를 끄고 키 인구를 맞추면 8.1 B/put. per-resident-key 적합은 전 arm 에서 노이즈(R² 0.002–0.020)이고, writer 를 멈추면 성장이 0(22 샘플·105초 바이트 단위로 동일) — 축은 연산이다.
★ 트리거를 RSS 로 잡은 이유 (이 브랜치 자신의 초안을 리그가 반증했다)
당연해 보이는 설계는 "
mallinfo2()로 free/held 비율을 보고 임계 넘으면 trim" 이다. 작동하지 않는다. 살아있는 master 에서 trim 한 번 전후:malloc_trim은 free 청크 아래의 페이지를 반납하고 청크 자체는 glibc bin 에 남긴다(여전히 allocator 가 나눠줄 몫이다). 그래서 비율은 자기가 트리거해야 할 동작에 대해 거의 불변이고, 루프는 trim → 67MB 회수 → 여전히 0.315 > 0.25 → 다음 hysteresis 창에 또 trim 을 반복한다. 워터마크의 논리를 입은 30초 타이머이고, #7 의 60초 타이머보다 나쁘다(논리까지 틀렸으므로). 프로덕션에서는 멀쩡해 보였을 것이다 — 이 쓰기 속도에서 30초 타이머도 RSS 를 묶기는 하니까. 결함은 순전히 추론에 있었고, 누군가 ratio 를 조정해보고 아무 변화가 없을 때 미스터리로 드러났을 것이다.trim 이 실제로 움직이는 양은 RSS 다:
rss_at_last_trim을 매 trim 후 다시 읽으므로 구조적으로 자기 리셋되고, 그 바닥값이 live 바로 위에 앉으므로 RSS ≤ rss_at_last_trim × (1 + ratio) 를 묶는다. 위 숫자로: trim 이 RSS 를 ~247 MB 에 내려놓으니 0.25 에서 다음은 ~308 MB 에서 발동, live ~240 MB 대비. 쓰기 속도와 무관하게 성립한다 — 빠른 writer 는 천장에 더 빨리 도달할 뿐이다. 그게 일정과 상한의 차이이고, 그래서 기본 ON 이다.판정 읽기도 더 싸졌다: arena 56개를 락 잡고 순회하는
mallinfo2()대신/proc/self/statm한 줄.mallinfo2는 로그 줄에만 남긴다 — trim 시점의 held/free 는 사후에 효과를 읽는 데 실제로 유용하다.설정 표면
기본 OFF 플래그 4개 → 작동하는 기본값을 가진 1개:
삭제:
--enable_periodic_malloc_trim,--malloc_trim_interval_ms(이 설계가 대체),--enable_metadata_rehash_on_erase/--metadata_rehash_erase_interval(rehash 레버 — 한 번도 측정된 적 없다. #7 에서 추측이었고, 미검증 레버를 출하하는 것은 선택지가 아니라 부채다). #7 의 jemallocmallctlepoch+purge 경로도 삭제 — 컴파일된 적이 없었다.STORE_USE_JEMALLOC빌드에서는 이제 "jemalloc 은 자기 decay 일정으로 반납하니MALLOC_CONF로 튜닝하라"고 한 번 말하고 빠진다.일부러 이 커밋에 없는 것
glibc arena 상한은 per-op 기울기를 실측 39% 줄였다(
glibc.malloc.arena_max=4에서 233.7 → 142.3 B/put). 하지만 상한이 아니다 — arm F 도 여전히 단조 증가하고 한 번도 안 내려온다 — 그리고 코드가 필요 없다: env 한 줄이고, 배포 manifest 에 속한다(거기서 glibc 가 cgroup 16 을 무시하고 호스트 코어 128 로 arena 상한을 잡는 것도 함께 방어한다). magma #312 에 넣었다.jemalloc via
LD_PRELOAD는 더 깊은 구조적 답으로 남는다(background_thread+ decay 튜닝, 재빌드 불요). 역시 환경 변경이고, 서둘러 넣을 것이 아니라 soak 할 것이다.검증 상태
Dockerfile.unittest) — 통과 전 merge 안 함세션 기록 = inference-playbook
autoresearch/2026-08-20-inf363-master-metadata-leak/(arm A–F 원자료, 소스 감사, upstream 스캔, glibc 선행연구 조사).
upstream(kvcache-ai#3452) 에는 게시하지 않는다 — 초안만(운영자 지시).