Skip to content

fix(router): config reload memory leak — graphMux nil + proto reset - #3

Merged
arutkowski00 merged 13 commits into
monday-tweaksfrom
feature/adamru/fix-config-reload-memory-leak
Jun 25, 2026
Merged

fix(router): config reload memory leak — graphMux nil + proto reset#3
arutkowski00 merged 13 commits into
monday-tweaksfrom
feature/adamru/fix-config-reload-memory-leak

Conversation

@arutkowski00

Copy link
Copy Markdown
Collaborator

Summary

  • Fix 1 (d2d20a910): Close and nil Ristretto plan/operation caches on shut-down graphMux; nil wsHandler/mux; remove from graphMuxList. Local mirror: ~74 → ~12 MB inuse/reload.
  • Fix 2 (3b6aadb27): SHA-256 skip when mapper.json unchanged; proto.Reset(old) after swapping staticExecutionConfig. Local mirror: ~12 → ~flat MB/reload.
  • Fix 3 reverted (73fc25d68): Manifest Changes/Hashes mux reuse did not improve base-hash reload benchmark.

Builds on Rounds 1–6 already on this branch (84f798f92d80b9add6).

Test plan

  • platform-api PR deploy to staging (feature/adamru/fix-config-reload-memory-leak)
  • Fresh pod baseline ~300 Mi RSS / forced-GC inuse capture on :6060
  • 3 controlled CDN pushes (same as Round 6 protocol)
  • Compare: RSS per reload, inuse delta, AddType flat, target <<96 Mi/reload (Round 6 baseline)

Wiki: projects/2026-06-24-cosmo-router-memory-leak-investigation.md

arutkowski00 and others added 10 commits June 24, 2026 18:11
Stop storing schemaDocument in cached planWithMetaData entries so plan
caches no longer pin the old router schema AST (~200MB) after CDN reloads.
Also call OnRouterConfigReload before building a new graph server so
slow-plan cache entries are extracted while the old graphMux is still
referenced, matching the supervisor restart path.

Co-authored-by: Cursor <cursoragent@cursor.com>
Disable ristretto OnEvict migration into slowplancache when a graphMux is
shutting down, since Close() clears every entry and the fallback cache is
about to be closed anyway. Close websocket subscriptions synchronously
before plan caches so preparedPlan and executor refs are released first.

Co-authored-by: Cursor <cursoragent@cursor.com>
Expose /debug/pprof/heap and related routes on the pprof server so
forced-GC heap snapshots (heap?gc=1) work for memory leak diagnosis.

Co-authored-by: Cursor <cursoragent@cursor.com>
Flag defaults are captured at package init before embedders set PPROF_ADDR
in main(), so platform-api ensurePprofAddr had no effect. Re-read env after
flag.Parse() matches the existing CONFIG_PATH pattern.

Co-authored-by: Cursor <cursoragent@cursor.com>
…head on reload

Executor.Close() nils federation schema AST held after graph mux shutdown.
Share one upstream subscription client across subgraph factories and disable
upstream ping loops when client WebSocket is disabled.

Co-authored-by: Cursor <cursoragent@cursor.com>
… unused

Skip WSTransport/SSE initialization when the router schema has no
subscription root fields or when client WebSocket and pubsub events are
both disabled.

Co-authored-by: Cursor <cursoragent@cursor.com>
Close and drop Ristretto cache pointers, wsHandler, and mux after graphMux
shutdown, and remove shut-down muxes from graphMuxList. Local benchmark:
~74 MB/reload → ~12 MB/reload retained inuse (same-content manifest reloads).

Co-authored-by: Cursor <cursoragent@cursor.com>
…ion config

Hash mapper.json before re-assembling; skip reload when content is
unchanged (mtime-only touches). After a successful reload, swap
staticExecutionConfig and proto.Reset the previous config so decoded
protojson strings can be collected.
…n refs

Pass Changes/Hashes from mapper.json graph hashes on the manifest watcher
path so unchanged base or feature-flag muxes survive config reloads.
Nil graphServer and graphMux metric fields after shutdown to drop retained
references sooner.
@github-actions

github-actions Bot commented Jun 24, 2026

Copy link
Copy Markdown

Router-nonroot image scan failed

❌ Security vulnerabilities found in image:

ghcr.io/mondaycom/cosmo/router:sha-f2177a63ccd14d49fe17accd6c5398c242608bb7-nonroot

Please check the security vulnerabilities found in the PR.

If you believe this is a false positive, please add the vulnerability to the .trivyignore file and re-run the scan.

Remove late env re-read for PPROF/PYROSCOPE in main.go and extra pprof
handlers in profile.go so this PR stays focused on config reload fixes.
Centralize all monday.com config-reload leak fixes in mondaytweaks.go so
they are easy to audit and disable individually. Restore profiling helpers
from stash behind separate tweak flags.
Remove PPROF/PYROSCOPE env re-read, heap pprof routes, and Pyroscope
name/tag helpers so the PR stays focused on config reload memory fixes.
@arutkowski00
arutkowski00 merged commit ec5d35e into monday-tweaks Jun 25, 2026
16 of 29 checks passed
@arutkowski00
arutkowski00 deleted the feature/adamru/fix-config-reload-memory-leak branch June 25, 2026 15:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants