fix(local-inference): RAM-class inference-memory policy — idle unload + pressure release so lmkd stops killing the app (#11760)

[lalalune]

Closes #11760.

Policy chosen

RAM-class-scoped footprint caps + idle unload + pressure-reactive release of the resident inference state — in the existing processes. No android:process isolation.

The #11506 forensics show lmkd killing ai.elizaos.app at ~3.2 GB RSS on the 5.7 GB Pixel 6a because the resident inference state (model weights + KV cache + compute buffers — 2.25 GB of GL mtrack on the Vulkan host, ~1.2 GB of anon heap on the CPU path) is pinned for the life of the process, keeping the app the largest resident process at all times. Three levers fix that root cause directly:

1. RAM-class defaults (issue ask b/d). A single classifier — CONSTRAINED below 7 GiB usable (Pixel 6a reports 5.7 GiB; 8 GB-nominal devices report ≥7.3 GiB) — drives the resident LLM context: ELIZA_LLM_N_CTX 4096 (constrained) vs 8192 (standard) on the bionic host, and the branded in-process default drops 16384→8192 on constrained devices. ELIZA_LLM_N_BATCH=128 (the compute-buffer lever) was already in place from the 8 GB-Mali work and is kept. Model tier needs no change: 2B Q4 is already both the catalog floor and the hardcoded mobile default.
2. Idle unload (issue ask a). The resident model + context + stream are freed after a RAM-class idle window — 5 min constrained / 30 min standard (debug.eliza.inference.idle_unload_ms / ELIZA_LOCAL_IDLE_UNLOAD_MS override; 0 disables). Reload cost is the documented ~7-8 s resident rebuild (ensureResidentCtx) or the in-process lifecycle reload (ensureChatLoaded) — both pre-existing, both lossless (agent/app state lives outside the inference context; per-turn KV is re-prefilled anyway).
3. Pressure release (issue ask c + the acceptance OR-arm).
   • App process (bionic host): onTrimMemory relay (RUNNING_LOW/RUNNING_CRITICAL/BACKGROUND+) + a 30 s ActivityManager.getMemoryInfo() poll that releases within 512 MiB of the lmkd threshold on constrained devices (trim callbacks are deprecated/unreliable on API 34+; the poll is the dependable path).
   • Bun agent (in-process leg): the agent process never receives onTrimMemory, so /proc/meminfo MemAvailable is its pressure signal — release below 12 % available (constrained) / 5 % (standard).

Why not process isolation: the Vulkan inference already runs in the only process that can reach the Mali driver (ElizaBionicInferenceServer, in-app JNI — the musl bun agent's linker namespace cannot load the Vulkan HIDL closure). A separate :inference process would duplicate the JNI host + UDS plumbing, add its own overhead, and complicate the #11738-hardened service lifecycle — while the reclaim semantics it buys ("inference memory dies, app survives") are exactly what explicit release already delivers: resetResident() frees the GL mtrack deterministically and the next turn reloads. If post-policy field data still shows lmkd taking the app while a session is actively hot, isolation is the documented escalation; it is not needed to meet the acceptance.

The #11738 FGS-restart behavior is untouched (no changes to the restart/denial paths; onTrimMemory is additive).

Implementation

Android (app process):

• InferenceMemoryPolicy.java (new) — pure decision logic: RAM-class classification (+ debug.eliza.inference.ram_class override for emulator verification), per-class n_ctx + idle windows, trim-level decisions, availMem-margin decisions. 17 JVM unit tests (InferenceMemoryPolicyTest.java, first tests in :app:testDebugUnitTest).
• ElizaBionicInferenceServer.java — RAM-class ELIZA_LLM_N_CTX default; single daemon scheduler (30 s tick) for idle + pressure release; releaseResident(reason) frees the resident model/ctx/stream under residentLock (an in-flight decode always finishes its turn first; the idle window is re-checked under the lock); loud structured release logs.
• ElizaAgentService.java — onTrimMemory relay (async; never blocks the main thread), ActivityManager-backed pressure probe, policy-decision log, and RAM-class env export to the bun agent (ELIZA_INFERENCE_RAM_CLASS, ELIZA_LOCAL_IDLE_UNLOAD_MS), constrained-branded n_ctx 8192.

Bun agent (in-process AOSP leg):

• plugins/plugin-aosp-local-inference/src/inference-memory-policy.ts (new) — mirrored classifier (env → /proc/meminfo), idle resolution, MemAvailable pressure check, and InferenceIdleUnloader (in-flight-aware; never unloads mid-generate).
• aosp-local-inference-bootstrap.ts — every loader call routes through idle tracking; unload flows through the existing unloadModel + markEvicted → ensureChatLoaded reload plumbing (same path the voice handlers' out-of-band eviction already uses). 24 policy tests + full plugin suite green (99).

Bug found + fixed by the on-device verification: the in-process AOSP chat load still requested the retired QJL/PolarQuant KV types (qjl1_256/q4_polar). Current fused libs route those into the stock llama.cpp V-quant check, which hard-fails without flash-attn (deliberately off on Android — the scalar-FA race), so every in-process generate died with llm_stream_open: failed to init llama context on the stock debug APK. Chat KV is now the Gemma path — q8_0 K (halves the K cache, no FA needed, matches ELIZA_1_KV_QUANT) / f16 V — with ELIZA_LLAMA_CACHE_TYPE_K/V env overrides.

Gate (issue ask 3):

• packages/benchmarks/mobile-resource: new idle-reclaim workload (runs last, after the chat workloads leave the model warm) + maxPostIdleUnloadRssMb budget — android-phone 1600 MB for both tiers. A regression that stops the unload leaves tail RSS at the warm footprint (≥2.3 GB measured) → fails loud. checkBudgets gained per-workload check selection; 13 node --test cases green.

Emulator verification (arm64 AVD, Android 15, CONSTRAINED — MemTotal ≈ 2.9 GiB at soak time)

Full bundle: .github/issue-evidence/11760-inference-memory-policy/. Highlights:

• Policy fires — three constrained decisions logged from both the Java probe and the TS bootstrap: ramClass=CONSTRAINED totalMem=2975MB lmkThreshold=216MB nCtx=4096 idleUnloadMs=120000.
• On-device generate works (after this soak surfaced + fixed the retired-QJL-KV bug): fast path done, latencyMs=3782, cacheTypeK=q8_0 cacheTypeV=f16.
• Release reclaims ~700 MB, lossless — the in-process pressure lever fired twice on-device (MemAvailable=233MB and 321MB, both ramClass=constrained). The second correlates exactly in meminfo-timeline.tsv: bun agent VmRSS 1.36 GB (loaded) → 640 MB (freed) → 1.5 GB (reloaded on the next request) while MemAvailable rose ~570 MB — the app was never killed.
• Acceptance met — during the soak lmkd fired 38 kills across 28 other packages under "low watermark is breached", and ai.elizaos.app was killed 0 times; exit-info is clean of reason=3 (LOW_MEMORY) (its only exits are this session's USER REQUESTED force-stops + PACKAGE UPDATED reinstalls). This is the on-device inference memory: LMK kills the app at ~3.2 GB RSS (in-process Vulkan pins 2.3-2.7 GB) — need an unload / smaller-tier / isolate-inference policy #11760 acceptance shape: the app is no longer lmkd's target under real ambient pressure.

Honest deltas vs the Pixel 6a target

• The session-shared AVD measured CONSTRAINED (2.9 GiB — harder than the 5.7 GB Pixel 6a) and runs the CPU path (no libggml-vulkan.so staged → no Mali GL-mtrack term; the identical resetResident/unloadModel release plumbing is unit-tested on both legs). The idle lever specifically kept losing the race to the pressure lever (correct: the emulator was genuinely pressured) and the app was restarted by sibling sessions every ~2-3 min, so the idle trigger on-device is covered by the unit tests rather than an isolated soak capture; the release mechanism it shares with pressure is proven on-device.
• The device-exact 10-min Pixel 6a soak (Mali GL mtrack + that device's lmkd tables) is the hardware-lab row → hardware-lab: on-device battery/power/thermal capture + flip mobile-resource-workbench device lanes to gating #11734.
• The budget (maxPostIdleUnloadRssMb=1600) is a first emulator-informed ceiling; ratchet per BASELINE.md once the Pixel row lands.

Verification

• bun test __tests__ (plugin-aosp-local-inference): 99 pass.
• :app:testDebugUnitTest: 17 pass; :app:compileDebugJavaWithJavac clean.
• node --test metrics.test.mjs: 13 pass.
• bunx tsc --noEmit (plugin): clean; biome clean on touched files.
• On-emulator soak: see evidence.

🤖 Generated with Claude Code

[greptile-apps]

Your trial has ended. Reactivate Greptile to resume code reviews.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: e8ff2955-178f-486c-9d9f-4781a7f910c3

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/11760-inference-memory-policy

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[greptile-apps]

Your trial has ended. Reactivate Greptile to resume code reviews.

[greptile-apps]

Your trial has ended. Reactivate Greptile to resume code reviews.

[lalalune]

Rebased and cleaned the PR branch on top of current origin/develop (after #11823). The accidental generated plugin artifacts from plugin-anthropic-proxy / plugin-hyperliquid are no longer in the diff; remaining scope is Android/app-core memory policy, AOSP local-inference memory policy, mobile-resource budget gate, docs, and #11760 evidence.

Validation on 64c030100c:

• bun test __tests__ in plugins/plugin-aosp-local-inference -> 100 pass
• node --test packages/benchmarks/mobile-resource/metrics.test.mjs -> 10 pass
• bun run --cwd plugins/plugin-aosp-local-inference typecheck -> pass
• bunx biome check on the touched TS/JS/Java files -> pass for supported files
• node packages/shared/scripts/generate-keywords.mjs --target ts -> completed with no tracked changes
• git diff --check origin/develop...HEAD -> pass
• JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64 ./gradlew :app:testDebugUnitTest :app:compileDebugJavaWithJavac -x copyForkLlamaLib from packages/app-core/platforms/android -> pass

Full Android Gradle without -x copyForkLlamaLib is blocked in this checkout by missing pre-staged fused inference lib for arm64-v8a (copyForkLlamaLib asks for ELIZA_MTP_ANDROID_LIBDIR / compile-libllama.mjs --target android-arm64-vulkan-fused). Java compile/test validation is green once that native-copy task is excluded and Java 21 is used.

[lalalune]

Local validation after rebase/scope cleanup:

• bun run install:light
• bun run --cwd packages/core build
• bun run --cwd packages/shared build
• bun run --cwd plugins/plugin-aosp-local-inference test — 100 pass
• bun run --cwd plugins/plugin-aosp-local-inference typecheck
• node --test packages/benchmarks/mobile-resource/metrics.test.mjs — 10 pass
• JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64 ./gradlew -PelizaCloudBuild=true :app:testDebugUnitTest — BUILD SUCCESSFUL
• bunx @biomejs/biome check plugins/plugin-aosp-local-inference packages/benchmarks/mobile-resource .github/issue-evidence/11760-inference-memory-policy — exit 0; existing warning-level mobile-resource env-var notices only
• git diff --check HEAD

Also verified the PR tree no longer includes generated .js/.map artifacts from unrelated plugins.

[github-actions]

LifeOps Multi-Tier Benchmark

Suite: — Tiers requested:

large — cancelled

frontier — cancelled

Artifacts: lifeops-multi-tier-large-28638415289, lifeops-multi-tier-frontier-28638415289

[claude]

Claude encountered an error —— View job

---

I'll analyze this and get back to you.