chore: faster builds, fzf lua - #74
Merged
Merged
Conversation
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
harivansh-afk
added a commit
that referenced
this pull request
May 15, 2026
The 121B Step-Flash-REAP model loaded with `-ngl 99` and `-c 131072` pinned ~76 GiB of GPU plus large NVIDIA-driver host pages on spark, exhausting host RAM and making forgejo (and everything else on the box) thrash on zram swap. Disable the service so it does not auto-start at boot. Run `systemctl start llama-cpp` manually when actually needed. Reviewed-on: https://git.harivan.sh/harivansh-afk/nix/pulls/74 Co-authored-by: Harivansh Rathi <rathiharivansh@gmail.com> Co-committed-by: Harivansh Rathi <rathiharivansh@gmail.com>
harivansh-afk
added a commit
that referenced
this pull request
May 16, 2026
## Summary Re-enable the llama-cpp service on spark but with auto-sleep so it stops pinning ~76 GiB of VRAM and NVIDIA driver host pages 24/7. That was the original reason the service was disabled in #74. ## Changes - `enable = true` so the systemd unit has a real `ExecStart` again - `-c 32768` (was `131072`): shrinks KV cache from ~6.5 GiB to ~1.6 GiB on GPU - `--sleep-idle-seconds 600`: weights + KV cache unload from VRAM after 10 min idle; next request reloads transparently (flag is present in the current nixpkgs llama-cpp build, verified via `llama-server --help`) - `OOMScoreAdjust = 1000` on the unit: llama-server is the first kill target if memory pressure ever spikes, protecting forgejo ## How the auto-sleep works `--sleep-idle-seconds` (PR ggml-org/llama.cpp#18228) keeps the server process alive at near-zero footprint after the timeout, then lazy-reloads on the next request. Different from `--standby-timeout`, which kills the whole server process. Sleep is the better fit here because pi keeps a long-lived OpenAI-compatible client against `127.0.0.1:8080`. ## Test plan - [ ] `nixos-rebuild switch --flake .#spark` on spark - [ ] `systemctl status llama-cpp` shows `active (running)` - [ ] `curl -s http://127.0.0.1:8080/health` returns `{"status":"ok"}` - [ ] First completion request loads the model; `nvidia-smi` shows VRAM usage climb to ~70 GiB - [ ] Wait ~11 minutes idle; `nvidia-smi` shows VRAM drop back to near-zero, process still alive - [ ] Next request reloads model and serves a completion - [ ] Pi (`dots/pi/models.json`) routes through unchanged - [ ] `systemctl status forgejo` shows no degradation while idle Reviewed-on: https://git.harivan.sh/harivansh-afk/nix/pulls/87 Co-authored-by: Harivansh Rathi <rathiharivansh@gmail.com> Co-committed-by: Harivansh Rathi <rathiharivansh@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.