feat(local-runtime): managed llama.cpp runtime — local models with one click - #85852
Closed
jquesnelle wants to merge 106 commits into
Closed
feat(local-runtime): managed llama.cpp runtime — local models with one click#85852jquesnelle wants to merge 106 commits into
jquesnelle wants to merge 106 commits into
Conversation
…esolve Hermes can now bring its own inference engine. New hermes_cli/local_runtime package: - binaries: resolve and download official llama.cpp release builds for the host platform (CUDA/Metal/Vulkan/HIP/CPU), sha256-verified, with N-1 tag retention for rollback and honest errors for platform gaps. - supervisor: spawn one llama-server in router mode with a generated API key; crash-restart with backoff (router only — child failures surface, never auto-retry); readiness proven by a real generation rather than a health probe; idle models unload after 15 minutes and reload on demand. - detect: fingerprint an already-running llama-server via /props so an external server is used instead of starting a second one. Servers that merely speak /v1 (Ollama, LM Studio) don't false-positive. - endpoint resolution: a llamacpp-flavored provider with no explicit base_url resolves managed-first, detected-external second; an explicit base_url always wins. No new provider surface — the existing custom provider aliases carry it. - lifecycle: the backend boots the server when the user has opted in and shuts it down with the app so no orphan pins GPU memory. Config lives in the local_runtime section; deliberately no context or VRAM knobs. The server binds 127.0.0.1 (never localhost — the name resolution adds ~2s per request on Windows) and models load on first inference rather than at boot.
Local models get one context contract: any model runs at any window up to its native max; hardware and session depth only change speed. No knobs. - gguf + estimator: a stdlib GGUF reader feeds a per-layer context-memory estimator that prices dense, sliding-window, and recurrent/hybrid layers separately. Per-architecture cost spreads ~40x (dense 144 KiB per token vs hybrid ~3 KiB), so per-layer pricing is what makes 1M-token windows a launch decision instead of a guess. Estimator accuracy vs real models: worst case ~8%. - context_policy: models launch at the largest window that fits GPU memory entirely, floored at 64K. On Windows/WDDM, over-allocating VRAM silently slows decode ~9x, so every window grant re-fits against live memory. When weights exceed VRAM, spill placement pins expert/FFN weights to host RAM so attention and KV stay resident (~1.75x over naive spill); speculative decoding turns on only for spilled configs. - growth: when a session reaches its window's edge, Hermes grows the window toward native max instead of compressing — both compression gates try growth first, and compression becomes the move of last resort at native max, at the ~6 tok/s speed floor, or when physics says stop. Growth re-prefills server-side; the conversation history never mutates, so prompt caching is unaffected. Grown windows persist per model and re-fit honestly on every boot. - presets: launch decisions travel to the router as a generated --models-preset INI; catalog sampling defaults merge under policy keys (policy wins), vision projectors and spec-decode drafts attach when present. - model_metadata: the context meter reads the granted window from the running server, per router child, so the UI shows the window the model actually has.
A curated model catalog priced for the machine it's viewed on, and the /api/local-models/* routes the desktop consumes. - catalog: each model ships a quant ladder (Q8 down to Q4, best first) with exact sizes and sha256s pinned from Hugging Face LFS metadata. Selection picks the highest-quality build whose weights + 64K-floor KV fit GPU memory entirely; machines that can't get Q4 spilled to system RAM; refusal only when even Q4 exceeds GPU + RAM. Nothing below Q4 ships — the quality loss is too severe for a first local-AI experience. Split-GGUF variants, vision projectors, and spec-decode draft models download as a unit, each file verified. - routes: status (cheap, poll-safe), hardware, catalog (each row carries plain-language fit facts the UI shows verbatim), download jobs with aggregate byte progress that survive the pane unmounting, activate (start server, set as main model — the click is the opt-in), eject, delete (removes every staged file), and server on/off. Downloads run 8 parallel ranged connections and verify sha256 before use; a running router only scans models at spawn, so staging changes bounce it. - inventory: staged local models appear as a provider row in the model picker payload every surface consumes; no credential — a local server is authenticated by reachability. Catalog reachability (repos, filenames, live-sha drift) is covered by an opt-in network test gated on HERMES_TEST_NETWORK=1.
The desktop surface for the managed local runtime: - Local Models pane (Settings -> Providers): install the runtime, browse the catalog with per-machine fit pills (green fits-your-GPU / amber uses-system-RAM / red too-big, plus context start/max and vision), download with live byte progress, Use to make a model the default, eject and delete. Rows show residency live while the pane is visible — a stale 'Not in memory' next to a full GPU reads as a broken feature. - Downloads and activations run through an app-level job store, so closing the pane (or reloading the app) never orphans a 20 GB download; completion and failure surface as toasts wherever the user is. - Onboarding and the providers Accounts page offer 'Run models locally — no account needed' alongside cloud providers. - System resources statusbar item (hidden by default): GPU utilization, GPU memory, and RAM, polled only while visible. - i18n for en/zh/zh-hant/ja; Badge gains a success variant so fit state reads as a real traffic light.
User documentation for the managed runtime: the install -> download -> use flow, how hardware-aware model selection works, the memory guarantees (fit pills, context growth, the 64K floor), the system resources statusbar item, the local_runtime config reference, and using an existing llama-server instead of the managed one. Cross-linked from configuring-models, the desktop guide, and both manual local-LLM guides (Ollama, Mac), which stay authoritative for manual setups.
Contributor
૮ >ﻌ< ა ci reviewran on f2ae1df — Merge branch 'feat/local-models' of github.com:NousResearch/ ❌ Job failuresPython tests / Run tests · View jobJob Python tests / Run tests failed.
|
alt-glitch
added
type/feature
…, subprocess encoding Three portability bugs the Windows development machine hid: - The version check's subprocess.run used text=True without an explicit encoding, which decodes with the locale codepage and crashes on non-UTF-8 bytes (the repo's Windows-footgun lint). Pass encoding='utf-8', errors='replace'. - test_sha256_mismatch_rejects hardcoded the Windows asset name (bin-win-*.zip); Linux runners resolve bin-ubuntu-*.tar.gz, so the poisoned download was never the file under verification and the expected rejection never fired. Resolve the asset name the way the installer does. - test_download_job_lifecycle_with_sha_failure let variant selection price against the host machine; a GPU-less CI runner honestly refuses every build (409) before the download path under test is reached. Pin a generous budget — the test is about hash failure, not selection.
…ded model Day-0 catalog swap on Qwen3.8's release. Same hybrid-attention family as its predecessor, verified against the published base config: 64 layers, 16 full-attention (full_attention_interval=4), 4 KV heads x 256 head_dim = 4 KiB/token per full layer — so the context-memory estimator carries over unchanged, and the GGUF header remains the authority after download. Vision (mmproj) and the 256K native window carry over too. Quant ladder Q8/Q6/Q5/Q4 with sizes and sha256s pinned from HF LFS metadata; live reachability test passes against the new repo. Tagged day-0 per the catalog's validation lifecycle: the readiness generation gates every first load until the rung is proven end-to-end on real hardware. The Qwen3.6-35B-A3B entry stays — the 27B is the recommended row and the swap is one-for-one.
…y, not live-free VRAM A model the pane promised '144K fully on GPU' could launch with its weights pinned to CPU — single-digit tokens/s, high CPU, the card 60% empty. Cause: preset generation runs during boot and refresh, while the OUTGOING server instance still holds the card. The live-free probe read the predecessor's residency as memory that doesn't exist, the fit concluded 'weights don't fit', and the spill placement did exactly what it was told: hold the 64K floor and pin FFN weights to host. Both launch (bootstrap preset generation) and growth re-fit now price against the capacity budget — total device memory minus margin — because both execute through a server bounce: the old instance's memory is freed before the new one loads a byte. Growth had the same bug in a nastier costume: the model being grown vetoed its own next rung by reading its own residency as unavailable. Live-free remains the right probe for telemetry (hardware route, statusbar), which reports the present, not a post-bounce future. New contract test asserts every probe_budget call in bootstrap and growth passes planning=True, with the symptom documented in the assertion message.
The fit computed weights + KV and nothing else. A real load also costs CUDA contexts, compute buffers (~1.5 GiB measured on a 32 GiB card), and the vision projector when one ships (~0.9 GiB). A decision that passes on paper by less than that margin spills in practice: the server's own allocator shaves layers to CPU after our math said zero-spill, and the user watches a 'fits your GPU' model decode on CPU cores. initial_window() gains overhead_bytes (default 0 keeps the decision tables pure physics); preset generation passes RUNTIME_OVERHEAD plus the staged mmproj's bytes; variant selection prices the same overhead so the quant picked is the quant that actually fits; the grown-window restore re-check includes it too. Visible consequence on a 32 GiB card: Qwen3.8-27B Q6 now grants the 64K floor zero-spill (24.1 GiB weights + overhead leaves ~3 GiB for KV) instead of promising 144K and spilling. Q5 grants 216K zero-spill — quality still wins at the floor per the ladder policy.
…ctually running The pane said 'fits your GPU' while a model decoded on CPU cores; the only way to notice was Task Manager. Placement is the difference between full speed and 'why is my CPU busy', so it's now inspectable in the app: - The status route reports, per loaded model, the launch plan read back from the preset INI (the INI is the record — it spawned the children) and the granted window from the running child itself. - Loaded rows replace the bare 'In memory' pill with a placement pill: green '144K · all on GPU' or amber '64K · partly in RAM', tooltip explaining the trade and the way out (more compact build / smaller context). i18n x4. The pane already polls status while visible, so placement stays live across loads, ejects, and growth bounces.
The catalog route still computed its 'Starts at NK' pill without the runtime-overhead term the launch decision now prices, so a row could advertise 144K while the launch policy grants 64K — the pane contradicting itself one pill apart. Same overhead, same numbers, one story.
…not just the floor
The selector treated the 64K floor as its goal: any quant that cleared
it won on quality alone, so a 32 GiB card got Q6 at a 64K window when
Q5 would have run a 216K window fully resident. The floor is a
guarantee, not a target — one quant step between adjacent dynamic-quant
rungs costs little, while 64K vs 216K changes what a session can do.
New constant TARGET_WINDOW = 144K, derived from data rather than taste:
across 161 real agentic sessions, 66% complete uncompressed in 64K, 91%
in 144K, and the marginal gain past 144K (+6 points at 216K) falls
below the quality cost of another quant step. The derivation lives in
the constant's comment so whoever revisits it knows what evidence to
beat.
Selection is now four ranked rules, each a guarantee the ones below may
not break: never below Q4; never below a 64K window; prefer reaching
the target window; then maximize quality. Concretely a two-pass pick:
highest quality that zero-spills at the target, else highest quality at
the floor (small cards keep their exact previous behavior), else
smallest-spilled, else refusal.
Catalog rows explain the trade in the quant reason ('best balance for
your GPU — a larger build would shrink the context window'). Decision
table on a 32 GiB card: Qwen3.8-27B Q5 @ 216K (was Q6 @ 64K),
Nemotron Q4 @ 1M, dense Muse honestly at Q4 @ 64K (nothing reaches the
target — dense KV is why hybrids are the recommended rows).
…s owns unified memory Two workarounds dated to bring-up on RTX Spark hardware whose driver misreported memory, and both survived the driver bug they were built around: - hardware.py treated any discrete GPU whose reported VRAM ~= system RAM as a lying UMA device and budgeted from OS memory instead. On healthy drivers the device query is the truth; a workstation card in a RAM-matched box would have been silently misbudgeted. - context_policy capped UMA context memory at a flat 25% of unified memory — a second, arbitrary ceiling on top of the real constraint. The budget already encodes unified memory correctly (usable = RAM minus headroom, ram_available = 0, so nothing can 'spill'): the ladder stops where weights + KV genuinely stop fitting, and a machine with room for a huge window gets it instead of a hardcoded fraction. The genuine UMA path stays: Apple-Silicon-class devices (no discrete NVIDIA query) budget from OS physical memory minus headroom. The uma flag stays too — placement semantics (kv_on_gpu, spill wording) still differ on unified memory. If a driver misreports again, the fix belongs in a vendor-specific probe quirk, not a policy-layer guess. The UMA context test now asserts the constraint arrives through the budget (resident decision, weights+KV within usable) rather than pinning the removed 25% constant.
…-models # Conflicts: # apps/desktop/src/app/shell/hooks/use-statusbar-items.tsx # apps/desktop/src/i18n/en.ts # hermes_cli/web_server.py
…e the staged models A model downloaded in one app session and used after a restart could load with NO policy at all: boot found the previous session's server still running (state-file adoption), adopted it, and never regenerated presets — so the new GGUF autoloaded with llama-server defaults: f16 KV at maximum context, no placement overrides. On Windows/WDDM that over-allocation silently demotes VRAM pages and the model decodes at a crawl with GPU utilization pinned at 100% doing no useful work (busy- wait on demoted memory), while every preset-covered model on the same server runs fine. Boot now adopts a running incumbent only while its preset file covers every staged model. A stale incumbent is stopped (state pid, SIGTERM, bounded wait) and replaced by a fresh boot with regenerated presets — sessions ride through on the stable port + persisted key like any other supervised restart. Contract tests: staleness detection (missing section = stale, covered = current, no models = never stale) and the adopt/replace decision itself.
…gine, models, and server Creating a second profile forced a full re-download of the llama.cpp engine and every staged model, and would have spawned a second server fighting the first for the stable port: models/ and runtimes/llamacpp/ resolved under the PROFILE's HERMES_HOME. They are machine assets, not profile state. A 20 GB GGUF describes nothing about a profile; the engine build describes this machine's hardware; the server state file describes the one managed server all profiles share. All three now resolve under the shared Hermes root (get_default_hermes_root — the profiles/<name> parent), so every profile sees the same catalog downloads, the same installed engine, and adopts the same running server. Per-profile decisions (default model, enabled) stay in each profile's config.yaml as before. Default-profile installs see byte-identical paths — no migration. The few sites that derived runtimes/llamacpp from HERMES_HOME directly now route through runtimes_root()/models_dir(), so scoping bugs cannot come back one file at a time. Contract tests: named-profile resolution lands at the shared root (for the dirs and for every state file), default-profile paths unchanged.
…with a vision projector find_entry_for_model returns (entry, variant); the mmproj-overhead branch treated the tuple as the entry and raised AttributeError — on every boot, for every REAL catalog model, because the synthetic models in the test suite never resolve to a catalog entry and so never executed the branch. The exception fed the fallback, which dropped the preset file entirely: the router ran stock fit (f16 KV at max context, no placement) for every model, reintroducing the silent busy-wait the policy exists to prevent. Unpack the tuple. New regression test stages a real catalog id with an mmproj so the branch executes under test. The fallback made the crash invisible, so it gets a degradation ladder: on generation failure, serve with the PREVIOUS policy file when one exists (a stale policy beats no policy; only models staged since the last successful run go unpoliced) and log at error level naming the consequence. Only a first boot with no INI at all falls to stock fit.
… cloud catalog Pasting an image at a vision-capable local model failed with 'can't see the image': capability lookup consults the user's config override, the models.dev catalog, and an Ollama probe — and a cloud catalog has never heard of a local GGUF, so every managed model read as text-only and images detoured to the auxiliary vision model (or nothing). Wrong twice for a local-first user: broken feature, and a screenshot silently leaving the machine. New hermes_cli/local_runtime/capabilities.py answers from ground truth, best source first: the RUNNING child's /props modalities block (the server that will receive the image says whether it can see), then the catalog entry's vision projector — required to actually be on disk, so a model staged without its mmproj honestly reads blind. Non-managed models return None and the chain falls through unchanged. Wired into _lookup_supports_vision between the config override (still root of trust) and the cloud catalog; _main_model_supports_vision and image routing inherit through the same resolver. No image conversion needed anywhere — llama-server accepts standard image_url content parts once its projector is loaded. Contract tests: not-ours passes through, projector-on-disk sees, projector-missing is blind, live /props beats the catalog, the chain never consults the cloud catalog for a managed model, and the user override still outranks everything.
…s decoder drops WebP silently Pasting a .webp at a local vision model produced fluent, completely wrong descriptions: llama.cpp decodes images with stb_image, which has no WebP support, and an undecodable image part fails SILENTLY — no HTTP error, no log line. The model receives a turn that mentions an image it never saw and confabulates. Measured against the live server with the same red square: PNG answered 'Red', WebP answered 'Unseen', and the model's own reasoning discussed being unable to see the image while the visible reply described an imaginary one. Image attachment already had a transcode-to-PNG path for formats some cloud providers reject (AVIF/HEIC/BMP); WebP was in the universal set because every cloud provider takes it. When the active main model is served by the managed runtime, narrow the accepted set to what its decoder actually handles (PNG/JPEG) so WebP transcodes here instead of vanishing server-side. Cloud providers keep native WebP — no transcode tax where none is needed. Live receipt: the transcoded WebP-as-PNG answers 'Red' through the real server. Contract tests: webp->png for managed, webp passthrough for cloud.
… for the managed server The WebP silent-drop fix covered direct image attachments but missed the second route to the same cliff: desktop drag-and-drop attaches a note telling the model to call vision_analyze, whose native fast path embeds the image into conversation history through its own normalization (_normalize_to_supported_image). That set was cloud-shaped — WebP 'supported' — so the WebP data-URL reached the managed server inside a multimodal tool result, was dropped silently by its decoder, and the model confabulated. Symptom: asked about a dragged-in medal photo, the model described the Hermes desktop app itself — its training-data prior for 'screenshot attached to a chat conversation'. Normalization now consults the same managed-runtime accepted set as attachment routing (one constant, hermes_cli.local_runtime.capabilities. ACCEPTED_IMAGE_MIMES): WebP converts to PNG before it enters history for a managed main model; cloud providers keep native WebP. This also protects the embedded-in-history bytes, so a session resumed later cannot replay an undecodable part. Contract test covers both providers through the real normalization function.
andrexibiza
reviewed
Aug 19, 2026
andrexibiza
left a comment
andrexibiza
left a comment
Contributor
There was a problem hiding this comment.
Reviewed exact head 663413e3bbd945a3b8f193c6e5e9b71d9854b223 against merge base 8911e2e0edf750b104edbdc106d63d6cdac88524, including the runtime binary installer, model catalog/download path, machine-scoped supervisor and endpoint state, Desktop lifecycle, context-growth policy, tests, current CI, and the adjacent process-ownership work already merged elsewhere in Hermes.
This is a serious and directionally strong subsystem. The model-download side pins Hugging Face LFS SHA-256 values; readiness uses a real generation rather than /health; granted n_ctx is reconciled from the child; boot does not surprise-download an engine; and the UI separates downloaded, loaded, placement, and active-model state. The no-new-provider approach and the deliberate machine/profile boundary are also coherent.
I found three class-level blockers before this can move out of draft.
1. The executable installer is trust-on-first-use, not an authenticated pinned install
The model assets are pinned correctly, but the executable runtime is not.
Both production entry points call:
local_models_runtime_install()→ensure_runtime_installed(tag, backend)ensure_local_runtime()→ensure_runtime_installed(tag, backend)
without expected_sha256.
Inside ensure_runtime_installed(), the first downloaded archive is hashed, but when no expected digest was supplied that hash is merely written into the new local manifest. A compromised or substituted first download is therefore accepted as the baseline so long as the extracted binary prints a version string containing the requested build number. On later boots, a manifest with any truthy verified_version returns immediately without re-hashing either the cached archive or the installed executable tree.
That is not a meaningful integrity boundary for code Hermes downloads and executes. It also differs from the established installer posture in merged #30035, where @teknium1 pinned bws and verified it against the upstream checksum file before execution.
Required shape:
- resolve immutable asset digests from an authenticated upstream checksum/release-metadata source, or carry a reviewed digest table with the selected rolling tag;
- make the production installer unable to omit those expected digests;
- verify cached archives on reuse and bind the installed manifest to the verified archive/content, not only
--versionoutput; - add a production-path test proving an unpinned install cannot become executable merely by self-recording its first observed hash.
The catalog’s pinned GGUF/mmproj/draft hashes are the right precedent and should remain unchanged.
2. server.json is being used as kill authority without process identity or ownership generation
The machine-scoped state file contains only base_url, api_key, and pid.
That weak record is used in several stronger ways:
_state_endpoint()treats a non-healthy endpoint as a legitimate starting managed server wheneverpsutil.pid_exists(pid)succeeds; when psutil is unavailable,_pid_alive()returnsTrueoptimistically._stop_state_server()sends SIGTERM to the stored PID when presets are stale.- the Desktop server-stop route also terminates the stored PID when this process has no local supervisor.
A stale file plus PID reuse is therefore enough to authorize killing an unrelated same-user process. There is no executable identity, creation time, command line, parent/supervisor identity, nonce/generation, or ownership token proving that the live PID is the process that wrote this record.
There is a second half to the same ownership defect. The runtime is intentionally machine-scoped and reusable by multiple Hermes profiles/processes, but the first process that owns _SUPERVISOR unconditionally stops the server from its web_server lifespan finalizer. Process B can be actively using the shared router that Process A started; closing A’s Desktop then tears down B’s model endpoint. A shared machine service cannot have its lifetime owned by whichever client happened to spawn it first.
Hermes has already paid for this lesson in the gateway layer. #86658—preserving the contributions from #85743 (@RelaxJonh / JonthanaHanh), #86100 (@arccat-114), and the direction pioneered in #83720—stopped the orphan reaper from killing supervised or recorded healthy gateways. #87113 by @monerostar then added the missing Windows Scheduled Task Ready/Queued ownership proof. This PR recreates that same “PID found therefore ours to kill” class for the local runtime.
Required shape:
- bind the state record to a durable runtime generation and verifiable process identity before any terminate/replace action;
- make writes atomic and owner-only;
- define process-independent lifetime ownership: a managed OS service, explicit lease/refcount/adoption protocol, or another design that prevents one client’s shutdown from killing a router still leased by another;
- test PID reuse, stale state, two profiles/processes sharing one router, and owner exit while the second client remains active.
The shared .api_key, presets.ini, and window_overrides.json also need to participate in the same generation/locking model rather than being independent uncoordinated machine-wide files.
3. Context growth proves one model idle, then restarts the entire shared router
maybe_grow_window() calls sup.is_idle(model_id), so the admission proof covers only the model being grown. When growth is granted, it writes the override and calls refresh_local_runtime(), which invokes shutdown_local_runtime() and restarts the entire router.
But this router can host up to models_max=4 models and is intentionally shared across profiles/processes. Model A can be idle while model B is actively generating for another session. A’s compression preflight can then bounce the whole process and terminate B’s in-flight request even though the growth gate said the server was safe to restart.
This is the other side of the otherwise sensible per-model memory policy: the decision is local, but the mutation authority is process-global.
Required shape:
- either grow/reload only the affected child model, or acquire a router-wide drain/lease proving every loaded model and every client is idle before restart;
- ensure another process cannot admit work between the idle proof and the restart;
- add a two-model witness where A is idle, B is busy, and A’s growth defers without stopping the router;
- add a cross-profile/process witness once the machine-scoped ownership contract is defined.
Using sup.is_idle() with no model argument would be a better local-process check, but it is not sufficient by itself because the current state model has no cross-process admission lease.
Current-head merge and CI state
This remains a draft and is currently non-mergeable. Current main during review is 23c1c9815ff44f28a576f9e11f46d6ce63d61026; it is 137 commits ahead of the merge base, while this PR has 20 unique commits.
At exact head:
- Docker Build, Test, and Publish: success;
- CI: failure;
- blocking failures visible in the executed matrix:
Python lints / Windows footgunsand Python test slice2/12; - ruff, Windows-only, macOS-only, E2E, and the visible remaining Python slices are green.
The job summaries do not expose the underlying failure text through the connector, so I am not guessing at those two causes. They remain independent current-head gates.
Re-review gate
- Authenticated/pinned executable installation, including cache/manifest reuse.
- Durable process-generation and multi-client lifetime ownership for the machine-scoped router.
- Router-wide concurrency safety for context growth/restart.
- Current-main rebase and an executed green exact-head matrix.
This is not a duplicate of the existing manual Ollama/MLX/BYO paths; it is a distinct managed-runtime product layer. The blockers are about the authority that comes with making installation, process ownership, and context resizing automatic—not about the value of that direction.
Setup flows should end at the action, not the settings pane: when the one-click setup completes while the user is still on the Local Models view watching it, the app navigates to a fresh chat with the new default model ready to try. Two guards keep it polite: - Only jobs this mount SAW running count — a finished quickstart already in the job list when the pane mounts (stale history) never navigates. - Unmount cancels the intent: a user who wandered off to another view mid-download keeps their place. No focus theft. Pane tests render under MemoryRouter now (the pane navigates); a route-probe test pins both directions: watched-running -> done navigates to the new-chat route, stale done on mount does not.
Asset resolution verified against the live release before the literal landed: both Windows CUDA targets (13.3 x64, 13.4 arm64) resolve name-for-name. b10679 carries the qwen4exp architecture (landed b10660, sparse-attention fix b10678) — the enabler for Qwen3.8-Flash-Next as a day-0 catalog entry. Installed engines are untouched; existing installs see the update card in Local Models and move on click.
…ehind an engine gate The qwen4exp architecture landed in llama.cpp b10660 (sparse-attention fix b10678); our new pin b10679 carries it. Entry fields from the live GGUF header (48 blocks at full-attention interval 4 -> 12 full + 36 recurrent-class layers; 512 experts, 10 used; 248320 vocab) and the HF tree (121.3 GB across 4 split parts + vision projector). min_engine b10678 exercises the day-0 gate for real: on engines at or past the pin the row is a normal downloadable entry; older installed engines see 'Needs engine update' and the download refuses with a plain message until the update card is clicked. Estimator inputs are header-derived priors pending a calibration pass on hardware that fits it (fleet: unmeasured until then — the entry ships unvalidated by design; touch generation still gates first load). Live reachability green across all six catalog repos.
… the catalog Slate decision: the catalog is Qwen3.8 27B (recommended), Qwen3.8 Flash Next (day-0, engine-gated), Qwen3.6 35B-A3B, and DeepSeek V4 Flash. Already-staged copies of cut models keep working — they become 'Added by you' rows with full management, the same treatment as any non-catalog model; only the curated listing changes. Tests that used the cut entries as subjects retarget to surviving models. The hybrid-KV-economics contract also generalizes: it now compares the 35B against a synthetic all-full-attention profile of the same shape and asserts KV tracks the full-attention layer share (x kv_scale), replacing a Nemotron-specific 5x ratio that was really a fact about one model's 6-of-52 layer mix.
…ill, no green on system RAM Rows order resident -> spilled -> too-big so what runs well leads. The start/grow pill pair collapses to a single 'Up to X context' pill, and 'Full X context' keeps its green only when the model earned that window resident on the GPU — a full window served from system RAM shows gray so the picker never advertises exactly the wrong model.
staged_models() counted a split GGUF the moment part 00001 landed, so a model mid-download leaked into staged_model_ids() -> the llamacpp provider row -> every model picker, as a selectable model that 400s on use. A split now counts only when every part is on disk. The catalog route reads the same helper instead of its own per-entry glob, so the pane's 'downloaded' flag and the pickers can never disagree.
A downloading model appears in the composer's model menu and the Cmd+K picker as a grayed, disabled row with the same byte progress the Local Models pane shows — inside the Local group when one exists, else under its own Local heading (first download: nothing staged, no provider row yet). Quickstart runs show the same way while fetching bytes. Opening a picker kicks the app-level job poller so work started before a reload is rediscovered, and when a download settles the catalog refetches so the placeholder becomes the real selectable row without closing the menu. The jobs store republishes every ~700ms with fresh byte counts while a download runs, so the menus subscribe only to download identity (via useStoreSelector) and each row selects its own percent scalar — a whole-store subscription re-rendered the entire Radix tree per tick, which broke the hover submenu's lifecycle (ghost poppers) and left the menu's focus guard stuck over the composer (#72163 class).
…light test references test_quickstart_is_single_flight shipped in e900b0d referencing a fixture that was never written, so the test errored at setup on every run. The fixture stubs installed_tags, select_variant, and _engine_too_old so preflight passes hermetically and the POST actually reaches the single-flight lock the test asserts on.
…d bandwidth The static recommended flag in catalog.json picked the dense 27B on every machine, including unified-memory boxes where it decodes at ~13 tok/s while the 35B-A3B MoE does ~60. Replace the flag with a per-machine derivation: decode is memory-bound, so predicted speed is bandwidth over bytes-read-per-token, and the pick is the highest-quality entry that runs resident and clears a 20 tok/s pleasant floor — else the fastest resident entry, else the least-painful spill. Catalog entries carry two authored fields in place of the flag: quality (AA-informed ordering, editorially owned — never fetched at runtime) and decode_fraction (share of weight bytes a token actually reads; 1.0 dense, the active-slice ratio for MoE). The bandwidth axis is the existing uma flag for now; measured per-machine bandwidth can replace the class constants without touching the rule. All three consumers derive: the pane badge and hero card through the catalog route, quickstart's default target through the same resolver, each gated on engine eligibility. The decision table lives on as a checked-in test pinning every memory-class x bandwidth cell — a catalog change flips cells in that file and the diff in review IS the editorial sign-off. scripts/aa_quality_sync.py proposes quality updates at authoring time; the commit decides. The quickstart fixture's select_variant stub now constructs a real VariantChoice — the resolver reads zero_spill, which the SimpleNamespace stub lacked.
Machines that could serve a local model but have nothing set up get one bubble on the composer's model pill: your hardware can run this, chats stay on your computer and cost nothing, with the Set it up button that lands on the Local Models quickstart hero. Eligibility is the backend's own physics check (a catalog model fits, nothing servable yet) read lazily at the first quiet moment and cached for the session; a local connection is required, because on a remote backend the promise would describe someone else's computer. Campaign tips are a new shape beside the rotation: conditional, actionable (they carry the one button a tip may have), perishable. The rotation consults campaigns first at each quiet due moment — a pending eligibility read holds the turn so a walk tip cannot spend the six-hour cooldown ahead of the campaign. An ignored bubble returns in a week rather than walking on; the X retires it forever, same ledger as every tip; completing setup makes the machine ineligible, so the campaign retires itself without bookkeeping. The campaign id stays out of the rotation's cursor — showTip writing an unknown id into lastTipId would restart the walk from the top — and out of TipId, so the walk can never land on it. The shown-at ledger is a new persisted store keyed by tip id; TipBubble's action button keeps the no-focus-steal contract and closes the bubble on its way to the pane.
# Conflicts: # website/docs/user-guide/features/memory.md
…f wedging the session Two halves of one field-reported failure. First, a llamacpp send with no server said 'HTTP 401 Invalid API key': resolution fell through to the generic custom path and a cloud provider rejected the placeholder key. The alias path now stops with a real answer — the server is turned off (config says disabled) or isn't running (enabled; likely still starting) — and points at Settings. An explicit base_url keeps its own error path. Second, the session that hit that error never recovered: prompt.submit's build kick is a no-op once agent_build_started is set, so every later send — including the error card's Retry — replayed the stored agent_error even after the server came back. Only new sessions worked, and the one hidden way to heal a wedged session was switching models, because only the model-switch path reset the failed generation. prompt.submit now routes through _restart_completed_failed_agent_build first: one completed failed build is cleared and rebuilt with fresh provider resolution; every other state falls through unchanged.
Test docstrings and comments carried names, dates, and partner references from internal review rounds. The failure modes stand on their own description; attribution belongs in the session notes, not the tree.
The comment dated a partner artifact instead of describing the validation; the published recipes are the durable reference.
A multi-asset engine plan (engine zip + cudart zip) restarted the byte
counters per asset and per stage, so the settings pane showed a parade
of small bars that each rewound to zero — indistinguishable from a
stalling install. The progress hook now banks finished assets' bytes
into a running base: one bar, one combined total, forward-only. Unpack
and verify narrate in the detail line without rewinding the counters.
The download rows also say what they're doing ("Downloading 4.2 of
16.1 GB") instead of a bare byte ratio.
The recommendation was already derived (quality-ranked, fit- and speed-gated), so the resolver knows why it chose — surface that as the badge's tooltip instead of leaving the pick unexplained. recommended_entry now returns (entry, reason); the reason names the branch that fired: best-quality-resident, speed-gated-quality (a higher-quality model fit but missed the speed floor — the unified- memory case), fastest-resident, or least-painful-spilled. The catalog route passes it through as recommended_reason on the recommended row, and the decision-table test pins the reason per cell alongside the pick, so a rationale flip shows up in review even when the pick holds. Copy stays qualitative (no predicted numbers — predictions order candidates, they are not promises). en/zh/ja/zh-hant; ar falls back. Also fixes Pill swallowing its rest props, which left every tooltip mounted on a pill silently dead: Tip works by asChild-cloning hover handlers onto its child, and the pill dropped them. Spread them through, and pin the regression with a test that actually hovers.
# Conflicts: # agent/auxiliary_client.py
…S_PAGES macOS getconf lacks _PHYS_PAGES/_AVPHYS_PAGES (exit 64), so _ram_bytes() returned (0, 0), probe_budget() priced every catalog row against a 0-byte budget, and the Local Models pane showed all models unavailable. Darwin branch ahead of the POSIX one: total from sysctl hw.memsize, available from vm_stat (free + inactive + purgeable + speculative pages), total//2 as the conservative fallback.
…ent into feat/local-models
Local models land on main without surfacing for everyone yet: the GUI shows them only when the app is launched with --local (hermes desktop --local, or the flag on Hermes.exe itself). The flag is strict — staged models on disk and a configured local server don't bring the UI back without it — and gates presentation only; backend routes and the CLI stay live, so a configured local model keeps serving. The flag rides existing seams: the desktop subcommand appends --local to the Electron argv, main answers a hermes:launch-flags sendSync (the translucency pattern), the preload exposes localModelsEnabled, and the renderer reads it once into a store atom. Gated surfaces (all eight, requests included — an unflagged renderer makes zero local-models API calls): - Settings nav entry + Local Models pane (stale ?pview=local renders nothing) - Providers-page and onboarding LocalModelsProviderRow - composer model dropdown: llamacpp group, download rows, load bars, status polling - Cmd+K picker: same, including the jobs-poller kick - local-setup campaign tip (declines before any eligibility read) - System resources statusbar item (absent from the bar and the customize menu, no hardware polling) - in-chat model-load progress bar Tests pin the strict contract per surface class; the flag store and argparse contract get their own suites.
On Windows os.kill(pid, 0) TERMINATES the process rather than probing — endpoint.py documents and avoids exactly this pitfall; the bootstrap stop path was the sibling call site still doing it. Reuse endpoint's _pid_alive (psutil with optimistic fallback).
status/hardware/catalog/eject were async def with fully sync bodies: blocking urlopen (up to 120s on eject), nvidia-smi subprocesses, and directory scans froze the whole dashboard event loop while they ran. Drop the async keyword — FastAPI serves plain sync handlers from its threadpool, so a slow probe stalls one worker, never the loop.
sup.start() can fail after the router process exists (health timeout, spawn error). ensure_local_runtime caught the exception and returned None without stopping the supervisor — an orphaned llama-server left holding the port and VRAM with the state file still advertising it. Stop it before re-raising; the outer handler still degrades to None.
_spawn() opened a fresh log handle each call; the crash-restart loop leaked one fd per restart and stop() only closed the last one. Close the existing handle before reopening.
The doc claimed SHA-256 verification of model downloads; the code checks byte-size against the catalog by design (catalog sizes may lag an upstream re-upload) and deletes incomplete transfers. Engine zips do carry sha256 pins — say which is which.
# Conflicts: # apps/desktop/src/store/tips.ts # run_agent.py
…ent into feat/local-models
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hermes can now run models entirely on the user's machine with zero configuration: it installs and supervises a llama.cpp server, picks the right build of each model for the hardware, and manages GPU memory end-to-end. The user picks a model; there are deliberately no context, VRAM, or quantization knobs anywhere.
The desktop flow: Settings → Providers → Local Models → install runtime (one click) → download a model (priced for your machine before you commit gigabytes) → Use → new chats run locally. Survives app restarts; switching back to a cloud provider is one click.
How it's built (one commit per layer)
custom/llamacppaliases carry resolution.Validation status — read before testing
The memory-policy constants (the 9× WDDM over-allocation penalty, per-architecture KV costs, spill placement, growth-by-re-prefill) were measured on real hardware, but on a single venue: RTX 5090 / Windows 11. The full loop — onboarding → download → chat → mid-session window growth → restart survival — is exercised end-to-end there, plus 118 contract tests and an opt-in live catalog check (
HERMES_TEST_NETWORK=1).What testers are most valuable for: macOS/Metal, AMD/Vulkan, CPU-only, and small-VRAM (8–12 GB) machines — the quant-ladder selector and spill path have receipts only at 32 GB.
Known behavior (deliberate or deferred, not surprises)