Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion docs/MODELS.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,18 @@ Retrieval quality did not pay for it at this corpus size.
Replaced `mlx-community/bge-small-en-v1.5-bf16` (384-dim, 2023) on 2026-08-09. Changing it requires
re-ingesting every collection and re-running `scripts/rag-calibrate.py --write`; see `docs/rag.md`.

## Local GLM — Machine 1 (in progress, 2026-08-09)
## Local GLM — Machine 1 (BLOCKED on mlx-lm, 2026-08-09)

> **Does not load on mlx-lm 0.31.3.** GLM-5.2's IndexShare reuses one attention indexer across
> every four layers, so the checkpoint carries indexer weights on 21 of 78 layers; stock mlx-lm
> builds one per layer and fails with `Missing 285 parameters`. Support is
> [mlx-lm PR #1410](https://github.com/ml-explore/mlx-lm/pull/1410) — **open, not merged**.
> Decision 2026-08-09: stay on the z.ai cloud endpoint and revisit when the PR lands. The weights
> and launcher below stay in place for that day.
>
> `mlx-community/GLM-5-4bit` (the June predecessor) *does* carry an indexer on every layer and
> would load on stock mlx-lm, if a local GLM becomes urgent before the PR merges.


```text
mlx-community/GLM-5.2-4bit (glm_moe_dsa, 743B total / 40B active, MIT, 1M ctx, 418 GB on disk)
Expand Down
30 changes: 28 additions & 2 deletions plans/2026-08-09-model-refresh.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,12 @@ the HF cache and silently kept the slow path. The meter now pins each local endp
weights (`forceModel` in `src/meter/upstreams.ts`, env `METER_REVIEW_MODEL` / `METER_DEV_MODEL`).
Verified: a request carrying the stale id is served by the q8 path.

### 2. Developer → DeepSeek-V4-Flash — BLOCKED (needs a decision)
### 2. Developer → DeepSeek-V4-Flash — BLOCKED on mlx-lm (deferred by decision)

**Decision (2026-08-09): wait for upstream.** The Developer stays on Qwen3.6-27B; the 4-bit
DeepSeek weights finish downloading to `/Users/admin/ai/models/DeepSeek-V4-Flash-4bit` and wait
there. No third-party model code goes into the serving venv.


284B/13B-active MoE, MIT, 1M context, 151 GB at 4-bit. Weights download fine, but
**`mlx-lm` 0.31.3 — the latest release — has no `deepseek_v4` module**, so the model cannot load
Expand All @@ -56,7 +61,28 @@ Options, in the user's hands:
GLM-5.2 (`glm_moe_dsa`) both runs on `mlx-lm` 0.31.3 and beats Qwen3.6-27B — but at 418 GB it
cannot share M2 with the Reviewer.

### 3. Local GLM-5.2 on Machine 1 — DOWNLOADED, WAITING ON ONE SUDO STEP
### 3. Local GLM-5.2 on Machine 1 — DOWNLOADED, BLOCKED ON mlx-lm (deferred by decision)

**Outcome:** the wired-memory ceiling was raised (`iogpu.wired_limit_mb = 491520`, verified) and the
server started, but the model will not load on mlx-lm 0.31.3:

```
ValueError: Missing 285 parameters: model.layers.{…}.self_attn.indexer.{k_norm,weights_proj,wk,wq_b}
```

GLM-5.2's IndexShare shares one attention indexer across every four layers — the checkpoint has
indexer weights on 21 of its 78 layers, while stock mlx-lm builds one per layer (57 × 5 = the 285).
Support is [mlx-lm PR #1410](https://github.com/ml-explore/mlx-lm/pull/1410), which is **open, not
merged**; the mlx-community quant's own card says it was converted with that PR applied.

**Decision (2026-08-09): stay on the z.ai cloud on `:9004` and revisit when the PR merges.** The
GLM LaunchAgent was removed so nothing crashloops at login; the 389.6 GiB of weights, the serve
script, and the meter's env flip all stay in place. `mlx-community/GLM-5-4bit` (June predecessor,
indexer on every layer) was verified as a stock-mlx-lm-compatible fallback if a local GLM becomes
urgent first.

Machine 1 keeps the 8-bit Orchestrator (37.2 GB instead of 65 GB) — revert with `ORCH_MODEL_PATH`
in the LaunchAgent plus `METER_ORCH_REVIEW_MODEL` in `dashboard.env` if the BF16 is wanted back.

`mlx-community/GLM-5.2-4bit`, 418 GB, MIT, 1M context, arch confirmed supported by the installed
`mlx-lm`. Download running; serve script (`scripts/start-glm.sh`, port **8005** — 8004 is the MTP
Expand Down