DeepSeek-V4-Flash-2bit-DQ on M4 Max: OOM fix, native tool calling, benchmarks, degeneration study - #2
Merged
Merged
Conversation
…l writeup DeepSeek-V4-Flash on Apple Silicon aborted with `[metal::malloc] Resource limit (499000) exceeded` after ~11.3K decode tokens (prompt-length independent), then wedged the server until restart. Root cause: the attention caches (PoolingCache concat-grow + RotatingKVCache slice-assign, single and batched) build un-detached per-step lazy graphs, so the cache pins every prior step's live Metal buffer — the live-resource COUNT hits the device residency cap (~1 buffer/layer/step, 43 layers ≈ 11.3K steps). Fix: patches/mlx-lm-deepseek-v4-cache-materialize.patch — one hunk in DeepseekV4Model.__call__ that mx.eval's all per-layer cache state once per forward, detaching the chains so the count stays bounded. Verified: forced-gen 19,989 tokens clean (was 11,314), 31.3 t/s (no regression), 40-case single-server tool-call sweep 40/40 with 0 OOMs (was 49 OOMs, aborted at case 20). Leak slope 205 → 7 KB/step. Also: corrected investigation + fix-plan docs (root cause was NOT per-command-buffer/pooled_seq), setup-guide patch promotion, upstream issue/PR drafts, and the .bench-logs probes used to diagnose and verify. Note: the 2-bit DQ quant separately degenerates in open-ended chat and 4-bit+ exceeds 128 GB, so this checkpoint stays benchmark-only on this rig. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-bench soak Submitted the Metal residency-leak fix upstream: - issue ml-explore/mlx-lm#1332 (root cause + reproducer + diagnostics) - PR Blaizzy/mlx-lm#25 against the #1192 head branch - heads-up comment on #1192 linking both Adds the pre-submission soak result (MMLU+GPQA+HumanEval, 300 requests on a single long-lived patched server: 0 metal::malloc, 0 errors, ~2h44m) to the writeup + PR-draft verification tables, plus the soak driver script. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-benches plan - Phase 3 #10 plan: add "Outcome (2026-05-30) — partial sweep complete" (MMLU 44 / GPQA 24 / HumanEval 48, 300-request/0-OOM soak, upstream links); mark retry preconditions satisfied. - New runbook docs/benchmark-plans/2026-05-30-deepseek-v4-flash-remaining-benches.md: remaining benches ordered shortest->biggest (Veerman, DROP, throughput, MATH, LiveCodeBench, Terminal-Bench) with an explicit "update docs/charts after each" protocol and the tool_combined data-hygiene note. - Bump local-llm-bench-m4-32gb submodule (charts + scoreboards + run data). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… N/A) Tick Veerman in the remaining-benches runbook; bump submodule (Veerman + tool_combined chart, missing-tool-template note). DROP/MATH/LiveCodeBench running. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The mlx-community 2-bit conversion ships a chat template with no tools branch, so
mlx_lm.server drops the tools array ("model does not support tool calling") and the
model never sees a tool schema -> prose-only on tool benches (jdhodges 8/40, Veerman
2/12, all prose-correct edge cases).
Fix = Hermes/json_tools-style template (DeepSeek <｜User｜>/<｜Assistant｜> turn markers
+ <tools> injection + <tool_call> rendering) plus tokenizer_config tool_parser_type=
json_tools, so mlx-lm detects tools, injects schemas, and parses <tool_call> output.
Plumbing verified in-process (temp tokenizer copy, no GPU/server): has_tool_calling
True, tool_call_start '<tool_call>', tools injected into the rendered prompt. Live
model-emission test deferred until the running DROP/MATH/LCB benchmark queue finishes
(install.sh/uninstall.sh manage the model dir; test-...sh runs the probe + Veerman).
Branch only; not merged. See docs/benchmark-plans/2026-05-30-deepseek-v4-flash-tool-template.md
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…arse (0/4 -> 4/4)
The tool template made the model EMIT perfect <tool_call>{json}</tool_call>, but
mlx-lm's stock json_tools parser never recognized them: it matches the start marker
by token sequence, and "<tool_call>" -> (<,tool,_call,>) while the model emits
"<tool_call>\n" where BPE merges ">"+"\n" into one token (>\n), so the 4-token
marker is never a subsequence and the state machine stays in "normal" -> the call
comes back as content, tool_calls null.
deepseek_json.py matches on the stable "<tool_call" prefix (no ">") and extracts the
first brace-balanced JSON object (tolerating the leftover ">" and trailing
</tool_call>). install.sh deploys it into the venv + sets tool_parser_type.
Verified: 4-prompt probe 0/4 -> 4/4, correct names + args, 0 OOMs. Full jdhodges +
Veerman benches running to quantify vs the 8/40, 2/12 prose floor.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… multi-tool tweak reverted Final tool-calling numbers WITH the template + deepseek_json parser: jdhodges 33/40 (82%, was 8/40), Veerman 6/12 (50%, was 2/12), combined 75%, 0 OOMs. Failure analysis: every miss is reasoning/coverage, not format/parse. Dominant mode is partial multi-tool (emits 1 of N parallel calls). Tried a template tweak demanding all parallel calls as consecutive <tool_call> blocks + a 2-block example -> NO improvement (jdhodges identical 33/40; all 4 parallel cases still 1-of-N; one single-call case regressed). Parallel multi-tool is a 2-bit capability ceiling, not a template gap. Reverted to the simpler original template. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…late notes) Tick MATH in the remaining-benches runbook (47%, 41 TRUNC, 0 OOM). LiveCodeBench running (last queue item). Submodule -> 045bdbb. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…merge bug
Detailed analysis (problem/diagnostic/solution/known-limitations) + a deterministic,
GPU-free reproducer for an mlx-lm bug: tool-call detection matches markers as exact
token-id sequences, but json_tools's "<tool_call>" encodes to a run ending in a
standalone ">" token that BPE merges with the following byte ("<tool_call>\n" -> ">\n"),
so the marker is never a subsequence and valid tool calls are returned as content
(tool_calls=null). Verified via mlx-lm's own SequenceStateMachine. Proposed fix: match
the "<tool_call" prefix + brace-balanced JSON extraction (backward-compatible). Took our
model 0 -> 82% on tool calling. For an issue+PR to ml-explore/mlx-lm.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ated before parse) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…indings) Data point for deepseek-ai/DeepSeek-V4-Flash#16: MLX 2-bit conversion ships no tool template; tool-aware template -> 82% jdhodges / 50% Veerman; flags the mlx-lm json_tools token-merge gotcha (#1335/#1336). Native format is DSML, not Hermes; for posting under the user's HF account. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ule (tool run data) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…the Hermes ceiling) Native DeepSeek-V4 DSML tool format (official chat_template.jinja from deepseek-ai/DeepSeek-V4-Flash PR #16 + a deepseek_dsml parser modeled on minimax_m2). DSML expresses multiple <｜DSML｜invoke> in one <｜DSML｜tool_calls> block = native parallel calls. Probe result: the 2-bit model emits BOTH parallel calls in DSML (get_weather Tokyo + get_weather Paris), 0 OOMs — confirming the partial-multi-tool failure under Hermes was a FORMAT TAX, not a quant ceiling (my earlier conclusion was too pessimistic). Parser unit tests pass (single/parallel/mixed-type). Marker uses the stable "<｜DSML｜tool_calls" prefix to dodge the same ">" BPE merge as json_tools. Full jdhodges+veerman bench running. thinking OFF via thinking_mode=chat. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nclusion; bump submodule Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…#1337) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…fix + looping = quant) Answers the open 'repeating output' question on the model repo's discussion #1: separates the two failure modes (Metal residency-leak crash, fixed + filed upstream #1332/#25; vs the 2-bit quant degeneration loop, inherent), with the diagnosis and links. For the user's HF account. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…t loop Investigate whether sampling can avoid DeepSeek-V4-Flash-2bit-DQ's repetition degeneration. Outcome vindicates the original "no setting fixes it" — now quantified with real loop rates. Findings (rate + LLM-judge validated): - The loop is stochastic, not deterministic: ~50% on a short open-ended trigger (8 seeds), ~94-100% on long-form (story/list, judged by Qwen3.6-35B-A3B). - No sampling config reliably reduces it; additive penalties land within noise. - XTC -- the most degeneration-specific knob -- BACKFIRES, ~doubling the short loop rate (37-50% -> 87%); it derails the model rather than rescuing it. Self-correction: an intermediate "XTC fixes it / impossibility result" write-up was wrong, an artifact of an mlx-lm server bug (#1245/#1331) where the compiled sampler ignored per-request seeds -- all 50 multi-seed groups were byte-identical, so each matrix cell was one deterministic sample. Ported the seed fix locally, verified seeds vary, re-ran with 8 real seeds; conclusions inverted. Section is retracted in the doc. Upstream bugs surfaced (both already tracked -- patches/notes only, no new filings): - XTC ragged xtc_special_tokens crashes server for all models (#1257 / #1258 / #1301 / #1176): patches/mlx-lm-xtc-special-tokens-flatten.patch. - Compiled sampler ignores per-request seed (#1245 / #1331): port documented in patches/mlx-lm-server-seed-fix-NOTE.md. Changes: - docs/benchmark-plans/2026-05-31-deepseek-v4-flash-degeneration-sampling.md: plan + rate tables + judge results + retraction + methodology notes. - docs/deepseek-v4-flash-hf-leak-post.md sec.2: replace over-absolute claim with the rate-backed version (no fix; XTC backfires; long-form ~94-100% collapse). - .bench-logs/: degeneration sweep harness, coherence judge, runners, raw data. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes
mlx-community/DeepSeek-V4-Flash-2bit-DQreliable and well-characterized on a Mac Studio M4 Max 128 GB, and contributes the fixes/findings upstream. 23 commits across four workstreams.1. Metal OOM fix (the headline runtime bug)
Long generations crashed with
[metal::malloc] Resource limit (499000) exceededat ~11.3K tokens, independent of prompt length, wedging the Metal queue. Root cause: the DeepSeek-V4 attention caches (PoolingCache/RotatingKVCache) build per-decode-step updates with un-detached concat/slice graphs, so every prior step's Metal buffer stays resident (~1/layer/step → 499000/43 ≈ 11.3K). Fix:mx.evalall per-layer cache state once per forward inDeepseekV4Model.__call__. After: clean to 19,989 tokens (was OOM at 11,314), 31.3 tok/s (no regression), 0 OOMs over a 300-request soak.patches/mlx-lm-deepseek-v4-cache-materialize.patch.2. Native tool calling (0 → 98%)
Out of the box
mlx_lm.serverlogged "model does not support tool calling" and dropped thetoolsarray (the conversion ships no tools branch + mlx-lm has no DeepSeek parser). Two root causes fixed:deepseek_dsmlparser.<tool_call>'s trailing>merges with\n) → prefix-anchored matching.Result on the 2-bit build: jdhodges 98% (39/40), Veerman 75%, parallel multi-tool 8/8 — native DSML beats a generic Hermes workaround (82%) and unlocks parallel calls. (Corrected an earlier "2-bit can't do parallel tools" conclusion — it was a format tax, not a quant ceiling.)
json_toolstoken-merge fix issue #1335 / PR #1336; pairs with official template PR deepseek-ai/DeepSeek-V4-Flash#16.3. Benchmark characterization
Knowledge + reasoning soak on this rig: MMLU 44, GPQA 24, HumanEval 48, MATH 47, DROP 71 (+ throughput/T-Bench). Charts + notes updated in the bench submodule.
4. Degeneration sampling study (newest)
Can sampling avoid the 2-bit repetition loop? Rate-measured answer: no — and XTC backfires.
xtc_special_tokens(#1257/#1258/#1301/#1176) and the seed bug (#1245/#1331).Notes
.patchfiles + a port note underpatches/.docs/deepseek-v4-flash-hf-*.md) are staged for the maintainers' threads..bench-logs/anddocs/benchmark-plans/.🤖 Generated with Claude Code