fix: strip mtp label for GLM-4 architecture to prevent MTP crash (#2451) - #2621
Closed
bong-water-water-bong wants to merge 5 commits into
Closed
bong-water-water-bong wants to merge 5 commits into
bong-water-water-bong wants to merge 5 commits into
Conversation
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
github-actions
Bot
added
engine::llamacpp
Adds a new mlx-engine backend wrapping lemon-mlx-engine (https://github.com/lemonade-sdk/lemon-mlx-engine) as a WrappedServer subprocess, enabling Apple Silicon MLX inference via Lemonade. Changes: - New mlx backend descriptor at src/cpp/include/lemon/backends/mlx/mlx.h - New MlxServer class (mlx_server.h, mlx_server.cpp) forwarding chat/completions + completions APIs, unsupported for responses - Registered in LEMON_BACKENDS (CMakeLists.txt) for auto-generated descriptor registry, factory, install specs, and support matrix - Backend version pin (b1013-stable for metal/rocm/cpu) in backend_versions.json with rocm-stable-runtime field - Two sample models (Qwen3-0.6B-MLX suggested, Qwen3-4B-MLX opt-in) in server_models.json - get_device_type_from_recipe updated in model_types.h for mlx-engine - Support matrix entries in mlx.h descriptor: metal (macOS), rocm/gfx1150-120X (linux), cpu fallback (linux/macos) - Test harness capabilities catalog updated for mlx-engine
bong-water-water-bong
force-pushed
the
split/glm4-mtp-fix
branch
from
August 3, 2026 21:15
e8a9bbe to
bff225a
Compare
… org
mlx_server.cpp: replace undefined SPEC with mlx::spec(), keep the
anonymous namespace and mlx namespace inside lemon::backends (the early
} // namespace backends put them at lemon:: scope), and include mlx.h
for the descriptor. The mlx backend is in LEMON_BACKENDS unconditionally,
so every Windows build has been broken since the backend was added -
first surfaced by the Validate New llama.cpp Release run.
build-container.yml: push to ghcr.io/${{ github.repository }} instead of
the stale ghcr.io/lemonade-sdk org, which denies the repo's GITHUB_TOKEN
(permission_denied: The requested installation does not exist).
fix: mlx backend fails to compile; publish containers to correct ghcr org
…onade-sdk#2451) GLM-4 MoE models (e.g. GLM-4.5-Air-UD-Q4K-XL-GGUF) embed MTP layers in the GGUF, triggering speculative decoding (draft-mtp). But the GLM-4 MoE graph builder in llama.cpp asserts on missing multimodal metadata when constructing the MTP draft context, crashing on first prompt. Fix: after apply_gguf_capability_labels(), strip the mtp label for models whose GGUF architecture starts with 'glm4' (case-insensitive). These models load and run correctly without MTP acceleration until upstream fixes glm4-moe.cpp. Fixes lemonade-sdk#2451.
bong-water-water-bong
force-pushed
the
split/glm4-mtp-fix
branch
from
August 16, 2026 23:28
bff225a to
e9cd20d
Compare
Contributor
Author
|
Closing: stacked on the closed mlx-engine backend work. The unique commit — strip |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
GLM-4 MoE models (e.g., GLM-4.5-Air-UD-Q4K-XL-GGUF) embed MTP layers in the GGUF, triggering speculative decoding (
--spec-type draft-mtp). The GLM-4 MoE graph builder in llama.cpp asserts on missing multimodal metadata when constructing the MTP draft context, crashing on the first prompt.Fix
After
apply_gguf_capability_labels()runs, check if the GGUF architecture starts with "glm4" (case-insensitive). If so, strip themtplabel so the model runs without speculative decoding. The model loads and runs correctly — just without the MTP optimization.Changes
src/cpp/server/backends/llamacpp/llamacpp_server.cpp—+14linesFixes #2451