Fix GLM 5.2 native MTP execution - #1049
Conversation
|
This pull request is currently a draft. Reviews will not take place until the PR is marked as ready for review. |
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughFinal-stage layer-package runtimes now retain embeddings for native MTP. llama.cpp adds GLM-DSA-specific fused-op handling, guarded attention inputs, and a dedicated MTP computation graph that produces logits. ChangesGLM-DSA native MTP
Estimated code review effort: 4 (Complex) | ~45 minutes Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
f295f68 to
4ec025e
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (2)
third_party/llama.cpp/patches/0019-ggml-add-GLM-DSA-sparse-execution-primitives.patch (2)
951-955: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueDangling TODO comment with no adjacent disabled code.
This TODO block describes "keep route packing disabled until..." but there is no visible disabled/guarded code immediately following it in this hunk — it reads as an orphaned note that could confuse future readers about what, exactly, is disabled and where.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@third_party/llama.cpp/patches/0019-ggml-add-GLM-DSA-sparse-execution-primitives.patch` around lines 951 - 955, Remove the orphaned GLM-DSA route-packing TODO immediately before ggml_metal_graph_optimize_reorder, since it has no adjacent disabled or guarded implementation to reference. Keep the optimization function and surrounding code unchanged.
974-977: 🚀 Performance & Scalability | 🔵 Trivial | 💤 Low valueFusion-scratch buffers are unconditionally allocated on every Metal init, regardless of whether the GLM-DSA private-scratch/dual-lane paths are ever exercised.
GGML_METAL_MAX_COMMAND_BUFFERS + 1(9) buffers ofGGML_METAL_FUSION_SCRATCH_SIZE(1 MiB) each are allocated inggml_metal_init, i.e. up to 9 MiB reserved for every Metal backend instance even when the consuming features (gated bySKIPPY_GLM_DSA_*/GGML_METAL_EXPERIMENTAL_GLM_MOE_*env vars) are disabled. The failure path is handled gracefully (falls back via nil buffer id), so this isn't a bug, but it's a permanent resource cost paid by all users of this backend.Also applies to: 1101-1125
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@third_party/llama.cpp/patches/0019-ggml-add-GLM-DSA-sparse-execution-primitives.patch` around lines 974 - 977, Make fusion-scratch allocation in ggml_metal_init conditional on whether the GLM-DSA/private-scratch or dual-lane paths are enabled by their existing SKIPPY_GLM_DSA_* or GGML_METAL_EXPERIMENTAL_GLM_MOE_* configuration. Avoid reserving the GGML_METAL_MAX_COMMAND_BUFFERS + 1 scratch buffers when those features are disabled, while preserving the current nil-buffer fallback and allocation behavior when they are enabled.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@third_party/llama.cpp/patches/0019-ggml-add-GLM-DSA-sparse-execution-primitives.patch`:
- Around line 27-39: Update RPC_PROTO_PATCH_VERSION in ggml-rpc.h alongside the
GGML_OP_COUNT change from 101 to 107, incrementing the protocol patch version so
RPC peers with mismatched operation sets are detected as incompatible.
In
`@third_party/llama.cpp/patches/0020-skippy-expose-GLM-DSA-staged-runtime-controls.patch`:
- Around line 2588-2598: Guard the graph reset in skippy_import_kv_page by
checking whether get_gf_res_prev() returns a non-null result before calling
reset(). Preserve the existing behavior of invalidating the previous graph when
it exists, while allowing KV page imports before any graph result has been
created.
In
`@third_party/llama.cpp/patches/0021-tests-cover-native-GLM-DSA-execution-paths.patch`:
- Around line 647-652: Update the routed-whole-graph sanity probe invoking
test_backend_ops with the backend_args array, forwarding the selected
GLM52_PHASE_E_BACKEND filter before the existing perf options. Keep the sanity
log and TOPK_MOE_GLM_CONSUMER_SANITY configuration unchanged so verdict()
evaluates rows from the requested backend.
In
`@third_party/llama.cpp/patches/0032-ggml-add-active-count-Q3-GLM-MoE-down-kernels.patch`:
- Around line 22-69: Update the Q3 down-kernel dispatch branches to call the
matching active-slot getters: use _active6, _active4, or _active2 for
q3_active_count values 6, 4, and 2. Apply this in
third_party/llama.cpp/patches/0032-ggml-add-active-count-Q3-GLM-MoE-down-kernels.patch
lines 161-215 and
third_party/llama.cpp/patches/0033-ggml-use-GLM-max-active-policy-for-Q3-down-kernels.patch
lines 75-84, replacing calls to
ggml_metal_library_get_pipeline_mul_mv_id_q3_weighted_reduce_slots_sg_r8_nb8_w0
while leaving other dispatch behavior unchanged.
---
Nitpick comments:
In
`@third_party/llama.cpp/patches/0019-ggml-add-GLM-DSA-sparse-execution-primitives.patch`:
- Around line 951-955: Remove the orphaned GLM-DSA route-packing TODO
immediately before ggml_metal_graph_optimize_reorder, since it has no adjacent
disabled or guarded implementation to reference. Keep the optimization function
and surrounding code unchanged.
- Around line 974-977: Make fusion-scratch allocation in ggml_metal_init
conditional on whether the GLM-DSA/private-scratch or dual-lane paths are
enabled by their existing SKIPPY_GLM_DSA_* or GGML_METAL_EXPERIMENTAL_GLM_MOE_*
configuration. Avoid reserving the GGML_METAL_MAX_COMMAND_BUFFERS + 1 scratch
buffers when those features are disabled, while preserving the current
nil-buffer fallback and allocation behavior when they are enabled.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 384b91a1-9f57-4794-b1bc-0cdd2e666875
📒 Files selected for processing (24)
crates/skippy-ffi/src/lib.rscrates/skippy-runtime/src/config.rscrates/skippy-server/src/runtime_state.rsthird_party/llama.cpp/patches/0019-ggml-add-GLM-DSA-sparse-execution-primitives.patchthird_party/llama.cpp/patches/0020-skippy-expose-GLM-DSA-staged-runtime-controls.patchthird_party/llama.cpp/patches/0021-tests-cover-native-GLM-DSA-execution-paths.patchthird_party/llama.cpp/patches/0022-Support-GLM-DSA-fused-KV_B-tensors.patchthird_party/llama.cpp/patches/0023-Run-GLM-DSA-through-IndexShare-graph.patchthird_party/llama.cpp/patches/0024-Use-DSA-KV-cache-for-GLM-DSA.patchthird_party/llama.cpp/patches/0025-Bump-Skippy-ABI-for-GLM-DSA-runtime-config.patchthird_party/llama.cpp/patches/0026-Fix-GLM-DSA-Metal-get_rows-placement.patchthird_party/llama.cpp/patches/0027-ggml-default-GLM-MoE-two-phase-Metal-path.patchthird_party/llama.cpp/patches/0028-ggml-add-GLM-MoE-Metal-selector-diagnostics.patchthird_party/llama.cpp/patches/0029-ggml-skip-zero-weight-GLM-MoE-gate-up-slots.patchthird_party/llama.cpp/patches/0030-tests-cover-full-GLM-MoE-selected-chain.patchthird_party/llama.cpp/patches/0031-tests-add-GLM-Q2Q3-selected-weight-roofline.patchthird_party/llama.cpp/patches/0032-ggml-add-active-count-Q3-GLM-MoE-down-kernels.patchthird_party/llama.cpp/patches/0033-ggml-use-GLM-max-active-policy-for-Q3-down-kernels.patchthird_party/llama.cpp/patches/0034-ggml-honor-explicit-Q2-gate-up-Metal-variant-flags.patchthird_party/llama.cpp/patches/0035-ggml-make-GLM-MoE-roofline-honor-active-experts.patchthird_party/llama.cpp/patches/0036-ggml-avoid-GLM-Q3-fused-tail-under-active-policy.patchthird_party/llama.cpp/patches/0037-ggml-shrink-GLM-Q2-gate-up-active-dispatch.patchthird_party/llama.cpp/patches/0038-ggml-skip-inactive-GLM-Q3-down-slots.patchthird_party/llama.cpp/patches/0039-Fix-GLM-DSA-native-MTP-execution.patch
🚧 Files skipped from review as they are similar to previous changes (1)
- crates/skippy-server/src/runtime_state.rs
There was a problem hiding this comment.
Caution
Inline review comments failed to post. This is likely due to GitHub's internal server error or limits when posting large numbers of comments. If you are seeing this consistently it is likely a permissions issue. Please check "Moderation" -> "Code review limits" under your organization settings.
Actionable comments posted: 4
🧹 Nitpick comments (2)
third_party/llama.cpp/patches/0019-ggml-add-GLM-DSA-sparse-execution-primitives.patch (2)
951-955: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueDangling TODO comment with no adjacent disabled code.
This TODO block describes "keep route packing disabled until..." but there is no visible disabled/guarded code immediately following it in this hunk — it reads as an orphaned note that could confuse future readers about what, exactly, is disabled and where.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@third_party/llama.cpp/patches/0019-ggml-add-GLM-DSA-sparse-execution-primitives.patch` around lines 951 - 955, Remove the orphaned GLM-DSA route-packing TODO immediately before ggml_metal_graph_optimize_reorder, since it has no adjacent disabled or guarded implementation to reference. Keep the optimization function and surrounding code unchanged.
974-977: 🚀 Performance & Scalability | 🔵 Trivial | 💤 Low valueFusion-scratch buffers are unconditionally allocated on every Metal init, regardless of whether the GLM-DSA private-scratch/dual-lane paths are ever exercised.
GGML_METAL_MAX_COMMAND_BUFFERS + 1(9) buffers ofGGML_METAL_FUSION_SCRATCH_SIZE(1 MiB) each are allocated inggml_metal_init, i.e. up to 9 MiB reserved for every Metal backend instance even when the consuming features (gated bySKIPPY_GLM_DSA_*/GGML_METAL_EXPERIMENTAL_GLM_MOE_*env vars) are disabled. The failure path is handled gracefully (falls back via nil buffer id), so this isn't a bug, but it's a permanent resource cost paid by all users of this backend.Also applies to: 1101-1125
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@third_party/llama.cpp/patches/0019-ggml-add-GLM-DSA-sparse-execution-primitives.patch` around lines 974 - 977, Make fusion-scratch allocation in ggml_metal_init conditional on whether the GLM-DSA/private-scratch or dual-lane paths are enabled by their existing SKIPPY_GLM_DSA_* or GGML_METAL_EXPERIMENTAL_GLM_MOE_* configuration. Avoid reserving the GGML_METAL_MAX_COMMAND_BUFFERS + 1 scratch buffers when those features are disabled, while preserving the current nil-buffer fallback and allocation behavior when they are enabled.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@third_party/llama.cpp/patches/0019-ggml-add-GLM-DSA-sparse-execution-primitives.patch`:
- Around line 27-39: Update RPC_PROTO_PATCH_VERSION in ggml-rpc.h alongside the
GGML_OP_COUNT change from 101 to 107, incrementing the protocol patch version so
RPC peers with mismatched operation sets are detected as incompatible.
In
`@third_party/llama.cpp/patches/0020-skippy-expose-GLM-DSA-staged-runtime-controls.patch`:
- Around line 2588-2598: Guard the graph reset in skippy_import_kv_page by
checking whether get_gf_res_prev() returns a non-null result before calling
reset(). Preserve the existing behavior of invalidating the previous graph when
it exists, while allowing KV page imports before any graph result has been
created.
In
`@third_party/llama.cpp/patches/0021-tests-cover-native-GLM-DSA-execution-paths.patch`:
- Around line 647-652: Update the routed-whole-graph sanity probe invoking
test_backend_ops with the backend_args array, forwarding the selected
GLM52_PHASE_E_BACKEND filter before the existing perf options. Keep the sanity
log and TOPK_MOE_GLM_CONSUMER_SANITY configuration unchanged so verdict()
evaluates rows from the requested backend.
In
`@third_party/llama.cpp/patches/0032-ggml-add-active-count-Q3-GLM-MoE-down-kernels.patch`:
- Around line 22-69: Update the Q3 down-kernel dispatch branches to call the
matching active-slot getters: use _active6, _active4, or _active2 for
q3_active_count values 6, 4, and 2. Apply this in
third_party/llama.cpp/patches/0032-ggml-add-active-count-Q3-GLM-MoE-down-kernels.patch
lines 161-215 and
third_party/llama.cpp/patches/0033-ggml-use-GLM-max-active-policy-for-Q3-down-kernels.patch
lines 75-84, replacing calls to
ggml_metal_library_get_pipeline_mul_mv_id_q3_weighted_reduce_slots_sg_r8_nb8_w0
while leaving other dispatch behavior unchanged.
---
Nitpick comments:
In
`@third_party/llama.cpp/patches/0019-ggml-add-GLM-DSA-sparse-execution-primitives.patch`:
- Around line 951-955: Remove the orphaned GLM-DSA route-packing TODO
immediately before ggml_metal_graph_optimize_reorder, since it has no adjacent
disabled or guarded implementation to reference. Keep the optimization function
and surrounding code unchanged.
- Around line 974-977: Make fusion-scratch allocation in ggml_metal_init
conditional on whether the GLM-DSA/private-scratch or dual-lane paths are
enabled by their existing SKIPPY_GLM_DSA_* or GGML_METAL_EXPERIMENTAL_GLM_MOE_*
configuration. Avoid reserving the GGML_METAL_MAX_COMMAND_BUFFERS + 1 scratch
buffers when those features are disabled, while preserving the current
nil-buffer fallback and allocation behavior when they are enabled.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 384b91a1-9f57-4794-b1bc-0cdd2e666875
📒 Files selected for processing (24)
crates/skippy-ffi/src/lib.rscrates/skippy-runtime/src/config.rscrates/skippy-server/src/runtime_state.rsthird_party/llama.cpp/patches/0019-ggml-add-GLM-DSA-sparse-execution-primitives.patchthird_party/llama.cpp/patches/0020-skippy-expose-GLM-DSA-staged-runtime-controls.patchthird_party/llama.cpp/patches/0021-tests-cover-native-GLM-DSA-execution-paths.patchthird_party/llama.cpp/patches/0022-Support-GLM-DSA-fused-KV_B-tensors.patchthird_party/llama.cpp/patches/0023-Run-GLM-DSA-through-IndexShare-graph.patchthird_party/llama.cpp/patches/0024-Use-DSA-KV-cache-for-GLM-DSA.patchthird_party/llama.cpp/patches/0025-Bump-Skippy-ABI-for-GLM-DSA-runtime-config.patchthird_party/llama.cpp/patches/0026-Fix-GLM-DSA-Metal-get_rows-placement.patchthird_party/llama.cpp/patches/0027-ggml-default-GLM-MoE-two-phase-Metal-path.patchthird_party/llama.cpp/patches/0028-ggml-add-GLM-MoE-Metal-selector-diagnostics.patchthird_party/llama.cpp/patches/0029-ggml-skip-zero-weight-GLM-MoE-gate-up-slots.patchthird_party/llama.cpp/patches/0030-tests-cover-full-GLM-MoE-selected-chain.patchthird_party/llama.cpp/patches/0031-tests-add-GLM-Q2Q3-selected-weight-roofline.patchthird_party/llama.cpp/patches/0032-ggml-add-active-count-Q3-GLM-MoE-down-kernels.patchthird_party/llama.cpp/patches/0033-ggml-use-GLM-max-active-policy-for-Q3-down-kernels.patchthird_party/llama.cpp/patches/0034-ggml-honor-explicit-Q2-gate-up-Metal-variant-flags.patchthird_party/llama.cpp/patches/0035-ggml-make-GLM-MoE-roofline-honor-active-experts.patchthird_party/llama.cpp/patches/0036-ggml-avoid-GLM-Q3-fused-tail-under-active-policy.patchthird_party/llama.cpp/patches/0037-ggml-shrink-GLM-Q2-gate-up-active-dispatch.patchthird_party/llama.cpp/patches/0038-ggml-skip-inactive-GLM-Q3-down-slots.patchthird_party/llama.cpp/patches/0039-Fix-GLM-DSA-native-MTP-execution.patch
🚧 Files skipped from review as they are similar to previous changes (1)
- crates/skippy-server/src/runtime_state.rs
🛑 Comments failed to post (4)
third_party/llama.cpp/patches/0019-ggml-add-GLM-DSA-sparse-execution-primitives.patch (1)
27-39: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
RPC protocol version not bumped despite
GGML_OP_COUNTchange.The patch bumps the
static_asserttoGGML_OP_COUNT == 107, butRPC_PROTO_PATCH_VERSIONis left at3. The assert's own message says to update this version wheneverGGML_OP_COUNTchanges, precisely because RPC clients/servers negotiate compatibility using this constant. Six new ops (GGML_OP_DSA_SPARSE_MASK/ATTN/TOP1_ATTN,GGML_OP_MOE_ROUTE_WEIGHTS/WEIGHTED_SUM/MUL_MAT_ID) were added but the RPC version string was not incremented, so RPC peers running mismatched op sets will not be detected as incompatible.
#defineRPC_PROTO_PATCH_VERSION 3🐛 Proposed fix
-#define RPC_PROTO_PATCH_VERSION 3 +#define RPC_PROTO_PATCH_VERSION 4📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.`#define` RPC_PROTO_PATCH_VERSION 4🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@third_party/llama.cpp/patches/0019-ggml-add-GLM-DSA-sparse-execution-primitives.patch` around lines 27 - 39, Update RPC_PROTO_PATCH_VERSION in ggml-rpc.h alongside the GGML_OP_COUNT change from 101 to 107, incrementing the protocol patch version so RPC peers with mismatched operation sets are detected as incompatible.third_party/llama.cpp/patches/0020-skippy-expose-GLM-DSA-staged-runtime-controls.patch (1)
2588-2598: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash # Confirm get_gf_res_prev() can return nullptr and how callers treat it. rg -nP 'get_gf_res_prev\s*\(' third_party/llama.cpp -C2 # Locate the definition/return contract. ast-grep run --pattern 'llm_graph_result * $_::get_gf_res_prev() { $$$ }' --lang cpp third_party/llama.cpp 2>/dev/null || \ rg -nP '\bget_gf_res_prev\b' third_party/llama.cpp -g '*.cpp' -g '*.h' -C3Repository: Mesh-LLM/mesh-llm
Length of output: 6444
🏁 Script executed:
#!/bin/bash set -euo pipefail # Inspect the accessor definition and the backing storage. sed -n '80,120p' third_party/llama.cpp/patches/0007-Expand-staged-execution-across-VL-and-broad-model-fa.patch printf '\n---\n' sed -n '1,140p' third_party/llama.cpp/patches/0007-Expand-staged-execution-across-VL-and-broad-model-fa.patch | rg -n 'gf_res_prev|get_gf_res_prev|reset\(' -n -C2 printf '\n=== other uses in patch 0020 ===\n' rg -n 'get_gf_res_prev\(\)->reset|get_gf_res_prev\(\)' third_party/llama.cpp/patches/0020-skippy-expose-GLM-DSA-staged-runtime-controls.patch -C2Repository: Mesh-LLM/mesh-llm
Length of output: 3811
🏁 Script executed:
#!/bin/bash set -euo pipefail # Look for the import_kv_page control flow and any surrounding preconditions. rg -n 'skippy_import_kv_page|import_kv_page|n_past = std::max|get_gf_res_prev\(\)' third_party/llama.cpp/patches/0020-skippy-expose-GLM-DSA-staged-runtime-controls.patch -C4Repository: Mesh-LLM/mesh-llm
Length of output: 3083
Guard
get_gf_res_prev()before callingreset().
get_gf_res_prev()returns the raw previous graph result and can be null; the other new uses in this patch already check for that. Callingreset()unconditionally here can crash if a KV page is imported before any prior graph result exists.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@third_party/llama.cpp/patches/0020-skippy-expose-GLM-DSA-staged-runtime-controls.patch` around lines 2588 - 2598, Guard the graph reset in skippy_import_kv_page by checking whether get_gf_res_prev() returns a non-null result before calling reset(). Preserve the existing behavior of invalidating the previous graph when it exists, while allowing KV page imports before any graph result has been created.third_party/llama.cpp/patches/0021-tests-cover-native-GLM-DSA-execution-paths.patch (1)
647-652: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Routed-whole-graph sanity probe ignores
${backend_args[@]}.Every other
test_backend_opsinvocation in this script forwards"${backend_args[@]}"(fromGLM52_PHASE_E_BACKEND), but this sanity probe does not. Its rows feed therequire_gatesplausibility check (verdict()), so when a backend is explicitly selected, the probe still runs against the default/all backends and can trip a spurious gate failure or produce mismatched rows.🔧 Forward the backend filter to the sanity probe
"${test_backend_ops}" perf \ + "${backend_args[@]}" \ -o TOPK_MOE_GLM_CONSUMER_SANITY \ -p 'n_embd=6144,n_ff=2048,.*fused_glu=1' \ --output console > "${sanity_log}" 2>&1📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.if [[ "${GLM52_PHASE_E_ROUTED_WHOLE_GRAPH_SANITY:-1}" != "0" ]]; then sanity_log="${out_dir}/routed-whole-graph-sanity.console.log" "${test_backend_ops}" perf \ "${backend_args[@]}" \ -o TOPK_MOE_GLM_CONSUMER_SANITY \ -p 'n_embd=6144,n_ff=2048,.*fused_glu=1' \ --output console > "${sanity_log}" 2>&1🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@third_party/llama.cpp/patches/0021-tests-cover-native-GLM-DSA-execution-paths.patch` around lines 647 - 652, Update the routed-whole-graph sanity probe invoking test_backend_ops with the backend_args array, forwarding the selected GLM52_PHASE_E_BACKEND filter before the existing perf options. Keep the sanity log and TOPK_MOE_GLM_CONSUMER_SANITY configuration unchanged so verdict() evaluates rows from the requested backend.third_party/llama.cpp/patches/0032-ggml-add-active-count-Q3-GLM-MoE-down-kernels.patch (1)
22-69: 🚀 Performance & Scalability | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash set -euo pipefail printf '\n== file list ==\n' git ls-files 'third_party/llama.cpp/patches/0032-ggml-add-active-count-Q3-GLM-MoE-down-kernels.patch' \ 'third_party/llama.cpp/patches/0033-ggml-use-GLM-max-active-policy-for-Q3-down-kernels.patch' printf '\n== relevant hunks in 0032 ==\n' grep -nE 'active6|active4|active2|weighted_reduce_slots_sg_r8_nb8_w0_active|weighted_reduce_slots_sg_r8_nb8_w0\(' \ third_party/llama.cpp/patches/0032-ggml-add-active-count-Q3-GLM-MoE-down-kernels.patch printf '\n== relevant hunks in 0033 ==\n' grep -nE 'q3_active_count == [642]|weighted_reduce_slots_sg_r8_nb8_w0_active|weighted_reduce_slots_sg_r8_nb8_w0\(' \ third_party/llama.cpp/patches/0033-ggml-use-GLM-max-active-policy-for-Q3-down-kernels.patch printf '\n== show context around 0032 helper and call sites ==\n' sed -n '1,260p' third_party/llama.cpp/patches/0032-ggml-add-active-count-Q3-GLM-MoE-down-kernels.patch | \ nl -ba | sed -n '1,260p' | grep -nE 'weighted_reduce_slots_sg_r8_nb8_w0_active|active6|active4|active2|kernel_mul_mv_id_q3_K_wr_slots_r8_a[642]|nsg_slot_parallel|_pad0' printf '\n== show context around 0033 call sites ==\n' sed -n '1,180p' third_party/llama.cpp/patches/0033-ggml-use-GLM-max-active-policy-for-Q3-down-kernels.patch | \ nl -ba | grep -nE 'q3_active_count == [642]|weighted_reduce_slots_sg_r8_nb8_w0_active|weighted_reduce_slots_sg_r8_nb8_w0\('Repository: Mesh-LLM/mesh-llm
Length of output: 4710
🏁 Script executed:
#!/bin/bash set -euo pipefail # Read the two patch files with line numbers in the relevant regions. echo '--- 0032 lines 1-260 ---' nl -ba third_party/llama.cpp/patches/0032-ggml-add-active-count-Q3-GLM-MoE-down-kernels.patch | sed -n '1,260p' echo '--- 0033 lines 1-180 ---' nl -ba third_party/llama.cpp/patches/0033-ggml-use-GLM-max-active-policy-for-Q3-down-kernels.patch | sed -n '1,180p'Repository: Mesh-LLM/mesh-llm
Length of output: 220
🏁 Script executed:
#!/bin/bash set -euo pipefail echo "probe"Repository: Mesh-LLM/mesh-llm
Length of output: 161
Use the
_active6/4/2getters in the Q3 down-kernel branchesThese branches still call
ggml_metal_library_get_pipeline_mul_mv_id_q3_weighted_reduce_slots_sg_r8_nb8_w0(...), so the 6/4/2-slot path never gets selected.
third_party/llama.cpp/patches/0032-ggml-add-active-count-Q3-GLM-MoE-down-kernels.patch#L161-L215: call the matching_active6,_active4, or_active2helper in each branch.third_party/llama.cpp/patches/0033-ggml-use-GLM-max-active-policy-for-Q3-down-kernels.patch#L75-L84: same fix for theq3_active_count == 6/4/2branches.📍 Affects 2 files
third_party/llama.cpp/patches/0032-ggml-add-active-count-Q3-GLM-MoE-down-kernels.patch#L22-L69(this comment)third_party/llama.cpp/patches/0033-ggml-use-GLM-max-active-policy-for-Q3-down-kernels.patch#L75-L84🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@third_party/llama.cpp/patches/0032-ggml-add-active-count-Q3-GLM-MoE-down-kernels.patch` around lines 22 - 69, Update the Q3 down-kernel dispatch branches to call the matching active-slot getters: use _active6, _active4, or _active2 for q3_active_count values 6, 4, and 2. Apply this in third_party/llama.cpp/patches/0032-ggml-add-active-count-Q3-GLM-MoE-down-kernels.patch lines 161-215 and third_party/llama.cpp/patches/0033-ggml-use-GLM-max-active-policy-for-Q3-down-kernels.patch lines 75-84, replacing calls to ggml_metal_library_get_pipeline_mul_mv_id_q3_weighted_reduce_slots_sg_r8_nb8_w0 while leaving other dispatch behavior unchanged.
e40ac0d to
f4c41ce
Compare
4ec025e to
00ea41a
Compare
f4c41ce to
09ee9ca
Compare
00ea41a to
49f0ab0
Compare
1 participant
What changed
nextn.embed_tokens.SKIPPY_GLM52_MTP_PACKAGE.Why
GLM-5.2's final MTP block is GLM-DSA shaped. Reusing the DeepSeek2 MTP graph and loading the final package stage without embeddings caused native MTP execution to fail for the published GLM 5.2 layer package.
Validation
LLAMA_BUILD_DIR=.deps/llama-build/build-stage-abi-static-cpu SKIPPY_FORCE_LLAMA_BUILD=1 scripts/build-llama.shcargo fmt --all --checkcargo test -p skippy-server runtime_config_keeps_package_embeddings_for_final_non_first_stage -- --nocaptureLLAMA_STAGE_BUILD_DIR=.deps/llama-build/build-stage-abi-static-cpu SKIPPY_GLM52_MTP_PACKAGE=/Users/jdumay/.cache/huggingface/hub/models--meshllm--GLM-5.2-Q2_K-MTP-Q8-layers/snapshots/main GGML_METAL_DEVICES=0 cargo test -p skippy-server glm52_final_stage_package_executes_native_mtp_when_fixture_is_set -- --nocaptureLLAMA_WORKDIR=$(mktemp -d /tmp/mesh-llm-llama.XXXXXX) scripts/prepare-llama.sh pinnedSummary by CodeRabbit
Bug Fixes
Testing