Skip to content

deepseek-ocr SAM ggml_conv_2d with the im2col kept in F32 #26727

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ngxson merged 2 commits into from
Aug 18, 2026
Merged

deepseek-ocr SAM ggml_conv_2d with the im2col kept in F32 #26727

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1000c37
deepseek-ocr SAM ggml_conv_2d with the im2col kept in F32
sfallah Aug 7, 2026
7667e17
Merge remote-tracking branch 'sfallah/master' into sf/dsocr-conv-f32
sfallah Aug 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 22 additions & 5 deletions tools/mtmd/models/deepseekocr.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,22 @@ static ggml_tensor * get_rel_pos(ggml_context * ctx0,
return cur; // [C, k_size, q_size]
}

// ggml_conv_2d with the im2col kept in F32: the F16 im2col it emits since #23660 degrades OCR
static ggml_tensor * conv_2d_f32(ggml_context * ctx0, ggml_tensor * a, ggml_tensor * b,

@ngxson ngxson Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we simply cast the kernel to F32 and run ggml_conv_2d, does it work?

@sfallah sfallah Aug 10, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No — the kernels are already F32, and ggml_conv_2d applies an F16 im2col on F32 kernels:

a->type == GGML_TYPE_BF16 ? GGML_TYPE_F32 : GGML_TYPE_F16

That F32-kernel + F16-im2col combination is exactly the regression this PR fixes.

Casting the kernels to BF16 would work; BF16 results in an F32 im2col, and for these weights the cast is lossless (HF ships them BF16; the converter upcasts to F32).
But it is a hack that would rely on the side-effect of a->type == GGML_TYPE_BF16 ? GGML_TYPE_F32 : GGML_TYPE_F16.
It will also cost either a cast node in the graph or a converter change.
But a converter change only helps newly converted GGUF files; every already-published mmproj keeps F32 kernels, so the model code has to handle them either way.

int s0, int s1, int p0, int p1, int d0, int d1) {
const ggml_type im2col_type = a->type == GGML_TYPE_F16 ? GGML_TYPE_F16 : GGML_TYPE_F32;
ggml_tensor * im2col = ggml_im2col(ctx0, a, b, s0, s1, p0, p1, d0, d1, true, im2col_type); // [N, OH, OW, IC * KH * KW]

ggml_tensor * result = ggml_mul_mat(ctx0,
ggml_reshape_2d(ctx0, im2col, im2col->ne[0], im2col->ne[3] * im2col->ne[2] * im2col->ne[1]),
ggml_reshape_2d(ctx0, a, (a->ne[0] * a->ne[1] * a->ne[2]), a->ne[3]));

result = ggml_reshape_4d(ctx0, result, im2col->ne[1], im2col->ne[2], im2col->ne[3], a->ne[3]); // [OC, N, OH, OW]
result = ggml_cont(ctx0, ggml_permute(ctx0, result, 0, 1, 3, 2)); // [N, OC, OH, OW]

return result;
}


ggml_tensor * clip_graph_deepseekocr::build_sam(ggml_tensor * inp_raw) {
// Building SAM
Expand All @@ -101,7 +117,8 @@ ggml_tensor * clip_graph_deepseekocr::build_sam(ggml_tensor * inp_raw) {

ggml_tensor * inpL;

inpL = ggml_conv_2d_sk_p0(ctx0, model.patch_embed_proj_w, inp_raw);
inpL = conv_2d_f32(ctx0, model.patch_embed_proj_w, inp_raw,
(int) model.patch_embed_proj_w->ne[0], (int) model.patch_embed_proj_w->ne[1], 0, 0, 1, 1);
inpL = ggml_add(ctx0, inpL, ggml_reshape_3d(ctx0, model.patch_embed_proj_b, 1, 1, n_embd));
inpL = ggml_cont(ctx0, ggml_permute(ctx0, inpL, 1, 2, 0, 3));

Expand Down Expand Up @@ -229,18 +246,18 @@ ggml_tensor * clip_graph_deepseekocr::build_sam(ggml_tensor * inp_raw) {

cur = ggml_cont(ctx0, ggml_permute(ctx0, cur, 2, 0, 1, 3));

cur = ggml_conv_2d(ctx0, model.neck_0_w, cur, 1, 1, 0, 0, 1, 1);
cur = conv_2d_f32(ctx0, model.neck_0_w, cur, 1, 1, 0, 0, 1, 1);
cur = ggml_cont(ctx0, ggml_permute(ctx0, cur, 1, 2, 0, 3));
cur = build_norm(cur, model.neck_1_w, model.neck_1_b, NORM_TYPE_NORMAL, sam_eps, -1);
cur = ggml_cont(ctx0, ggml_permute(ctx0, cur, 2, 0, 1, 3));

cur = ggml_conv_2d(ctx0, model.neck_2_w, cur, 1, 1, 1, 1, 1, 1);
cur = conv_2d_f32(ctx0, model.neck_2_w, cur, 1, 1, 1, 1, 1, 1);
cur = ggml_cont(ctx0, ggml_permute(ctx0, cur, 1, 2, 0, 3));
cur = build_norm(cur, model.neck_3_w, model.neck_3_b, NORM_TYPE_NORMAL, sam_eps, -1);
cur = ggml_cont(ctx0, ggml_permute(ctx0, cur, 2, 0, 1, 3));

cur = ggml_conv_2d(ctx0, model.net_2, cur, 2, 2, 1, 1, 1, 1);
cur = ggml_conv_2d(ctx0, model.net_3, cur, 2, 2, 1, 1, 1, 1);
cur = conv_2d_f32(ctx0, model.net_2, cur, 2, 2, 1, 1, 1, 1);
cur = conv_2d_f32(ctx0, model.net_3, cur, 2, 2, 1, 1, 1, 1);
cb(cur, "sam_output", -1);

ggml_build_forward_expand(gf, cur);
Expand Down
Loading