model: Support DeepSeek-V4-Flash-Vision-Exp

[tarruda]

Overview

Adds support for DeepSeek V4 Flash Vision Exp.

Additional information

Not sure if anyone else is already working, but I had GPT/Codex work on this overnight so leaving it here as a draft/reference. I've had it split into digestible commits to simplify reviewing. Hopefully it will be useful to someone working on this.

This required a new chat template, which should be handled automatically in the conversion script.

Test weights/mmproj currently being uploaded here: https://huggingface.co/tarruda/DeepSeek-V4-Flash-Vision-Exp-GGUF

Did some smoke tests on vision and appears to be working well and supports output of bounding boxes in the same format as qwen:

Details

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: Yes this was 100% generated by GPT 5.6 Sol. Did some manual smoke tests and seems to be correct, with text part being similar to 0731.

[tarruda]

Just saw @ngxson PRs #28133 and #28154, so closing this.

[ngxson]

• Yes this was 100% generated by GPT 5.6 Sol

classic "let's add 10 more machinery" move from gpt, claude would not even suggest that

[tarruda]

classic "let's add 10 more machinery" move from gpt, claude would not even suggest that

Do you mean that a lot of the changes here are not necessary? Other than the chat template, I don't really understand most of the changes in this PR, but I'd appreciate if you could point which parts are you referring to.

[ngxson]

chat template part is already redundant to start with, and among some more redundant things that I spotted so far:

• reinvented hash inside test cases
• the round trip mtmd_helper_chunk_required_ubatch_internal, not redundant but not clever either
• hacks to kv cache

if you generated this whole PR with just one single /goal, this could be a good example of the paper clip optimizer problem - pursuing one goal while giving up all other constraints

[tarruda]

chat template part is already redundant to start with

The chat template is based on the existing template which was largely based on the original encoding.py reference code provided by deepseek, plus some changes to support image input. Here's the diff:

--- models/templates/deepseek-ai-DeepSeek-V4-Flash-0731.jinja   2026-08-31 16:21:09
+++ models/templates/deepseek-ai-DeepSeek-V4-Flash-Vision-Exp.jinja     2026-09-01 06:29:13
@@ -12,6 +12,31 @@
   {%- set drop_thinking = true -%}
 {%- endif -%}
 {%- set dsml_token = '｜DSML｜' -%}
+{%- set image_placeholder = '<' + (dsml_token | replace('DSML', 'deepseek_image')) + '>' -%}
+{%- macro checked_text(text) -%}
+  {%- if image_placeholder in text -%}
+    {{- raise_exception('Images must be provided as media content blocks.') -}}
+  {%- endif -%}
+  {{- text -}}
+{%- endmacro -%}
+{%- macro render_content(content) -%}
+  {%- if content is string -%}
+    {{- checked_text(content) -}}
+  {%- elif content is not none -%}
+    {%- for part in content -%}
+      {%- if not loop.first -%}
+        {{- '\n\n' -}}
+      {%- endif -%}
+      {%- if part['type'] == 'text' -%}
+        {{- checked_text(part['text'] or '') -}}
+      {%- elif part['type'] == 'media_marker' -%}
+        {{- part['text'] -}}
+      {%- else -%}
+        {{- raise_exception('Unsupported message content type.') -}}
+      {%- endif -%}
+    {%- endfor -%}
+  {%- endif -%}
+{%- endmacro -%}
 {%- set thinking_start_token = '<think>' -%}
 {%- set thinking_end_token = '</think>' -%}
 {%- set reasoning_effort_high = 'Reasoning Effort: Absolute maximum with no shortcuts permitted.\nYou MUST be very thorough in your thinking and comprehensively decompose the problem to resolve the root cause, rigorously stress-testing your logic against all potential paths, edge cases, and adversarial scenarios.\nExplicitly write out your entire deliberation process, documenting every intermediate step, considered alternative, and rejected hypothesis to ensure absolutely no assumption is left unchecked.\n\n' -%}
@@ -24,10 +49,10 @@
 {%- for message in messages -%}
   {%- if message['role'] == 'system' -%}
     {%- if ns.is_first_sp -%}
-      {%- set ns.system_prompt = ns.system_prompt + (message['content'] or '') -%}
+      {%- set ns.system_prompt = ns.system_prompt + render_content(message['content']) -%}
       {%- set ns.is_first_sp = false -%}
     {%- else -%}
-      {%- set ns.system_prompt = ns.system_prompt + '\n\n' + (message['content'] or '') -%}
+      {%- set ns.system_prompt = ns.system_prompt + '\n\n' + render_content(message['content']) -%}
     {%- endif -%}
   {%- endif -%}
 {%- endfor -%}
@@ -78,7 +103,7 @@
       {{- '<｜User｜>' -}}
       {%- set state.in_user = true -%}
     {%- endif -%}
-    {{- message['content'] or '' -}}
+    {{- render_content(message['content']) -}}
   {%- elif message['role'] == 'tool' -%}
     {%- if state.in_user -%}
       {{- '\n\n' -}}
@@ -86,7 +111,7 @@
       {{- '<｜User｜>' -}}
       {%- set state.in_user = true -%}
     {%- endif -%}
-    {{- '<tool_result>' + (message['content'] or '') + '</tool_result>' -}}
+    {{- '<tool_result>' + render_content(message['content']) + '</tool_result>' -}}
   {%- elif message['role'] == 'assistant' -%}
     {%- set state.in_user = false -%}
     {{- '<｜Assistant｜>' -}}
@@ -95,14 +120,14 @@
     {%- if keep_reasoning -%}
       {{- thinking_start_token -}}
       {%- if message['reasoning_content'] is defined and message['reasoning_content'] -%}
-        {{- message['reasoning_content'] -}}
+        {{- checked_text(message['reasoning_content']) -}}
       {%- endif -%}
       {{- thinking_end_token -}}
     {%- else -%}
       {{- thinking_end_token -}}
     {%- endif -%}
     {%- if message['content'] is defined and message['content'] -%}
-      {{- message['content'] -}}
+      {{- render_content(message['content']) -}}
     {%- endif -%}
     {%- if message['tool_calls'] -%}
       {{- '\n\n<' + dsml_token + 'tool_calls>\n' -}}

What is redundant about this?

[ngxson]

a good instinct is to ask your agent: why other models don't need this patch?