[Model] Add BerryLM

[DanilSmorchkov]

Purpose

Adds BerryLM (BerryLMForCausalLM, model_type: berrylm), a hybrid linear/full-attention MoE decoder with a
thinking mode, released as rwb-ai/BerryLM-OS (~3.2B active / ~18.3B total, 40 layers, 180k text-only vocabulary,
256k context). transformers support: huggingface/transformers#48901 (in review; the checkpoint ships the same modeling
as remote code, so --trust-remote-code is not needed once this PR is in).

Architecture: three linear-attention layers per full-attention layer, the linear layers being a gated delta rule
with a per-channel forget gate (rank-128 projection of the layer input → one log-decay per key channel per head,
the Kimi Delta Attention recurrence, GVA 16 key / 32 value heads); GQA full attention with a sigmoid output gate and
partial rotary; Gated Block AttnRes (every layer reads a softmax mixture of the residual streams committed at
block boundaries, gated toward the identity by a per-layer scalar — token-local, transparent to the KV / recurrent
caches); 128-expert top-8 MoE plus a shared expert.

What the PR adds (self-contained model file, no dependency on other model files):

• vllm/model_executor/models/berrylm.py: attention block (Attention + rotary + output gate), KDA linear-attention
  block with one merged in_proj_qkvzb GEMM and the low-rank gate head, decode / prefill through
  fla.ops.kda (chunk_kda_with_fused_gate / fused_recurrent_kda) on the mamba-style state cache
  (MambaBase / HybridAttentionMambaModelConfig), FusedMoE + shared expert, the AttnRes mixer as a fused Triton
  kernel (one launch per layer; torch fallback off-GPU). Two custom ops: berrylm_kda_attention (a graph-splitting
  op, registered in CompilationConfig._attention_ops) and berrylm_gated_attnres (opaque, not splitting). Full
  CUDA graphs (FULL_AND_PIECEWISE), prefix caching with recurrent-state blocks, chunked prefill, TP (heads and
  experts sharded), quantized weight loading through the standard packed_modules_mapping.
• vllm/transformers_utils/configs/berrylm.py: in-tree config (model_type resolves without remote code).
• vllm/parser/berrylm.py + adapters: --reasoning-parser berrylm (<think> blocks) and --tool-call-parser berrylm
  (XML <tool_call> with typed JSON arguments), streaming and non-streaming.
• Registry, tests/models/registry.py (is_available_online=False until the Hub repo is public),
  tests/parser/engine/test_berrylm.py (reasoning / tool grammar, streaming, adapters). No test_hybrid.py case: there is
  no small checkpoint, and the 18B release one is too large for that test.
• Docs: docs/models/supported_models.md, the reasoning-parser table in docs/features/reasoning_outputs.md, a section in
  docs/features/tool_calling.md.

Test Plan

pytest tests/parser/engine/test_berrylm.py
pytest tests/models/test_registry.py -k BerryLM
vllm serve rwb-ai/BerryLM-OS --reasoning-parser berrylm --tool-call-parser berrylm --enable-auto-tool-choice

Serving-path parity (our harness, H200, release checkpoint): the served engine scores fixed token ids (prompt logprobs +
greedy decode) and transformers teacher-forces the same ids; TP1 / TP2, full CUDA graphs, prefix caching on, short prompts
and 6k–8k-token documents, cache-hit and multi-turn continuations. Parser check over the chat API: reasoning on/off, tool
call with typed arguments, streaming (17/17). The commands of the model card were also replayed from a clean environment
(this tree installed over the nightly wheel of its base commit) on H200 and H100.

Test Result

Parity vs transformers on the release checkpoint (bf16, fla kernels on the transformers side):

	6k–8k prompts, mean|Δlogprob|	greedy decode = transformers argmax	greedy-decode max|Δlogprob|	prefix-cache pass
TP1	0.082	87.5–100 %	0.35	identical ids
TP2	0.079	87.5–100 %	0.22	identical ids

0.08 is the bf16 floor of this model rather than a port error: two transformers builds (5.18.0 and main of 17.09) differ
from each other by the same mean on the same ids. The worst single prompt position is 2.6 at TP1 and 2.3 at TP2 (at TP1,
2 of 6044 positions are above 2); those positions move by up to 2.8 between the two transformers builds as well.

Serving throughput on H200 (vllm bench serve, random prompts, ignore_eos, default settings). TP1 measured on this
branch's tree with the release checkpoint; TP2 is from the 17.09 head of the branch (same model file, vLLM main of 17.09,
median of 3 runs on an exclusive node), not re-measured:

	1k→512 c=1 TPOT ms	8k→512 c=1 TPOT ms	1k→512 c=64 tok/s	1k→2k c=64 tok/s	8k→512 c=64 tok/s
TP1	4.06	4.11	4231	4788	2032
TP2 (17.09)	3.71	3.75	6103	—	—

Long context was checked on the 17.09 head of this branch (before the fused decode kernels, earlier checkpoint): 32k and
64k-token documents, per-8k-window mean |Δlogprob| 0.06–0.08 from 6k to 64k, 65k-token prefill 4.6 s with chunked
prefill 2048. On the current head the longest prompt in the parity run is 8k tokens.

Parser check: 17/17 (reasoning, tool calls, streaming). Upstream tests (H200): tests/parser/engine/test_berrylm.py
19 passed, tests/models/test_registry.py -k "BerryLM or coverage" 2 passed.

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[mergify]

Documentation preview: https://vllm--57384.org.readthedocs.build/en/57384/

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

PRs do not trigger a full CI run by default. Reviewers with write access and configured trusted contributors can comment /ci run for upstream CI or /amd-ci run for AMD CI only whenever CI signals are needed.

Once the PR is approved or has the ready label, the PR author can also use the corresponding /ci run, /ci retry, and /ci cancel commands, or their /amd-ci variants. New commits do not start upstream CI automatically.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

Agent Guidelines

IMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban.

🚀