diff --git a/docs/guides/llm/dsv4-flash.md b/docs/guides/llm/dsv4-flash.md new file mode 100644 index 0000000000..81a5e5c162 --- /dev/null +++ b/docs/guides/llm/dsv4-flash.md @@ -0,0 +1,111 @@ +# Fine-Tune DeepSeek V4 Flash + +## Introduction + +[deepseek-ai/DeepSeek-V4-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) is the latest fine-grained Mixture-of-Experts language model from DeepSeek. It uses a 43-layer all-MoE backbone (no dense MLP layers) with 256 routed experts plus one shared expert per block and top-6 routing. The architecture introduces a hybrid per-layer attention zoo — Sliding-Window Attention (SWA), Compressed Sparse Attention (CSA, Compressor + Indexer), and Hierarchical Compressed Attention (HCA, Compressor only) — selectable per layer through `compress_ratios`. The first `num_hash_layers` blocks use a hash-clustering gate (`DeepseekV4HashGate`) for token-to-expert routing, and every block maintains `hc_mult=4` Hyper-Connection streams mixed via a learned col-norm-first Sinkhorn router. + +This guide walks you through fine-tuning DeepSeek V4 Flash on HellaSwag using NVIDIA NeMo Automodel. You will learn how to configure the recipe, launch training, and inspect the results. + +To set up your environment to run NeMo Automodel, follow the [installation guide](https://github.com/NVIDIA-NeMo/Automodel#-install-nemo-automodel). + +## Data + +### HellaSwag + +We use [HellaSwag](https://huggingface.co/datasets/Rowan/hellaswag), a commonsense natural-language-inference dataset consisting of context + four candidate continuations. The version used here is the standard `rowan/hellaswag` HuggingFace split, formatted for next-token-prediction fine-tuning. + +- **Train / validation splits** taken directly from the HuggingFace dataset. +- **Tokenizer**: shared with the base model (`AutoTokenizer.from_pretrained` on the DeepSeek V4 Flash checkpoint). +- **Padding**: `pad_seq_len_divisible=64` via the default collater. + +For the full HellaSwag dataset wrapper used in NeMo Automodel, see [`nemo_automodel.components.datasets.llm.hellaswag.HellaSwag`](https://github.com/NVIDIA-NeMo/Automodel/blob/main/nemo_automodel/components/datasets/llm/hellaswag.py). + +## Architecture Notes + +DeepSeek V4 Flash differs from V3 / V3.2 in several load-bearing ways. The state-dict adapter and pipeline-parallel forward in NeMo Automodel handle each of these transparently: + +- **Attention**: GQA with a single KV head broadcast to all 64 attention heads, Q-LoRA (`q_lora_rank=1024`), and grouped O-LoRA (`o_lora_rank=1024`, `o_groups=8`) — not MLA. Per-head non-learnable rsqrt on Q after `wq_b` matches the inference reference. +- **Hybrid attention via `compress_ratios`**: + - `compress_ratio=0` → pure SWA with a learned per-head attention sink. + - `compress_ratio=4` → CSA: Compressor (overlap mode, pools `2 * ratio` raw tokens per compressed token) plus Indexer (selects top-k compressed positions per query). An explicit additive `[B, 1, S, P_total]` mask enforces per-query causal correctness. + - `compress_ratio=128` → HCA: Compressor only (non-overlap pooling), deterministic `p < (q + 1) // ratio` causal mask. +- **Dual RoPE bases**: `theta=10000` for `compress_ratio==0` layers; `theta=160000` (with YaRN scaling) for `compress_ratio>0` layers, applied to both the main attention Q/KV and the Compressor sub-module on those layers. RoPE is encoded as INTERLEAVED pairs (`view_as_complex` style) to match the released checkpoint. +- **Hash-routing first layers**: the first `num_hash_layers` (default 3) blocks use a `DeepseekV4HashGate` with a `tid2eid` lookup table. `input_ids` is threaded through the model and the V4-aware pipeline forward; under pipeline parallelism, hash layers live on stage 0 where `input_ids` is available. +- **Hyper-Connections (HC)**: every block maintains `hc_mult=4` streams of the hidden state. The mixer follows the released `hc_split_sinkhorn` formulas: `pre = sigmoid + eps`, `post = 2 * sigmoid` (no `+eps`), `comb = softmax(dim=-1) + eps` followed by a col-norm-first Sinkhorn (`iters - 1` alternating row/col passes), producing a doubly-stochastic mixing matrix per block. +- **MoE routing**: `sqrtsoftplus` scoring with `noaux_tc` topk method and clamped SwiGLU on routed experts (`swiglu_limit=10.0`). +- **Optional MTP layers** via `num_nextn_predict_layers`. + +### Checkpoint format + +The released DSV4-Flash safetensors mix several quantization formats. The state-dict adapter handles all of them transparently: + +- **Routed experts**: FP4 `e2m1fn` packed two values per int8 byte, with per-row 32-col FP8 `e8m0fnu` scales — unpacked on load, re-emitted in matching packed placeholders on `to_hf` so DCP shape/dtype validation lines up with the on-disk layout. +- **Shared experts + non-expert weights**: standard FP8 `e4m3fn` 128×128 block scales. +- **Hash layers' gate has no bias on disk**: the adapter reads `num_hash_layers` from the checkpoint's `config.json` and drops the corresponding bias keys before DCP load. +- **Indexer / Compressor key flattening**: on disk the Indexer sits as a sibling of the Compressor with its own nested compressor (`indexer.compressor.{ape,norm,wgate,wkv}` + `indexer.{wq_b,weights_proj}`); the adapter renames these to land at the flat `compressor.indexer.*` layout. + +A new in-tree `HuggingFaceStorageReader` recognizes `F8_E8M0` and `F8_E5M2` dtypes (the upstream reader silently dropped them), restoring DCP metadata on every rank for these checkpoints. + +## Launch Training + +Two recipes ship under [`examples/llm_finetune/deepseek_v4/`](https://github.com/NVIDIA-NeMo/Automodel/tree/main/examples/llm_finetune/deepseek_v4): + +- [`deepseek_v4_flash_validate.yaml`](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/llm_finetune/deepseek_v4/deepseek_v4_flash_validate.yaml) — single-node 8 × A100-80G infrastructure validation on a 4-layer truncated harness exercising the full attention zoo (`compress_ratios=[0, 0, 4, 128]` → SWA / SWA / CSA / HCA), `num_hash_layers=2`, `pp_size=2`, `ep_size=4`. Use this first to confirm the environment and checkpoint paths before scaling out. +- [`deepseek_v4_flash_hellaswag.yaml`](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/llm_finetune/deepseek_v4/deepseek_v4_flash_hellaswag.yaml) — HellaSwag finetune recipe. The yaml header documents how to scale `num_hidden_layers` and `ep_size` for the full 43-layer multi-node run. + +NeMo Automodel supports several ways to launch training — via the Automodel CLI with Slurm, interactive sessions, `torchrun`, and more. For full details on all launch options (Slurm batch jobs, multi-node configuration, environment variables, etc.), see the [Run on a Cluster](https://github.com/NVIDIA-NeMo/Automodel/blob/main/docs/launcher/slurm.md) guide. + +### Quick infrastructure validation (single node, 8 × A100-80G) + +Run the 4-layer validate harness to confirm forward / backward / PP / DCP health end-to-end before launching the full schedule: + +```bash +PYTHONPATH=/path/to/Automodel:$PYTHONPATH automodel \ + examples/llm_finetune/deepseek_v4/deepseek_v4_flash_validate.yaml \ + --nproc-per-node 8 \ + --model.config.pretrained_model_name_or_path=/your/local/dsv4-flash \ + --model.config.name_or_path=/your/local/dsv4-flash +``` + +The validate yaml uses `num_hidden_layers=4` with `compress_ratios=[0, 0, 4, 128]`, exercising SWA / SWA / CSA / HCA on every forward, and `num_hash_layers=2` so both stage-0 layers are hash-routed. + +### Standalone Slurm Script + +Below is a standalone Slurm script example for the HellaSwag recipe. Before running it, ensure your cluster environment is configured following the [Run on a Cluster](https://github.com/NVIDIA-NeMo/Automodel/blob/main/docs/launcher/slurm.md) guide. Then submit the job: + +```bash +export TRANSFORMERS_OFFLINE=1 +export HF_HOME=your/path/to/hf_cache +export HF_DATASETS_OFFLINE=1 +export WANDB_API_KEY=your_wandb_key + +srun --output=output.out \ + --error=output.err \ + --container-image /your/path/to/automodel.image.sqsh --no-container-mount-home bash -c " + CUDA_DEVICE_MAX_CONNECTIONS=1 automodel \ + examples/llm_finetune/deepseek_v4/deepseek_v4_flash_hellaswag.yaml \ + --nproc-per-node=8 \ + --model.config.pretrained_model_name_or_path=/your/local/dsv4-flash \ + --model.config.name_or_path=/your/local/dsv4-flash " +``` + +**Before you start**: +- Hugging Face applies rate limits on downloads. We recommend cloning the model repository to your local filesystem beforehand. +- Ensure your Hugging Face cache (`HF_HOME`) is configured and that the dataset is already cached locally. +- To enable Weights & Biases logging, set your `WANDB_API_KEY` and configure the `wandb` section in the YAML file. +- For the full 43-layer schedule, increase `ep_size` (and add `pp_size`) per the cluster you are running on; see the yaml header for guidance. + +## Layer-Parity Validation + +The bringup was validated against the official DeepSeek inference reference (`dsv4flash/inference/model.py`) by per-tensor dump bisection. On the 4-layer parity harness (`compress_ratios=[0, 0, 4, 128]`, `num_hash_layers=2`, PP=1, EP=8): + +- **Final-logits cosine similarity: 0.998 vs reference, top-1 token matches.** +- Every block cosine similarity ≥ 0.987. + +## Training Results + +The training loss curve below is from a 43-layer full-finetune run on HellaSwag with the full attention zoo (SWA + CSA + HCA) live. + +

+ DeepSeek V4 Flash Training Loss Curve +

diff --git a/docs/index.md b/docs/index.md index f4e8560d80..a2d2d33b5c 100644 --- a/docs/index.md +++ b/docs/index.md @@ -98,6 +98,7 @@ Find the right guide for your task -- fine-tuning, pretraining, distillation, di | **Fine-tune VLM-MoE** | You need large-scale vision-language training with sparse MoE efficiency | Image + text dataset | VLM (MoE) | [Fine-tune VLM-MoE](guides/vlm/qwen3-5.md) | | **Embedding fine-tune** | You want to improve text similarity for search, retrieval, or RAG | Text pairs / retrieval corpus | LLM | {bdg-info}`Coming Soon` | | **Fine-tune a large MoE** | You are adapting a large sparse MoE model (DeepSeek-V3, GLM-5, etc.) to your domain | Text dataset (e.g., HellaSwag) | LLM (MoE) | [Fine-tune MoE](guides/llm/large-moe-finetune.md) | +| **Fine-tune DeepSeek V4 Flash** | You want to fine-tune the DeepSeek V4 Flash hybrid-attention MoE (SWA / CSA / HCA + hash-routing) | Text dataset (e.g., HellaSwag) | LLM (MoE) | [Fine-tune DeepSeek V4 Flash](guides/llm/dsv4-flash.md) | | **Sequence classification** | You need to classify text into categories (sentiment, topic, NLI) | Text + labels (e.g., GLUE MRPC) | LLM | [Train classifier](guides/llm/sequence-classification.md) | | **QAT fine-tune** | You want a quantized model that keeps accuracy for efficient deployment | Text dataset | LLM | [Enable QAT](guides/quantization-aware-training.md) | | **Knowledge distillation** | You want a smaller, faster model that retains most of the teacher's quality | Instruction dataset + teacher model | LLM | [Distill a model](guides/llm/knowledge-distillation.md) | @@ -244,6 +245,7 @@ SFT & PEFT Function Calling guides/llm/knowledge-distillation.md Large MoE Fine-Tuning +DeepSeek V4 Flash Pretraining NanoGPT Pretraining Sequence Classification diff --git a/docs/model-coverage/latest-models.md b/docs/model-coverage/latest-models.md index 69f96f942c..efbdbe5ba1 100644 --- a/docs/model-coverage/latest-models.md +++ b/docs/model-coverage/latest-models.md @@ -6,6 +6,7 @@ See the [Model Coverage Overview](overview.md) for release summaries, and the [L | Date | Model | HF Model ID | Modality | Recipe | Try on Brev | |------|-------|-------------|----------|--------|------| +| 2026-04-25 | DeepSeek V4 Flash | [`deepseek-ai/DeepSeek-V4-Flash`](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) | LLM | [deepseek_v4_flash_hellaswag.yaml](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/llm_finetune/deepseek_v4/deepseek_v4_flash_hellaswag.yaml) | 🚧 | | 2026-04-22 | Qwen3.6-27B | [`Qwen/Qwen3.6-27B`](https://huggingface.co/Qwen/Qwen3.6-27B) | VLM | [qwen3_6_27b.yaml](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/vlm_finetune/qwen3_5/qwen3_6_27b.yaml) | 🚧 | | 2026-04-16 | LLaVA-OneVision-1.5 (4B / 8B) | [`lmms-lab/LLaVA-OneVision-1.5-4B-Instruct`](https://huggingface.co/lmms-lab/LLaVA-OneVision-1.5-4B-Instruct) | VLM | [llava_ov_1_5_4b_finetune.yaml](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/vlm_finetune/llava_onevision/llava_ov_1_5_4b_finetune.yaml) | 🚧 | | 2026-04-16 | Qwen3.6 MoE | [`Qwen/Qwen3.6-35B-A3B`](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) | VLM | [qwen3_6_35b.yaml](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/vlm_finetune/qwen3_5_moe/qwen3_6_35b.yaml) | 🚧 |