diff --git a/README.md b/README.md index ec62901918..a623d96f84 100644 --- a/README.md +++ b/README.md @@ -21,6 +21,7 @@ ## 📣 News and Discussions +- [04/25/2026][**DeepSeek V4 Flash**](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) We now support finetuning `deepseek-ai/DeepSeek-V4-Flash`. Check out our [recipe](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/llm_finetune/deepseek_v4/deepseek_v4_flash_hellaswag.yaml) and [guide](https://github.com/NVIDIA-NeMo/Automodel/blob/main/docs/guides/llm/dsv4-flash.md). - [04/22/2026][**Qwen3.6-27B**](https://huggingface.co/Qwen/Qwen3.6-27B) We now support finetuning `Qwen/Qwen3.6-27B`. Check out our [recipe](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/vlm_finetune/qwen3_5/qwen3_6_27b.yaml). - [04/20/2026][**Qwen-Image**](https://huggingface.co/Qwen/Qwen-Image) We now support finetuning `Qwen/Qwen-Image`, thanks to [@harshareddy832](https://github.com/harshareddy832). Check out our [recipe](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/diffusion/finetune/qwen_image_t2i_flow.yaml). - [04/16/2026][**Qwen3.6 MoE**](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) We now support finetuning `Qwen/Qwen3.6-35B-A3B`. Check out our [recipe](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/vlm_finetune/qwen3_5_moe/qwen3_6_35b.yaml). diff --git a/docs/guides/llm/dsv4-flash.md b/docs/guides/llm/dsv4-flash.md index 81a5e5c162..bd3872fa49 100644 --- a/docs/guides/llm/dsv4-flash.md +++ b/docs/guides/llm/dsv4-flash.md @@ -48,27 +48,10 @@ A new in-tree `HuggingFaceStorageReader` recognizes `F8_E8M0` and `F8_E5M2` dtyp ## Launch Training -Two recipes ship under [`examples/llm_finetune/deepseek_v4/`](https://github.com/NVIDIA-NeMo/Automodel/tree/main/examples/llm_finetune/deepseek_v4): - -- [`deepseek_v4_flash_validate.yaml`](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/llm_finetune/deepseek_v4/deepseek_v4_flash_validate.yaml) — single-node 8 × A100-80G infrastructure validation on a 4-layer truncated harness exercising the full attention zoo (`compress_ratios=[0, 0, 4, 128]` → SWA / SWA / CSA / HCA), `num_hash_layers=2`, `pp_size=2`, `ep_size=4`. Use this first to confirm the environment and checkpoint paths before scaling out. -- [`deepseek_v4_flash_hellaswag.yaml`](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/llm_finetune/deepseek_v4/deepseek_v4_flash_hellaswag.yaml) — HellaSwag finetune recipe. The yaml header documents how to scale `num_hidden_layers` and `ep_size` for the full 43-layer multi-node run. +A ready-to-use recipe ships at [`examples/llm_finetune/deepseek_v4/deepseek_v4_flash_hellaswag.yaml`](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/llm_finetune/deepseek_v4/deepseek_v4_flash_hellaswag.yaml). The yaml header documents how to scale `num_hidden_layers` and `ep_size` for the full 43-layer multi-node run. NeMo Automodel supports several ways to launch training — via the Automodel CLI with Slurm, interactive sessions, `torchrun`, and more. For full details on all launch options (Slurm batch jobs, multi-node configuration, environment variables, etc.), see the [Run on a Cluster](https://github.com/NVIDIA-NeMo/Automodel/blob/main/docs/launcher/slurm.md) guide. -### Quick infrastructure validation (single node, 8 × A100-80G) - -Run the 4-layer validate harness to confirm forward / backward / PP / DCP health end-to-end before launching the full schedule: - -```bash -PYTHONPATH=/path/to/Automodel:$PYTHONPATH automodel \ - examples/llm_finetune/deepseek_v4/deepseek_v4_flash_validate.yaml \ - --nproc-per-node 8 \ - --model.config.pretrained_model_name_or_path=/your/local/dsv4-flash \ - --model.config.name_or_path=/your/local/dsv4-flash -``` - -The validate yaml uses `num_hidden_layers=4` with `compress_ratios=[0, 0, 4, 128]`, exercising SWA / SWA / CSA / HCA on every forward, and `num_hash_layers=2` so both stage-0 layers are hash-routed. - ### Standalone Slurm Script Below is a standalone Slurm script example for the HellaSwag recipe. Before running it, ensure your cluster environment is configured following the [Run on a Cluster](https://github.com/NVIDIA-NeMo/Automodel/blob/main/docs/launcher/slurm.md) guide. Then submit the job: diff --git a/docs/model-coverage/llm/deepseek-ai/dsv4-flash.md b/docs/model-coverage/llm/deepseek-ai/dsv4-flash.md new file mode 100644 index 0000000000..4b4d8aacc4 --- /dev/null +++ b/docs/model-coverage/llm/deepseek-ai/dsv4-flash.md @@ -0,0 +1,91 @@ +# DeepSeek V4 Flash + +[DeepSeek V4 Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) is DeepSeek's latest fine-grained Mixture-of-Experts language model. It uses a 43-layer all-MoE backbone with 256 routed experts plus one shared expert per block, top-6 routing, and a hybrid per-layer attention zoo (SWA / CSA / HCA) selectable through `compress_ratios`. The first `num_hash_layers` blocks use a hash-clustering gate, and every block maintains `hc_mult=4` Hyper-Connection streams mixed via a learned col-norm-first Sinkhorn router. + +:::{card} +| | | +|---|---| +| **Task** | Text Generation (MoE) | +| **Architecture** | `DeepseekV4ForCausalLM` | +| **Parameters** | fine-grained MoE, 256 routed + 1 shared expert | +| **HF Org** | [deepseek-ai](https://huggingface.co/deepseek-ai) | +::: + +## Available Models + +- **DeepSeek-V4-Flash** + +## Architecture + +- `DeepseekV4ForCausalLM` + +## Example HF Models + +| Model | HF ID | +|---|---| +| DeepSeek V4 Flash | [`deepseek-ai/DeepSeek-V4-Flash`](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) | + +## Example Recipes + +| Recipe | Description | +|---|---| +| [`deepseek_v4_flash_hellaswag.yaml`](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/llm_finetune/deepseek_v4/deepseek_v4_flash_hellaswag.yaml) | SFT — DeepSeek V4 Flash on HellaSwag with pipeline parallelism | + + +## Try with NeMo AutoModel + +**1. Install** ([full instructions](../../../guides/installation.md)): + +```bash +pip install nemo-automodel +``` + +**2. Clone the repo** to get the example recipes: + +```bash +git clone https://github.com/NVIDIA-NeMo/Automodel.git +cd Automodel +``` + +:::{note} +The full 43-layer schedule requires a multi-node run; see the recipe yaml header for `ep_size` / `pp_size` guidance. See the [Launcher Guide](../../../launcher/slurm.md) for multi-node setup. +::: + +**3. Run the recipe** from inside the repo: + +```bash +automodel --nproc-per-node=8 examples/llm_finetune/deepseek_v4/deepseek_v4_flash_hellaswag.yaml +``` + +:::{dropdown} Run with Docker +**1. Pull the container** and mount a checkpoint directory: + +```bash +docker run --gpus all -it --rm \ + --shm-size=8g \ + -v $(pwd)/checkpoints:/opt/Automodel/checkpoints \ + nvcr.io/nvidia/nemo-automodel:26.02.00 +``` + +**2.** Navigate to the AutoModel directory (where the recipes are): + +```bash +cd /opt/Automodel +``` + +**3. Run the recipe**: + +```bash +automodel --nproc-per-node=8 examples/llm_finetune/deepseek_v4/deepseek_v4_flash_hellaswag.yaml +``` +::: + +See the [Installation Guide](../../../guides/installation.md) and [LLM Fine-Tuning Guide](../../../guides/llm/finetune.md). + +## Fine-Tuning + +See the [Fine-Tune DeepSeek V4 Flash](../../../guides/llm/dsv4-flash.md) guide and the [Large MoE Fine-Tuning Guide](../../../guides/llm/large-moe-finetune.md). + +## Hugging Face Model Cards + +- [deepseek-ai/DeepSeek-V4-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) diff --git a/docs/model-coverage/llm/index.md b/docs/model-coverage/llm/index.md index 0b5093bae5..6122aaf10b 100644 --- a/docs/model-coverage/llm/index.md +++ b/docs/model-coverage/llm/index.md @@ -28,6 +28,7 @@ NeMo AutoModel supports the [AutoModelForCausalLM](https://huggingface.co/transf | Qwen / Alibaba Cloud | [Qwen3-Next](qwen/qwen3-next.md) | `Qwen3NextForCausalLM` | | DeepSeek | [DeepSeek](deepseek-ai/deepseek.md) | `DeepseekForCausalLM` | | DeepSeek | [DeepSeek-V3](deepseek-ai/deepseek-v3.md) | `DeepseekV3ForCausalLM`, `DeepseekV32ForCausalLM` | +| DeepSeek | [DeepSeek V4 Flash](deepseek-ai/dsv4-flash.md) | `DeepseekV4ForCausalLM` | | Mistral AI | [Mistral](mistralai/mistral.md) | `MistralForCausalLM` | | Mistral AI | [Mixtral](mistralai/mixtral.md) | `MixtralForCausalLM` | | Mistral AI | [Ministral3 / Devstral](mistralai/ministral3.md) | `Mistral3ForConditionalGeneration` | @@ -96,6 +97,7 @@ qwen/qwen3-moe qwen/qwen3-next deepseek-ai/deepseek deepseek-ai/deepseek-v3 +deepseek-ai/dsv4-flash mistralai/mistral mistralai/mixtral mistralai/ministral3