diff --git a/docs/broken_links_false_positives.json b/docs/broken_links_false_positives.json new file mode 100644 index 0000000000..694f732a72 --- /dev/null +++ b/docs/broken_links_false_positives.json @@ -0,0 +1 @@ +{"uri": "https://www.llama.com/models/llama-3/"} diff --git a/docs/index.md b/docs/index.md index 9cda991c7b..cf9a9529bd 100644 --- a/docs/index.md +++ b/docs/index.md @@ -22,6 +22,14 @@ bridge-guide.md bridge-tech-details.md ``` +```{toctree} +:caption: Supported Models +:hidden: + +models/llm/index.md +models/vlm/index.md +``` + ```{toctree} :caption: Training and Customization :hidden: diff --git a/docs/models/llm/index.md b/docs/models/llm/index.md new file mode 100644 index 0000000000..336c1ea134 --- /dev/null +++ b/docs/models/llm/index.md @@ -0,0 +1,10 @@ +# Large Language Models + +This section documents Large Language Models supported by Megatron Bridge, with examples for converting to/from 🤗 Hugging Face and links to training recipes. + +```{toctree} +:hidden: + +llama3.md +qwen.md +``` \ No newline at end of file diff --git a/docs/models/llm/llama3.md b/docs/models/llm/llama3.md new file mode 100644 index 0000000000..0ea0963e76 --- /dev/null +++ b/docs/models/llm/llama3.md @@ -0,0 +1,43 @@ +# Llama 3 + +[Meta’s Llama](https://www.llama.com/models/llama-3/) builds on the general transformer decoder framework with some key additions such as pre-normalization, SwiGLU activations, and Rotary Positional Embeddings (RoPE). More information is available in the companion paper [“Llama: Open and Efficient Foundation Language Models”](https://arxiv.org/abs/2302.13971). With a wide variety of model sizes - Llama has options for every inference budget. + +Llama family models are supported via the Bridge system with auto-detected configuration and weight mapping. + +## Conversion with 🤗 Hugging Face + +### Load HF → Megatron +```python +from megatron.bridge import AutoBridge + +# Example: Llama 3.1 8B +bridge = AutoBridge.from_hf_pretrained("meta-llama/Llama-3.1-8B-Instruct") +provider = bridge.to_megatron_provider() + +# Configure parallelism before instantiating the model +provider.tensor_model_parallel_size = 8 +provider.pipeline_model_parallel_size = 1 + +model = provider.provide_distributed_model(wrap_with_ddp=False) +``` + +### Export Megatron → HF +```python +# Convert from a Megatron checkpoint directory to HF format +bridge.export_ckpt( + megatron_path="/results/llama3_8b/checkpoints/iter_00002000", + hf_path="./llama-hf-export", +) +``` + +## Examples +- Checkpoint import/export: [examples/conversion/convert_checkpoints.py](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/examples/conversion/convert_checkpoints.py) +- Generate text (HF→Megatron): [examples/conversion/hf_to_megatron_generate_text.py](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/examples/conversion/hf_to_megatron_generate_text.py) + +## Pretrain recipes +- See: [bridge.recipes.llama.llama3](../../apidocs/bridge/bridge.recipes.llama.llama3.md) + +## Related docs +- Recipe usage: [Recipe usage](../../recipe-usage.md) +- Customizing the training recipe configuration: [Configuration overview](../../training/config-container-overview.md) +- Training entry points: [Entry points](../../training/entry-points.md) diff --git a/docs/models/llm/qwen.md b/docs/models/llm/qwen.md new file mode 100644 index 0000000000..852672243b --- /dev/null +++ b/docs/models/llm/qwen.md @@ -0,0 +1,62 @@ +# Qwen + +Qwen2/2.5/3 models are supported via the Bridge with QK layernorm handling (Qwen3) and bias in QKV (Qwen2). + +## Conversion with 🤗 Hugging Face + +### Load HF → Megatron +```python +from megatron.bridge import AutoBridge + +# Example: Qwen3 7B +bridge = AutoBridge.from_hf_pretrained("Qwen/Qwen3-7B") +provider = bridge.to_megatron_provider() + +provider.tensor_model_parallel_size = 8 +model = provider.provide_distributed_model(wrap_with_ddp=False) +``` + +### Export Megatron → HF +```python +bridge.save_hf_pretrained(model, "./qwen-hf-export") +# or convert a checkpoint directory +bridge.export_ckpt( + megatron_path="/results/qwen3_8b/checkpoints/iter_00002000", + hf_path="./qwen-hf-export", +) +``` + +### Examples +- Checkpoint import/export: `examples/conversion/convert_checkpoints.py` +- Generate text (HF→Megatron): `examples/conversion/hf_to_megatron_generate_text.py` + +## Pretrain recipes +- Example usage (Qwen3 8B) +```python +from megatron.bridge.recipes.qwen import qwen3_8b_pretrain_config + +cfg = qwen3_8b_pretrain_config( + hf_path="Qwen/Qwen3-8B", + data_paths=["/path/to/dataset.nvjsonl"], + dir="/results/qwen3_8b", +) +``` + +- API reference for Qwen recipes: + - Qwen recipes overview: [bridge.recipes.qwen](../../apidocs/bridge/bridge.recipes.qwen.md) + - Qwen2 recipes: [bridge.recipes.qwen.qwen2](../../apidocs/bridge/bridge.recipes.qwen.qwen2.md) + - Qwen3 recipes: [bridge.recipes.qwen.qwen3](../../apidocs/bridge/bridge.recipes.qwen.qwen3.md) + - Qwen3 MoE recipes: [bridge.recipes.qwen.qwen3_moe](../../apidocs/bridge/bridge.recipes.qwen.qwen3_moe.md) + +## Finetuning recipes +- Coming soon + +## Hugging Face model cards +- Qwen2: `https://huggingface.co/Qwen/Qwen2-7B` +- Qwen2.5: `https://huggingface.co/Qwen/Qwen2.5-7B` +- Qwen3: `https://huggingface.co/Qwen/Qwen3-7B` + +## Related docs +- Recipe usage and customization: [Recipe usage](../../recipe-usage.md) +- Training configuration: [Configuration overview](../../training/config-container-overview.md) +- Training entry points: [Entry points](../../training/entry-points.md) diff --git a/docs/models/vlm/index.md b/docs/models/vlm/index.md new file mode 100644 index 0000000000..fbec0fdd48 --- /dev/null +++ b/docs/models/vlm/index.md @@ -0,0 +1,8 @@ +# Vision Language Models + +This section documents Vision Language Models supported by Megatron Bridge. + +```{toctree} +:hidden: + +``` diff --git a/docs/recipe-usage.md b/docs/recipe-usage.md index 075f4be5d6..a115c66573 100644 --- a/docs/recipe-usage.md +++ b/docs/recipe-usage.md @@ -3,6 +3,13 @@ Megatron Bridge provides production-ready training recipes for several popular models. You can find an overview of supported recipes and 🤗 HuggingFace bridges [here](index.md#supported-models). This guide will cover the next steps to make use of a training recipe, including how to [override configuration](#overriding-configuration) and how to [launch a job](#launch-methods). +## Overview + +- **Coverage**: We provide recipes across select model families and sizes, including Llama, Qwen, DeepSeek, and Nemotron-H (Mamba-based). +- **Defaults**: Each recipe sets defaults meant for convergence and performance across parallelisms, precision data types, and optimizer & scheduler choices. These recipes can be used as a high-quality starting point. +- **Integration**: Recipes return a single `ConfigContainer` that plugs directly into our training [entry points](training/entry-points.md) (see the published docs as well: https://docs.nvidia.com/nemo/megatron-bridge/latest/training/entry-points.html). +- **Customization**: You can override any part of the recipe (Python, YAML, CLI) to adapt to your data, scale, and objectives. + ## Overriding configuration Recipes are provided through a {py:class}`~bridge.training.config.ConfigContainer` object. This is a dataclass that holds all configuration objects needed for training. You can find a more detailed overview of the `ConfigContainer` [here](training/config-container-overview.md).