Best practice for training LLaMA models in Megatron-LM
-
Updated
Jan 2, 2024 - Python
#
megatron-lm
Here are 12 public repositories matching this topic...
Super-Efficient RLHF Training of LLMs with Parameter Reallocation
distributed-systems reinforcement-learning distributed-computing transformers large-scale-machine-learning deepspeed megatron-lm large-language-models llm reinforcement-learning-from-human-feedback llm-training llm-framework
-
Updated
Jan 13, 2025 - Python
Annotations of the interesting ML papers I read
nlp machine-learning deep-learning transformers gpt research-paper bert gpt-2 xlnet annotated-paper megatron-lm papers-annotations
-
Updated
Mar 8, 2025
Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*
transformers moe data-parallelism distributed-optimizers model-parallelism megatron mixture-of-experts pipeline-parallelism huggingface-transformers megatron-lm tensor-parallelism large-scale-language-modeling 3d-parallelism zero-1 sequence-parallelism
-
Updated
Dec 14, 2023 - Python
Odysseus: Playground of LLM Sequence Parallelism
-
Updated
Jun 17, 2024 - Python
A LLaMA1/LLaMA12 Megatron implement.
-
Updated
Dec 13, 2023 - Python
Training NVIDIA NeMo Megatron Large Language Model (LLM) using NeMo Framework on Google Kubernetes Engine
-
Updated
Nov 19, 2024 - HCL
Minimal yet high performant code for pretraining llms. Attempts to implement some SOTA features. Implements training through: Deepspeed, Megatron-LM, and FSDP. WIP
-
Updated
Feb 6, 2024 - Python
Megatron-LM/GPT-NeoX compatible Text Encoder with 🤗Transformers AutoTokenizer.
-
Updated
Nov 16, 2023 - Python
Running Large Language Model easily.
-
Updated
Mar 18, 2025 - Jupyter Notebook
Wrapped Megatron: As User-Friendly as HuggingFace, As Powerful as Megatron-LM | Megatron封装：和HuggingFace一样方便，和Megatron-LM一样强大
wrapper simplification transformer wrap llama gpt sft megatron huggingface-transformers deepspeed megatron-lm llm llm-training
-
Updated
Dec 23, 2024 - Python
Improve this page
Add a description, image, and links to the megatron-lm topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the megatron-lm topic, visit your repo's landing page and select "manage topics."