diff --git a/README.md b/README.md index 58658430c3..1ed2e83d1d 100644 --- a/README.md +++ b/README.md @@ -7,165 +7,186 @@ WeChat Group |

-ReaL +ReaL -AReaL (Ant Reasoning RL) is an open-source **fully asynchronous reinforcement learning training system** for large reasoning models developed at **the RL Lab, Ant Research**. Built upon the open-source project [RealHF](https://github.com/openpsi-project/ReaLHF), we are fully committed to open-source by providing training details, data, and infrastructure required to reproduce results along with the model itself. AReaL aims to help everyone build their own AI agents easily and affordably. Our team loves milk tea because it's delicious, customizable, and affordable. We hope you enjoy our project just like how you enjoy real-world milk tea (cheers). +AReaL (Ant Reasoning RL) is an open-source **fully asynchronous reinforcement learning +training system** for large reasoning models developed at **the RL Lab, Ant Research**. +Built upon the open-source project [ReaLHF](https://github.com/openpsi-project/ReaLHF), +we are fully committed to open-source by providing training details, data, and +infrastructure required to reproduce results along with the model itself. AReaL aims to +help everyone build their own AI agents easily and affordably. Our team loves milk tea +because it's delicious, customizable, and affordable. We hope you enjoy our project just +like how you enjoy real-world milk tea (cheers). **AReaL Highlights** -+ 🔥 **[NEW] Asynchronous RL:** With algorithm-system co-design, AReaL supports fully asynchronous RL for **the fastest training**! Experimental support for multi-turn agentic RL is also provided. -+ 🛠️ **Open & Reproducible**: We continuously release _all code, datasets, and training recipes_ for RL training of LLMs. -+ 🚀 **Scalability**: AReaL can seamlessly adapt to different computational resource settings, ranging from a single node to 1K GPUs. -+ 🔪 **Cutting-Edge Performance:** AReaL can produce models with cutting-edge reasoning capabilities in math and coding. We are also actively working on agentic tasks. +- ⚡ **\[NEW\] AReaL-lite:** Our new + release AReaL-lite is a **light-weight** and **algorithm-first** codebase that + prioritizes better development experiences for AI researchers. As a result, AReaL-lite + delivers most AReaL functionalities while maintains its high performance with much + fewer lines of code. This allows users to build their own **agentic** training + workflows with minimal efforts. +- 🔥 **Asynchronous RL**: With algorithm-system co-design, AReaL supports fully + asynchronous RL for **the fastest training speed**! Experimental support for + multi-turn agentic RL is also provided. +- 🛠️ **Open & Reproducible**: We continuously release _all code, datasets, and training + recipes_ for RL training of LLMs. +- 🚀 **Scalability**: AReaL can seamlessly adapt to different computational resource + settings, ranging from a single node to 1K GPUs. +- 🔪 **Cutting-Edge Performance:** AReaL can produce models with cutting-edge reasoning + capabilities in math and coding. We are also actively working on agentic tasks. ## News -**[2025/06/03] (v0.3, boba²)** We release **boba²** (double-boba) for fully asynchronous RL training, which achieves a **2.77x speedup while obtaining on-par or even better training performance** compared to synchronous systems. Moreover, asynchronous RL makes it extremely easy to set up multi-turn agentic RL training! Check out [our v0.3 overview blog](/blog/AReaL_v0_3.md) and the [research paper](https://arxiv.org/pdf/2505.24298). +**\[2025/07/31\] (AReaL-lite)** We introduce AReaL-lite, a **light-weight** version of +AReaL designed specifically for AI researchers and rapid prototyping. AReaL-lite +features an **algorithm-first** API design that prioritizes ease of use and algorithm +development, while inherently supporting **fully asynchronous agentic RL**. With 80% +fewer lines of code, AReaL-lite maintains 90% of AReaL's high performance and core +functionality. Check out [our AReaL-lite design doc](/areal/README.md) and +[the quickstart guide](https://inclusionai.github.io/AReaL/tutorial/quickstart.html) to +begin your journey with **AReaL-lite**! -**[2025/03/31] (v0.2, boba)** Here comes our next milestone release - boba! Please call it A-ReaL-boba! This release includes much faster training with SGLang support and SOTA 7B and 32B models on math reasoning. Check our [v0.2 technical blog](/blog/AReaL_v0_2.md). +**\[2025/06/03\] (v0.3, boba²)** We release **boba²** (double-boba) for fully +asynchronous RL training, which achieves a **2.77x speedup while obtaining on-par or +even better training performance** compared to synchronous systems. Moreover, +asynchronous RL makes it extremely easy to set up multi-turn agentic RL training! Check +out [our v0.3 overview blog](/blog/AReaL_v0_3.md) and the +[research paper](https://arxiv.org/pdf/2505.24298). -**[2025/02/24] (v0.1)** Our initial release includes reproducible results for 1.5B and 7B LRMs. Check our [v0.1 technical blog](/blog/AReaL_v0_1.md). +**\[2025/03/31\] (v0.2, boba)** Here comes our next milestone release - boba! Please +call it A-ReaL-boba! This release includes much faster training with SGLang support and +SOTA 7B and 32B models on math reasoning. Check our +[v0.2 technical blog](/blog/AReaL_v0_2.md). -## Release Highlights +**\[2025/02/24\] (v0.1)** Our initial release includes reproducible results for 1.5B and +7B LRMs. Check our [v0.1 technical blog](/blog/AReaL_v0_1.md). -In our AReaL-boba² (A-ReaL-double-boba) release, we highlight the top 3 most important features: +## AReaL-lite Release Highlights -+ A fully asynchronous RL training pipeline with **system and RL algorithm co-design**, achieving over 2.77x speedup without any performance drop. Check the [benchmark scripts and instructions here](https://github.com/inclusionAI/AReaL/tree/main/benchmark/verl_v0_3_0_post1_76084d3). +New highlights in AReaL-lite: -+ SOTA coding models, i.e., a 14B model with a **69.1 score on LCB-v5**. To reproduce, check the [configs](https://github.com/inclusionAI/AReaL/tree/main/examples/configs/v0.3-qwen3-code) and [instructions](https://inclusionai.github.io/AReaL/references/reproduce.html). +- Instead of the *system-first* architecture in old AReaL, AReaL-lite follows an + **algorithm-first** API design that aims to provide the following key features: -+ Experimental support for **multi-turn** agentic RL training. Check our [complete example](https://inclusionai.github.io/AReaL/customization/agent.html). + - **Light-weight** & **easy-to-write** algorithm and training workflow customization. + - **Easy to scale up** without knowing system and infrastructure details. + - **Adaptable and plugable:** Smooth to integrate with other modern AI applications. -For the complete system design and more training details, please check [our v0.3 blog](/blog/AReaL_v0_3.md) and our [research paper](https://arxiv.org/pdf/2505.24298). + These features make AReaL-lite easy for AI researchers to adopt, understand, and + develop effectively and efficiently. To learn more about the design principles of + AReaL, please read the [AReaL-lite design doc](/areal/README.md)! -**Jump to the [quickstart section](https://github.com/inclusionAI/AReaL?tab=readme-ov-file#getting-started) if you want to quickly run an experiment and get your hands dirty!** 😈 +- A much more *light-weight* codebase compared to old AReaL codebase with only **20%** # + lines of code, with a detailed + [code walkthrough](https://inclusionai.github.io/AReaL/areal/gsm8k_grpo.html) on an + GRPO-on-GSM8K example. Save your time & efforts for code reading! -### Overview of Asynchronous RL Training +- Smoother customization for your own **algorithms** and **agentic & RLVR rollout** RL + within a single file! Check + [here](https://inclusionai.github.io/AReaL/customization/agent.html) for agent & RLVR + customization and [here](https://inclusionai.github.io/AReaL/tutorial/algorithm.html) + for algorithm customization. -During the synchronous RL training process, a generation step must wait until the longest sequence completes within the batch of LLM outputs. Due to the varying output lengths for LRMs, a synchronous RL system suffers from massive GPU idle time, leading to training inefficiency. Some recent works ([DeepCoder](https://pretty-radio-b75.notion.site/DeepCoder-A-Fully-Open-Source-14B-Coder-at-O3-mini-Level-1cf81902c14680b3bee5eb349a512a51), [Intellect](https://www.primeintellect.ai/blog/intellect-2)) propose overlapping a single training step with a single generation step to accelerate training. However, the largest bottleneck remains unchanged: the samples within a batch are still from the same model version, leading to waiting and GPU idle time. +Good old stuff from AReaL: -![Synchronous vs One-step Overlap RL](/assets/sync_one_step_gen.png) +- High performance and scalability with fully asynchronous RL training. Check our + [boba² (v0.3) blog](/blog/AReaL_v0_3.md) for details. -*Fig.1. Left: Execution timeline of synchronous RL training. Right: Execution timeline of one-step overlap RL system.* +- A single command line to launch an experiment, no matter on a single node or a + large-scale distributed cluster. -AReaL adopts a fully asynchronous RL training framework that completely decouples generation from training. In AReaL, LLM generation runs in a streaming manner, with each rollout worker continuously producing outputs without waiting. Meanwhile, trainer workers perform parallel model updates upon receiving training batches. +Now, let us run an example experiment with AReaL-lite following the quickstart guide +below! -![Asynchronous RL Training](/assets/async_timeline.png) +## Getting Started with AReaL-lite -*Fig 2. Execution timeline of our fully asynchronous RL system.* +Our training scripts will automatically download the dataset (openai/gsm8k) and model +(Qwen/Qwen2-1.5B-Instruct). On a single node, runs: -AReaL follows a system-algorithm co-design principle: on the system side, AReaL efficiently syncs model parameters and carefully controls the staleness of each training sample; on the algorithm side, AReaL improves the objective of PPO to make async-RL stable. - -We compare the scalability of **asynchronous RL** training based on our AReaL-boba² system with **classical synchronous RL** training (we adopt the fastest open-source system veRL, main branch on 05/07/2025) across different model sizes and different numbers of H800 GPUs. AReaL demonstrates much improved scaling capabilities with respect to training throughput. This is also partially due to AReaL decoupling training and generation, leading to much fewer GPU memory fragments. - -![Scaling Comparison](/assets/async_scaling_vs_verl.png) - -*Fig.3 The scaling trend of asynchronous RL (based on AReaL-boba2) and classical synchronous RL (based on veRL) with different model sizes. Dotted lines indicate ideal linear scaling.* - -### SOTA Code Generation Model by AReaL-boba² - -We use **Qwen3** as our base model. After asynchronous RL training, we achieve SOTA results on LiveCodeBench, Codeforces, and CodeContests benchmarks. - -| **Model (8B)** | **LiveCodeBench v5**
**(2024.10-2025.2)** | **Codeforces** | **CodeContests** | -| :---: | :---: | :---: | :---: | -| Qwen3-8B | 58.8 | 1879/96.7% | 31.4 | -| DeepSeek-R1-0528-Qwen3-8B | 58.4 | 1945/97.3% | 31.0 | -| [🤗 AReaL-boba²-8B-Open](https://huggingface.co/inclusionAI/AReaL-boba-2-8B-subset) | 62.0 | 1933/97.2% | **41.4** | -| [🤗 AReaL-boba²-8B](https://huggingface.co/inclusionAI/AReaL-boba-2-8B) | **63.0** | **1962/97.5%** | 40.8 | - -| **Model (14B)** | **LiveCodeBench v5**
**(2024.10-2025.2)** | **Codeforces** | **CodeContests** | -| :---: | :---: | :---: | :---: | -| Qwen3-14B | 65.4 | 1978/97.7% | 38.3 | -| DeepCoder-14B-Preview | 60.6 | 1936/95.3% | 40.1 | -| [🤗 AReaL-boba²-14B-Open](https://huggingface.co/inclusionAI/AReaL-boba-2-14B-subset) | 67.3 | 1990/97.8% | **46.2** | -| [🤗 AReal-boba²-14B](https://huggingface.co/inclusionAI/AReaL-boba-2-14B) | **69.1** | **2044/98.2%** | 46.1 | +```bash +python3 -m arealite.launcher.local examples/arealite/gsm8k_grpo.py --config examples/arealite/configs/gsm8k_grpo.yaml +``` -| **Larger Models** | **LiveCodeBench v5**
**(2024.10-2025.2)** | **Codeforces** | **CodeContests** | -| :---: | :---: | :---: | :---: | -| Qwen3-235B | 70.7 | 2056 | - | -| DeepSeek-R1 | 64.3 | 2029 | - | -| OpenAI-o3-mini (Medium) | 66.3 | 2036 | - | +On a Ray cluster with 2 nodes & 8 GPUs each node, runs (Remember to change paths in the +YAML file to your own shared storage): -*Table 1: Coding Task Performance Comparison. AReaL-boba²-8B/14B-Open denotes training results on open-source data. AReaL-boba²-8B/14B models are trained with an additional small amount of internal data and achieve SOTA performance on LiveCodeBench, Codeforces & CodeContests.* +```bash +python3 -m arealite.launcher.ray examples/arealite/gsm8k_grpo.py --config examples/arealite/configs/gsm8k_grpo.yaml \ + cluster.n_nodes=2 \ + cluster.n_gpus_per_node=8 +``` -We highlight the [tutorials](https://inclusionai.github.io/AReaL/customization/dataset.html) and [code walkthroughs](https://inclusionai.github.io/AReaL/developer/overview.html) about the following key features for asynchronous training: + -+ [Streaming generation and reward computation](https://inclusionai.github.io/AReaL/developer/rollout/rollout_worker.html) -+ [Interruptible rollout](https://inclusionai.github.io/AReaL/developer/rollout/gserver.html) -+ [Data staleness control with the rollout controller](https://inclusionai.github.io/AReaL/developer/rollout/gserver.html) -+ [The adoption of decoupled PPO loss](https://inclusionai.github.io/AReaL/customization/algorithm.html) + -AReaL-boba² allows you to independently customize the [dataset](https://inclusionai.github.io/AReaL/customization/dataset.html), [rollout behavior](https://inclusionai.github.io/AReaL/customization/agent.html), and the [training algorithm](https://inclusionai.github.io/AReaL/customization/algorithm.html), without needing to modify the heavy system-level code. +For more detailed guide on how to run experiments in AReaL-lite, please check out +[our quickstart guide](https://inclusionai.github.io/AReaL/tutorial/quickstart.html)! -In particular, we show a simple example to develop a multi-turn math agent for RL training. Please see the learning curve below and reference the [step-by-step guide](https://inclusionai.github.io/AReaL/customization/agent.html) if you want to implement your own agentic RL project. +## Switching from legacy AReaL to AReaL-lite -## Getting Started -Obtain the training data: -- [Math](https://huggingface.co/datasets/inclusionAI/AReaL-boba-Data) -- [Code](https://huggingface.co/datasets/inclusionAI/AReaL-boba-2-RL-Code) +We also provide a convenient script to convert your AReaL YAML config into AReaL-lite +config in one command line. First you need to locate your AReaL config either modified +from files from `examples` folder, or generated when you run your experiments in +`//` folder. Runs: -For code training data, a simple preprocessing script was provided in `examples/data_preprocess/preprocess_training_data.py`: -```bash -python3 preprocess_training_data.py --data_path $original_data_path --output_path $training_data_path ``` - -Train Qwen3 1.7B locally (Remember to modify `dataset.path` in the script below): -```bash -bash examples/run_async_ppo.sh +python arealite/utils/convert.py --convert_src AReaL --src_config_path --template_path examples/arealite/configs/gsm8k_grpo.yaml --output_path ``` -Evaluation: +Then you should be able to run experiments with your old settings on AReaL-lite! -```bash -cd evaluation -# Evaluate the model -python eval_and_aggregate.py \ - --model_path ${MODEL_PATH} \ - --output_path ${OUTPUT_PATH} \ - --data_names aime24,aime25 \ - --max_gen_tokens 32768 \ - --data_names codeforces,lcb_v5 \ - --prompt_type qwen3-think-pure \ - --temperature 1.0 -``` +## AReaL-lite vs legacy AReaL + +AReaL-lite is an initiative to fully refactor AReaL, addressing historical issues such +as redundant code and unnecessary system-level abstractions. Currently, AReaL-lite +provides a lightweight codebase that enables fast prototyping for new RL training +workflows and algorithms on a relatively small scale. For large-scale experiments (1K+ +GPUs), we recommend using the battle-tested legacy AReaL to ensure stability. In the +future, we will continue developing AReaL-lite by expanding its APIs, migrating legacy +features, introducing new functionality, and validating the system through large-scale +experiments. ## Resources -+ [Documentation](https://inclusionai.github.io/AReaL/) -+ [Contributing](https://inclusionai.github.io/AReaL/contrib.html) +- [Documentation](https://inclusionai.github.io/AReaL/) +- [Contributing](https://inclusionai.github.io/AReaL/contrib.html) ### Quickstart -+ [Installation](https://inclusionai.github.io/AReaL/tutorial/installation.html) -+ [Example: Improving the math capability of Qwen3 with PPO](https://inclusionai.github.io/AReaL/tutorial/quickstart.html) +- [Installation](https://inclusionai.github.io/AReaL/tutorial/installation.html) +- [AReaL-lite Quickstart](https://inclusionai.github.io/AReaL/tutorial/quickstart.html) -### Benchmark and Reproduction +### Code Walkthrough -+ **Reproduce boba² Code Models** - - 🤗 **Model weights**: [8B-code](https://huggingface.co/inclusionAI/AReaL-boba-2-8B), [14B-code](https://huggingface.co/inclusionAI/AReaL-boba-2-14B), [8B-code-open](https://huggingface.co/inclusionAI/AReaL-boba-2-8B-subset), [14B-code-open](https://huggingface.co/inclusionAI/AReaL-boba-2-14B-subset) - - [Evaluation Guide](https://inclusionai.github.io/AReaL/tutorial/eval.html) - - [Training configs](https://github.com/inclusionAI/AReaL/tree/main/examples/configs/v0.3-qwen3-code) and [instructions](https://inclusionai.github.io/AReaL/references/reproduce.html) -+ [Scripts for Benchmark Training Throughput](https://github.com/inclusionAI/AReaL/tree/main/benchmark/verl_v0_3_0_post1_76084d3) +- [Running GRPO on GSM8K dataset with AReaL-lite](https://inclusionai.github.io/AReaL/arealite/gsm8k_grpo.html) -### Customization Guide +### Customization -- [Use your own dataset](https://inclusionai.github.io/AReaL/customization/dataset.html) -- [Modifying the reward function and rollout behavior (multi-turn agentic RL)](https://inclusionai.github.io/AReaL/customization/agent.html) -- [Modifying PPO to GRPO](https://inclusionai.github.io/AReaL/customization/algorithm.html#grouped-advantage-normalization) -- [Developing the decoupled PPO loss](https://inclusionai.github.io/AReaL/customization/algorithm.html#the-decoupled-ppo-loss) +- [Customize dataset with AReaL-lite](https://inclusionai.github.io/AReaL/customization/dataset.html) +- [Customize Agentic/RVLR rollout workflows with AReaL-lite](https://inclusionai.github.io/AReaL/customization/agent.html) +- [Customize algorithms with AReaL-lite](https://inclusionai.github.io/AReaL/customization/algorithm.html) -### System Code Walkthrough +### AReaL Legacy -+ [Trainer](https://inclusionai.github.io/AReaL/developer/trainer/model_worker.html) -+ [Model Backend and Algorithm Interface](https://inclusionai.github.io/AReaL/developer/trainer/algo_interface.html) -+ [Rollout Controller](https://inclusionai.github.io/AReaL/developer/rollout/gserver.html) -+ [Streaming generation and reward computation](https://inclusionai.github.io/AReaL/developer/rollout/rollout_worker.html) +For old AReaL documentation, check the legacy sections in our +[documentation](https://inclusionai.github.io/AReaL/). To reproduce AReaL boba & boba² +results, check our +[reproduction guide with legacy AReaL](https://inclusionai.github.io/AReaL/references/reproduce.html). ## Future Plan -AReaL is under active development. We plan to have minor releases weekly and major releases monthly. Community engagement and contributions are extremely welcome. We are also **hiring interns and full-time employees** with open positions in both the US and China. +AReaL is under active development. We plan to have minor releases weekly and major +releases monthly. Community engagement and contributions are extremely welcome. We are +also **hiring interns and full-time employees** with open positions in both the US and +China. For the research and development plan already in place, please see the following list: @@ -174,26 +195,46 @@ For the research and development plan already in place, please see the following - [x] Support for SGLang - [x] RL training with coding problems - [x] Asynchronous generation and RL training -- [ ] Optimizations for distributed training: expert parallel for MOE and zero-bubble pipelining -- [ ] RL for vision-language models (VLM) +- [ ] Optimizations for distributed training: expert parallel for MOE and zero-bubble + pipelining +- [x] RL for vision-language models (VLM) - [x] Multi-turn agentic RL -- [ ] Function calling and tool use +- [x] Function calling and tool use ### Algorithm Development - [x] RL training recipes for 1.5B and 7B models - [x] A complete RL training recipe for 32B models - [ ] Sample-efficient multi-task RL algorithms -- [ ] Agentic capabilities with end-to-end RL +- [x] Agentic capabilities with end-to-end RL - [ ] Stable RL training for larger MOE models ## Acknowledgement -We would like to note that major contributors are from the RL Lab at Ant Research and the Institute for Interdisciplinary Information Sciences, Tsinghua University. - -Our team has also received invaluable assistance from the Data Intelligence Lab at Ant Research for data support and from the Super Computing Technology (SCT) team at Ant Group, particularly in the realm of large-scale cluster operations and maintenance. - -We also appreciate all the pioneering works from the community, particularly the [ReaLHF](https://github.com/openpsi-project/ReaLHF) project from OpenPsi Inc. and other projects, including but not limited to [DeepScaleR](https://github.com/agentica-project/deepscaler), [Open-Reasoner-Zero](https://github.com/Open-Reasoner-Zero/Open-Reasoner-Zero/tree/main), [OpenRLHF](https://github.com/OpenRLHF/OpenRLHF), [VeRL](https://github.com/volcengine/verl), [SGLang](https://github.com/sgl-project/sglang), [QwQ](https://github.com/QwenLM/QwQ), [Light-R1](https://github.com/Qihoo360/Light-R1) and [DAPO](https://github.com/BytedTsinghua-SIA/DAPO). +We would like to note that major contributors are from the RL Lab at Ant Research and +the Institute for Interdisciplinary Information Sciences, Tsinghua University. + +Our team has also received invaluable assistance from the following groups (listed in +alphabetical order): + +• The Data Intelligence Lab at Ant Research for their data support • The +[Relaxed System Lab](https://github.com/Relaxed-System-Lab) from HKUST for seamless +cooperation on many system-related aspects • The +[SGLang team](https://github.com/sgl-project/sglang) for supporting customized features +for asynchronous RL training and their contributions during the development of +AReaL-lite • The Super Computing Technology (SCT) team at Ant Group, particularly for +their expertise in large-scale cluster operations and maintenance + +We also appreciate all the pioneering works from the community, particularly the +[ReaLHF](https://github.com/openpsi-project/ReaLHF) project from OpenPsi Inc. and other +projects, including but not limited to +[DeepScaleR](https://github.com/agentica-project/deepscaler), +[Open-Reasoner-Zero](https://github.com/Open-Reasoner-Zero/Open-Reasoner-Zero/tree/main), +[OpenRLHF](https://github.com/OpenRLHF/OpenRLHF), +[VeRL](https://github.com/volcengine/verl), +[SGLang](https://github.com/sgl-project/sglang), [QwQ](https://github.com/QwenLM/QwQ), +[Light-R1](https://github.com/Qihoo360/Light-R1) and +[DAPO](https://github.com/BytedTsinghua-SIA/DAPO). ## Citation @@ -210,12 +251,12 @@ We also appreciate all the pioneering works from the community, particularly the ```bibtex @misc{fu2025areal, - title={AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning}, + title={AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning}, author={Wei Fu and Jiaxuan Gao and Xujie Shen and Chen Zhu and Zhiyu Mei and Chuyi He and Shusheng Xu and Guo Wei and Jun Mei and Jiashu Wang and Tongkai Yang and Binhang Yuan and Yi Wu}, year={2025}, eprint={2505.24298}, archivePrefix={arXiv}, primaryClass={cs.LG}, - url={https://arxiv.org/abs/2505.24298}, + url={https://arxiv.org/abs/2505.24298}, } ``` diff --git a/areal/README.md b/areal/README.md index 8b092d2939..3b179ef7ea 100644 --- a/areal/README.md +++ b/areal/README.md @@ -1,19 +1,34 @@ -# AReaL-Lite Design Doc +# AReaL-lite Design Doc ## TL;DR -Follow our [step-by-step code walk-through](../docs/areal/gsm8k_grpo.md) to immediately -get started with AReaL-Lite! +Follow our +[step-by-step code walk-through](https://inclusionai.github.io/AReaL/lite/gsm8k_grpo.html) +to immediately get started with AReaL-lite! ## Motivation AReaL presents several challenges that make it difficult for AI researchers to adopt, understand, and develop with effectively. The primary issue stems from its -*system-centric* rather than *AI-centric* architecture. The reinforcement learning -algorithm workflow is built around multiple *workers* executing consecutive *model -function calls* — concepts that are unfamiliar to most AI researchers. This forces users -to first master these system-level abstractions before they can implement workflows and -algorithms for their specific research needs. +*system-first* rather than *algorithm-first* architecture and API design. An +*algorithm-first* design aims to provide three key features: + +- **Light-weight & easy-to-write customization:** Users can implement their algorithms + and training workflows with minimal and concentrated code, often in just a few files + or even a single file. +- **Easy to scale up:** Experiments can be scaled up seamlessly without requiring + knowledge of underlying system or infrastructure details. +- **Adaptable and plugable:** Users is free to integrate the system with code or APIs + from other AI libraries, or plug APIs from the system into other frameworks. + +We believe that AReaL, as well as other existing RL systems, falls short in fully +delivering these features. For example, the RL training workflow in AReaL is built +around multiple *workers* executing *model function calls* in a *DAG* (Directed Acyclic +Graph). To customize a training workflow, researchers first need to understand these +system-level concepts. Then they are forced to find code to modify, which is scattered +around in the codebase. It is also nearly impossible to exploit packages like `datasets` +since it is not compatible with the workers. This gap is the core motivation behind +AReaL-lite: rebuilding AReaL with an algorithm-first architecture and APIs. Beyond architectural concerns, AReaL suffers from accumulated technical debt. The codebase contains substantial legacy code inherited from previous projects that no @@ -26,22 +41,23 @@ possible to achieve comparable efficiency with significantly fewer lines of code presents an ideal opportunity to redesign the API and distill the massive codebase into something clean and maintainable. Rather than pursuing maximum efficiency, our goal is to deliver 90% of AReaL's functionality while dramatically reducing code complexity and -user burden. This philosophy drives AReaL-Lite — the lightweight version of AReaL. +user burden. This philosophy drives AReaL-lite — the lightweight version of AReaL. -AReaL-Lite serves as the first phase in AReaL's broader refactoring initiative. It +AReaL-lite serves as the first phase in AReaL's broader refactoring initiative. It functions both as a standalone training library with intuitive interfaces and as the foundation for AReaL's future core API definitions. The plan is to transform AReaL's -current worker-based architecture into an AI-centric architecture similar to AReaL-Lite, -where AReaL will **extend** AReaL-Lite's APIs and implementations to support additional -backends for efficient large-scale training. +current worker-based architecture into an algorithm-first architecture similar to +AReaL-lite, where AReaL will **extend** AReaL-lite's APIs and implementations to support +additional backends for efficient large-scale training. ## Design Principles -Our design is guided by seven core principles: +To achieve *algorithm-first* and *light-weight* while maintaining the efficiency, our +design is guided by seven core principles: 1. **Native asynchronous RL training support** — Built from the ground up for disentangled generation and training -1. **AI-centric design** — Minimize exposure to system concepts like "PlacementGroup" +1. **System-less design** — Minimize exposure to system concepts like "PlacementGroup" 1. **PyTorch-centric approach** — Use raw PyTorch types without unnecessary abstractions 1. **Transparent algorithm orchestration** — Make the flow of operations clear and understandable @@ -65,6 +81,16 @@ areal/ ### Component Overview +The AReaL-lite codebase is structured into four distinct layers: the API layer, backend +layer, customization layer, and entry point layer. As illustrated in the figure below, +workflow and algorithm customization logic resides in separate layers above the backend. +We prioritize keeping the entry point and customization layers clean and intuitive, +isolating them from the complex backend implementation. With AReaL-lite, users can +define their custom training workflows and algorithms entirely within a single entry +point file. + +![arealite-layers](../assets/areal_lite_layers.png) + #### 1. API Layer (`api/`) The API layer establishes clean contracts between components through abstract interfaces @@ -141,9 +167,9 @@ python3 -m areal.launcher.slurm examples/lite/gsm8k_grpo.py \ For detailed customization instructions, please refer to our documentation: -- [Adding new agents](../docs/customization/agent.md) -- [Adding new datasets](../docs/customization/dataset.md) -- [Adding new algorithms](../docs/customization/algorithm.md) +- [Adding new agents](https://inclusionai.github.io/AReaL/customization/agent.html) +- [Adding new datasets](https://inclusionai.github.io/AReaL/customization/dataset.html) +- [Adding new algorithms](https://inclusionai.github.io/AReaL/customization/algorithm.html) ## Implementation Details diff --git a/assets/areal_lite_layers.png b/assets/areal_lite_layers.png new file mode 100644 index 0000000000..f57fdab3b5 Binary files /dev/null and b/assets/areal_lite_layers.png differ diff --git a/assets/logo_lite.png b/assets/logo_lite.png new file mode 100644 index 0000000000..fd9c61bebe Binary files /dev/null and b/assets/logo_lite.png differ diff --git a/blog/AReaL_v0_3.md b/blog/AReaL_v0_3.md index 0222343c6a..9ae8334f43 100644 --- a/blog/AReaL_v0_3.md +++ b/blog/AReaL_v0_3.md @@ -6,59 +6,81 @@ We now release AReaL v0.3, featuring three major milestones: -- **A fully asynchronous RL training pipeline with system and RL algorithm co-design**, achieving over 2.77x speedup without any performance drop -- **SOTA coding models**, i.e., a 14B model with a **69.1 score on LCB-v5**. Reproducible results with fully open-sourced datasets are also provided +- **A fully asynchronous RL training pipeline with system and RL algorithm co-design**, + achieving over 2.77x speedup without any performance drop +- **SOTA coding models**, i.e., a 14B model with a **69.1 score on LCB-v5**. + Reproducible results with fully open-sourced datasets are also provided - Experimental support for **multi-turn agentic RL training** ### Performance Results -| **Model (8B)** | **LiveCodeBench v5** (2024.10-2025.2) | **Codeforce** | **CodeContests** | -|:---:|:---:|:---:|:---:| -| Qwen3-8B | 58.8 | 1879/96.7% | 31.4 | -| DeepSeek-R1-0528-Qwen3-8B | 58.4 | 1945/97.3% | 31.0 | -| [🤗 AReaL-boba²-8B-Open](https://huggingface.co/inclusionAI/AReaL-boba-2-8B-subset) | 62.0 | 1933/97.2% | **41.4** | -| [🤗 AReaL-boba²-8B](https://huggingface.co/inclusionAI/AReaL-boba-2-8B) | **63.0** | **1962/97.5%** | 40.8 | - -| **Model (14B)** | **LiveCodeBench v5** (2024.10-2025.2) | **Codeforce** | **CodeContests** | -|:---:|:---:|:---:|:---:| -| Qwen3-14B | 65.4 | 1978/97.7% | 38.3 | -| DeepCoder-14B-Preview | 60.6 | 1936/95.3% | 40.1 | -| [🤗 AReaL-boba²-14B-Open](https://huggingface.co/inclusionAI/AReaL-boba-2-14B-subset) | 67.3 | 1990/97.8% | **46.2** | -| [🤗 AReal-boba²-14B](https://huggingface.co/inclusionAI/AReaL-boba-2-14B) | **69.1** | **2044/98.2%** | 46.1 | - -| **Larger Models** | **LiveCodeBench v5** (2024.10-2025.2) | **Codeforce** | **CodeContests** | -|:---:|:---:|:---:|:---:| -| Qwen3-235B | 70.7 | 2056 | - | -| DeepSeek-R1 | 64.3 | 2029 | - | -| OpenAI-o3-mini (Medium) | 66.3 | 2036 | - | - -*Table 1: Coding Task Performance Comparison. AReaL-boba²-8B/14B-Open denotes training results on open-sourced data. AReaL-boba²-8B/14B models are trained with an additional small amount of internal data and achieve SOTA performance on LiveCodeBench, Codeforce & CodeContests.* - -To access our latest models and training data, please visit this [Huggingface Link](https://huggingface.co/collections/inclusionAI/areal-boba-2-683f0e819ccb7bb2e1b2f2d5). +| **Model (8B)** | **LiveCodeBench v5** (2024.10-2025.2) | **Codeforce** | **CodeContests** | +| :---------------------------------------------------------------------------------: | :-----------------------------------: | :------------: | :--------------: | +| Qwen3-8B | 58.8 | 1879/96.7% | 31.4 | +| DeepSeek-R1-0528-Qwen3-8B | 58.4 | 1945/97.3% | 31.0 | +| [🤗 AReaL-boba²-8B-Open](https://huggingface.co/inclusionAI/AReaL-boba-2-8B-subset) | 62.0 | 1933/97.2% | **41.4** | +| [🤗 AReaL-boba²-8B](https://huggingface.co/inclusionAI/AReaL-boba-2-8B) | **63.0** | **1962/97.5%** | 40.8 | + +| **Model (14B)** | **LiveCodeBench v5** (2024.10-2025.2) | **Codeforce** | **CodeContests** | +| :-----------------------------------------------------------------------------------: | :-----------------------------------: | :------------: | :--------------: | +| Qwen3-14B | 65.4 | 1978/97.7% | 38.3 | +| DeepCoder-14B-Preview | 60.6 | 1936/95.3% | 40.1 | +| [🤗 AReaL-boba²-14B-Open](https://huggingface.co/inclusionAI/AReaL-boba-2-14B-subset) | 67.3 | 1990/97.8% | **46.2** | +| [🤗 AReal-boba²-14B](https://huggingface.co/inclusionAI/AReaL-boba-2-14B) | **69.1** | **2044/98.2%** | 46.1 | + +| **Larger Models** | **LiveCodeBench v5** (2024.10-2025.2) | **Codeforce** | **CodeContests** | +| :---------------------: | :-----------------------------------: | :-----------: | :--------------: | +| Qwen3-235B | 70.7 | 2056 | - | +| DeepSeek-R1 | 64.3 | 2029 | - | +| OpenAI-o3-mini (Medium) | 66.3 | 2036 | - | + +*Table 1: Coding Task Performance Comparison. AReaL-boba²-8B/14B-Open denotes training +results on open-sourced data. AReaL-boba²-8B/14B models are trained with an additional +small amount of internal data and achieve SOTA performance on LiveCodeBench, Codeforce & +CodeContests.* + +To access our latest models and training data, please visit this +[Huggingface Link](https://huggingface.co/collections/inclusionAI/areal-boba-2-683f0e819ccb7bb2e1b2f2d5). ## Motivation for Asynchronous RL System ### Inference devices are underutilized -During the synchronous RL training process, the generation step must wait until the finish of the longest output within a batch. Due to the varying output lengths for LRM, a synchronous RL system suffers from severe training inefficiency. +During the synchronous RL training process, the generation step must wait until the +finish of the longest output within a batch. Due to the varying output lengths for LRM, +a synchronous RL system suffers from severe training inefficiency. -Some works ([DeepCoder](https://pretty-radio-b75.notion.site/DeepCoder-A-Fully-Open-Source-14B-Coder-at-O3-mini-Level-1cf81902c14680b3bee5eb349a512a51), [Intellect](https://www.primeintellect.ai/blog/intellect-2)) use outputs generated from a previous model version to update the current model. For the best performances, the model version used for rollout generation is limited to only one or two steps older. However, all these systems still follow a batched generation setting, the issue of system inefficiency during the generation phase still remains unaddressed. +Some works +([DeepCoder](https://pretty-radio-b75.notion.site/DeepCoder-A-Fully-Open-Source-14B-Coder-at-O3-mini-Level-1cf81902c14680b3bee5eb349a512a51), +[Intellect](https://www.primeintellect.ai/blog/intellect-2)) use outputs generated from +a previous model version to update the current model. For the best performances, the +model version used for rollout generation is limited to only one or two steps older. +However, all these systems still follow a batched generation setting, the issue of +system inefficiency during the generation phase still remains unaddressed. ![](/assets/sync_one_step_gen.png) -*Fig.1. Left: Execution timeline of a synchronous RL training. Right: Execution timeline of one-step overlap RL system.* +*Fig.1. Left: Execution timeline of a synchronous RL training. Right: Execution timeline +of one-step overlap RL system.* ### Scalability is poor in synchronous RL systems -Synchronous systems distribute generation across all devices, reducing the per-GPU decoding batch size. This pushes the decoding process into a memory-IO-bound regime where additional devices fail to improve throughput. +Synchronous systems distribute generation across all devices, reducing the per-GPU +decoding batch size. This pushes the decoding process into a memory-IO-bound regime +where additional devices fail to improve throughput. ![](/assets/gen_scaling_trend.png) -*Fig2. Left: Strong scaling of batched generation throughput for a 1.5B LRM. Right: Generation becomes memory-IO bound as GPU count increases.* +*Fig2. Left: Strong scaling of batched generation throughput for a 1.5B LRM. Right: +Generation becomes memory-IO bound as GPU count increases.* ## Asynchronous AReaL Overview -We design a system that fully decouples generation and training across separate GPU clusters. This system should be hardware-efficient, scalable, and equipped with the flexibility for a customized RL workflow. We implement these principles in AReaL. Fig.3 presents the architecture and data flow of AREAL. The system comprises 4 core components: +We design a system that fully decouples generation and training across separate GPU +clusters. This system should be hardware-efficient, scalable, and equipped with the +flexibility for a customized RL workflow. We implement these principles in AReaL. Fig.3 +presents the architecture and data flow of AREAL. The system comprises 4 core +components: ![](/assets/arch.png) @@ -66,13 +88,34 @@ We design a system that fully decouples generation and training across separate ### Core Components -- **Interruptible Rollout Worker** handles two types of requests: (1) The *generate* request generates responses given prompts. (2) The *update_weights* request interrupts all ongoing generations and loads parameters of new versions. Upon the interruption, the rollout workers discard KV caches computed by old weights, and re-compute them using the new weights. Afterwards, the rollout workers continue to decode the unfinished sequences until the next interruption or termination. We emphasize that such interruptions and in-flight weight updates would result in trajectories composed of segments produced by different model versions. This introduces a novel algorithmic challenge, which will be addressed in the next section. - -- **Reward Service** evaluates the accuracy of the responses generated by the model. For example, in the coding task, this service extracts the code and executes unit tests to verify its accuracy. - -- **Trainer Workers** continuously sample from the replay buffer, accumulating data until reaching the configured training batch size. They then perform PPO updates and store the resulting parameters in distributed storage. To ensure data freshness, data from the replay buffer is used only once. - -- **Rollout Controller** serves as a critical bridge between the rollout workers, reward service, and the model workers. During the training process, it reads data from the dataset and invokes the rollout worker's *generate* request. The received response is then sent to the reward service to obtain the reward. The trajectory, along with the reward, is stored in the replay buffer, waiting to be trained by the model worker. After the model worker updates the parameters, the controller calls the rollout worker's *update_weight*. We illustrate the generation and training management in Fig.4. This asynchronous pipeline ensures continuous full utilization of both generation and training resources. +- **Interruptible Rollout Worker** handles two types of requests: (1) The *generate* + request generates responses given prompts. (2) The *update_weights* request interrupts + all ongoing generations and loads parameters of new versions. Upon the interruption, + the rollout workers discard KV caches computed by old weights, and re-compute them + using the new weights. Afterwards, the rollout workers continue to decode the + unfinished sequences until the next interruption or termination. We emphasize that + such interruptions and in-flight weight updates would result in trajectories composed + of segments produced by different model versions. This introduces a novel algorithmic + challenge, which will be addressed in the next section. + +- **Reward Service** evaluates the accuracy of the responses generated by the model. For + example, in the coding task, this service extracts the code and executes unit tests to + verify its accuracy. + +- **Trainer Workers** continuously sample from the replay buffer, accumulating data + until reaching the configured training batch size. They then perform PPO updates and + store the resulting parameters in distributed storage. To ensure data freshness, data + from the replay buffer is used only once. + +- **Rollout Controller** serves as a critical bridge between the rollout workers, reward + service, and the model workers. During the training process, it reads data from the + dataset and invokes the rollout worker's *generate* request. The received response is + then sent to the reward service to obtain the reward. The trajectory, along with the + reward, is stored in the replay buffer, waiting to be trained by the model worker. + After the model worker updates the parameters, the controller calls the rollout + worker's *update_weight*. We illustrate the generation and training management in + Fig.4. This asynchronous pipeline ensures continuous full utilization of both + generation and training resources. ![](/assets/async_timeline.png) @@ -82,49 +125,80 @@ We design a system that fully decouples generation and training across separate ### Challenges -While the asynchronous system design offers significant acceleration through improved device utilization, it introduces several technical challenges that require algorithmic considerations. +While the asynchronous system design offers significant acceleration through improved +device utilization, it introduces several technical challenges that require algorithmic +considerations. -- **Challenge 1: Data Staleness** - Due to the asynchronous nature of AREAL, each training batch contains data from multiple prior policy versions. Data staleness would lead to a distribution gap between the training data and the latest model. In asynchronous RL training for LRMs, this issue could be even more severe for long trajectories due to extended decoding time. +- **Challenge 1: Data Staleness** - Due to the asynchronous nature of AREAL, each + training batch contains data from multiple prior policy versions. Data staleness would + lead to a distribution gap between the training data and the latest model. In + asynchronous RL training for LRMs, this issue could be even more severe for long + trajectories due to extended decoding time. -- **Challenge 2: Inconsistent Policy Versions** - As shown in Fig.4, a single generated trajectory may involve segments produced by different policy versions. This inconsistency fundamentally violates the formulation of standard PPO that assumes all actions being generated by a single policy. +- **Challenge 2: Inconsistent Policy Versions** - As shown in Fig.4, a single generated + trajectory may involve segments produced by different policy versions. This + inconsistency fundamentally violates the formulation of standard PPO that assumes all + actions being generated by a single policy. ### Solutions To overcome these two challenges, we propose two solutions: -- **Solution 1: Staleness-Aware Training** - To avoid negative impact of the data staleness issue, we constrain the version discrepancy between the policy version of generated trajectories and the training policy. We introduce a hyperparameter η representing the maximum permitted staleness. When η = 0, the system degenerates to the synchronous RL setting where generation exactly matches training batches. When η = 1, the system recovers to the previous one-step overlap methods. +- **Solution 1: Staleness-Aware Training** - To avoid negative impact of the data + staleness issue, we constrain the version discrepancy between the policy version of + generated trajectories and the training policy. We introduce a hyperparameter η + representing the maximum permitted staleness. When η = 0, the system degenerates to + the synchronous RL setting where generation exactly matches training batches. When η = + 1, the system recovers to the previous one-step overlap methods. -- **Solution 2: Decoupled PPO Objective** - We apply a decoupled PPO objective that disentangles the behavior policy and the proximal policy. The behavior policy represents the policy used for sampling trajectories and the proxy policy is a proximal policy serving as a recent target to regularize the update of online policy. +- **Solution 2: Decoupled PPO Objective** - We apply a decoupled PPO objective that + disentangles the behavior policy and the proximal policy. The behavior policy + represents the policy used for sampling trajectories and the proxy policy is a + proximal policy serving as a recent target to regularize the update of online policy. ![](/assets/decoupled_ppo_obj.png) ## Validating Asynchronous AReaL -We first evaluate Asynchronous AReaL on a math task to validate our system and algorithmic choices. Building on our previous release, AReaL-boba, we adopt R1-Distill-Qwen as the base model and [AReaL-boba-106k](https://huggingface.co/datasets/inclusionAI/AReaL-boba-Data/blob/main/AReaL-boba-106k.jsonl) as the training dataset. +We first evaluate Asynchronous AReaL on a math task to validate our system and +algorithmic choices. Building on our previous release, AReaL-boba, we adopt +R1-Distill-Qwen as the base model and +[AReaL-boba-106k](https://huggingface.co/datasets/inclusionAI/AReaL-boba-Data/blob/main/AReaL-boba-106k.jsonl) +as the training dataset. ### End-to-End Performance Comparison -We compared synchronous and asynchronous training on 1.5B and 7B models. Under equal resource constraints and training steps, the asynchronous system was over twice as fast as the synchronous one. Evaluations on AIME24 confirmed that this acceleration does not compromise performance. - -| **Model (1.5B)** | **AIME24** | **# Nodes** | **PPO Steps** | **Training Hours** | -|:---:|:---:|:---:|:---:|:---:| -| basemodel | 29.3 | - | - | - | -| Synchronous AReaL | 42.0 | 16 | 1000 | 41.0 | -| Asynchronous AReaL | 42.2 | 16 | 1000 | **14.8** | - -| **Model (7B)** | **AIME24** | **# Nodes** | **PPO Steps** | **Training Hours** | -|:---:|:---:|:---:|:---:|:---:| -| basemodel | 54.3 | - | - | - | -| Synchronous AReaL | 63.0 | 24 | 1000 | 57.7 | -| Asynchronous AReaL | 63.1 | 24 | 1000 | **25.4** | - -*Table 2: End-to-end comparison between asynchronous AReaL and synchronous AReaL. Following last release AReaL-boba, we adopt R1-Distill-Qwen as the basemodel and [AReaL-boba-106k](https://huggingface.co/datasets/inclusionAI/AReaL-boba-Data/blob/main/AReaL-boba-106k.jsonl) as the training data. Each performance number represents the average results of 32 sampling responses.* +We compared synchronous and asynchronous training on 1.5B and 7B models. Under equal +resource constraints and training steps, the asynchronous system was over twice as fast +as the synchronous one. Evaluations on AIME24 confirmed that this acceleration does not +compromise performance. + +| **Model (1.5B)** | **AIME24** | **# Nodes** | **PPO Steps** | **Training Hours** | +| :----------------: | :--------: | :---------: | :-----------: | :----------------: | +| basemodel | 29.3 | - | - | - | +| Synchronous AReaL | 42.0 | 16 | 1000 | 41.0 | +| Asynchronous AReaL | 42.2 | 16 | 1000 | **14.8** | + +| **Model (7B)** | **AIME24** | **# Nodes** | **PPO Steps** | **Training Hours** | +| :----------------: | :--------: | :---------: | :-----------: | :----------------: | +| basemodel | 54.3 | - | - | - | +| Synchronous AReaL | 63.0 | 24 | 1000 | 57.7 | +| Asynchronous AReaL | 63.1 | 24 | 1000 | **25.4** | + +*Table 2: End-to-end comparison between asynchronous AReaL and synchronous AReaL. +Following last release AReaL-boba, we adopt R1-Distill-Qwen as the basemodel and +[AReaL-boba-106k](https://huggingface.co/datasets/inclusionAI/AReaL-boba-Data/blob/main/AReaL-boba-106k.jsonl) +as the training data. Each performance number represents the average results of 32 +sampling responses.* ### Ablation Study #### System Ablations -We ablate interruptible generation and present the resulting generation throughput in Fig.5. Without interruptible generation, the controller must wait for the longest response. In particular, interruptible generation leads to a 12% and 17% throughput increase for 1.5B and 7B models respectively on 4 nodes. +We ablate interruptible generation and present the resulting generation throughput in +Fig.5. Without interruptible generation, the controller must wait for the longest +response. In particular, interruptible generation leads to a 12% and 17% throughput +increase for 1.5B and 7B models respectively on 4 nodes. ![](/assets/interrupt_gen_ablation.png) @@ -132,58 +206,92 @@ We ablate interruptible generation and present the resulting generation throughp #### Algorithm Ablations -We vary the maximum allowed staleness η and compare configurations with and without the decoupled PPO objective. Fig.6 shows the learning curves after 1000 PPO updates, demonstrating that naive PPO fails to match the performance of the synchronous RL oracle (i.e., the performance when η = 0). Even slight staleness can significantly degrade final performance due to the improper clipping center and policy changes during interruptible generation. Furthermore, increasing data staleness consistently degrades learning performance. +We vary the maximum allowed staleness η and compare configurations with and without the +decoupled PPO objective. Fig.6 shows the learning curves after 1000 PPO updates, +demonstrating that naive PPO fails to match the performance of the synchronous RL oracle +(i.e., the performance when η = 0). Even slight staleness can significantly degrade +final performance due to the improper clipping center and policy changes during +interruptible generation. Furthermore, increasing data staleness consistently degrades +learning performance. ![](/assets/algo_ablation.png) -*Fig.6 Ablation Study on Decoupled PPO Objective with DeepSeek-R1-Distill-Qwen-1.5B. Left: Learning curves with naive PPO. Right: Learning curves with decoupled PPO objective.* - -| η | **AIME 24** | | **AIME 25** | | -|:---:|:---:|:---:|:---:|:---:| -| | naive PPO | + Decoupled Objective | naive PPO | + Decoupled Objective | -| 0 | 42.0 | | 32.9 | | -| 1 | 41.8 | 42.1 | 30.7 | 31.9 | -| 2 | 40.0 | 41.8 | 32.1 | 32.5 | -| 4 | 23.3 | 42.2 | 23.1 | 32.0 | -| 8 | 35.7 | 41.0 | 27.8 | 31.1 | -| 16 | 35.8 | 38.7 | 26.2 | 32.5 | -| ∞ | 34.0 | 36.9 | 26.9 | 29.9 | +*Fig.6 Ablation Study on Decoupled PPO Objective with DeepSeek-R1-Distill-Qwen-1.5B. +Left: Learning curves with naive PPO. Right: Learning curves with decoupled PPO +objective.* -*Table 3: Evaluation scores when varying data staleness, comparing performance with and without the decoupled objective.* +| η | **AIME 24** | | **AIME 25** | | +| :-: | :---------: | :-------------------: | :---------: | :-------------------: | +| | naive PPO | + Decoupled Objective | naive PPO | + Decoupled Objective | +| 0 | 42.0 | | 32.9 | | +| 1 | 41.8 | 42.1 | 30.7 | 31.9 | +| 2 | 40.0 | 41.8 | 32.1 | 32.5 | +| 4 | 23.3 | 42.2 | 23.1 | 32.0 | +| 8 | 35.7 | 41.0 | 27.8 | 31.1 | +| 16 | 35.8 | 38.7 | 26.2 | 32.5 | +| ∞ | 34.0 | 36.9 | 26.9 | 29.9 | +*Table 3: Evaluation scores when varying data staleness, comparing performance with and +without the decoupled objective.* ```{note} In practice, the effect of staleness could depend on various factors, such as learning rate, batch size, the number of mini batches, generation length, and also the RL algorithm. Here we aim to illustrate the impact of staleness on the training performance of a synchronous RL system, and show that stalenss control and decoupled PPO could well mitigate the negative impact of staleness. Here we follow a large batch size setting with `batch_size=512`, `n_rollouts=16`, `n_mini_batch=4`, `max_new_tokens=8192`, and `lr=2e-5`. The staleness is defined as the gap of model versions, where the model version increases by 1 when a batch of `batch_size` X `n_rollouts` samples are trained on. ``` -However, the decoupled PPO objective substantially improves training stability when handling stale data. Notably, even with the decoupled objective, unbounded staleness (maximum staleness → ∞) still results in inferior performance compared to the zero-staleness oracle. When properly constrained, moderate staleness (e.g., η ≤ 4) has minimal impact on final performance while significantly accelerating training through the asynchronous pipeline, as demonstrated in Tab.3 and Fig.7. +However, the decoupled PPO objective substantially improves training stability when +handling stale data. Notably, even with the decoupled objective, unbounded staleness +(maximum staleness → ∞) still results in inferior performance compared to the +zero-staleness oracle. When properly constrained, moderate staleness (e.g., η ≤ 4) has +minimal impact on final performance while significantly accelerating training through +the asynchronous pipeline, as demonstrated in Tab.3 and Fig.7. ![](/assets/staleness_throughput.png) -*Fig.7 The relationship between η and training throughput. Larger η leads to higher throughput.* +*Fig.7 The relationship between η and training throughput. Larger η leads to higher +throughput.* ## Training Recipe with Asynchronous AReaL ### Dataset Curation -We train the 7B and 14B model using open-source data from [DeepCoder](https://pretty-radio-b75.notion.site/DeepCoder-A-Fully-Open-Source-14B-Coder-at-O3-mini-Level-1cf81902c14680b3bee5eb349a512a51). After deduplication and quality filtering, approximately 7,600 coding problems remained. Additionally, we have around 10,000 internal data entries that can further improve the model's performance. We are actively working on open-sourcing this portion of the data. +We train the 7B and 14B model using open-source data from +[DeepCoder](https://pretty-radio-b75.notion.site/DeepCoder-A-Fully-Open-Source-14B-Coder-at-O3-mini-Level-1cf81902c14680b3bee5eb349a512a51). +After deduplication and quality filtering, approximately 7,600 coding problems remained. +Additionally, we have around 10,000 internal data entries that can further improve the +model's performance. We are actively working on open-sourcing this portion of the data. ### Rollout Strategy -During the rollout phase, the LLM generates 16 responses per question. To minimize output truncation, we set the maximum generation length to 27K tokens. +During the rollout phase, the LLM generates 16 responses per question. To minimize +output truncation, we set the maximum generation length to 27K tokens. ### Filtering Strategy -In the design of our asynchronous system, we can naturally incorporate strategies to filter some rollouts. Specifically, we exclude the questions where all 16 answers are either entirely correct or entirely incorrect. +In the design of our asynchronous system, we can naturally incorporate strategies to +filter some rollouts. Specifically, we exclude the questions where all 16 answers are +either entirely correct or entirely incorrect. ### Key Hyperparameters -| **Parameter** | **Value** | -|:---:|:---:| -| PPO Minibatches | 1 | -| Learning Rate | 2e-5 | -| Adam ε | 1e-5 | -| Batch Size | 2,048 | -| max_head_offpolicyness η | 16 | -| success_rate_ub | 0.95 | -| success_rate_lb | 0.05 | \ No newline at end of file +| **Parameter** | **Value** | +| :----------------------: | :-------: | +| PPO Minibatches | 1 | +| Learning Rate | 2e-5 | +| Adam ε | 1e-5 | +| Batch Size | 2,048 | +| max_head_offpolicyness η | 16 | +| success_rate_ub | 0.95 | +| success_rate_lb | 0.05 | + +## Resources + +- [boba² Code dataset](https://huggingface.co/datasets/inclusionAI/AReaL-boba-2-RL-Code) +- **Reproduce boba² Code Models** + - 🤗 **Model weights**: [8B-code](https://huggingface.co/inclusionAI/AReaL-boba-2-8B), + [14B-code](https://huggingface.co/inclusionAI/AReaL-boba-2-14B), + [8B-code-open](https://huggingface.co/inclusionAI/AReaL-boba-2-8B-subset), + [14B-code-open](https://huggingface.co/inclusionAI/AReaL-boba-2-14B-subset) + - [Evaluation Guide](https://inclusionai.github.io/AReaL/tutorial/eval.html) + - [Training configs](https://github.com/inclusionAI/AReaL/tree/main/examples/configs/v0.3-qwen3-code) + and [instructions](https://inclusionai.github.io/AReaL/references/reproduce.html) +- [Scripts for Benchmark Training Throughput](https://github.com/inclusionAI/AReaL/tree/main/benchmark/verl_v0_3_0_post1_76084d3) diff --git a/docs/_toc.yml b/docs/_toc.yml index a705494286..eaa524219c 100644 --- a/docs/_toc.yml +++ b/docs/_toc.yml @@ -11,7 +11,7 @@ parts: - file: tutorial/quickstart_legacy - file: tutorial/eval - file: tutorial/troubleshooting - - caption: Getting Started with AReaL-Lite + - caption: Getting Started with AReaL-lite chapters: - file: lite/gsm8k_grpo - caption: References diff --git a/docs/customization/agent.md b/docs/customization/agent.md index 8ee9e25708..8a96bbe955 100644 --- a/docs/customization/agent.md +++ b/docs/customization/agent.md @@ -1,17 +1,19 @@ # Rollout and Agentic RL This guide shows you how to create custom rollout behaviors for RL training by building -a multi-turn math agent with AReaL. This agent keeps trying to solve math problems until -it finds the correct answer. +a multi-turn math agent with **AReaL-lite**. This agent keeps trying to solve math +problems until it finds the correct answer. You can find the complete implementation in `areal/workflow/multi_turn.py`. ## Step 1: Define Your Workflow -AReaL gives you flexibility in how you design your agents. Instead of rigid `Agent` -classes that might constrain your agent's capabilities, AReaL captures all rollout -behavior in a `RolloutWorkflow` class. This approach lets you customize your agent's -behavior however you need. +AReaL-lite gives you flexibility in how you design your agents to run **an episode**. +**An episode** defines how your agent rollouts a complete training sample from an input +prompt, using tools, reward functions, and (multi-turn) generation. Instead of rigid +`Agent` classes that might constrain your agent's capabilities, AReaL-lite captures all +rollout behavior in a `RolloutWorkflow` class. This approach allows you to customize +your agent's behavior however you need. ```python # areal/api/workflow_api.py @@ -40,7 +42,7 @@ interact. > generated from that prompt—it's not batched. However, you can generate multiple > trajectories from a single prompt (for example, with GRPO or tree search). -### Setting Up the Multi-Turn Math Workflow +### Setting Up the Multi-turn Math Workflow Let's build a multi-turn rollout workflow for solving math problems. First, we'll define the `__init__` method to set up what we need during rollout: @@ -77,15 +79,16 @@ and converting it into an `LLMRequest` object for the inference engine: class MultiTurnWorkflow(RolloutWorkflow): # ... __init__ method above ... - async def arun_episode(self, engine: InferenceEngine, data): + async def arun_episode(self, engine: InferenceEngine, data) -> TensorDict: # Initialize result containers seq, logprobs, loss_mask, versions = [], [], [], [] messages = data["messages"] # Run multi-turn rollout until we get the correct answer - t = reward = 0 + turn_index = 0 + reward = 0 discount = 1.0 rid = uuid.uuid4().hex - while reward == 0 and t < self.max_turns: + while reward == 0 and turn_index < self.max_turns: # Convert the conversation into input tokens input_ids = self.tokenizer.apply_chat_template( messages, @@ -111,7 +114,7 @@ class MultiTurnWorkflow(RolloutWorkflow): > **Note**: The `rid` field in `LLMRequest` is the request ID. Requests with the same ID > will reuse the LLM inference server's KV caches for better efficiency. -### Handling Multi-Turn Conversations +### Handling Multi-turn Conversations Next, we'll check if the current answer is correct using our `reward_fn`. This function should return 1 for correct answers and 0 otherwise. When the answer is wrong, we'll @@ -121,7 +124,7 @@ apply a discount, add feedback to the conversation, and let the model try again: class MultiTurnWorkflow(RolloutWorkflow): # ... previous methods ... - async def arun_episode(self, engine: InferenceEngine, data): + async def arun_episode(self, engine: InferenceEngine, data) -> TensorDict: # ... initialization code ... while reward == 0 and t < self.max_turns: # Add feedback if the previous answer was incorrect @@ -192,7 +195,7 @@ Finally, let's complete the implementation by collecting trajectories in the class MultiTurnWorkflow(RolloutWorkflow): # ... previous methods ... - async def arun_episode(self, engine: InferenceEngine, data): + async def arun_episode(self, engine: InferenceEngine, data) -> TensorDict: # ... episode logic above ... while reward == 0 and t < self.max_turns: @@ -219,9 +222,9 @@ class MultiTurnWorkflow(RolloutWorkflow): ``` > **Important**: The returned `TensorDict` must follow HuggingFace's padded data format, -> where each tensor has shape `[batch_size, sequence_length, *]`. This allows AReaL to -> automatically batch multiple trajectories for training. Since this example returns a -> single trajectory, we use `unsqueeze(0)` to create a batch of size 1. +> where each tensor has shape `[batch_size, sequence_length, *]`. This allows AReaL-lite +> to automatically batch multiple trajectories for training. Since this example returns +> a single trajectory, we use `unsqueeze(0)` to create a batch of size 1. > **Note**: You're not restricted to specific keys in your `TensorDict`—different > algorithms need different keys. This example targets the GRPO algorithm, so we include @@ -247,18 +250,13 @@ def main(args): ) # Run training—no other changes needed! - data_generator = iter(train_dataloader) + data_generator = itertools.cycle(train_dataloader) for global_step in range(max_steps): with stats_tracker.record_timing("rollout"): if config.async_training: batch = rollout.prepare_batch(train_dataloader, workflow=workflow) else: - try: - data = next(data_generator) - except StopIteration: - data_generator = iter(train_dataloader) - data = next(data_generator) - batch = rollout.rollout_batch(data, workflow=workflow) + batch = rollout.rollout_batch(next(data_generator), workflow=workflow) # ... continue with training loop ... ``` diff --git a/docs/customization/algorithm.md b/docs/customization/algorithm.md index 139d03330b..b15a3fd168 100644 --- a/docs/customization/algorithm.md +++ b/docs/customization/algorithm.md @@ -3,7 +3,7 @@ > **Note**: We recommend the user to first read the > [agent customization guide](agent.md). -AReaL structures RL algorithms around two core components: +**AReaL-lite** structures RL algorithms around two core components: - **RolloutWorkflow**: Defines what data to generate during rollouts - **TrainEngine**: Defines how to process the generated data for training @@ -108,7 +108,7 @@ def reinforce_loss_fn(logits, data): ``` ```{note} -To decrease memory usage, AReaL automatically packs multiple sequences in an 1D tensor before forward passes. Hence, the loss function should assume handling 1D *packed* tensors instead of *padded* tensors. +To decrease memory usage, AReaL-lite automatically packs multiple sequences in an 1D tensor before forward passes. Hence, the loss function should assume handling 1D *packed* tensors instead of *padded* tensors. ``` Next, we implement the training engine. We use a two-class design to maintain backend diff --git a/docs/customization/dataset.md b/docs/customization/dataset.md index 84f2d540c2..5e0d066d84 100644 --- a/docs/customization/dataset.md +++ b/docs/customization/dataset.md @@ -1,8 +1,8 @@ # Dataset -AReaL directly integrates with the `Dataset` class from the HuggingFace `datasets` -package. This gives you full flexibility to load, process, and filter your data before -training. +**AReaL-lite** directly integrates with the `Dataset` class from the HuggingFace +`datasets` package. This gives you full flexibility to load, process, and filter your +data before training. The required columns in your dataset depend on the specific implementation of the `RolloutWorkflow` (for online reinforcement learning) or the training engines (for diff --git a/docs/legacy/customization/agent.md b/docs/legacy/customization/agent.md index e54eb89404..1e46228042 100644 --- a/docs/legacy/customization/agent.md +++ b/docs/legacy/customization/agent.md @@ -1,7 +1,7 @@ # Rollout and Agentic RL (Legacy) -> **Note**: While this legacy approach works, we strongly recommend using the new AReaL -> for new projects. It provides better flexibility, cleaner abstractions, and easier +> **Note**: While this legacy approach works, we strongly recommend using AReaL-lite for +> new projects. It provides better flexibility, cleaner abstractions, and easier > maintenance. ## Step 1: Define Your Agent Class diff --git a/docs/legacy/customization/algorithm.md b/docs/legacy/customization/algorithm.md index f8ebfb7b90..1bf0768772 100644 --- a/docs/legacy/customization/algorithm.md +++ b/docs/legacy/customization/algorithm.md @@ -1,6 +1,6 @@ # Training Algorithm (Legacy) -> **Note**: The AReaL's new approach is more recommended for new implementations due to +> **Note**: The AReaL-lite approach is more recommended for new implementations due to > its cleaner separation of concerns and better maintainability. The legacy approach encapsulates algorithms in a `ModelInterface` with three core diff --git a/docs/legacy/customization/dataset.md b/docs/legacy/customization/dataset.md index 4ab8849d2c..36d20f35b1 100644 --- a/docs/legacy/customization/dataset.md +++ b/docs/legacy/customization/dataset.md @@ -1,7 +1,7 @@ # Dataset (Legacy) -> **Note**: While this legacy approach works, we strongly recommend using the new AReaL -> for new projects. It provides better flexibility, cleaner abstractions, and easier +> **Note**: While this legacy approach works, we strongly recommend using AReaL-lite for +> new projects. It provides better flexibility, cleaner abstractions, and easier > maintenance. ## Define Your Dataset diff --git a/docs/lite/gsm8k_grpo.md b/docs/lite/gsm8k_grpo.md index 9e47282486..e41db2d1ce 100644 --- a/docs/lite/gsm8k_grpo.md +++ b/docs/lite/gsm8k_grpo.md @@ -1,16 +1,17 @@ # Running GRPO on GSM8K Dataset -This guide introduces how AReaL-Lite runs the GRPO algorithm on the GSM8K dataset, using -the training script [examples/lite/gsm8k_grpo.py](../../examples/lite/gsm8k_grpo.py) and -configuration file -[examples/lite/configs/gsm8k_grpo.yaml](../../examples/lite/configs/gsm8k_grpo.yaml). +This guide introduces how AReaL-lite runs the GRPO algorithm on the GSM8K dataset, using +the training script +[examples/lite/gsm8k_grpo.py](https://github.com/inclusionAI/AReaL/blob/main/examples/lite/gsm8k_grpo.py) +and configuration file +[examples/lite/configs/gsm8k_grpo.yaml](https://github.com/inclusionAI/AReaL/blob/main/examples/lite/configs/gsm8k_grpo.yaml). -## How AReaL-Lite Works +## How AReaL-lite Works The following figure illustrates the launching and one asynchronous training step of the -GRPO algorithm on the GSM8K dataset on AReaL-Lite. Compared with the old AReaL -implementation, AReaL-Lite runs inference servers and a SPMD training script instead of -a bunch of various workers. In a training step, AReaL-Lite: +GRPO algorithm on the GSM8K dataset on AReaL-lite. Compared with the old AReaL +implementation, AReaL-lite runs inference servers and a SPMD training script instead of +a bunch of various workers. In a training step, AReaL-lite: 1. Submits prompts from the dataset to `RemoteSGLangEngine`, who runs `RLVRWorkflow` in a streaming manner. @@ -21,15 +22,15 @@ a bunch of various workers. In a training step, AReaL-Lite: 1. Computes losses and update weights in `FSDPPPOActor`. 1. Transfers the updated weights to remote `SGLangServer` instances. -![AReaL-Lite-gsm8k-example](gsm8k_grpo.png) +![AReaL-lite-gsm8k-example](gsm8k_grpo.png) In the following sections, we will walk you through the code to explain concepts and show you how these steps are done in details. ## Launching the Experiment -As shown in [Quickstart Guide](../tutorial/quickstart.md), experiments in AReaL-Lite are -launched using standalone launchers with the following commands: +As shown in the [quickstart guide](../tutorial/quickstart.md), experiments in AReaL-lite +are launched using standalone launchers with the following commands: ``` # Local Launcher @@ -40,7 +41,7 @@ python -m areal.launcher.ray --config --config ``` -In AReaL-Lite: +In AReaL-lite: - The **training script** is an SPMD python script that serves as the experiment entry point. @@ -48,20 +49,22 @@ In AReaL-Lite: `LocalLauncher`, `ray.remote` for `RayLauncher`, `srun` for `SlurmLauncher`). - The launcher also manages inference servers (currently only supporting `SGLangServer`). The number and parallelization strategies (e.g. tensor parallel) are - determined by the option [allocation_mode](../../areal/api/cli_args.py#L797). + determined by the option + [allocation_mode](https://github.com/inclusionAI/AReaL/blob/main/areal/api/cli_args.py#L797). - For distributed launchers (`RayLauncher` and `SlurmLauncher`), inference servers run with a wrapper - [areal/launcher/sglang_server.py](../../areal/launcher/sglang_server.py) to handle - addresses and ports in distributed settings. + [areal/launcher/sglang_server.py](https://github.com/inclusionAI/AReaL/blob/main/areal/launcher/sglang_server.py) + to handle addresses and ports in distributed settings. - After `SGLangServer` instances are started, launchers collect their addresses and ports to set the `AREAL_LLM_SERVER_ADDRS` environment variable for training scripts to access these inference servers. The **configuration file** is a YAML file that sets the options provided in -[areal/api/cli_args.py](../../areal/api/cli_args.py). It could be modified via CLI -arguments such as `actor.path=Qwen/Qwen3-1.7B` and `+sglang.attention_backend=triton`. -The training scripts parse the config with CLI arguments into the config class defined -in [areal/api/cli_args.py](../../areal/api/cli_args.py). +[areal/api/cli_args.py](https://github.com/inclusionAI/AReaL/blob/main/areal/api/cli_args.py). +It could be modified via CLI arguments such as `actor.path=Qwen/Qwen3-1.7B` and +`+sglang.attention_backend=triton`. The training scripts parse the config with CLI +arguments into the config class defined in +[areal/api/cli_args.py](https://github.com/inclusionAI/AReaL/blob/main/areal/api/cli_args.py). ``` config, _ = load_expr_config(args, GRPOConfig) @@ -71,7 +74,7 @@ config: GRPOConfig ## Loading and Preprocessing Dataset We use the `datasets` and `torchdata` packages to load and preprocess the dataset into -our dataloader. First, we download `openai/gsm8k` from Huggingface and split it by data +our dataloader. First, we download `openai/gsm8k` from Hugging Face and split it by data parallel ranks, then map it to our desired format: ```python @@ -108,12 +111,106 @@ details. ## Rollout -The data lifecycle is controlled by an `RLVRWorkflow`, which defines how data progresses -from prompts to complete rollout data containing all fields required for training. Our -example shows a single-turn RLVR workflow with a math reward function. The core logic of -the workflow is implemented in an async method `arun_episode`, which takes a prompt, -generate answers with `RemoteSGLangEngine`, computes rewards, and populates additional -fields to produce finalized training data. +### Inference Engine: `RemoteSGLangEngine` + +In AReaL-lite, generation tasks are offloaded to remote inference servers, which operate +on separate GPUs from those used for training. The `RemoteSGLangEngine` acts as a client +that interacts with the servers. `RemoteSGLangEngine` runs in a SPMD manner on every +training process, without occupying any GPUs. + +`RemoteSGLangEngine` provides two core APIs that access the remote servers, `agenerate` +and `update_weights_async`. It is worth mentioning that, in asynchronous RL experiment +in AReaL-lite, inference-side weight update could happen **in the middle of** generation +of one prompt. With that being said, one output sequence could be generated by multiple +versions of models. Let us glimpse into code of `agenerate` and `update_weights_async` +for a better understanding. + +In `update_weights_async`, the engine first send `pause_generation` requests to all +inference servers, notifying them a weight update is about to happen. Upon receiveing +`pause_generation`, inference servers will immediately stop generating and respond with +already generated tokens. Then, the engine sends `update_weights_from_distributed` (for +NCCL update) or `update_weights_from_disk` (for disk update). After the update is +finished, the engine sends `continue_generation` to inference server telling them to +start working again. + +```python +class RemoteSGLangEngine: + ... + def update_weights_async(self, meta: WeightUpdateMeta): + # `update_weights_async` is completely async. + # It submits task to a ProcessPoolExecutor and returns a future + for addr in self.addresses: + res = requests.post(f"http://{addr}/pause_generation") + if meta.type == "nccl": + future = self.executor.submit( + # a function that send `update_weights_from_distributed` request + update_weights_from_distributed, + ) + elif meta.type == "disk": + ... + + def callback(future): + for addr in self.addresses + requests.post(f"http://{addr}/continue_generation") + + future.add_done_callback(callback) + return future +``` + +`agenerate` takes an `LLMRequest` with `input_ids` of **a single prompt** and generation +hyperparameters, and returns the final generation result, an `LLMResponse` with +`output_tokens` and other outputs. Since the generation could be interrupted, +`agenerate` iteratively prepares payload, sends requests and receives responses until +the generation finishes. + +```python +class RemoteSGLangEngine: + ... + async def agenerate(self, req: LLMRequest): + payload = ... # prepare payload for request + # If request is from the same workflow, choose old server + # to allow KVCache reuse. Otherwise choose server in a round + # robin manner. + server_addr = self.choose_server(req) + stop_reason = None + # other outputs are omitted for simplicity + output_tokens = [] + while (stop_reason != "stop" and len(output_tokens) < max_new_tokens): + # Request is interrupted, wait to avoid contention + if stop_reason is not None: + await asyncio.sleep(0.5) + # send request to remote sever + result = await arequest_with_retry( + addr=server_addr, + endpoint="/generate", + payload=payload, + method="POST" + ) + output_tokens.extend(result["output_ids"]) + # prepare payload for the next request + payload["input_ids"] += results["output_ids"] + payload["sample_params"]["max_new_tokens"] -= len(results["output_ids"]) + return LLMResponse( + input_tokens=req.input_ids, + output_tokens=output_tokens, + ... + ) + +``` + +The `InferenceEngine` class is designed to be extensible, supporting not just SGLang but +also other backends like vLLM. While different inference engines may be used, the +rollout management logic remains consistent. This common functionality is abstracted +into the `WorkflowExecutor`, which will be introduced in the following section. + +### `RLVRWorkflow` and `WorkflowExecutor` + +The rollout data lifecycle is controlled by an `RLVRWorkflow`, which defines how data +progresses from prompts to complete rollout data containing all fields required for +training. Our example shows a single-turn RLVR workflow with a math reward function. The +core logic of the workflow is implemented in an async method `arun_episode`, which takes +a prompt, generate answers with `RemoteSGLangEngine`, computes rewards, and populates +additional fields to produce finalized training data. ```python class RLVRWorkflow(RolloutWorkflow): @@ -157,26 +254,22 @@ workflow = RLVRWorkflow( ) ``` -In AReaL-Lite, generation tasks are offloaded to remote inference servers, which operate -on separate GPUs from those used for training. The `RemoteSGLangEngine` acts as a client -that interacts with the servers. `RemoteSGLangEngine` runs in a SPMD manner on every -training process, without occupying any GPUs. - -`RemoteSGLangEngine` is responsible for managing the data streaming through rollout +`WorkflowExecutor` is responsible for managing the data streaming through rollout workflows, and collates completed rollout data into batched training samples. When initializing, it launches a rollout thread that runs rollout workflows as `asyncio` tasks. The following code shows the simplified version of rollout thread implementation, which iteratively: - Checks available capacity. The capacity controls current number of rollout workflows - to limit concurrency and data off-policyness. + to limit concurrency and **data off-policyness** (The difference between the model + version used by generation and the model version updated by the trainer). - If there is capacity left and rollout is not paused for weight update, continuously obtains data from `input_queue` and creates `asyncio` tasks to run the workflows. - Waits for rollout workflows to finish. - Gathers data from finished workflows and puts them into `output_queue` ```python -class RemoteSGLangEngine(InferenceEngine): +class WorkflowExecutor: ... async def _rollout_thread_async(self): rid = 0 @@ -201,15 +294,15 @@ class RemoteSGLangEngine(InferenceEngine): rid += 1 # Wait for rollout completion tasks = list(rollout_tasks.values()) - done = [] + completed_tasks = [] if tasks: - done, _ = await asyncio.wait( + completed_tasks, _ = await asyncio.wait( tasks, timeout=ROLLOUT_POLL_WAIT_TIME, return_when=asyncio.FIRST_COMPLETED, ) # Collect done results, put the results into output queue - for task in done: + for task in completed_tasks: traj = await task task_rid = task.get_name() rollout_tasks.pop(task_rid) @@ -252,14 +345,24 @@ def prepare_batch( pass ``` +The `RemoteSGLangEngine` exposes `rollout_batch` and `prepare_batch` by calling them in +the workflow executor: + +```python +class RemoteSGLangEngine(InferenceEngine): + ... + def prepare_batch(self, *args, **kwargs): + return self.workflow_executor.prepare_batch(*args, **kwargs) +``` + The usage of `RemoteSGLangEngine` in the training script is simple: ```python -rollout = RemoteSGLangEngine(config.rollout) +rollout = RemoteSGLangEngine(config.inf_engine) rollout.initialize() eval_rollout = ... -data_generator = iterools.cycle(train_dataloader) +data_generator = itertools.cycle(train_dataloader) for global_step in range(max_steps): # rollout batched training data for current step if config.async_training: @@ -308,7 +411,8 @@ actor.step_lr_scheduler() ``` If you want to customize your own training algorithm, see -[Customize algorithms](../customization/algorithm.md) for more details. +[Customize algorithms](https://inclusionai.github.io/AReaL/customization/algorithm.html) +for more details. ## Transferring Weights to Inference Servers @@ -338,30 +442,29 @@ weight_update_meta = WeightUpdateMeta.from_disk(config.saver) ``` After a training step is finished, we transfer new weights from actor engine to remote -inference servers with steps shown in the following code: +inference servers: + +1. The rollout engine needs to stop sending generation requests to remote servers + (`rollout.pause()`) before weight update to avoid server-side congestion. +1. Since we need to invoke weight update on the trainer engine and remote inference + servers at the same time, in the training script, we asynchronously send requests to + remote inference servers, and then immediately upload weights on the trainer engine. ```python -# 1. Pause rollout on remote inference servers rollout.pause() -# 2. Send requests to remote servers, tell them to update weights if dist.get_rank() == 0: - future = rollout.update_weights(weight_update_meta) -# 3. Actor begins to transfer weights + future = rollout.update_weights_async(weight_update_meta) actor.upload_weights(weight_update_meta) -# 4. Wait for remote servers to return after finishing updates if dist.get_rank() == 0: future.result() -# 5. Synchronize rollout processes for model version update dist.barrier(device_ids=[actor.device.index]) torch.cuda.synchronize() -# 6. Resume rollout on remote inference servers rollout.resume() -# 7. Set version, ensures versions on actor and rollout engine are identical actor.set_version(global_step + 1) rollout.set_version(global_step + 1) ``` -Now a complete GRPO training step in AReaL-Lite is done! The core logic of our example +Now a complete GRPO training step in AReaL-lite is done! The core logic of our example training script can be summarized as: ```python @@ -383,7 +486,7 @@ for global_step in range(max_steps): rollout.pause() if dist.get_rank() == 0: - future = rollout.update_weights(weight_update_meta) + future = rollout.update_weights_async(weight_update_meta) actor.upload_weights(weight_update_meta) if dist.get_rank() == 0: future.result() @@ -394,14 +497,16 @@ for global_step in range(max_steps): ## Utilities -In AReaL-Lite, we provide a wide range of utilities for basic functionalities required +In AReaL-lite, we provide a wide range of utilities for basic functionalities required for observing and tuning your experiments. ### `Saver` and `Evaluator` -`Saver` ([areal/utils/saver.py](../../areal/utils/saver.py)) and `Evaluator` -([areal/utils/evaluator.py](../../areal/utils/evaluator.py)) manage the frequency to -save and evaluate the model with the train engine. +`Saver` +([areal/utils/saver.py](https://github.com/inclusionAI/AReaL/blob/main/areal/utils/saver.py)) +and `Evaluator` +([areal/utils/evaluator.py](https://github.com/inclusionAI/AReaL/blob/main/areal/utils/evaluator.py)) +manage the frequency to save and evaluate the model with the train engine. In our example, we call `saver.save` and `evaluator.evaluate` after every training step. these two methods will automatically check if it is time to save or evaluate the model, @@ -409,7 +514,8 @@ according to the experiment configuration. ### `stats_tracker` -`stats_tracker` ([realhf/base/stats_tracker.py](../../realhf/base/stats_tracker.py)) +`stats_tracker` +([realhf/base/stats_tracker.py](https://github.com/inclusionAI/AReaL/blob/main/realhf/base/stats_tracker.py)) gathers training statistics across parallel ranks and reduce them. 1. **Scalar-type statistics** are recorded by `stats_tracker.scalar(key=value)` and will @@ -468,9 +574,10 @@ stats = stats_tracker.export() ### `StatsLogger` -`StatsLogger` ([areal/utils/stats_logger.py](../../areal/utils/stats_logger.py)) logs -gathered training data to recorders like `wandb` and `tensorboard` on rank 0. In our -example script, after finishing a training step, +`StatsLogger` +([areal/utils/stats_logger.py](https://github.com/inclusionAI/AReaL/blob/main/areal/utils/stats_logger.py)) +logs gathered training data to recorders like `wandb` and `tensorboard` on rank 0. In +our example script, after finishing a training step, `logger.commit(epoch, step, global_step, stats)` is called to record all statistics from `stats_tracker` to print them as well as log them into the recorders set by the configuration. diff --git a/docs/lite/gsm8k_grpo.png b/docs/lite/gsm8k_grpo.png index 2a89b47286..51e5bb7244 100644 Binary files a/docs/lite/gsm8k_grpo.png and b/docs/lite/gsm8k_grpo.png differ diff --git a/docs/tutorial/installation.md b/docs/tutorial/installation.md index ce563b5858..16e968584e 100644 --- a/docs/tutorial/installation.md +++ b/docs/tutorial/installation.md @@ -10,31 +10,36 @@ The following hardware configuration has been extensively tested: - **CPU**: 64 cores per node - **Memory**: 1TB per node - **Network**: NVSwitch + RoCE 3.2 Tbps -- **Storage**: +- **Storage**: - 1TB local storage for single-node experiments - 10TB shared storage (NAS) for distributed experiments ### Software Requirements -| Component | Version | -|---|:---:| -| Operating System | CentOS 7 / Ubuntu 22.04 or any system meeting the requirements below | -| NVIDIA Driver | 550.127.08 | -| CUDA | 12.8 | -| Git LFS | Required for downloading models, datasets, and AReaL code. See [installation guide](https://docs.github.com/en/repositories/working-with-files/managing-large-files/installing-git-large-file-storage) | -| Docker | 27.5.1 | -| NVIDIA Container Toolkit | See [installation guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) | -| AReaL Image | `ghcr.io/inclusionai/areal-runtime:v0.3.0.post1` (includes runtime dependencies and Ray components) | +| Component | Version | +| ------------------------ | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | +| Operating System | CentOS 7 / Ubuntu 22.04 or any system meeting the requirements below | +| NVIDIA Driver | 550.127.08 | +| CUDA | 12.8 | +| Git LFS | Required for downloading models, datasets, and AReaL code. See [installation guide](https://docs.github.com/en/repositories/working-with-files/managing-large-files/installing-git-large-file-storage) | +| Docker | 27.5.1 | +| NVIDIA Container Toolkit | See [installation guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) | +| AReaL Image | `ghcr.io/inclusionai/areal-runtime:v0.3.0.post2` (includes runtime dependencies and Ray components) | -**Note**: This tutorial does not cover the installation of NVIDIA Drivers, CUDA, or shared storage mounting, as these depend on your specific node configuration and system version. Please complete these installations independently. +**Note**: This tutorial does not cover the installation of NVIDIA Drivers, CUDA, or +shared storage mounting, as these depend on your specific node configuration and system +version. Please complete these installations independently. ## Runtime Environment -**For multi-node training**: Ensure a shared storage path is mounted on every node (and mounted to the container if you are using Docker). This path will be used to save checkpoints and logs. +**For multi-node training**: Ensure a shared storage path is mounted on every node (and +mounted to the container if you are using Docker). This path will be used to save +checkpoints and logs. ### Option 1: Docker (Recommended) -We recommend using Docker with our provided image. The Dockerfile is available in the top-level directory of the AReaL repository. +We recommend using Docker with our provided image. The Dockerfile is available in the +top-level directory of the AReaL repository. ```bash docker pull ghcr.io/inclusionai/areal-runtime:v0.3.0.post1 @@ -50,9 +55,10 @@ bash examples/env/scripts/setup-container-deps.sh ### Option 2: Custom Environment Installation -1. Install [Miniconda](https://www.anaconda.com/docs/getting-started/miniconda/install) or [Anaconda](https://www.anaconda.com/docs/getting-started/anaconda/install). +1. Install [Miniconda](https://www.anaconda.com/docs/getting-started/miniconda/install) + or [Anaconda](https://www.anaconda.com/docs/getting-started/anaconda/install). -2. Create a conda virtual environment: +1. Create a conda virtual environment: ```bash conda create -n areal python=3.12 @@ -67,9 +73,11 @@ cd AReaL bash examples/env/scripts/setup-pip-deps.sh ``` + ## (Optional) Launch Ray Cluster for Distributed Training @@ -89,8 +97,9 @@ ray start --address $RAY_HEAD_IP You should see the Ray resource status displayed when running `ray status`. -Properly set the `n_nodes` argument in AReaL's training command, then AReaL's training script will automatically detect the resources and allocate workers to the cluster. +Properly set the `n_nodes` argument in AReaL's training command, then AReaL's training +script will automatically detect the resources and allocate workers to the cluster. ## Next Steps -Check the [quickstart section](quickstart.md) to launch your first AReaL job. \ No newline at end of file +Check the [quickstart section](quickstart.md) to launch your first AReaL job. diff --git a/docs/tutorial/quickstart.md b/docs/tutorial/quickstart.md index 3bc4cae57c..776b31aaf1 100644 --- a/docs/tutorial/quickstart.md +++ b/docs/tutorial/quickstart.md @@ -1,7 +1,7 @@ # Quickstart -Welcome to the **AReaL** Quickstart Guide! This guide demonstrates how to run an AReaL -experiment training an LLM on the GSM8K dataset using the GRPO algorithm with +Welcome to the **AReaL-lite** Quickstart Guide! This guide demonstrates how to run an +AReaL-lite experiment training an LLM on the GSM8K dataset using the GRPO algorithm with function-based rewards. Ensure you've completed [the installation and environment setup](installation.md) before proceeding. @@ -9,9 +9,10 @@ function-based rewards. Ensure you've completed To run the experiment, you will need: -- Training script: [examples/lite/gsm8k_grpo.py](../../examples/lite/gsm8k_grpo.py) +- Training script: + [examples/lite/gsm8k_grpo.py](https://github.com/inclusionAI/AReaL/blob/main/examples/lite/gsm8k_grpo.py) - Config YAML: - [examples/lite/configs/gsm8k_grpo.yaml](../../examples/lite/configs/gsm8k_grpo.yaml) + [examples/lite/configs/gsm8k_grpo.yaml](https://github.com/inclusionAI/AReaL/blob/main/examples/lite/configs/gsm8k_grpo.yaml) Our training scripts will automatically download the dataset (openai/gsm8k) and model (Qwen/Qwen2-1.5B-Instruct). To run the example with default configuration, execute from @@ -32,7 +33,7 @@ All available configuration options are listed in To customize the experiment (models, resources, algorithm options), you can: 1. Edit the YAML file directly at - [examples/lite/configs/gsm8k_grpo.yaml](../../examples/lite/configs/gsm8k_grpo.yaml). + [examples/lite/configs/gsm8k_grpo.yaml](https://github.com/inclusionAI/AReaL/blob/main/examples/lite/configs/gsm8k_grpo.yaml). 1. Add command-line options: - For existing options in the YAML file, directly add the option: `actor.path=Qwen/Qwen3-1.7B`. @@ -55,15 +56,16 @@ python3 -m areal.launcher.local examples/lite/gsm8k_grpo.py \ +sglang.attention_backend=triton ``` -::::{important} We're currently refactoring from legacy ReaLHF to AReaL, which -introduces some configuration differences. We provide a -[**config converter**](../../examples/lite/config_converter.py) to transfer old config -into AReaL YAML file for users' convenience. :::: +::::{important} We're currently refactoring from legacy AReaL to AReaL-lite, which +introduces some configuration differences. We provide a **config converter** to transfer +old AReaL config into AReaL-lite YAML file for users' convenience. +[Click here](#switching-from-legacy-areal-to-areal-lite) to learn how to use the +**config converter**. :::: ## Distributed Experiments with Ray or Slurm -AReaL provides standalone launchers for distributed experiments. After setting up your -Ray or Slurm cluster, launch experiments similarly to `LocalLauncher`: +AReaL-lite provides standalone launchers for distributed experiments. After setting up +your Ray or Slurm cluster, launch experiments similarly to `LocalLauncher`: ``` # Launch with Ray launcher. 4 nodes (4 GPUs each), 3 nodes for generation, 1 node for training. @@ -74,7 +76,6 @@ python3 -m areal.launcher.ray examples/lite/gsm8k_grpo.py \ allocation_mode=sglang.d12p1t1+d4p1t1 \ cluster.n_nodes=4 \ cluster.n_gpus_per_node=4 \ - ... # Launch with Slurm launcher. 16 nodes (8 GPUs each), 12 nodes for generation, 4 nodes for training python3 -m areal.launcher.slurm examples/lite/gsm8k_grpo.py \ @@ -84,7 +85,6 @@ python3 -m areal.launcher.slurm examples/lite/gsm8k_grpo.py \ allocation_mode=sglang.d96p1t1+d32p1t1 \ cluster.n_nodes=16 \ cluster.n_gpus_per_node=8 \ - ... ``` Additional references: @@ -98,19 +98,33 @@ Additional references: > > 1. Ensure `allocation_mode` matches your cluster configuration > (`#GPUs == cluster.n_nodes * cluster.n_gpus_per_node`) -> 1. Ray/Slurm launchers only works for more than 1 node (`cluster.n_nodes > 1`). For +> 1. Ray or Slurm launchers only works for more than 1 node (`cluster.n_nodes > 1`). For > single node scenario, please use `LocalLauncher`. -> 1. In Ray/Slurm launchers, GPUs are allocated at node granularity, which means #GPUs -> for generation or training must be integer multiples of `cluster.n_gpus_per_node`. +> 1. In Ray or Slurm launchers, GPUs are allocated at node granularity, which means +> #GPUs for generation or training must be integer multiples of +> `cluster.n_gpus_per_node`. +## Switching from legacy AReaL to AReaL-lite + +We also provide a convenient script to convert your AReaL YAML config into AReaL-lite +config in one command line. First you need to locate your AReaL config either modified +from files from `examples` folder, or generated when you run your experiments in +`//` folder. Runs: + +``` +python arealite/utils/convert.py --convert_src AReaL --src_config_path --template_path examples/arealite/configs/gsm8k_grpo.yaml --output_path +``` + +Then you should be able to run experiments with your old settings on AReaL-lite! + ## Next Steps -Check [Getting Started with AReaL](../lite/gsm8k_grpo.md) for a complete code +Check [Getting Started with AReaL-lite](../lite/gsm8k_grpo.md) for a complete code walkthrough on the GRPO GSM8K Example. Customization guides: diff --git a/docs/tutorial/quickstart_legacy.md b/docs/tutorial/quickstart_legacy.md index 3fd28c7038..f4e792330c 100644 --- a/docs/tutorial/quickstart_legacy.md +++ b/docs/tutorial/quickstart_legacy.md @@ -1,8 +1,8 @@ # Quickstart (Legacy) > **Note**: This is a quickstart guide for launching AReaL experiment with legacy code -> in `realhf/`. We strongly recommend users to try the new AReaL for better experiences. -> [Click here](quickstart.md) for the quickstart guide! +> in `realhf/`. We strongly recommend users to try AReaL-lite for better experiences. +> [Click here](quickstart.md) for AReaL-lite quickstart guide! This guide walks you through a simple example of training an LLM to solve math problems. Please ensure you have properly diff --git a/examples/lite/config_converter.py b/examples/lite/config_converter.py index edf9903a59..13749886c0 100644 --- a/examples/lite/config_converter.py +++ b/examples/lite/config_converter.py @@ -461,7 +461,7 @@ def post_process_args(args: dict): class AReaLConverter(Converter): """ - Convert realhf arguments to AReaL-Lite arguments. + Convert realhf arguments to AReaL-lite arguments. """ ARG_MAP = {