From f3f4de3a379cc18af5f1691067ed2ecae0869ef5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E6=B8=A9=E5=B7=AE?= Date: Mon, 31 Mar 2025 17:57:48 +0800 Subject: [PATCH 1/5] update readme --- README.md | 2 +- blog/AReaL_v0_2.md | 8 ++++++++ evaluation/README.md | 4 ++-- 3 files changed, 11 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index a3828e7f5b..40b40d2c47 100644 --- a/README.md +++ b/README.md @@ -52,7 +52,7 @@ In the following table, we show the convergence time under different resource se We use **R1-Distill-Qwen-7B** as our base model. After RL training, the pass@1 scores on AIME 2024 and AIME 2025 improve by 6.9 and 8.6 points, respectively, achieving SOTA performance among 7B models in mathematical reasoning. We have released the training data at [AReaL-boba-106k](https://huggingface.co/datasets/inclusionAI/AReaL-boba-Data/blob/main/AReaL-boba-106k.jsonl). -Although our dataset primarily consists of math and logic problems, we observed that RL training led to measurable improvements on the challenging STEM benchmark GPQA. We plan to open-source more datasets in the future, including code, STEM, and other domains. +Although our dataset primarily consists of math and logic problems, we observed that RL training led to measurable improvements on the challenging STEM benchmark GPQA. We plan to open-source more datasets in the future, including code, STEM, and other domains. All the reported numbers are re-evaluated using our evaluation code with more details in our [blog](blog/AReaL_v0_2.md#eval_detail). #### Approaching QwQ-32B performances using only 200 data samples | **Model** | **AIME 2024** | diff --git a/blog/AReaL_v0_2.md b/blog/AReaL_v0_2.md index a35f8f1194..71c95bbe3f 100644 --- a/blog/AReaL_v0_2.md +++ b/blog/AReaL_v0_2.md @@ -27,6 +27,14 @@ We are excited to release AReaL v0.2 (boba), featuring three major milestones: *Table 1: The performance of AReaL-boba-RL-7B and AReaL-boba-SFT-32B. We obtain SOTA 7B model using RL on math reasoning. Although our dataset primarily consists of math and logic problems, we observed that RL training led to measurable improvements on the challenging STEM benchmark GPQA. Additionally, We train a highly competitive 32B model using only 200 data samples, replicating **QwQ-32B's** inference performance on AIME 2024.* + +We declare that the reported numbers are based on our re-testing, with each number representing the average results of 32 sampling responses. The evaluation code is available in our [evaluation](/evaluation/README.md) folder. + +For the baselines and the SFT model AReaL-boba-SFT-32B, we followed the recommended configuration (temperature = 0.6, top_p = 0.95, Suggested by DeepSeek) and used the default R1-Distill-Qwen template for prompts. For the RL-trained models, we retained the same temperature as during RL rollout (1.0). + +Notably, during GPQA testing, we found that all answers are "A." To address this bias, we randomized the answer options. + + ### Training Speed Comparison ![throughput_comparision_with_v0.1.0.png](/assets/thpt_comparison.png) diff --git a/evaluation/README.md b/evaluation/README.md index 8caa1a6f73..051e9b1ff6 100644 --- a/evaluation/README.md +++ b/evaluation/README.md @@ -29,7 +29,7 @@ Evaluate AReaL-boba-RL-7B: python eval_and_aggregate.py \ --model_path ${MODEL_PATH} \ --output_path ${OUTPUT_PATH} \ ---data_names aime24,aime25 \ +--data_names aime24,aime25,gpqa_diamond \ --prompt_type AReaL-boba \ --output_path outputs --temperature 1.0 ``` @@ -39,7 +39,7 @@ Evaluate AReaL-boba-SFT-32B: python eval_and_aggregate.py \ --model_path ${MODEL_PATH} \ --output_path ${OUTPUT_PATH} \ ---data_names aime24,aime25 \ +--data_names aime24,aime25,gpqa_diamond \ --prompt_type AReaL-boba-SFT \ --samples_per_node 2 --num_sample_nodes 16 \ --output_path outputs --temperature 0.6 From 4d38d94d4c47d37f29f8cb8efaf9000725f50ee3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E6=B8=A9=E5=B7=AE?= Date: Mon, 31 Mar 2025 18:48:54 +0800 Subject: [PATCH 2/5] update readme --- blog/AReaL_v0_2.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/blog/AReaL_v0_2.md b/blog/AReaL_v0_2.md index 71c95bbe3f..8e7b2fecd0 100644 --- a/blog/AReaL_v0_2.md +++ b/blog/AReaL_v0_2.md @@ -27,13 +27,13 @@ We are excited to release AReaL v0.2 (boba), featuring three major milestones: *Table 1: The performance of AReaL-boba-RL-7B and AReaL-boba-SFT-32B. We obtain SOTA 7B model using RL on math reasoning. Although our dataset primarily consists of math and logic problems, we observed that RL training led to measurable improvements on the challenging STEM benchmark GPQA. Additionally, We train a highly competitive 32B model using only 200 data samples, replicating **QwQ-32B's** inference performance on AIME 2024.* - + We declare that the reported numbers are based on our re-testing, with each number representing the average results of 32 sampling responses. The evaluation code is available in our [evaluation](/evaluation/README.md) folder. For the baselines and the SFT model AReaL-boba-SFT-32B, we followed the recommended configuration (temperature = 0.6, top_p = 0.95, Suggested by DeepSeek) and used the default R1-Distill-Qwen template for prompts. For the RL-trained models, we retained the same temperature as during RL rollout (1.0). Notably, during GPQA testing, we found that all answers are "A." To address this bias, we randomized the answer options. - + ### Training Speed Comparison From 9b1359eb35e21fc39d9a1a1f67476dbd33cfd54d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E6=B8=A9=E5=B7=AE?= Date: Mon, 31 Mar 2025 18:49:41 +0800 Subject: [PATCH 3/5] update readme --- blog/AReaL_v0_2.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/blog/AReaL_v0_2.md b/blog/AReaL_v0_2.md index 8e7b2fecd0..4ad0a0a7d8 100644 --- a/blog/AReaL_v0_2.md +++ b/blog/AReaL_v0_2.md @@ -28,7 +28,7 @@ We are excited to release AReaL v0.2 (boba), featuring three major milestones: *Table 1: The performance of AReaL-boba-RL-7B and AReaL-boba-SFT-32B. We obtain SOTA 7B model using RL on math reasoning. Although our dataset primarily consists of math and logic problems, we observed that RL training led to measurable improvements on the challenging STEM benchmark GPQA. Additionally, We train a highly competitive 32B model using only 200 data samples, replicating **QwQ-32B's** inference performance on AIME 2024.* -We declare that the reported numbers are based on our re-testing, with each number representing the average results of 32 sampling responses. The evaluation code is available in our [evaluation](/evaluation/README.md) folder. +We declare that the reported numbers are based on our re-testing, with each number representing the average results of 32 sampling responses. The evaluation code is available in our [evaluation](/evaluation/) folder. For the baselines and the SFT model AReaL-boba-SFT-32B, we followed the recommended configuration (temperature = 0.6, top_p = 0.95, Suggested by DeepSeek) and used the default R1-Distill-Qwen template for prompts. For the RL-trained models, we retained the same temperature as during RL rollout (1.0). From 9b8ce0527070e7f25767f55524f8f27c3af75e76 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E6=B8=A9=E5=B7=AE?= Date: Mon, 31 Mar 2025 18:51:13 +0800 Subject: [PATCH 4/5] update readme --- blog/AReaL_v0_2.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/blog/AReaL_v0_2.md b/blog/AReaL_v0_2.md index 4ad0a0a7d8..56f25be235 100644 --- a/blog/AReaL_v0_2.md +++ b/blog/AReaL_v0_2.md @@ -30,7 +30,7 @@ We are excited to release AReaL v0.2 (boba), featuring three major milestones: We declare that the reported numbers are based on our re-testing, with each number representing the average results of 32 sampling responses. The evaluation code is available in our [evaluation](/evaluation/) folder. -For the baselines and the SFT model AReaL-boba-SFT-32B, we followed the recommended configuration (temperature = 0.6, top_p = 0.95, Suggested by DeepSeek) and used the default R1-Distill-Qwen template for prompts. For the RL-trained models, we retained the same temperature as during RL rollout (1.0). +For the baselines and the SFT model AReaL-boba-SFT-32B, we follow the recommended configuration (temperature = 0.6, top_p = 0.95, Suggested by DeepSeek) and use the default R1-Distill-Qwen template for prompts. For the RL-trained models, we retaine the same temperature as during RL rollout (1.0). Notably, during GPQA testing, we found that all answers are "A." To address this bias, we randomized the answer options. From 52e70752d5c1e32409f2c7812c91cadac3f391d4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E6=B8=A9=E5=B7=AE?= Date: Mon, 31 Mar 2025 18:52:18 +0800 Subject: [PATCH 5/5] update readme --- blog/AReaL_v0_2.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/blog/AReaL_v0_2.md b/blog/AReaL_v0_2.md index 56f25be235..aa03561ceb 100644 --- a/blog/AReaL_v0_2.md +++ b/blog/AReaL_v0_2.md @@ -30,7 +30,7 @@ We are excited to release AReaL v0.2 (boba), featuring three major milestones: We declare that the reported numbers are based on our re-testing, with each number representing the average results of 32 sampling responses. The evaluation code is available in our [evaluation](/evaluation/) folder. -For the baselines and the SFT model AReaL-boba-SFT-32B, we follow the recommended configuration (temperature = 0.6, top_p = 0.95, Suggested by DeepSeek) and use the default R1-Distill-Qwen template for prompts. For the RL-trained models, we retaine the same temperature as during RL rollout (1.0). +For the baselines and the SFT model AReaL-boba-SFT-32B, we follow the recommended configuration (temperature = 0.6, top_p = 0.95, suggested by DeepSeek) and use the default R1-Distill-Qwen template for prompts. For the RL-trained models, we retaine the same temperature as during RL rollout (1.0). Notably, during GPQA testing, we found that all answers are "A." To address this bias, we randomized the answer options.