Sync fork with upstream huggingface/trl (1307 commits) by himanshushukla12 · Pull Request #1 · himanshushukla12/trl

himanshushukla12 · 2026-05-07T19:33:05Z

Summary

Fast-forwards this fork's main from e0dd525 to upstream huggingface/trl@47b3778, picking up 1307 commits of fixes, features, and refactors. The fork had no unique commits, so this is a clean fast-forward (no merge conflicts).

How this was done

Added upstream remote → https://github.com/huggingface/trl.git
git fetch upstream main
git rev-list --left-right --count origin/main...upstream/main → 0 1307 (fork purely behind)
git merge --ff-only upstream/main on claude/update-trl-fixes-35dkl

Notable upstream changes included

Reward submodule refactor (🎁 Reward submodule huggingface/trl#3430)
Two-sided clipping for GRPO (feat: Implement Two-Sided Clipping for GRPO Trainer huggingface/trl#3434)
Activation offloading on XPU (enable activation offloading on XPU huggingface/trl#3444), trl env on XPU (enable trl env on xpu huggingface/trl#3438)
vLLM kv-cache dtype quantization (🏹 Support kv_cache_dtype to quantize kv-cache in vllm huggingface/trl#3422)
Many trainer-level deletions/restructures (DDPO, GKD, IterativeSFT, Online DPO, PPO config consolidation under base_trainer.py)
MFU helpers (Add MFU helpers huggingface/trl#5698), activation-offloading memory-leak fix (fix: prevent 5 GB+ CUDA memory leak in activation offloading by syncing streams and clear stashes in OffloadActivations.__exit__ huggingface/trl#5700), KTO/DPO ref-logp alignment (Align KTO with DPO: Align _precompute_ref_logps huggingface/trl#5714)

Test plan

CI runs against the synced tree
Spot-check that imports still resolve in downstream code that depends on this fork
Verify trainers that were removed/renamed upstream don't break local consumers

https://claude.ai/code/session_01PLT7bqaLbhounDpvGzYs8r

Generated by Claude Code

Co-authored-by: lewtun <lewis.c.tunstall@gmail.com>

…uggingface#5286)

…ward model (huggingface#5295)

…S.md` (huggingface#5294)

…gface#5305)

…ist (huggingface#5297)

…e#5307) Co-authored-by: Quentin Gallouédec <gallouedec.quentin@gmail.com> Co-authored-by: Quentin Gallouédec <45557362+qgallouedec@users.noreply.github.com>

…ngface#5233) Co-authored-by: Quentin Gallouédec <45557362+qgallouedec@users.noreply.github.com> Co-authored-by: Quentin Gallouédec <gallouedec.quentin@gmail.com>

)

Co-authored-by: aminediro <amine.dirhoussi@huggingface.co>

… dataset format (huggingface#5310)

…gface#5311)

…token IDs (huggingface#5242) Co-authored-by: Albert Villanova del Moral <8515462+albertvillanova@users.noreply.github.com>

…ace#5117)

Co-authored-by: Quentin Gallouédec <45557362+qgallouedec@users.noreply.github.com>

…n3/3.5 type of chat templates) (huggingface#5330) Co-authored-by: Quentin Gallouédec <45557362+qgallouedec@users.noreply.github.com>

…o_trainer.py` (huggingface#5332)

Co-authored-by: Quentin Gallouédec <45557362+qgallouedec@users.noreply.github.com>

…#4935) Co-authored-by: Kashif Rasul <kashif.rasul@gmail.com> Co-authored-by: cmpatino <carlos.patino@huggingface.co> Co-authored-by: Quentin Gallouédec <45557362+qgallouedec@users.noreply.github.com>

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Quentin Gallouédec <45557362+qgallouedec@users.noreply.github.com> Co-authored-by: Quentin Gallouédec <gallouedec.quentin@gmail.com>

huggingface#5327)

…ingface#5333)

…ner (huggingface#5347) Co-authored-by: Quentin Gallouédec <45557362+qgallouedec@users.noreply.github.com>

…uggingface#5353)

…ngface#5689)

Co-authored-by: Quentin Gallouédec <45557362+qgallouedec@users.noreply.github.com> Co-authored-by: Quentin Gallouédec <gallouedec.quentin@gmail.com>

…#5675)

)

Co-authored-by: Quentin Gallouédec <45557362+qgallouedec@users.noreply.github.com> Co-authored-by: Quentin Gallouédec <gallouedec.quentin@gmail.com>

…gingface#5701)

…ngface#5702)

Co-authored-by: Carlos Miguel Patiño <carlos.patino@huggingface.co> Co-authored-by: Albert Villanova del Moral <8515462+albertvillanova@users.noreply.github.com>

…ts (huggingface#5550) Signed-off-by: Liu, Kaixuan <kaixuan.liu@intel.com> Co-authored-by: Quentin Gallouédec <45557362+qgallouedec@users.noreply.github.com>

…ce#5696)

…uggingface#4688) Co-authored-by: Sunghyun Cho <roycho@lgcns.com> Co-authored-by: Kashif Rasul <kashif.rasul@gmail.com>

…uggingface#5699)

…gingface#5710)

…ng streams and clear stashes in OffloadActivations.__exit__ (huggingface#5700) Co-authored-by: Claude Opus 4.6 <noreply@openclaude.dev> Co-authored-by: Kashif Rasul <kashif.rasul@gmail.com>

Co-authored-by: aminediro <amine.dirhoussi@huggingface.co>

Copilot

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

cmpatino and others added 30 commits March 17, 2026 14:01

[GKD] Buffer Implementation for Distillation Trainer (huggingface#5137)

ee339a0

Co-authored-by: lewtun <lewis.c.tunstall@gmail.com>

Support max_length in DPO VLM training (huggingface#5284)

3acb8e8

Prevent corruption of DPO VLM training if "keep_end" truncation_mode (h…

7b42fc4

…uggingface#5286)

Fix UNEXPECTED lm_head.weight warning when loading a CausalLM as a re…

52cd0cc

…ward model (huggingface#5295)

Apply docstyle (huggingface#5296)

26ce6a3

Add guidance to avoid hasattr and getattr with defaults in `AGENT…

435c2ae

…S.md` (huggingface#5294)

Fix DPOTrainer collators to truncate sequences before padding (huggin…

ee96845

…gface#5305)

Update RewardFunc type annotation to allow Nonevalues in reward l…

5c6e915

…ist (huggingface#5297)

Suggest the Json() type for tool calling dataset format (huggingfac…

3972d66

…e#5307) Co-authored-by: Quentin Gallouédec <gallouedec.quentin@gmail.com> Co-authored-by: Quentin Gallouédec <45557362+qgallouedec@users.noreply.github.com>

Allow reward functions to log extra columns and scalar metrics (huggi…

91583cd

…ngface#5233) Co-authored-by: Quentin Gallouédec <45557362+qgallouedec@users.noreply.github.com> Co-authored-by: Quentin Gallouédec <gallouedec.quentin@gmail.com>

Fix GRPOTrainer attribute access for vLLM model config (huggingface#5302

116fc42

)

Support truncation_mode in SFT (huggingface#5306)

bc60040

Asynchronous GRPO (huggingface#5293)

b86b760

Co-authored-by: aminediro <amine.dirhoussi@huggingface.co>

Fix datasets version supporting Json dtype in docs about tool calling…

b5af98d

… dataset format (huggingface#5310)

Align docs about tool calling in trainers with dataset format (huggin…

321a3cd

…gface#5311)

[GRPO] Fix re-tokenization bug in tool-calling loop by concatenating …

ebdfe82

…token IDs (huggingface#5242) Co-authored-by: Albert Villanova del Moral <8515462+albertvillanova@users.noreply.github.com>

feat(experimental): Divergence Proximal Policy Optimization (huggingf…

8e6e062

…ace#5117)

Clean up model update group on worker exit (huggingface#5325)

e8dcece

Co-authored-by: Quentin Gallouédec <45557362+qgallouedec@users.noreply.github.com>

Fix style in DPPO docstrings (huggingface#5326)

4fea6d1

GRPOTrainer/async: fix prefix EOS slicing for tool suffix (with Qwe…

5bb083c

…n3/3.5 type of chat templates) (huggingface#5330) Co-authored-by: Quentin Gallouédec <45557362+qgallouedec@users.noreply.github.com>

refactor(async_rollout_worker): renamed tool variables to mirror `grp…

e923a9a

…o_trainer.py` (huggingface#5332)

Add truncation to SFT DataCollatorForLanguageModeling (huggingface#5315)

d8a2dd5

Co-authored-by: Quentin Gallouédec <45557362+qgallouedec@users.noreply.github.com>

Enhance print_prompt_completions_sample to include reasoning content (

4fda23e

huggingface#5327)

Add Cursor Bugbot rules from AGENTS.md (huggingface#5280)

bad33f2

Change model dtype from bfloat16 to float32 in AsyncGRPOTrainer (hugg…

05cfc90

…ingface#5333)

docs: Add "It Takes Two: Your GRPO Is Secretly DPO" paper to GRPOTrai…

5635466

…ner (huggingface#5347) Co-authored-by: Quentin Gallouédec <45557362+qgallouedec@users.noreply.github.com>

fix: apply reward_weights to logged reward/reward_std in GRPOTrainer (h…

eff9242

…uggingface#5353)

Remove post-collation truncation from DPO (huggingface#5350)

ec1802e

qgallouedec and others added 26 commits May 4, 2026 11:03

Extend invariant suite with gradient-checkpointing equivalence (huggi…

e55d788

…ngface#5689)

Add Gemma 3 training chat template (huggingface#5685)

909d0c1

Co-authored-by: Quentin Gallouédec <45557362+qgallouedec@users.noreply.github.com> Co-authored-by: Quentin Gallouédec <gallouedec.quentin@gmail.com>

Add {% generation %} markers for Cohere2 chat template (huggingface…

0a3d956

…#5675)

Add length-normalized sigmoid loss type to DPO trainer (huggingface#5406

20b2489

)

Add training chat template for Qwen3-2507 (huggingface#5574)

e5677da

Co-authored-by: Quentin Gallouédec <45557362+qgallouedec@users.noreply.github.com> Co-authored-by: Quentin Gallouédec <gallouedec.quentin@gmail.com>

Align KTO with DPO: Remove enforcement of causal language models (hug…

babb16b

…gingface#5701)

Align KTO with DPO: Remove duplicate import of PreTrainedModel (huggi…

17f8aac

…ngface#5702)

Align KTO with DPO: Simplify max_length init logic (huggingface#5703)

d2b80e0

Align KTO with DPO: Group training arguments (huggingface#5704)

f39373e

Align KTO with DPO: Use _metrics attribute (huggingface#5705)

df6ae2a

Reduce inconsistency across trainer test files (huggingface#5678)

a01bf61

Refactor tiny-model generation scripts (huggingface#5637)

e7c0019

Co-authored-by: Carlos Miguel Patiño <carlos.patino@huggingface.co> Co-authored-by: Albert Villanova del Moral <8515462+albertvillanova@users.noreply.github.com>

Accept processor in get_training_chat_template (huggingface#5560)

225a234

Enable chunked NLL loss with PEFT in SFT (huggingface#5676)

b8e6fc0

Fix GRPO VLM tests: Multimodal training requires conversational promp…

ca8d909

…ts (huggingface#5550) Signed-off-by: Liu, Kaixuan <kaixuan.liu@intel.com> Co-authored-by: Quentin Gallouédec <45557362+qgallouedec@users.noreply.github.com>

[experimental] Add OpenReward Standard environment adapter (huggingfa…

6ad4f30

…ce#5696)

GKDTrainer: Fix return_outputs in Liger kernel path and update tests (h…

1240ecf

…uggingface#4688) Co-authored-by: Sunghyun Cho <roycho@lgcns.com> Co-authored-by: Kashif Rasul <kashif.rasul@gmail.com>

Reject parallelism_config with cp_size>1 or sp_size>1 in GRPO/RLOO (h…

733f3b1

…uggingface#5699)

Fix typo in model name in README (huggingface#5711)

afae06f

Explicitly set model_accepts_loss_kwargs=False in DPO and Reward (hug…

dddac4c

…gingface#5710)

Fail early for unsupported PEFT + Liger Kernel in DPO (huggingface#5709)

19d007e

Revert VLM support in parse_response (huggingface#5561)

de7efdc

Align KTO with DPO: Align _precompute_ref_logps (huggingface#5714)

acbd53f

fix: prevent 5 GB+ CUDA memory leak in activation offloading by synci…

8a6cc03

…ng streams and clear stashes in OffloadActivations.__exit__ (huggingface#5700) Co-authored-by: Claude Opus 4.6 <noreply@openclaude.dev> Co-authored-by: Kashif Rasul <kashif.rasul@gmail.com>

Align tiny Qwen3 MoE config with Qwen/Qwen3-30B-A3B (huggingface#5716)

4601166

Co-authored-by: aminediro <amine.dirhoussi@huggingface.co>

Add MFU helpers (huggingface#5698)

47b3778

Copilot AI review requested due to automatic review settings May 7, 2026 19:33

Copilot AI reviewed May 7, 2026

himanshushukla12 merged commit efa96b8 into main May 7, 2026
4 of 14 checks passed

himanshushukla12 deleted the claude/update-trl-fixes-35dkl branch May 7, 2026 19:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sync fork with upstream huggingface/trl (1307 commits)#1

Sync fork with upstream huggingface/trl (1307 commits)#1
himanshushukla12 merged 1307 commits into
mainfrom
claude/update-trl-fixes-35dkl

himanshushukla12 commented May 7, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Conversation

himanshushukla12 commented May 7, 2026

Summary

How this was done

Notable upstream changes included

Test plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants