😷 Refactor GRPO/RLOO to isolate _generate

[qgallouedec]

This PR belongs to a sequence of PR that aims to refactor the generation part of GRPO/RLOO to allow for easier customization and ultimately tool calling

Previous:

• 🟩 Drop image_split_sizes in favour of image_grid_thw #4111
• 📽 Multi image support for GRPO/RLOO #4113

Next:

• 🧺 [1/N] Refactor _generate in GRPO/RLOO: list of ints instead of tensors #4146
• 🧺 [2/N] Refactor _generate in GRPO/RLOO: Use prompt_ids from generation #4152
• 🧺 [3/N] Refactor _generate in GRPO/RLOO: Rely on generator for prompt truncation #4153
• 🧺 [4/N] Refactor _generate in GRPO/RLOO: Move forward_kwargs outside generation method #4154
• 🧺 [5/N] Refactor _generate in GRPO/RLOO: Insert images in the prompt #4155

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[qgallouedec]

cc @Peter-Chou, customization is made easier with this one

[lewtun]

LGTM once the tests pass and a question about whether we should split _generate from scoring entirely

[Peter-Chou]

@qgallouedec Yes. The original _generate_and_score_completions method was way too lengthy.
Breaking it down into finer-grained sub-methods and chaining them together like this is an excellent approach!

[LeonMalteW]

I'm quite interested in following the refactoring process, as the biggest problem for me is scaling to context sizes of up to 32K, and a lot of OOM bombs get hit right now.
Will the final refactor turn _generate_and_score_completions and all the subfunctions into a pipeline so that we don't need to keep all the data on all GPUs the same, and only the necessary data gets synced? Or is this too specific and would it not hit the target of the refactor?

[qgallouedec]

This refactoring is actually for enabling multi-turn RL (with tool calling)