Update wordle.py example with masking of env tokens

[sergiopaniego]

What does this PR do?

Some results:

The position_reward could be renamed to something like partial_reward.

One idea:
The model is Qwen3-1.7B and it achieves a correct reward of around 0.3. This does not mean that it answers correctly 30% of the time, but rather that, for a given answer, about 30% of the letters are correct—rather than winning 30% of the games.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a GitHub issue? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

Who can review?

@kashif @qgallouedec

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[qgallouedec]

would something like this work?:

tool_mask = extra_fields.pop("env_mask", None)

these tokens are basically processed similarly, maybe it can simplify things

[sergiopaniego]

I've reviewed the rest of the scripts+notebooks using OpenEnv and they seem to be ok since they add the env feedback to the prompt.

I still need to upload the updated Wordle notebook (wip)

[qgallouedec]

@albertvillanova moved everything related to vLLM in a separate class in #4700, you'll have a to deal with some conflicts 😬

[albertvillanova]

@albertvillanova moved everything related to vLLM in a separate class in #4700, you'll have a to deal with some conflicts 😬

Thanks for the ping, @qgallouedec!

@sergiopaniego, since I am familiar with the changes introduced in #4700, I can take care of resolving the conflicts in this PR if that helps. 🤗

[sergiopaniego]

I'd really appreciate some help with that @albertvillanova!! 😄

[albertvillanova]

Done!

[albertvillanova]

Thanks! Just some comments below.

[albertvillanova]

Why the change from trackio to wandb?

[albertvillanova]

This new _build_server_generation_kwargs function duplicates logic from _build_colocate_sampling_params. Maybe we could refactor these to reduce duplication.

[albertvillanova]

As privately discussed, please note that

• Trainer no longer has structured_output_regex attribute
• Instead, it is an attribute of VLLMGeneration

So, in these code lines:

trl/trl/experimental/openenv/utils.py

Lines 35 to 36 in 25c5d51

	if trainer.structured_outputs_regex:
	structured_outputs = StructuredOutputsParams(regex=trainer.structured_outputs_regex)

maybe better using:

    if trainer.vllm_generation.structured_outputs_regex:
        structured_outputs = StructuredOutputsParams(regex=trainer.vllm_generation.structured_outputs_regex)

[qgallouedec]

Everything related to GRPO and vLLM lgtm! nice work @sergiopaniego!

[sergiopaniego]

Results using trackio in with the latest changes: https://huggingface.co/spaces/sergiopaniego/Wordle-GRPO

TODO: Upload the updated wordle notebook