Fix OpenRewardSpec omitting task‑scoped tools during rollout binding (fixes #5727)

[rycerzes]

Summary

Fix bug where OpenRewardSpec omitted task‑scoped tools during rollout binding.

Changes

• Core fix (addresses OpenReward binding only discovers shared /tools, misses task-specific /task_tools #5727) – trl/experimental/openreward/_spec.py now correctly discovers and binds task‑specific tools
  • Previously, only shared tools (from GET /{env_name}/tools) were bound.
  • This PR adds a discover_task_tools flag (default True) that opens a short‑lived session and calls session.list_tools() (which returns the union of shared and task‑specific tools).
  • If discovery fails, it falls back to the shared‑only list (non‑breaking).
  • The new task_tools_discovery_index parameter allows manual control over which task index is used for the probe.
• Supporting changes
  • New helper _tool_specs_from_session_list_tools() to normalise the SDK output.
  • Extended docstrings to explain the discovery behaviour.
  • Updated tests (tests/experimental/_openreward_echo_env.py, tests/experimental/test_openreward.py) to verify that task‑scoped tools are bound and that disabling discovery works as expected.

Related Issues

Fixes #5727

Testing

• Verified that a GRPO trainer now sees and can call both echo and hint task‑scoped tools.
• Added unit tests to validate the discovery fallback and the configurable discover_task_tools=False behaviour.

CC: @adithya-s-k

---

Note

Medium Risk
Changes tool discovery/binding logic to open probe sessions and merge tool specs, which could affect rollout initialization behavior and performance or expose differing tool schemas across tasks.

Overview
Fixes OpenRewardSpec tool binding to include task-scoped ORS tools by optionally probing session.list_tools() (ORS /task_tools) during spec initialization and merging discovered ToolSpecs by name, with fallback to shared-only environment.list_tools() on errors.

Adds discover_task_tools (default True) and task_tools_discovery_index to control/limit probing (including multi-index probing when indices is provided), and updates the experimental echo env + tests to validate binding of a session-only hint tool and the opt-out behavior.

^{Reviewed by Cursor Bugbot for commit 8f3d02c. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit f48d157. Configure here.}

[adithya-s-k]

Amazing work @rycerzes 🙌 — nice catch on the /tools vs /task_tools distinction, that's a detail I missed when reading the SDK. Repro in #5727 is super clean too. The default-on discover_task_tools with a graceful shared-only fallback is exactly the right shape.

Two small suggestions if you want to tighten it before merge (otherwise this LGTM):

1. When indices= is set and task_tools_discovery_index isn't, discovery opens one probe session per distinct index — could be O(N) HTTP sessions if someone passes a large indices= list. Might be worth defaulting to single-probe (index 0) and gating the multi-probe behavior behind an explicit flag.
2. The discovery-failure warning logs the raw exception, and the probe session is opened with secrets= (which includes the API key). Low-risk, but if the SDK ever embeds request headers in error reprs the key could surface in logs — narrower exception handling or scrubbing the message would be defensive.

Neither is blocking. Thanks for cleaning this up so quickly 🚀

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[rycerzes]

Thanks for the review @adithya-s-k!

1. Multi-probe is intentional since the ORS contract makes list_task_tools() per-task, so tool sets can genuinely differ across indices. Single-probe would silently miss tools in heterogeneous rollouts. Users with uniform tasks can already opt in via task_tools_discovery_index=.
2. The SDK doesn't embed secrets in exception reprs today, so we're holding off on defensive scrubbing for now, easy to revisit if it ever becomes real.

I have fixed the failing lint issue, the other tests failing is pre-existing and is not introduced by this branch. All good to merge as-is. 🙌

[adithya-s-k]

@qgallouedec , have a look at this and let me know your thoughts