Integration with LLM-based agents (for example through tlr)

[mainrs]

Hello! I want to do some comparison between different algorithms for my agent. However, I also want to give LLMs a try. Is there a goto way to integrate it with rl_zoo3?

This library provides means of fine-tuning huggingface models: https://huggingface.co/docs/trl/en/index.

[qgallouedec]

Hey thanks for raising this point.
LLMs are language models, can you explain how you think about training and evaluation on RL tasks?
Also, TRL is for LLM finetuning, which is a slightly different paradigm than classical RL. Can you clarify the link you see?

[mainrs]

There are basically two things I am curious about.

For one, I wanted to use an LLM to plan the agent's actions. The LLM gets a description of the environment and its action set and is allowed to derive a plan. For example for FrozenLake-v1. The problem is that there is no "learning" besides the token context.

(Second) That's why I thought about fine-tuning/aligning the LLM to include a way to learn. One could use tlr for that. It also works by using a reward to fine-tune the model. So my idea was to ask the LLM what it would do, run the simulation and get a final reward. That reward is used to fine-tune it. With the hopes that it would produce a better working LLM for my specific task.

I am unsure about how to tie it all together though. If it even makes sense to use rl_zoo for this. I really liked the utility scripts that the repository provides. Like video generation, plotting and the likes. And, if possible, I want to re-use as much as I can.

[qgallouedec]

Ok I see what you want to do. If I were you, I think I wouldn't try to integrate it into rl-zoo, but rather start from scratch and cherry pick the utilities you want from rl-zoo.
If, on the contrary, you decide to integrate it into rl-zoo3, let us know here and link your branch, or any other material that could benefit the community.