Refactor stage #1: AReaL-lite

[garrett4wade]

No description provided.

[garrett4wade]

Hi @tsaoyu, after three weeks of development, we’ve finalized AReaLite and addressed the issues you highlighted in #125 . We’ve carefully incorporated your feedback and would appreciate it if you could review the latest changes when you have time. :)

Some direct references:

• Design Doc
• Getting Started with AReaLite
• Customization Guide: Adding new agent, dataset, or algorithm

[tsaoyu]

I have go through the code base from the gsk8k example. In general, the design is very clean and easy for RL developers for now. The overhead due to RL system is much less than before. Here are few comments that may help the project to be better understand by related parties.

[tsaoyu]

Can we have a __post_init__(self): to verifiy these critical hyper-parameter. The pros of dataclass is its efficiency over pydantic but it sometimes lacks of validations

[garrett4wade]

Omegaconf is not compatible with the __post_init__. The __post_init__ is executed when initializing the dataclass with default values, rather than the values provided in the yaml config.

Totally agree that we need data validation here, but we need some more discussions how we can do that.

[tsaoyu]

def post_init(self):
if self.c_clip is not None and self.c_clip <= 1.0:
raise ValueError("PPOActorConfig.c_clip must be greater than 1.0")
if self.use_decoupled_loss and not self.recompute_logprob:
raise ValueError("use_decoupled_loss requires recompute_logprob to be True")

[tsaoyu]

After the arg pop up, it would be better to emit some info to the user that due to the version of sglang, the arg is not used. Now it is done in a silent way that user may not aware of the env issue.

[garrett4wade]

The code should enforce sglang>=0.4.9.post2. Relevant code removed.

[tsaoyu]

How this mixed precision setup is determined? Suppose we have float16 as the default setting how will this prop. through the whole project without manual setups? If I undestand correclty, it is now using float16 as the the default dytpe and float32 as the optimiser dtype. This should be more related to the model itself, e.g. qwen 2.5 sereis shows little performance degradation using float16 but qwen 3 series is more sensentive to that. So how about give an option for this?

[garrett4wade]

1. Defaults to bfloat16 following the common practice.
2. Added an additional field called "grad_reduce_dtype" in TrainEngineConfig to explicitly control gradient dtype
3. dtypes are propagated through yaml config. sglang.dtype by default uses actor.dtype.

[tsaoyu]

Same as previous comment, does this have to be validate? May not for this hyper parameter?

[tsaoyu]

Does it possible to support trainer only or inference only mode for debug like slime?
When trainer_n_nodes is 0, skip the trainer launcher? Or it is better to have it as two entry points?

[tsaoyu]

Does it possible to support trainer only or inference only mode for debug like slime?
When trainer_n_nodes is 0, skip the trainer launcher? Or it is better to have it as two entry points?

[tsaoyu]

When will hsdp happens?

[tsaoyu]

(for future) To show the advantage of this decoupled design, it can be demonstrate that the sample level async is easy for the multi-turn. This can be useful to evenly distribute the load on reward_fn (that may CPU bound) to further acclerate the performance during the rollout.

[tsaoyu]

After discount *= self.turn_discount will always be 0 ?

[garrett4wade]

This is a typo. The initial value should be 1.