fix: convergence issue by adding use_inductor=False in vllm compilation_config

[ZhiyuLi-Nvidia]

What does this PR do ?

Closes #998.

Looks like it can be resolved with the compilation flag {"use_inductor": False}.
"With this flag, vllm will use the custom CUDA kernels instead of the Triton kernels generated by torch.compile "which might cause numerical issue here.

There's no logprob error spikes in 140 steps and rewards were increasing stably. The speed performance looks similar.
https://wandb.ai/nvidia/grpo-dev-zhiyul/workspace?nw=nwuserzhiyul

• Rewards of 140 steps

• logprob error w/ and w/o the change

Issues

List issues that this PR closes (syntax):

Usage

• You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
• Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

• ...

[parthchadha]

@ZhiyuLi-Nvidia good find! Can you share performance on larger qwen models as well? Also, please attach the plots to the PR description since not everyone can access internal wandb reports.

[terrykong]

nice find @ZhiyuLi-Nvidia !

is it possible to construct a model diagnostic test for this?

https://github.com/NVIDIA-NeMo/RL/tree/main/tools/model_diagnostics

might be helpful for others who are debugging their model run

[ZhiyuLi-Nvidia]

Thank you @parthchadha

@ZhiyuLi-Nvidia good find! Can you share performance on larger qwen models as well?

Which model do you recommend?

Also, please attach the plots to the PR description since not everyone can access internal wandb reports.

Added the key screenshots.

[parthchadha]

Thank you @parthchadha

@ZhiyuLi-Nvidia good find! Can you share performance on larger qwen models as well?

Which model do you recommend?

Also, please attach the plots to the PR description since not everyone can access internal wandb reports.

Added the key screenshots.

Let's run qwen 32b from #957 (we can try with 32k osl)

[ZhiyuLi-Nvidia]

is it possible to construct a model diagnostic test for this?

https://github.com/NVIDIA-NeMo/RL/tree/main/tools/model_diagnostics

might be helpful for others who are debugging their model run

Good suggestion. Added.

[ZhiyuLi-Nvidia]

@terrykong added output example 2ba5e3e

[ZhiyuLi-Nvidia]

Thank you @parthchadha

@ZhiyuLi-Nvidia good find! Can you share performance on larger qwen models as well?

Which model do you recommend?

Also, please attach the plots to the PR description since not everyone can access internal wandb reports.

Added the key screenshots.

Let's run qwen 32b from #957 (we can try with 32k osl)

@parthchadha I kept get OOM in the middle of training. Shall we go back to it once merged or in a more stable state?