[RL] Fix PPO and add GRPO

[greycooker]

Before submitting

• Lint code. If there are lint issues, please format the code first.

# Install and register `pre-commit` in the project folder
pip install pre-commit && pre-commit install

# Process previous code files separately
pre-commit run --file XXXX.py

• Add test cases into tests folder. If there are codecov issues, please add tests cases first.

PR types

New features

PR changes

Others

Description

Fix PPO and add GRPO

[paddle-bot]

Thanks for your contribution!

[codecov]

Codecov Report

Attention: Patch coverage is 8.69565% with 21 lines in your changes missing coverage. Please review.

Project coverage is 51.33%. Comparing base (347d77c) to head (8ac7fc1).
Report is 6 commits behind head on develop.

Files with missing lines	Patch %	Lines
paddlenlp/trl/llm_utils.py	0.00%	16 Missing ⚠️
...dlenlp/experimental/transformers/llama/modeling.py	0.00%	5 Missing ⚠️

❌ Your patch status has failed because the patch coverage (8.69%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.
❌ Your project status has failed because the head coverage (51.33%) is below the target coverage (58.00%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #9931      +/-   ##
===========================================
- Coverage    51.34%   51.33%   -0.01%     
===========================================
  Files          745      745              
  Lines       118567   118602      +35     
===========================================
+ Hits         60877    60889      +12     
- Misses       57690    57713      +23     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.