feat(workflow): add Anthropic Messages API support for RL training

[fishcrap]

Description

Add Anthropic Messages API compatibility to AReaL's proxy server, enabling Claude-style agents to participate in reinforcement learning workflows.

What changed:

• New /v1/messages endpoint in the proxy rollout server that accepts Anthropic-format requests
• MathAgent workflow class demonstrating single-turn math reasoning with Anthropic Messages API
• LiteLLM-based bidirectional conversion between Anthropic and OpenAI message formats

Why this change was needed:
Anthropic's Claude models use a different API format than OpenAI. This change allows researchers to use Anthropic-compatible agents (including Claude) in RL training loops without modifying their agent code. The proxy server handles format conversion transparently.

Architecture flow:

Anthropic Agent -> /v1/messages (proxy) -> LiteLLM conversion -> OpenAI endpoint -> RL engine
                                        <- LiteLLM conversion <- OpenAI response <-

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not
  work as expected)
• Documentation update
• Code refactoring (no functional changes)
• Performance improvement
• Test coverage improvement

Checklist

• I have read the Contributing Guide
• I have run formatting tools (pre-commit or manual)
• I have run relevant unit tests and they pass
• I have added tests for new functionality
• I have updated documentation if needed
• My branch is up to date with main
• This PR introduces breaking changes (if yes, fill out details below)
• If this PR changes documentation, I have built and previewed it locally with
  jb build docs
• No critical issues raised by AI reviewers (/gemini review)

Breaking Change Details (if applicable):

This is a backward-compatible addition. Existing usage remains unchanged. The new /v1/messages endpoint is additive and does not affect existing /chat/completions or /responses endpoints.

Additional Context

Files changed:

• areal/experimental/openai/proxy/proxy_rollout_server.py - New /v1/messages endpoint with request/response conversion
• areal/experimental/openai/proxy/server.py - Added ANTHROPIC_MESSAGES_PATHNAME constant
• areal/workflow/anthropic/math_agent.py - Example agent using Anthropic Messages API
• areal/workflow/anthropic/__init__.py - Module exports
• pyproject.toml - Added litellm and anthropic dependencies

Known limitations:

• Only non-streaming requests are currently supported
• Tool use / function calling conversion not yet implemented
• MathAgent is a simple single-turn example; multi-turn conversations would need state management

[gemini-code-assist]

Summary of Changes

Hello @fishcrap, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the AReaL framework by integrating support for the Anthropic Messages API. This allows users to train agents compatible with Claude models, broadening the ecosystem of large language models that can be leveraged within the platform. The changes include a new proxy endpoint for seamless format conversion and a comprehensive example demonstrating the training workflow for Anthropic-based agents.

Highlights

• Anthropic Messages API Support: A new v1/messages endpoint has been added to the proxy rollout server. This endpoint accepts Anthropic-format requests, converts them to OpenAI format using LiteLLM's AnthropicAdapter for processing through the existing OpenAI-compatible pipeline, and then converts responses back to Anthropic Message format.
• Claude Agent Example for RL Training: A new anthropic_agents example directory has been introduced, demonstrating how to train Claude-compatible agents. This includes math_agent.py (a simple single-turn math agent using the Anthropic Client), train_agents.py (a training script following the openai_agents pattern), and config.yaml (a GRPO training configuration for GSM8K).

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces support for the Anthropic Messages API within the proxy rollout server and provides an example for training Claude-compatible agents. The changes involve adding necessary imports, a new FastAPI endpoint to handle Anthropic requests by converting them to OpenAI format and back using LiteLLM's AnthropicAdapter, and new example files for an Anthropic math agent and its training configuration. The implementation appears generally sound, but there are a few areas where robustness and clarity could be improved, particularly around error handling and configuration management.

[garrett4wade]

/gemini review

[gemini-code-assist]

Code Review

This pull request adds support for the Anthropic Messages API by introducing a new proxy endpoint and an example agent. The implementation is well-structured. My review focuses on improving error handling and logging for better debuggability, and some minor refactoring for robustness and clarity in the new agent workflow.