Add comprehensive PlantUML system architecture diagram for AReaL

[Copilot]

This PR adds a comprehensive PlantUML diagram that illustrates the high-level architecture of the AReaL (Ant Reasoning Reinforcement Learning) system, addressing the need for clear documentation of the system's innovative asynchronous RL training architecture.

What's Added

Core Documentation

• docs/architecture/system-overview.puml - Complete PlantUML source diagram showing the system architecture
• docs/architecture/system-overview.png - Generated visual diagram (256KB)
• docs/architecture/README.md - Comprehensive documentation explaining the diagram and its usage

Architecture Coverage

The diagram comprehensively covers all major system components:

API Layer (realhf/api/): CLI arguments, core APIs (System, Model, Data, Agent, Environment), and quickstart functionality

Model Implementation (realhf/impl/model/): Backend integrations including SGLang, vLLM, and Megatron, neural network modules (attention, MLP, embeddings, MOE), and parallelism support (pipeline parallel, tensor parallel, parameter reallocation)

Training System (realhf/system/): Core workers (Model Worker, Rollout Worker, Generation Server), distributed control systems, data management components, and communication infrastructure

Infrastructure (realhf/base/): Logging, name resolution, topology management, GPU utilities, monitoring, and networking

Scheduler (realhf/scheduler/): Job scheduling clients, evaluators, and execution backends (local/SLURM)

Implementation & Experiments: Concrete implementations for agents, datasets, environments, and both asynchronous and traditional experiment configurations

Key Architectural Insights

The diagram specifically highlights AReaL's fully asynchronous RL training architecture, showing:

1. Decoupled Generation and Training: How the Generation Server operates continuously without waiting for training completion
2. Asynchronous Data Flow: The streaming architecture that enables maximum throughput through Push/Pull streams and buffering
3. Distributed Control: How the Controller manages generation and training workers independently
4. Component Relationships: Clear visualization of how different system layers interact

Benefits

• Developer Onboarding: New contributors can quickly understand the system architecture
• System Understanding: Visualizes the innovative async RL design that achieves 2.77x speedup
• Documentation: Provides a living document that reflects the actual codebase structure
• Reference: Serves as a comprehensive overview for both technical and non-technical stakeholders

The diagram is generated using industry-standard PlantUML and can be easily updated as the system evolves. It successfully captures the essence of AReaL's breakthrough asynchronous reinforcement learning architecture for large language models.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

• esm.ubuntu.com
  • Triggering command: /usr/lib/apt/methods/https (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

This pull request was created as a result of the following prompt from Copilot chat.

Create a comprehensive PlantUML diagram that illustrates the high-level architecture of the AReaL (Ant Reasoning Reinforcement Learning) system. The diagram should include:

Requirements:

1. Core Components: Show the main system components including:

   • API layer (CLI args, core APIs, quickstart)
   • Model implementations (backends like SGLang, modules, parallelism)
   • Training systems (schedulers, experiments)
   • Base infrastructure (logging, topology, name resolution)
   • Evaluation systems

2. System Architecture: Visualize the decoupled generation and training architecture that AReaL implements for RL training of LLMs

3. Key Relationships: Show how different components interact, particularly:

   • The relationship between model backends and training systems
   • How the scheduler manages distributed training
   • The flow from API to implementation layers

4. File Organization: The diagram should reflect the actual code structure found in the repository, including the main realhf/ package structure

Implementation Details:

• Use PlantUML component diagram syntax
• Include appropriate groupings and packages to show system layers
• Add brief descriptions for major components
• Use consistent styling and clear component relationships
• Save as docs/architecture/system-overview.puml

This diagram will help developers and users understand the overall system architecture and how different parts of AReaL work together to provide reinforcement learning capabilities for large language models.

---

💬 Share your feedback on Copilot coding agent for the chance to win a $200 gift card! Click here to start the survey.