Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions Dockerfile.cuda
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,6 @@ COPY packages/ /app/packages/
COPY deps/ /app/deps/
COPY configs /app/configs
COPY examples /app/examples
COPY k8s /app/k8s
COPY benchmarks/scripts /app/benchmarks/scripts
COPY scripts/docker-editable-pretend-versions.sh /app/scripts/docker-editable-pretend-versions.sh

Expand Down Expand Up @@ -304,6 +303,6 @@ ENV LD_LIBRARY_PATH="/opt/ucx/lib:/opt/ucx/lib/ucx${LD_LIBRARY_PATH:+:${LD_LIBRA
ENV HF_HUB_ETAG_TIMEOUT=500
ENV HF_HUB_DOWNLOAD_TIMEOUT=300

# Use entrypoint for setup (ulimit, etc) but default to sleep infinity for K8s
# Use entrypoint for setup (ulimit, etc) but default to sleep infinity
ENTRYPOINT ["/app/docker-entrypoint.sh"]
CMD ["sleep", "infinity"]
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ prime-rl is a framework for large-scale reinforcement learning. It is designed t
2. Performant: built to train 1T+ MoE models on 1000+ GPUs with [FSDP2](https://docs.pytorch.org/tutorials/intermediate/FSDP_tutorial.html) for training and [vLLM](https://github.com/vllm-project/vllm) for inference, with FP8 inference, PD disaggregation, EP and CP parallelism, and more.
3. Native integration with [`verifiers`](https://github.com/PrimeIntellect-ai/verifiers) environments through the [Environments Hub](https://app.primeintellect.ai/dashboard/environments?ex_sort=most_stars), including built-in support for SWE and agentic environments.
4. End-to-end post-training: SFT, RL training, and evals.
5. Multi-node deployment with Slurm and Kubernetes support.
5. Multi-node deployment with Slurm support.
6. Multimodal support for VLMs such as Qwen3.5.
7. Hackable, modular, and extensible by design.
8. One-line SLURM deployment for frontier models — e.g. [`GLM-5` FP8 with P/D disaggregation, the `llm-d` router, and Mooncake KV offload](examples/advanced/glm-5.3/).
Expand Down Expand Up @@ -181,7 +181,7 @@ Follow this guide to learn the basics of prime-rl. You can train your own models
### Advanced Training: 32 - 2048 GPUs:

Follow this guide to train large models on hard reasoning and agentic tasks.
These guides are designed to be run from a Slurm cluster but can also be adapted to k8s deployments.
These guides are designed to be run from a Slurm cluster.

1. [**Qwen3-30B-A3B**](examples/advanced/qwen3-30b-a3b/): Train `Qwen3-30B-A3B` on math, SWE, and agentic tool use.
2. [**GLM-4.5-Air**](examples/advanced/glm-4.5-air/): Train `GLM-4.5-Air` on search, SWE, and terminal tasks.
Expand Down
2 changes: 1 addition & 1 deletion docs/training.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ This page covers everything you need to launch, observe, checkpoint, and recover
| `uv run inference` | vLLM server. | Always use this entrypoint over `vllm serve` — it adds `/update_weights`, `/load_lora_adapter`, and `/init_broadcaster`. |
| `uv run orchestrator` | Standalone orchestrator process. | Pair with a separately-launched inference server and one `env-server` per source. |
| `uv run eval` | Multi-env evals against a live inference server. | One epoch per source, pinned (or adaptive) concurrency, cursor checkpoints + `--resume`, dashboard + optional platform upload; see [Eval](eval.md). |
| `uv run env-server` | Standalone env server for one environment. | The `rl` launcher starts these automatically (one per train/eval source; each binds an OS-assigned loopback port and publishes it to `configs/attempt_N/resolved/envs/<split>/<name>.address` for the orchestrator); only needed when running the orchestrator standalone, or for sources with an explicit `serve.address` — those are externally managed (e.g. their own k8s pod) and the launcher expects the server to already run there. |
| `uv run env-server` | Standalone env server for one environment. | The `rl` launcher starts these automatically (one per train/eval source; each binds an OS-assigned loopback port and publishes it to `configs/attempt_N/resolved/envs/<split>/<name>.address` for the orchestrator); only needed when running the orchestrator standalone, or for sources with an explicit `serve.address` — those are externally managed (e.g. on their own host) and the launcher expects the server to already run there. |

## RL Trainer

Expand Down
120 changes: 0 additions & 120 deletions examples/basic/reverse-text/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,123 +98,3 @@ uv run eval @ examples/basic/reverse-text/eval.toml -m PrimeIntellect/Qwen3-0.6B
```

Way better! Now we get an **average reward of ~0.8**.

## Kubernetes Deployment

If you're running on Kubernetes, you can deploy this example using the provided Helm chart. The Helm chart automatically configures all components with proper networking and shared storage.

### Step 1: Deploy for SFT Training

The reverse-text example is configured with `autoStart: true` for RL training, but we need to run SFT first. Deploy with autoStart disabled:

```bash
cd k8s
helm install my-exp ./prime-rl -f ./prime-rl/examples/reverse-text.yaml \
--set orchestrator.autoStart=false \
--set inference.autoStart=false \
--set trainer.autoStart=false

# Check pods are ready
kubectl get pods -l app=prime-rl,example=reverse-text
# All pods should show 1/1 Running
```

### Step 2: Run SFT Training

Exec into the trainer pod and run SFT:

```bash
kubectl exec -it my-exp-trainer-0 -- bash
uv run sft @ /app/examples/basic/reverse-text/sft.toml --output-dir /data/outputs --run.name sft
# This will save DCP checkpoints to /data/outputs/sft/checkpoints/step_100
```

### Step 3: Deploy RL Training

Now upgrade the deployment to enable autoStart (uses the default `autoStart: true` from reverse-text.yaml):

```bash
# Exit the pod first (Ctrl+D)
cd k8s
helm upgrade my-exp ./prime-rl -f ./prime-rl/examples/reverse-text.yaml

# Check pods have restarted
kubectl get pods -l app=prime-rl,example=reverse-text
```

The RL components will automatically start. Monitor the logs:

```bash
# View inference server logs
kubectl logs -f my-exp-inference-0

# View orchestrator logs
kubectl logs -f my-exp-orchestrator-0

# View trainer logs
kubectl logs -f my-exp-trainer-0
```

### Alternative: Manual RL Training

If you prefer to run RL components manually, keep `autoStart: false` and exec into each pod:

```bash
# Terminal 1 - Inference
kubectl exec -it my-exp-inference-0 -- bash
uv run inference @ /app/k8s/prime-rl/examples/reverse-text/infer.toml

# Terminal 2 - Orchestrator
kubectl exec -it my-exp-orchestrator-0 -- bash
uv run orchestrator @ /app/k8s/prime-rl/examples/reverse-text/orch.toml --output-dir /data/outputs --client.base-url '["'$INFERENCE_URL'"]'

# Terminal 3 - Trainer
kubectl exec -it my-exp-trainer-0 -- bash
uv run trainer @ /app/k8s/prime-rl/examples/reverse-text/train.toml --output-dir /data/outputs
```

### Access Outputs and Checkpoints

All outputs are written to `/data/outputs` on the shared NFS storage:

```bash
# Exec into any pod to access outputs
kubectl exec -it my-exp-trainer-0 -- bash
ls -lh /data/outputs/weights/
ls -lh /data/outputs/rollouts/
```

### Step 4: Run Evaluation

To evaluate the trained RL model:

```bash
# Exec into trainer pod
kubectl exec -it my-exp-trainer-0 -- bash

# If inference server isn't running with the RL model, start it in another terminal:
kubectl exec -it my-exp-inference-0 -- bash
uv run inference --vllm.model /data/outputs/weights_hf/step_20

# Back in trainer pod, run evaluation
uv run eval @ examples/basic/reverse-text/eval.toml \
-m /data/outputs/weights_hf/step_20 \
--client.base-url $INFERENCE_URL
```

### Clean Up

```bash
helm uninstall my-exp
# Optionally delete shared data. The PVC is named <release-name>-shared-data:
kubectl delete pvc my-exp-shared-data
```

**What the Helm chart provides:**
- Predictably named pods: `<release-name>-trainer-0`, `<release-name>-inference-0`, `<release-name>-orchestrator-0`
- Shared NFS storage at `/data` for model checkpoints and outputs
- GPU resources (1 GPU per inference/trainer pod)
- Kubernetes services for component communication
- Auto-start capability with proper networking configured

See the [K8s deployment guide](../../../k8s/README.md) for more details on scaling, distributed training, and advanced configurations.
72 changes: 0 additions & 72 deletions k8s/README.md

This file was deleted.

16 changes: 0 additions & 16 deletions k8s/prime-rl/Chart.yaml

This file was deleted.

44 changes: 0 additions & 44 deletions k8s/prime-rl/examples/reverse-text.yaml

This file was deleted.

2 changes: 0 additions & 2 deletions k8s/prime-rl/examples/reverse-text/infer.toml

This file was deleted.

27 changes: 0 additions & 27 deletions k8s/prime-rl/examples/reverse-text/orch.toml

This file was deleted.

14 changes: 0 additions & 14 deletions k8s/prime-rl/examples/reverse-text/train.toml

This file was deleted.

Loading
Loading