[Spike] Sync vs Fully-Async training: Qwen3-4B GRPO on GPU

[CalvinXKY]

Type: Spike / experiment log
Scope: GPU (A800), non-colocate
Related code / docs:

• train_async.py + --rollout-function-path vime.rollout.fully_async_rollout.generate_rollout_fully_async
• Examples: examples/fully_async/
• Docs: docs/en/examples/qwen3-4B.md (async section)

---

1. Motivation

When train and rollout are disaggregated (non-colocate), sync GRPO naturally bubbles:
train GPUs wait for rollout, then rollout GPUs wait for train. Official docs already
recommend switching train.py → train_async.py so the next rollout is generated while
the current batch is trained.

We ran a controlled A/B (same model / data / steps / GPU layout) to answer:

1. How much wall-clock does the fully-async combo (train_async.py + generate_rollout_fully_async) save vs sync under a matched step budget?
2. Does accuracy suffer relative to sync, stay flat, or improve?
3. How should “vime fully async” be described vs common “RL fully async” (stale-weight /
   producer–consumer) designs — and vs train_async.py with the default sync rollout?

---

2. Terminology note (important)

Term	What we mean
train_async.py (default rollout)	Outer-loop prefetch / one-step pipeline: while training step N, start generate(N+1); sync that generate before update_weights (default --update-weights-interval 1). Each generate() still finishes a full batch via default vime.rollout.vllm_rollout.generate_rollout.
vime fully_async (this spike)	train_async.py + --rollout-function-path vime.rollout.fully_async_rollout.generate_rollout_fully_async. Same outer cadence, plus a continuous rollout worker that keeps generations in flight across generate() calls (long-tail hiding / warm queue). See examples/fully_async/README.md.
RL “fully async” (elsewhere)	Train and inference are decoupled: a continuous rollout server fills a buffer; train consumes from the buffer; weight sync is periodic → multi-step stale behavior policy is allowed.

This spike’s Async arm is vime fully_async (both layers), not train_async.py alone.
We did not A/B train_async+default vs fully_async_rollout; isolating that extra gain needs a three-way run.

Relative to a fully decoupled async design, vime’s path still ties weight updates to the step loop (typically ~one-step off-policy under the default interval), rather than allowing arbitrary multi-step staleness.

---

3. Setup

Item	Value
Hardware	A800
Model	Qwen3-4B
Dataset (train)	dapo-math-17k
Method	GRPO
Steps	200
TP	2
Actor GPUs	2
Rollout GPUs	2 (disaggregated)
rollout-batch-size / n-samples / gbs	8 / 4 / 32
Eval	dapo-math-17k first 500 prompts; vLLM greedy; thinking off; max_tokens=1024

Mode	Entry	Rollout path
Sync	train.py	default sync rollout
Async	train_async.py	vime.rollout.fully_async_rollout.generate_rollout_fully_async

---

4. Results

4.1 Training speed

Metric	Sync	Async
step_time	~15.5 s	~6 s
train_wait_time	~10.3 s (~66%)	~0–1 s
train_time (compute)	~5.3 s	~5.3 s
update_weights	~0.3 s	~0.3–0.7 s
Wall clock	~52 min	~20 min
Speedup	1.0×	~2.6×

Async mainly removes the train-wait bubble; per-step train compute stays similar.

4.2 Downstream accuracy (greedy eval, 500 prompts)

Model	Accuracy	Correct / Total	vs Baseline
Baseline (pretrained Qwen3-4B)	16.60%	83 / 500	—
Sync	17.20%	86 / 500	+0.6 pp
Async	20.60%	103 / 500	+4.0 pp

Rough efficiency (accuracy gain / wall-clock minute): ~0.012 pp/min (sync) vs ~0.20 pp/min (async).

Eval parsing: prefer \boxed{}, then Answer: line, else trailing number.

4.3 Qualitative observations

1. Async is faster and higher-scoring than sync under this matched setup.
2. Sync early eval slices (e.g. first 50–100 prompts) dip below baseline (8–9% vs ~12–13%); async does not show the same early dip in our curve.
3. Same nominal step count ≠ same “data freshness” timeline: async overlaps generation, so the effective exposure schedule differs from sync even when step count matches.

---

5. References (in-tree)

• examples/fully_async/README.md
• examples/fully_async/run-qwen2.5-0.5B-fully-async.sh
• vime/rollout/fully_async_rollout.py
• docs/en/examples/qwen3-4B.md — “asynchronous training” note

[CalvinXKY]

[floatlibai]

I wonder what's the difference btween vime-full-async and train_async.py with default sync rollout, since you mentioned that vime-full-async still keeps train loop steps 1:1.

In other words, is vime-full-async essentially the same thing as one-step-off-policy async?

[CalvinXKY]

I wonder what's the difference btween vime-full-async and train_async.py with default sync rollout, since you mentioned that vime-full-async still keeps train loop steps 1:1.

In other words, is vime-full-async essentially the same thing as one-step-off-policy async?

vime-full-async is not the same as one-step-off, but it builds on train_async.py. One-step-off ≈ train_async + default rollout; fully-async ≈ train_async + fully_async_rollout.

This spike’s Async run used the fully-async combo (both). The “1:1 prefetch” wording mainly describes the train_async outer loop; isolating the extra gain from fully_async_rollout would need a three-way A/B.

Description has been changed

[floatlibai]

Hi there, could you provide the actual eval script/command? Also, is it expected to use the same dataset (dapo-math) for both training and evaluation?

[chenchao9999]

hello, Could you tell me the version of the Docker image you pulled? Adapt it to the A800.thanks