Generic RemoteBackend abstraction + Arctic adapter (parallel to PR #1) by sfc-gh-kganesan · Pull Request #2 · Snowflake-AI-Research/verl

sfc-gh-kganesan · 2026-05-19T23:07:10Z

Summary

This PR introduces a generic RemoteBackend abstraction in verl/remote_backend/ plus the Arctic adapter as a reference implementation, rebased onto upstream verl v0.7.1 (bec9ef74).

It has been iterated on with @sfc-gh-truwase's review and now lands here as a single, review-friendly diff.

Why

Today verl's training loop talks directly to in-process workers (FSDP/Megatron). For RL backends that already own their own training engine, sampling engine, log-prob engine, and weight-sync plumbing (Arctic RL, Tinker, etc.) the integration cost is large because there is no documented contract for "run this op out-of-process and give me back tensors / metrics".

RemoteBackend is that contract.

What

verl/remote_backend/base.py — 9-method ABC: from_config (sole public constructor; takes an optional handle= to re-attach), reconnect_handle, destroy, compute_log_prob, update_actor, generate, update_weights, save_checkpoint, requires_single_forwarder. All abstract — backends declare every behaviour explicitly, no silent defaults.
RemoteBackendRegistry — name -> dotted path map; backends register themselves at import time. main_ppo.py never imports a concrete backend.
verl/remote_backend/trainer.py — RayPPOTrainer subclass that creates the backend on the driver, threads the reconnect_handle() into wg_kwargs, and enforces single-forwarder constraints when the backend opts in.
verl/remote_backend/worker.py — CPU-only forwarder worker group. Owns dispatch annotations, FlopsCounter MFU, NJT reconstruction, and metric normalisation. Forwards every op to the backend.
verl/trainer/ppo/arctic_rl_client.py — Arctic adapter, registered as "arctic". Implements all 9 ABC methods on top of ArcticTraining.ArcticRLClient.
verl/workers/rollout/arctic_rollout/ — Arctic vLLM rollout replica that re-attaches via from_config(handle=...).
trainer.remote_backend: null field in ppo_trainer.yaml and a one-line conditional in main_ppo.py that selects the RemoteBackendTrainer when the field is set. The standard in-process verl path is untouched when it isn't.

Plug-in shape

Adding a new backend (e.g. tinker) is two edits:

Add "tinker": "verl.trainer.ppo.tinker_backend" to RemoteBackendRegistry.MODULES.
Decorate the adapter class with @RemoteBackendRegistry.register("tinker") and implement the 9 ABC methods.

Test plan

4-step Arctic E2E smoke against this branch (MFU≈880, all metrics flowing).
Repeat smoke against this branch (rebase target).
Review pass with the verl maintainers.

Notes

This PR does not remove or rename anything on the upstream verl path; the abstraction is purely additive.
The Arctic adapter is included as a reference implementation. Adapters for other backends are out of scope for this PR.

Made with Cursor

sfc-gh-truwase · 2026-05-20T17:53:23Z

 from ..utils import enable_full_determinism, postprocess_batch_func, prepare_micro_batches
 from .utils import create_device_mesh, get_sharding_strategy

+from arctic_training.debug import pr, pr0


Changes in this file were for debugging. File should be unchanged.

sfc-gh-truwase · 2026-05-20T17:53:50Z

    else:
        total_data_size = len(data)
        micro_batch_size_per_gpu = data["micro_batch_size_per_gpu"]
+        # assert total_data_size % (force_group_size * micro_batch_size_per_gpu) == 0, (


File should be unchanged. Revert changes.

sfc-gh-truwase · 2026-05-20T17:54:17Z

    Returns:
        tensor: sliced response tensor of shape [bsz, max_response_len]
    """
+    # print(f"{tensor.is_nested=}")


File should be unchanged.

sfc-gh-truwase · 2026-05-20T17:54:35Z

 logger = logging.getLogger(__file__)
 logger.setLevel(os.getenv("VERL_LOGGING_LEVEL", "WARN"))

+ENABLE_TIMERS = False


File should be unchanged.

sfc-gh-truwase · 2026-05-20T17:56:26Z

@@ -0,0 +1,272 @@
+"""


Remove this recipe file. Let's use gsm8k example instead.

sfc-gh-truwase · 2026-05-20T17:56:50Z

@@ -0,0 +1,164 @@
+#!/bin/bash


Replace with gsm8k equivalent.

sfc-gh-truwase · 2026-05-20T17:57:55Z

+ENABLE_TIMERS = False
+if ENABLE_TIMERS:
+    from arctic_training.debug import SynchronizedWallClockTimerSimple
+    timers = SynchronizedWallClockTimerSimple(wall_clock_breakdown=True)
+else:
+    from arctic_training.debug import SynchronizedWallClockTimerSimpleDummy
+    timers = SynchronizedWallClockTimerSimpleDummy(wall_clock_breakdown=True)
+


Remove debugging and timer codes

sfc-gh-truwase · 2026-05-20T17:58:09Z

                dataloader_kwargs={"shuffle": shuffle},
            )

+            tname_update_actor = timers.start("xyz ray_trainer._update_actor actor_rollout_wg.update_actor")


Remove timer codes.

sfc-gh-truwase · 2026-05-20T17:58:17Z


+            tname_update_actor = timers.start("xyz ray_trainer._update_actor actor_rollout_wg.update_actor")
            actor_output = self.actor_rollout_wg.update_actor(batch_td)
+            timers.stop_and_print_elapsed(tname_update_actor)


Remove timer code.

sfc-gh-truwase · 2026-05-20T17:58:25Z

        else:
            actor_output = self.actor_rollout_wg.update_actor(batch)

+        timers.stop_and_print_elapsed(tname_e2e)


Remove timer codes.

sfc-gh-truwase · 2026-05-20T21:16:43Z

+    trainer.experiment_name=$experiment_name \
+    trainer.project_name=arctic_rl_gsm8k_public \
+    trainer.val_before_train=False \
+    trainer.n_gpus_per_node=$NGPU_PER_NODE \


Suggested change

trainer.n_gpus_per_node=$NGPU_PER_NODE \

trainer.n_gpus_per_node=1 \

sfc-gh-truwase · 2026-05-20T21:17:41Z

+    algorithm.adv_estimator=grpo \
+    data.train_files=/code/shared/gsm8k/train.parquet \
+    data.val_files=/code/shared/gsm8k/test.parquet \
+    data.train_batch_size=$BSZ \


Simplify for readability by using consts instead of bash vars.

Suggested change

data.train_batch_size=$BSZ \

data.train_batch_size=16 \

Mirrors snowflake-eng/arctic-verl#22, rebased onto upstream verl v0.7.1 (bec9ef7) so the verl team can review the generic remote-backend abstraction in isolation. Adds: * `verl/remote_backend/{base,trainer,worker,worker_utils}.py` — a 9- method `RemoteBackend` ABC, `RemoteBackendRegistry` (lazy-imports adapters via a `name -> dotted-path` map), `RemoteBackendTrainer` (RayPPOTrainer subclass that drives a backend via `from_config` and threads a `reconnect_handle()` into every forwarder), and a CPU-only forwarder worker that delegates compute_log_prob / update_actor / update_weights / save_checkpoint to the backend and rebuilds NJTs + FlopsCounter MFU on the way back. * `verl/trainer/ppo/arctic_rl_client.py` — the Arctic adapter, registered as `"arctic"` via `@RemoteBackendRegistry.register(...)`, and `verl/workers/rollout/arctic_rollout/` — the Arctic vLLM rollout replica that re-attaches via `from_config(handle=...)`. * `trainer.remote_backend` config field in `ppo_trainer.yaml` and the one-line conditional in `main_ppo.py` that selects `RemoteBackendTrainer` + the generic forwarder when the field is set. The standard in-process verl path is untouched when it isn't. * One canonical example (`examples/arctic_rl/run_bird_grpo_arl_zorro_yes.sh`) plus baseline scripts and a reward helper for parity with the Arctic public PR. Reviewer notes: * Backend payload schemas, wire formats, and loss-function plumbing stay inside each backend; the ABC says nothing about them. * `RemoteBackend.requires_single_forwarder() -> True` for Arctic; the trainer asserts `n_gpus_per_node * nnodes == 1` and a single rollout replica because the backend owns its own training / sampling parallelism downstream. * Adapters live in `RemoteBackendRegistry.MODULES`; adding a new backend (e.g. `"tinker"`) is one map entry + one decorated class. This branch is the parallel of snowflake-eng/arctic-verl#22 rebased onto upstream v0.7.1. Co-authored-by: Cursor <cursoragent@cursor.com>

sfc-gh-kganesan · 2026-05-29T21:41:00Z

cc @sfc-gh-truwase @zw0610 — opened the upstream RFC for this abstraction at verl-project#6537 (modeled after the [RFC] issues on verl-project, includes architecture diagrams, contract, worked Arctic example, migration path, and open questions). Comments + alternatives welcome there.

@zw0610

Addresses @zw0610's structural review on verl-project#6422: - #2: verl/trainer/ppo/arctic_rl_client.py -> verl/workers/remote_client/arctic_rl_client.py - #3: verl/workers/rollout/arctic_rollout/ -> verl/workers/rollout/remote_rollout/arctic_rollout/ - #4: TODO(@zw0610) on RolloutReplicaRegistry registrations in verl/workers/rollout/replica.py:360 about lazy-init each option to avoid pulling in vLLM + SGLang + TRT-LLM transitive deps eagerly. Pure code-motion + a TODO; no behavior change. Imports inside the moved files and in workers/rollout/replica.py updated. Co-authored-by: Cursor <cursoragent@cursor.com>

sfc-gh-kganesan force-pushed the karthik/remote-backend-abstraction-public branch from 1becf47 to d71d8a6 Compare May 19, 2026 23:19

sfc-gh-truwase reviewed May 20, 2026

View reviewed changes

sfc-gh-kganesan force-pushed the karthik/remote-backend-abstraction-public branch 2 times, most recently from 0da961c to 34d4c31 Compare May 20, 2026 20:16

sfc-gh-truwase reviewed May 20, 2026

View reviewed changes

sfc-gh-kganesan force-pushed the karthik/remote-backend-abstraction-public branch from 34d4c31 to d55fe02 Compare May 20, 2026 21:27

sfc-gh-kganesan changed the base branch from v0.7.1 to arctic_rl_share_v0.7.1 May 20, 2026 21:50

sfc-gh-truwase approved these changes May 20, 2026

View reviewed changes

sfc-gh-truwase merged commit 61ed850 into Snowflake-AI-Research:arctic_rl_share_v0.7.1 May 20, 2026
2 of 3 checks passed

This was referenced May 23, 2026

Restructure remote_backend per @zw0610 review (stacked on #3) #4

Merged

[RFC] Generic RemoteBackend abstraction for out-of-process RL backends verl-project/verl#6537

Open

	trainer.n_gpus_per_node=$NGPU_PER_NODE \
	trainer.n_gpus_per_node=1 \

Conversation

sfc-gh-kganesan commented May 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why

What

Plug-in shape

Test plan

Notes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sfc-gh-kganesan commented May 29, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

sfc-gh-kganesan commented May 19, 2026 •

edited

Loading