Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
c5c9f0a
[Reload] Observe the required key set at first load, not predict it
new-TonyWang Jul 24, 2026
0c264e1
[Model Loader] Validate reload weight manifests
new-TonyWang Jul 25, 2026
cc5975a
[Model Loader] Make reload completion manifest-driven
new-TonyWang Jul 25, 2026
d055412
[Model Loader] Add structured load receipts
new-TonyWang Jul 25, 2026
98d7a95
[Model Loader] Detect load receipt key collisions
new-TonyWang Jul 25, 2026
61aac7c
[Reload] Document weight update transaction design
new-TonyWang Jul 25, 2026
7370766
[Reload] Translate transaction design to English
new-TonyWang Jul 25, 2026
594a5ec
[Reload] Refocus transaction design on correctness invariants
new-TonyWang Jul 25, 2026
152c165
[Reload] Map transaction checks to RFC failure categories
new-TonyWang Jul 25, 2026
4ee39b4
[Docs] Clarify reload transaction coverage and entry points
new-TonyWang Jul 25, 2026
7f259cb
[Reload] Add explicit update scopes
new-TonyWang Jul 28, 2026
34f82d9
[Reload] Probe dummy load manifests
new-TonyWang Jul 28, 2026
6d7fc21
[Reload] Gate dummy load probing with config
new-TonyWang Jul 28, 2026
8f70b0e
[Reload] Expose partial update baseline
new-TonyWang Jul 29, 2026
1196cc7
[Reload] Decouple receipt flow from arena
new-TonyWang Jul 29, 2026
895eec0
[Reload] Clean up receipt branch after split
new-TonyWang Jul 29, 2026
8dbfbda
[Docs] Describe load receipt and update scope flows
new-TonyWang Jul 29, 2026
0e8409d
[Docs] Fix update scope Mermaid rendering
new-TonyWang Jul 29, 2026
9517b60
[Reload] Add partial LoRA patch manifests
new-TonyWang Jul 29, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
527 changes: 527 additions & 0 deletions docs/design/TRANSACTION.md

Large diffs are not rendered by default.

442 changes: 442 additions & 0 deletions docs/design/UPDATE_SCOPE.md

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions docs/serving/offline_inference.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ For further details on metrics, please refer to [this page](../design/metrics.md
For further details on Weight Transfer, please refer to [this page](../training/weight_transfer/README.md).

- `LLM.init_weight_transfer_engine` - Initializes the weight transfer engine for RL training.
- `LLM.get_weight_update_manifest` - Lists updatable model weights and LoRA adapters.
- `LLM.start_weight_update` - Starts a new weight update cycle.
- `LLM.update_weights` - Updates the model weights.
- `LLM.finish_weight_update` - Finishes the current weight update cycle.
Expand Down
1 change: 1 addition & 0 deletions docs/serving/online_serving/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -171,6 +171,7 @@ For further details on Weight Transfer, please refer to [this page](../../traini
- `/resume` - Resume generation
- `/is_paused` - Check if generation is paused
- `/init_weight_transfer_engine` - Initialize weight transfer engine for RLHF
- `/weight_update_manifest` - List updatable model weights and LoRA adapters
- `/start_weight_update` - Prepares the inference engine for a weight update.
- `/update_weights` - Update model weights (can alter model behavior)
- `/finish_weight_update` - Finalizes the weight update
Expand Down
32 changes: 32 additions & 0 deletions docs/training/weight_transfer/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ When running vLLM as an HTTP server, the following endpoints are available for w
| Endpoint | Method | Description |
| -------- | ------ | ----------- |
| `/init_weight_transfer_engine` | POST | Initialize the weight transfer engine with backend-specific info |
| `/weight_update_manifest` | GET | Get updatable model-weight and LoRA manifests |
| `/start_weight_update` | POST | Start a weight update |
| `/update_weights` | POST | Transfer a batch of weights with backend-specific metadata |
| `/finish_weight_update` | POST | Finish the weight update and run post-processing |
Expand All @@ -61,6 +62,37 @@ When running vLLM as an HTTP server, the following endpoints are available for w
!!! note
The HTTP weight transfer endpoints require `VLLM_SERVER_DEV_MODE=1` to be set.

### Discovering legal partial-update scopes

Call `GET /weight_update_manifest` after model loading. The response includes
separate `model_weights` and `lora_adapters` fields. `model_weights` contains
all source names observed during the initial checkpoint load and
`atomic_source_groups`. The corresponding `atomic_update_scopes` entries can be
sent directly to `/start_weight_update`. A larger legal partial scope is a union
of complete atomic groups:

```json
{
"kind": "base_checkpoint",
"mode": "partial",
"source_names": [
"model.layers.0.input_layernorm.weight",
"model.layers.0.post_attention_layernorm.weight"
]
}
```

The groups are merged across workers, so TP, PP, and EP closure constraints
are preserved. Use the scope only when `model_weights.ready` is `true`.
A dummy model without metadata probing returns a provisional baseline until it
has completed one real full base-weight update.

Each entry in `lora_adapters` includes adapter identity, generation, the union
of rank-local runtime module names, and templates for replacement, partial
patch, and removal. A patch must use the current `base_generation`, select
complete runtime modules, and provide complete A/B pairs for every selected
module. Unselected modules retain their current weights.

## Trainer-Side API

Both backends provide static methods that the trainer calls to send weights. The general pattern is:
Expand Down
46 changes: 46 additions & 0 deletions tests/distributed/test_weight_transfer.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
"""

import pickle
from types import SimpleNamespace
from unittest.mock import MagicMock

import pybase64 as base64
Expand All @@ -18,6 +19,7 @@
from vllm.config.parallel import ParallelConfig
from vllm.config.weight_transfer import WeightTransferConfig
from vllm.distributed.weight_transfer import WeightTransferEngineFactory
from vllm.distributed.weight_transfer.base import WeightTransferEngine
from vllm.distributed.weight_transfer.ipc_engine import (
IPCWeightTransferEngine,
IPCWeightTransferInitInfo,
Expand Down Expand Up @@ -91,6 +93,50 @@ def create_mock_vllm_config(
return vllm_config


class TestDeclaredSourceManifest:
def _state(self, *, required: bool = True):
return SimpleNamespace(
_requires_declared_source_manifest=required,
_expected_source_names=None,
_received_source_names=set(),
)

def test_complete_chunked_manifest(self):
state = self._state()
WeightTransferEngine.observe_source_manifest(
state, ["q_proj.weight"], ["q_proj.weight", "k_proj.weight"]
)
WeightTransferEngine.observe_source_manifest(
state, ["k_proj.weight"], None
)
WeightTransferEngine.finish_source_manifest_validation(state)

def test_missing_packed_fragment_fails(self):
state = self._state()
WeightTransferEngine.observe_source_manifest(
state,
["q_proj.weight", "v_proj.weight"],
["q_proj.weight", "k_proj.weight", "v_proj.weight"],
)
with pytest.raises(RuntimeError, match=r"k_proj\.weight"):
WeightTransferEngine.finish_source_manifest_validation(state)

def test_dummy_first_transfer_requires_declaration(self):
state = self._state()
WeightTransferEngine.observe_source_manifest(
state, ["q_proj.weight"], None
)
with pytest.raises(RuntimeError, match="authoritative expected"):
WeightTransferEngine.finish_source_manifest_validation(state)

def test_later_transfer_keeps_backward_compatibility(self):
state = self._state(required=False)
WeightTransferEngine.observe_source_manifest(
state, ["q_proj.weight"], None
)
WeightTransferEngine.finish_source_manifest_validation(state)


# --- Unit Tests: NCCLWeightTransferUpdateInfo Validation ---


Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project

import asyncio
import json
from types import SimpleNamespace
from unittest.mock import AsyncMock

from vllm.entrypoints.serve.dev.rlhf.api_router import (
get_weight_update_manifest,
)


def test_weight_update_manifest_endpoint_returns_engine_manifest() -> None:
manifest = {
"model_weights": {
"ready": True,
"scope_template": {
"kind": "base_checkpoint",
"mode": "partial",
"source_names": [],
},
"source_names": ["layer.weight"],
"atomic_source_groups": [["layer.weight"]],
"workers": [],
"reason": None,
},
"lora_adapters": [],
}
engine = SimpleNamespace(
get_weight_update_manifest=AsyncMock(return_value=manifest)
)
request = SimpleNamespace(
app=SimpleNamespace(state=SimpleNamespace(engine_client=engine))
)

response = asyncio.run(get_weight_update_manifest(request))

assert response.status_code == 200
assert json.loads(response.body) == manifest
engine.get_weight_update_manifest.assert_awaited_once_with()
Loading
Loading