[docs] Add docs for new RL flows

[hao-aaron]

Purpose

Introduce new docs detailing weight sync and new pause resume, move old rlhf examples to legacy

Test Plan

Test Result

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[mergify]

Documentation preview: https://vllm--36188.org.readthedocs.build/en/36188/

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @hao-aaron.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[mergify]

Hi @hao-aaron, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

[gemini-code-assist]

Code Review

This pull request introduces comprehensive documentation for new Reinforcement Learning (RL) features, including asynchronous RL flows and weight transfer mechanisms. The new documentation is well-structured, covering the overview, API usage, and different backends like NCCL and IPC. The PR also reorganizes the example files by moving older examples to a legacy directory. The documentation additions are clear and provide important details, such as required environment variables and API behavior, which will be very helpful for users. I have reviewed the changes and found no issues.

_{Note: Security Review is unavailable for this PR.}

[mergify]

Hi @hao-aaron, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

[hmellor]

Feel free to put all the RLHF examples in examples/rlhf. It would make them nice and discoverable like the Pooling examples:

[hmellor]

You could instead link to the API docs with [`NCCLTrainerSendWeightsArgs`][vllm.distributed.weight_transfer.nccl_engine.NCCLTrainerSendWeightsArgs]

[mergify]

Hi @hao-aaron, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy failing?

mypy is run differently in CI. If the failure is related to this check, please use the following command to run it locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @hao-aaron.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[hmellor]

I just pushed a couple of changes to make the documentation generation a bit better.

Is there any reason to keep the legacy examples? They will still be present in older versions of the docs

[hao-aaron]

they could be good to have so people have an idea of how to use worker extension in case the custom weight sync doesn't work for their use case, but otherwise its not necessary. I can remove it

[mergify]

Hi @hao-aaron, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy failing?

mypy is run differently in CI. If the failure is related to this check, please use the following command to run it locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10

[hmellor]

@hao-aaron I just wanted to give you the option to recommend this, I think it looks nicer but it's up to you

Suggested change

	--weight-transfer-config '{"backend": "nccl"}'
	--weight-transfer-config.backend nccl

[hmellor]

This is a really nice improvement!

I'll start merging now and any tweaks to syntax/API reference linking can be follow ups if we want.