Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions .github/workflows/pr-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -556,10 +556,10 @@ jobs:
env:
GITHUB_COMMIT_NAME: ${{ github.sha }}_${{ github.event.pull_request.number || 'non-pr' }}
WANDB_API_KEY: ${{ secrets.WANDB_API_KEY }}
SLIME_TEST_ENABLE_INFINITE_RUN: ${{ (github.event_name == 'workflow_dispatch' && github.event.inputs.infinite_run) || 'false' }}
SLIME_TEST_USE_DEEPEP: ${{ matrix.info.use_deepep || '0' }}
SLIME_TEST_USE_FP8_ROLLOUT: ${{ matrix.info.use_fp8_rollout || '0' }}
SLIME_TEST_ENABLE_EVAL: ${{ matrix.info.enable_eval || '1' }}
VIME_TEST_ENABLE_INFINITE_RUN: ${{ (github.event_name == 'workflow_dispatch' && github.event.inputs.infinite_run) || 'false' }}
VIME_TEST_USE_DEEPEP: ${{ matrix.info.use_deepep || '0' }}
VIME_TEST_USE_FP8_ROLLOUT: ${{ matrix.info.use_fp8_rollout || '0' }}
VIME_TEST_ENABLE_EVAL: ${{ matrix.info.enable_eval || '1' }}

steps:
- name: Checkout repository
Expand Down
14 changes: 3 additions & 11 deletions docs/en/advanced/speculative-decoding.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,22 +12,14 @@ which vime forwards via `--vllm-speculative-config`. For models with MTP layers
--vllm-speculative-config '{"method":"mtp","num_speculative_tokens":3}'
```

To use a separately trained draft model, set `model` (and optionally `draft_tensor_parallel_size`)
in the same JSON:
To use a separately trained draft model (e.g., one trained with [vllm-project/speculators](https://github.com/vllm-project/speculators)),
set `model` (and optionally `draft_tensor_parallel_size`) in the same JSON:

```bash
--vllm-speculative-config '{"method":"eagle","num_speculative_tokens":3,"model":"/your/draft/model/path"}'
```

To train a draft model from scratch, see [TorchSpec](https://github.com/lightseekorg/TorchSpec)
and [vllm-project/speculators](https://github.com/vllm-project/speculators).
TorchSpec provides torch-native, disaggregated draft training.
Speculators supports EAGLE-3, DFlash, and MTP-style drafts, ships pre-trained
checkpoints on Hugging Face (see the `RedHatAI/*-speculator.*` collection), and
saves drafts in a format that `vllm serve <speculator_model>` can deploy directly.

For the full list of `SpeculativeConfig` fields (including `disable_by_batch_size`,
`acceptance_method`, draft TP, etc.), see vLLM's speculative-decoding
For detailed parameter meanings and configuration, see vLLM's speculative-decoding
[documentation](https://docs.vllm.ai/en/latest/features/speculative_decoding/).

## Online SFT for the Draft Model
Expand Down
45 changes: 0 additions & 45 deletions docs/en/developer_guide/debug.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,48 +67,3 @@ When running large scale RL, we will occationally meet the IMA in vLLM, there ar
4. Try CUDA Core Dump to find the error kernel

We recommend reading the blog from the vLLM team: [CUDA Core Dump: An Effective Tool to Debug Memory Access Issues and Beyond](https://blog.vllm.ai/2025/08/11/cuda-debugging.html)

## Step-by-Step Debugging with Ray Distributed Debugger

Ray provides a [distributed debugger](https://docs.ray.io/en/latest/ray-observability/ray-distributed-debugger.html) based on debugpy that lets you set breakpoints in the driver process and step through code interactively.

1. Install debugpy:

```bash
pip install debugpy==1.8.0
```

2. Enable `RAY_DEBUG_POSTMORTEM` in your launch script:

```bash
export RAY_DEBUG_POSTMORTEM=1

RUNTIME_ENV_JSON="{
\"env_vars\": {
...
\"RAY_DEBUG_POSTMORTEM\": \"${RAY_DEBUG_POSTMORTEM:-0}\"
}
}"

ray job submit --address="http://127.0.0.1:8265" \
--runtime-env-json="${RUNTIME_ENV_JSON}" \
-- python3 train.py [args...]
```

3. Add `ray.init()` before `breakpoint()` in `train.py`:

```python
if __name__ == "__main__":
ray.init()
breakpoint()
args = parse_args()
train(args)
```

`ray.init()` is required because the distributed debugger depends on `core_worker`, which is only available after Ray initialization. Without it, `breakpoint()` raises `AttributeError: 'Worker' object has no attribute 'core_worker'`.

4. Connect via VS Code:

Install the [Ray Distributed Debugger](https://marketplace.visualstudio.com/items?itemName=ray-project.ray-distributed-debugger) extension in VS Code. Run your launch script to submit the job. Once the job hits `breakpoint()`, open the Ray Dashboard panel in VS Code and click the active breakpoint to attach the debugger. You can then step through code, inspect variables, and set additional breakpoints directly in the editor.

> **Note**: Remove `ray.init()` and `breakpoint()` after debugging. An explicit `ray.init()` without arguments may cause issues in multi-node training where Ray injects specific namespace and runtime environment configurations via `ray job submit`.
Loading
Loading