[6/N] (Elastic EP) Recover failed ranks

[UNIDY2002]

Motivation

As a follow-up to #11657, this PR enables SGLang to dynamically add back previously failed processes, recovering the optimal throughput. This also marks the completion of our planned support for elasticity and fault tolerance, as outlined in the roadmap in #8961.

The core idea is as follows. When a rank failure is detected, the data-parallel controller automatically re-launches the corresponding process. Meanwhile, the remaining healthy processes continue serving ongoing inference requests and periodically poll the status of the relaunched process. Once the new process becomes ready, it is able to seamlessly rejoin the existing process group. With this design, disruption to ongoing inference is minimized.

flowchart TD
    X[Healthy processes] --> Y[Failed process found]
    Y --> A
    A[Normal Inference Iteration] --> B{Is new process ready?}
    B -- No --> C[Run inference normally]
    C --> A
    B -- Yes --> D[Join new process into process group]
    D --> C

    U[New process] -->E
    E[Process Relaunched] --> F[Setup Python modules, CUDA, etc.]
    F --> H[Initialize distributed state]
    H --> D

Loading

Modifications

• Add a store-true flag --mooncake-extend-group to server-args, indicating whether the process is relaunched to extend the ongoing group
• In parallel-state, pass the flag to Mooncake during process group initialization
• During inference, poll the status of new processes, and extend the ongoing process groups when necessary
• In data-parallel-controller, relaunch processes after failure is detected.

Accuracy Tests

test_mooncake_ep_small.py should pass.

Benchmarking and Profiling

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.
• Work with maintainers to merge your PR. See the PR Merge Process

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!