Skip to content

[Feature] Add fault tolerance with retry and MC2 scale-down on Ascend NPU #16145

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
fangyuchu wants to merge 42 commits into
base: main
Choose a base branch
from
Open

[Feature] Add fault tolerance with retry and MC2 scale-down on Ascend NPU #16145

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
00a9238
[Feature] Add retry-based fault tolerance for transient NPU fault rec…
zWaNg3 Jul 27, 2026
22617e0
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jul 31, 2026
d2665de
[Refactor] Reuse upstream GPUWorkerSentinel in NPU fault tolerance
205150940 Aug 6, 2026
a08b4ad
[BugFix] Add the fault_tolerance test and specify the environment var…
205150940 Aug 11, 2026
eb89fc0
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 11, 2026
9cb4e2a
[CI] Add the YAML file required for fault_tolerance CI tests.
205150940 Aug 12, 2026
672ec39
[CI] Move the FT test file.
205150940 Aug 13, 2026
a09160b
[BugFix] Add sentinel __init__.py and drop init_logger to fix pre-com…
205150940 Aug 13, 2026
67176c7
[Refactor] Drop FT_COMMUNICATION_OPS_ABORT_TIMEOUT_MS env var (#106)
205150940 Aug 21, 2026
b413f11
[Refactor] rebase main
205150940 Aug 24, 2026
e888e2c
[CI] Use v2 model runner for FT fault injection after DP allreduce
205150940 Aug 24, 2026
d1974f7
[CI] Edit test_fault_tolerance_e2e code comments (#108)
205150940 Aug 25, 2026
06d955e
Add accuracy guards for fault-tolerance retry recovery (#112)
205150940 Aug 27, 2026
80b9007
[CI] Fixed pre-commit bug (#113)
205150940 Aug 27, 2026
bb30f3a
[CI] Update the estimated_times for test_fault_tolerance
205150940 Aug 27, 2026
395849a
[CI] Fix bugs in CI tests. (#114)
205150940 Aug 27, 2026
cf06122
[Refactor] Add gpu-memory-utilization in test_fault_tolerance_e2e and…
205150940 Aug 28, 2026
b493aea
[CI] Perform accuracy testing by comparing model outputs. (#118)
205150940 Aug 29, 2026
7b9befd
[CI] Fix pre-commit bug (#119)
205150940 Aug 29, 2026
7b721b3
check factual answers in ft ci test
fangyuchu Aug 31, 2026
b0f1f4b
[Refactor] Apply cpu distributed timeout in worker instead of patch (…
205150940 Sep 4, 2026
4f3e0cf
feat(ft): add scale-down fault recovery for MC2 on Ascend NPU
fangyuchu Aug 24, 2026
9474646
fix(ft): clear a poisoned NPU stream on the first device error
zWaNg3 Aug 25, 2026
79e2f52
refactor(ft): simplify elastic_info rebuild in the MC2 all2all manager
zWaNg3 Aug 26, 2026
32b6164
refactor(ft): share the orig-to-dense rank table helper
zWaNg3 Aug 28, 2026
90300e3
refactor(ft): reuse the upstream scale_down redistribution flow
zWaNg3 Aug 31, 2026
629e4f9
refactor(ft): derive the shrunk MC2 expert width from the dead mask
zWaNg3 Aug 31, 2026
d116ca2
refactor(ft): extract routing densification and document the fault ba…
zWaNg3 Aug 31, 2026
204bc11
fix(ft): update w8a8_dynamic import path
zWaNg3 Sep 7, 2026
3ccbcc7
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 7, 2026
4110307
[Refactor] Adjust to upstream timeout settings
205150940 Sep 7, 2026
484d257
refactor(ft): simplify comments per review feedback
zWaNg3 Sep 8, 2026
ee5d67c
refactor(ft): initialize per-rank expert count at model load
zWaNg3 Sep 8, 2026
08378e8
test(ft): add scale-down e2e test for MC2 on Ascend NPU (#128)
205150940 Sep 10, 2026
5f32c4a
test(ft) fix deepseekv4-flash bugs (#130)
205150940 Sep 14, 2026
a6fa6ae
[Refactor] fix(ft): bound scale_down dummy batch with gloo cpu group …
205150940 Sep 18, 2026
3295849
fix(ft): drop unused fork-only re-export in eplb_redistribute
zWaNg3 Sep 20, 2026
f0e0820
remove mrv1 support (#136)
fangyuchu Sep 21, 2026
269028e
refactor name normalization to use upstream hf_to_vllm_mapper functio…
fangyuchu Sep 22, 2026
1826b87
fix(ft): route AsyncOutput.get_output through the fault barrier (#135)
205150940 Sep 22, 2026
007542b
Speed up weights reloading for scaling down (#133)
zWaNg3 Sep 22, 2026
00c1bdc
feat(worker): warn when HCCL timeout env vars override operator timeo…
205150940 Sep 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/scripts/estimated_times.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -144,3 +144,4 @@ estimated_times:
tests/e2e/pull_request/one_card/test_thinking_budget.py: 80
tests/e2e/pull_request/one_card/test_model_runner_v1_with_device.py: 70
tests/e2e/pull_request/one_card/test_msa_index_score.py: 20
tests/e2e/pull_request/four_card/fault_tolerance/test_fault_tolerance_e2e.py: 900
Empty file added tests/e2e/pull_request/four_card/fault_tolerance/__init__.py
View file
Open in desktop
Empty file.
Loading