Skip to content

ci: keep nightly suites running after a test fails - #282

Merged
Rockdu merged 1 commit into
mainfrom
ci/nightly-continue-on-error
Oct 4, 2026
Merged

Rockdu merged 1 commit into
mainfrom
ci/nightly-continue-on-error

Conversation

@Rockdu

@Rockdu Rockdu commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator

What

  • Nightly (schedule) runs pass --continue-on-error to every run_suite.py suite, so a failing test no longer stops the rest of its suite.
  • New workflow_dispatch input continue_on_error (default false) lets a manual run do the same.

Why

Each suite stops at its first failing test. Since the 2026-09-27 nightly, test_krea2_nft_ocr_2xGPU has failed in stage-c-3-gpu-h200, so test_sd3_ocr_grpo_2xGPU, which runs after it, has not run on any nightly since. A nightly exists to report every drift at once; fail-fast hides everything behind the first failure.

The job still fails: run_unittest_files marks the suite failed on any failing test, and --continue-on-error only skips the break.

Files

  • .github/workflows/pr-test.yml: the new dispatch input, the flag on all six suite commands, and the header note on nightly behaviour.

Checklist

  • pre-commit run --all-files passes — not run
  • Added/updated tests for new behaviour — no tests: workflow-only change; the workflow YAML parses and all six suite jobs carry the flag
  • pytest -x is green — not run, no Python changed
  • If launch flags changed, python3 train.py --help still parses — n/a
  • If a public flag was added, it appears in the CLI reference docs — n/a, no CLI flag added
  • If an example was added, it has a real walkthrough — n/a

Signed-off-by: rockdu <kangrdu@gmail.com>
@Rockdu
Rockdu merged commit f8c8780 into main Oct 4, 2026
11 of 12 checks passed
OnePunchMonk pushed a commit to OnePunchMonk/miles_diffusion that referenced this pull request Oct 5, 2026
OnePunchMonk pushed a commit to OnePunchMonk/miles_diffusion that referenced this pull request Oct 5, 2026
OnePunchMonk pushed a commit to OnePunchMonk/miles_diffusion that referenced this pull request Oct 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant