[Fix] retract-mode flush_cache no-op crash

[Zhichenzzz]

Summary

Companion to sgl-project/sglang#31962. sglang's flush_cache() refuses to run while
waiting_queue is non-empty — but pause_generation(mode="retract") re-queues every retracted
request into exactly that queue, and under high concurrency the queue also always holds
never-scheduled requests. So the flush that weight updates depend on could never succeed, and
retract mode was effectively unusable for RL training. The real fix is sglang-side (exempt the
paused-engine waiting_queue from the idle check); this PR carries the small miles-side
companions:

• sglang_engine.py flush_cache(): back off on 400 responses too — "Cache not flushed" is
  a normal 400 response, not an exception, so without the sleep the 60 retries burned through in
  under a second instead of giving in-flight generation ~60s to drain. Also surface the last
  response body in the eventual TimeoutError instead of discarding it (Timeout while flushing cache: Flush cache failed. instead of a bare timeout).
• update_weight_from_distributed/{delta.py,mixin.py}: if mode not in ("in_place"): is a
  parenthesized string, not a tuple — a substring check, not membership. Harmless today (no valid
  mode is a proper substring of "in_place") but a latent footgun; fixed to mode != "in_place".

For the megatron backend the sglang bug was a crash, not a silent no-op: the bucket-transfer
weight update leaves flush_cache=True at its default, hits sglang's internal
assert flush_cache_success, and the whole scheduler process dies with SIGQUIT.

A third equivalent fix (torchtitan_utils hardcodes mode="retract" instead of reading
args.pause_generation_mode) is tracked separately on feat/torchtitan_exp since that file
doesn't exist on main yet.

Test plan

• Baseline reproduction (megatron backend, fully-async, Qwen3.5-4B on 8xH200): without the
  sglang fix, the first deep-queue update_weights under --pause-generation-mode retract
  dies exactly as described (Cache not flushed because there are pending requests retried
  until TimeoutError); with [sglang-miles] Fix flush_cache() no-op after pause_generation in retract sgl-project/sglang#31962 applied, the same run completes.
• Production-scale stress (megatron, fully-async, 512 in-flight requests, global-batch 128
  → 4 weight broadcasts per rollout, 20 rollouts ≈ 80 retract-mode broadcasts, checkpoint
  persistence + pause-the-world eval): zero flush warnings, eval version pinning intact,
  training curve statistically identical to an in_place control run.
• MoE + TP4/EP4 (Qwen3-30B-A3B, megatron actor TP4/EP4 + sglang --tp 4 --ep-size 4):
  retract-mode weight updates and pause-the-world eval green end to end.
• sglang-side regression tests live in the paired PR (5 scheduler cases incl. the
  never-scheduled-backlog scenario).

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.