Adding GPT-NeoX by aflah02 · Pull Request #164 · sgl-project/sglang

aflah02 · 2024-02-07T21:08:11Z

I followed along the instructions here to add GPT-NeoX support which would bring support for the Pythia model family and other similar architecture models.

Reference: #157 (comment)

FIXED (Keeping Logs for Future Reference):
I was able to debug most errors but I'm stuck on this particular error which happens once I start requesting on the endpoint (i.e. it loads correctly I assume) -

INFO:     Started server process [1344199]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:30013 (Press CTRL+C to quit)
INFO:     127.0.0.1:37998 - "GET /get_model_info HTTP/1.1" 200 OK
new fill batch. #seq: 1. #cached_token: 0. #new_token: 17. #remaining_req: 0. #running_req: 0
Exception in ModelRpcClient:
Traceback (most recent call last):
  File "/NS/llm-1/work/afkhan/sglang/python/sglang/srt/managers/router/model_rpc.py", line 165, in exposed_step
    self.forward_step()
  File "/NS/llm-1/nobackup/afkhan/anaconda3/envs/sglangfact/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/NS/llm-1/work/afkhan/sglang/python/sglang/srt/managers/router/model_rpc.py", line 180, in forward_step
    self.forward_fill_batch(new_batch)
  File "/NS/llm-1/work/afkhan/sglang/python/sglang/srt/managers/router/model_rpc.py", line 369, in forward_fill_batch
    logits, (logprobs, normalized_logprobs) = self.model_runner.forward(
  File "/NS/llm-1/work/afkhan/sglang/python/sglang/srt/managers/router/model_runner.py", line 486, in forward
    return self.forward_extend(**kwargs)
  File "/NS/llm-1/nobackup/afkhan/anaconda3/envs/sglangfact/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/NS/llm-1/work/afkhan/sglang/python/sglang/srt/managers/router/model_runner.py", line 391, in forward_extend
    return self.model.forward(input_ids, input_metadata.positions, input_metadata)
  File "/NS/llm-1/work/afkhan/sglang/python/sglang/srt/models/gpt_neox.py", line 236, in forward
    return self.logits_processor(
  File "/NS/llm-1/nobackup/afkhan/anaconda3/envs/sglangfact/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/NS/llm-1/nobackup/afkhan/anaconda3/envs/sglangfact/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/NS/llm-1/work/afkhan/sglang/python/sglang/srt/layers/logits_processor.py", line 32, in forward
    last_logits = torch.matmul(last_hidden, weight.T)
  File "/NS/llm-1/nobackup/afkhan/anaconda3/envs/sglangfact/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1695, in __getattr__
    raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'ParallelLMHead' object has no attribute 'T'

Exception in ModelRpcClient:
Traceback (most recent call last):
  File "/NS/llm-1/work/afkhan/sglang/python/sglang/srt/managers/router/model_rpc.py", line 165, in exposed_step
    self.forward_step()
  File "/NS/llm-1/nobackup/afkhan/anaconda3/envs/sglangfact/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/NS/llm-1/work/afkhan/sglang/python/sglang/srt/managers/router/model_rpc.py", line 180, in forward_step
    self.forward_fill_batch(new_batch)
  File "/NS/llm-1/work/afkhan/sglang/python/sglang/srt/managers/router/model_rpc.py", line 369, in forward_fill_batch
    logits, (logprobs, normalized_logprobs) = self.model_runner.forward(
  File "/NS/llm-1/work/afkhan/sglang/python/sglang/srt/managers/router/model_runner.py", line 486, in forward
    return self.forward_extend(**kwargs)
  File "/NS/llm-1/nobackup/afkhan/anaconda3/envs/sglangfact/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/NS/llm-1/work/afkhan/sglang/python/sglang/srt/managers/router/model_runner.py", line 391, in forward_extend
    return self.model.forward(input_ids, input_metadata.positions, input_metadata)
  File "/NS/llm-1/work/afkhan/sglang/python/sglang/srt/models/gpt_neox.py", line 236, in forward
    return self.logits_processor(
  File "/NS/llm-1/nobackup/afkhan/anaconda3/envs/sglangfact/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/NS/llm-1/nobackup/afkhan/anaconda3/envs/sglangfact/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/NS/llm-1/work/afkhan/sglang/python/sglang/srt/layers/logits_processor.py", line 32, in forward
    last_logits = torch.matmul(last_hidden, weight.T)
  File "/NS/llm-1/nobackup/afkhan/anaconda3/envs/sglangfact/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1695, in __getattr__
    raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'ParallelLMHead' object has no attribute 'T'

/NS/llm-1/work/afkhan/sglang/python/sglang/srt/managers/router/model_rpc.py:204: UserWarning: Warning: available_size=714944, max_total_num_token=714961
KV cache pool leak detected!
  warnings.warn(
/NS/llm-1/work/afkhan/sglang/python/sglang/srt/managers/router/model_rpc.py:204: UserWarning: Warning: available_size=714944, max_total_num_token=714961
KV cache pool leak detected!
  warnings.warn(

Any idea what might be going wrong? It seems that the error is related to the LogitProcessor which I'm not very familiar with. I've tried to copy the logic from the llama implementation for the same

aflah02 · 2024-02-07T21:15:13Z

Update: I just noticed the missing part and changed that which fixes the old issue but now I get a new error -

Exception in ModelRpcClient:
Traceback (most recent call last):
  File "/NS/llm-1/work/afkhan/sglang/python/sglang/srt/managers/router/model_rpc.py", line 165, in exposed_step
    self.forward_step()
  File "/NS/llm-1/nobackup/afkhan/anaconda3/envs/sglangfact/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/NS/llm-1/work/afkhan/sglang/python/sglang/srt/managers/router/model_rpc.py", line 192, in forward_step
    self.forward_decode_batch(self.running_batch)
  File "/NS/llm-1/work/afkhan/sglang/python/sglang/srt/managers/router/model_rpc.py", line 429, in forward_decode_batch
    next_token_ids, next_token_probs = batch.sample(logits)
  File "/NS/llm-1/work/afkhan/sglang/python/sglang/srt/managers/router/infer_batch.py", line 452, in sample
    sampled_index = torch.multinomial(probs_sort, num_samples=1)
RuntimeError: probability tensor contains either `inf`, `nan` or element < 0

aflah02 · 2024-02-07T21:24:45Z

I did some further testing. It runs perfectly for - https://github.com/aflah02/sglang/blob/main/examples/usage/choices_logprob.py
But fails for https://github.com/aflah02/sglang/blob/main/examples/quick_start/srt_example_chat.py with the error above
Seems like the issue might be elsewhere

aflah02 · 2024-02-09T20:00:03Z

@merrymercy Any thoughts? Not sure why one tutorial works while the other doesn't

merrymercy · 2024-02-11T14:14:18Z

@aflah02

Can you try this tutorial? https://github.com/sgl-project/sglang/blob/main/examples/quick_start/srt_example_complete.py
The chat example does not work properly, possibly due to the vicuna chat template. The default chat template is vicuna, but GPT-NeoX has not been tuned on that template.
Can you add more print statements to see where the nan comes from? Does it occur in early transformers layers? Does it only occur in the last layer?

aflah02 · 2024-02-20T18:38:32Z

@merrymercy
For Part 2 It seems that the error mainly occurs in the last few layers/last layer. Some of the logs are here for the chat example - logs.txt
The first tutorial also gives a similar error

aflah02 · 2024-03-01T18:11:23Z

@merrymercy Any thoughts on what might be going wrong here? I don't know whether a template can make such breaking issues

merrymercy · 2024-03-11T02:19:36Z

@aflah02 I have no idea. I typically debug these kinds of wired bugs by comparing intermediate tensors layer by layer between sglang and huggingface/vllm implementations, similar to your print statements.

aflah02 · 2024-06-12T14:30:28Z

@merrymercy Sorry for being inactive, life got really busy the past few months. I don't have the bandwidth nowadays to take this on and if you want to then feel free to work on this

merrymercy · 2024-06-12T22:39:49Z

I will close this for now

…project#164)

… doc, outbound observability Subagent-driven audit of the T36 cleanup commit + answering the follow-up question 'is unbounded outbound queue OK?' (yes, by design — but only with monitoring + sustained-escalation backstop). A-block (immediate fixes): - verify/admission_controller/verify.py:181 + regression_probe.py:311 removed orphaned sched.aclose() calls (KvScheduler no longer owns an httpx client post-T36 cleanup) - verify/t42/verify.py: import Optional (was working only via 'from __future__ import annotations' stringification) - verify/t36/verify.py A6: removed dead 'no sync POST' assertion (structurally unreachable post-cleanup) - verify/t42/README.md + verify/t23_t37_apply_failed/README.md: updated to describe post-cleanup state (no more _record_skips, no more migrate_skipped sync line — apply_failed is the per- event grep target) B-block (doc + observability for the unbounded queue): - daemon/outbound.py: 'unbounded by design' docstring explaining the choice — DESIGN §6 non-blocking producer, arrival rate capped by sglang throughput, escalation in sgl-project#164 is the hard backstop. No magic in the impl, just decision documentation. - daemon/_observability.py: two new metric rings outbound_queue_depth + outbound_oldest_age_ms, sampled by OutboundQueue worker per pop. Emitted on daemon_obs_summary line (outbound_queue_depth_{p50,p99,max}, outbound_oldest_age_ ms_{p50,p99,max}) so operator alerts on quantiles. No hard- coded thresholds in the daemon — operator tunes in alerting tool. - daemon/outbound.py worker: per-pop record_outbound() call to the observability instance. NO in-process watchdog warning log (kept impl minimal — operator's metric pipeline already has depth+age; hardcoded code-level threshold would lock deploys out of tuning). DESIGN.md (sustained-load escalation tier, in support of sgl-project#164): - L2301 'Two fault classes' entry amended to add 'sustained- escalation tier'. Per-event the two classes (deployment-bug fatal, load absorb) don't blur; but in aggregate, sustained load failure crosses to fatal via the crash-only-software pattern. Daemon's silently-running-degraded forever is NOT a valid state. Test updates (all pass after trim — no watchdog warning => no A7 stage to test it): verify/t42: stage A3 expanded to include the two new ring keys in the required summary set; new stage B12 exercises record_outbound + asserts the new emit_summary fields. verify/t36: 8 stages green (A7 watchdog stage removed when the watchdog warning was removed for minimalism). verify/t23_t37_apply_failed: 7 stages green (A4 now tests the apply_failed structured-metric line; production handler used via attach_apply_failed_handler instead of inline replica). verify/admission_controller: orphan-ref fixes only. The sustained-escalate impl itself is opened as sgl-project#164 (CLI flags, counter, fatal call site, /health degraded reporting, verify subprocess test). This commit closes the audit's documentation + observability requirements so sgl-project#164 can land cleanly on top. Logs: verify/t36/results/20260601_t36_audit_163_pass.log (8/8) verify/t42/results/20260601_t42_audit_163_pass.log (21/21) verify/t23_t37_apply_failed/results/20260601_t23_t37_audit_163_pass.log (7/7)

…sustained tier) Closes the "running degraded forever" anti-pattern. When BOTH OutboundQueue.consecutive_failures >= escalate_failures (default 100) AND oldest_pending_batch_age_ms >= escalate_oldest_age_s * 1000 (default 300 s) hold simultaneously, the daemon self-kills via fatal('sglang_sustained_unreachable', ...) so the supervisor restarts. Crash-only software (Candea & Fox); silently eating memory + doing no useful work is not a valid state. Both-axis trigger guards against false positives: * Low-traffic dead-sglang: consec rockets but oldest_age stays small because failed POSTs drain as fast as they fail → no fatal * All-success but slow POSTs: oldest_age can grow under burst but consec stays 0 → no fatal Counted as failures (increment consec): 4xx / 5xx / transport exception. 2xx resets consec to 0. CLI: --sustained-escalate-fails (default 100), --sustained-escalate-age-s (default 300.0). /health body now carries outbound_consecutive_failures + outbound_oldest_age_ms so k8s readiness probes (and ops) can grep current state without waiting for the crash. HTTP status stays 200 — the restart signal is the fatal() exit, not health failure. verify/t164/: 6 stages, all GREEN. * A0 consec resets on 2xx * A1 consec increments on 5xx / 4xx / transport-exc * A2 high consec alone does NOT escalate (low-traffic safe) * A3 high oldest_age alone does NOT escalate (success path) * A4 /health body carries outbound counters * B0 subprocess: both thresholds trip → fatal + forensic dump Regression sanity: T22 / T36 / T42 / T23+T37 / T43 all still green.

…s against real sglang+daemon stack The ONLY verify that launches a full sglang + daemon process pair and pushes traffic through. Per-task verifies (T7/T8/T36/T42/T164/…) mock sglang; this one tests the actual wire path Run K uses. Six flavors (60s shared sglang+daemon stack except F): A proxy hot-path under load — 24 concurrent chats × 60s B state-dump under sustained traffic — /aginfer/state @ 2 Hz under chat load C event-router fan-in throughput — 200 webhook events D migrate under traffic — outbound + chat concurrently E threshold PUT atomicity — PUT/GET concurrent with traffic F dead-sglang resilience — separate daemon vs dead sglang port Run4 (2026-06-01): all 6 PASS. Headline: A: 10488 req, 100% success, p99=272 ms (Qwen3-0.6B) B: 159 samples, p99_max=33.9 ms — UNDER sgl-project#160's 50 ms F3-revisit ceiling C: 200/200 events accepted D: 200/200 migrate batches OK E: 268/268 PUTs OK, 0 torn reads of daemon's GET /aginfer/thresholds F: daemon stays alive, /health responsive, no false-positive fatal State worth recording (caught during wire-up): * SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 required; default HiRadixCache returns {"unsupported_tree_cache":...} for /aginfer/state * --aginfer-notify-url required for sglang's update_aginfer_thresholds to do anything (otherwise returns ok=False reason="no webhook firer") * Daemon MUST launch before sglang — sglang's T22 sgl-project#165 bootstrap GETs daemon/aginfer/thresholds at launch and halts on failure (deployment-ordering bug per daemon/aginfer_webhook.py) F's positive-case integration is structurally hard from the event path (state-fetch fails before outbound queue populates); T164 stages B0/C0 own the positive integration of escalate-to-fatal. F here covers the complementary negative (low-traffic dead-sglang must NOT escalate) — DESIGN §10 contract that sgl-project#164's both-axis trigger was designed to prevent false-positives on. Side-finding worth a follow-on: B's p99_max=34 ms beats sgl-project#160's 50 ms F3-revisit threshold under realistic load. Earlier sgl-project#160 trigger may have been a different config; worth re-examining whether sgl-project#160 is still load-bearing.

… — un-clog the hints flood Root cause of the residual migrate reject rate (post-sgl-project#224): the daemon pushes a hints PUT every event (T40), so hints are ~99% of outbound traffic (~11.7k vs ~130 migrate/cycle). The outbound channel is single-flight at sglang, so every time-sensitive migrate waited ~1.8s behind that idempotent flood in the FIFO queue, ageing until the radix tree diverged and the apply-time leaf check rejected it. Transport is not the bottleneck (localhost sub-ms) and daemon-side concurrency does not help (sglang serialises apply via one communicator) -- the fix is to stop the flood from blocking migrates. sgl-project#228 (latency, the relief lever): the worker drains the queued burst each wake and emits at most ONE dispatch per endpoint by temporal semantics -- hints merge to one PUT (highest stamp per hash; sglang is itself overwrite-by-stamp), migrates stay individual (rare; FIFO + per-POST escalation preserved), program_paused coalesced by pid. Order: program_paused -> migrate -> hints, so time-sensitive intents never queue behind the flood. current_oldest_pending_age_ms() now also counts the in-flight drained burst so /health backlog stays accurate. sgl-project#227 (correctness bound, generous): a migrate decided on a state snapshot is a prediction with a validity horizon; the worker drops one aged past AGINFER_MIGRATE_FRESHNESS_MS (default 30s -- a pathological- spike floor, NOT a tight knob; the coalescing removes normal-operation latency). Proactive complement to the reactive APPLY_FAILED re-issue. Latency proof (verify/outbound_coalesce B0/B1): with 200 hints queued ahead of a migrate, the migrate is dispatch call #0 (not sgl-project#201) and the 200 hints collapse to 1 PUT; wall-clock migrate latency mean 0.2ms over 3 trials vs the N*apply serial floor of 1000ms. Audit (no Critical/Major): hardened hints merge to max-stamp (order- independent); documented that the sgl-project#164 stall backstop relies on the always-on hints canary (a stale-dropped migrate emits no failure signal). task_done accounting, escalation, /health live-peek all preserved (t164/t36 green). Tests: new verify/outbound_coalesce (7 stages: coalesce correctness + latency proof). 17/17 touched suites green (t36 t164 t41 t187 t30 t22 t24 t23_t37 kv_scheduler_value_rule joint_decide admission_controller t34 t16 dump_leaf_flags program_tracker t21). DESIGN §6 updated. verify/t40 left as-is (pre-existing stub staleness, tracked sgl-project#229).

…gged single-flight) Live a3 cycle-2 evidence: a migrate BURST (134 in one cycle, vs cycle-1's 6) dispatched INDIVIDUALLY re-clogged the single-flight outbound channel exactly like the pre-sgl-project#228 hint flood — oldest-age climbed to ~700ms and remove_hbm_not_device_leaf rejects returned (the migrates aged in the queue). sgl-project#228 had coalesced only hints; under high migrate volume the imperative path's own traffic saturates the channel. Fix: migrates now coalesce like hints — the worker merges a wake's fresh migrate actions into ONE POST (latest decision per unit hash) after the sgl-project#227 stale-drop. A burst is one round-trip, not N; the single-flight ceiling now bounds how OFTEN a (coalesced) batch is sent, not how many actions it carries — which is what makes the burst case land. This also unifies the design (all three endpoints coalesce). The deeper limit (the single-flight ceiling itself) is the imperative path's structural bound, characterized in sgl-project#230 — coalescing does not remove it, it just stops bursts stacking N round-trips. Escalation (sgl-project#164): now counts failed POSTs per WAKE, not per enqueued batch. The sustained-unreachable fatal still fires (sustained failed dispatches + backlog age) — coalescing reduces POST count, not the signal. t164 B0/C0 prove fatal-on-sustained-failure via a 3-endpoint wake (3 failed POSTs → consec=3 → exit 1 + forensic dump). Tests (all rewritten for the coalesced contract, comprehensive): * outbound_coalesce A1/A2: migrate burst→1 POST, latest-per-hash, stale-drop-before-merge, freshness=0 keeps both. * t164 A0/A1/A2: escalation accounting tested per-dispatch via _dispatch_one (decoupled from drain); B0/C0: 3-endpoint wake → fatal; C3: slow stub proves live-peek in-flight age. * t36 A1/A2/A3/A4: content-delivered (not POST-count), cross-endpoint order, survive-errors-across-wakes. 18/18 touched suites green. DESIGN §6 updated.

) MAIN-RIG PART ONLY. The second host (gfx900 + 2080 Ti) was unavailable this window (NIC hardware fault), so every sm75/gfx900 arm of the recipe is SKIPPED and still owed -- see below. What was validated here, on the main rig (5090 + 2x3080), is the regression direction: 'literally no change where sgl-kernel exists'. Impl selection, verified statically in the venv rather than assumed: _select_weak_ref_tensor() -> sgl_kernel.memory.weak_ref_tensor is native fallback: False non-owning result, data_ptr identical to the source tensor So on a card that has the wheel, the same object is returned as before and the new pure-Python CUDA-array-interface path is never entered. CPU suites: 23 passed / 11 subtests test/registered/unit/compilation/test_weak_ref_tensor_fallback.py test/registered/unit/model_executor/test_breakable_graph_error_masking.py test/registered/unit/distributed/test_htccl_calibration_dtype.py Short boot, 27B FP8, uneven TP=3, spec on, CUDA graphs on: up in 55 s, capture target verify 3.52 / 3.52 / 3.54 s (established band 3.76-3.81 for the DCP arm; this is the dcp-off arm), health 200, max_total_num_tokens=446257 -- the same budget regime as the reference boots (448089 / 447173). Generation is token-identical to the established dcp-off regime on all four probe prompts, accepts identical (3.200 / 2.895 / 3.556 / 2.667). Semantic check before merging, since htccl_device.py is touched by both this branch and sgl-project#181, which was merged one commit earlier: the two hunks are disjoint and independent -- sgl-project#181 works at module level (JIT cache hygiene, _ext_cache_guarded / _load_ext, around lines 31 and 701-840), this branch changes the link-calibration dtype INSIDE class HTCCLDeviceTransport (around line 889). Neither reads the other's state. STILL OWED, second host -- none of this ran: - the actual sm75 (2080 Ti) solo boot that sgl-project#164 is for: the ModuleNotFoundError inside a breakable-graph break point must become a working native fallback; - the gfx900 (Vega) arm of the same; - the mixed-vendor pair arms. The native fallback path therefore rests on its CPU tests alone; this merge only establishes that sm80+ is unchanged by construction and in measurement.

…t#182/sgl-project#181 green and merged, sgl-project#96 RED Main rig only; the second host was unavailable, so every sm75/gfx900 arm is skipped and listed as owed rather than passed. Three merges taken (a620d94, 81902b6, af162cd) with their gate readings in the merge messages. sgl-project#96 (feat/swa-dcp-triton) is NOT merged. sgl-project#96 root cause, found by instrumenting the raise rather than inferring it: Gemma-4 mutates the attention backend's forward_metadata IN PLACE after init_forward_metadata (gemma4_mm.py:428) to install its bidirectional mask, and it does so on EVERY prefill -- the per-request append at :406 sits outside the mm_inputs guard, so a pure-text request installs a plain causal mask too. sgl-project#96 opens the geometry guard for SWA-hybrid models and thereby routes a Gemma-4 full-attention layer into _forward_extend_dcp, which refuses any custom mask. Explicitly NOT the kv>=tp write-gather sublane from sgl-project#173 that the briefing flagged as the likely confound: the failure is a precondition check on the extend read path, before any write gather, and the mask comes from the model file. Also registered: the long-prompt cold-vs-warm output difference seen while running sgl-project#180's V4 is NOT sgl-project#180 -- it reproduces with spec off, with --dcp-size 1 and on sgl-project#180's parent commit, and flashinfer is stable across the same transition. Plus the harness trap that made it look worse than it is: SGLANG_MEASURED_KV_BUDGET sizes from the PREVIOUS boot's leftover, so a first boot in a cold budget record is not comparable to a warm one. Integrated regression on the merge tip is green (health 200, calibration identity [2, 1, 2] unchanged, self-determinism 3x on the short prompts), with capture at 3.93-3.95 s recorded as slightly above the asked band rather than rounded into it, and "plan differs" reported as absent rather than as 0.

…e it was given Regression from 769f0b9 (sgl-project#164). That commit changed the signature `__exit__(self, *args)` -> `__exit__(self, exc_type=None, exc_val=None, exc_tb=None)` and left one call site on the old name: self._stream_ctx.__exit__(*args) `args` no longer exists, so the line raises `NameError: name 'args' is not defined` -- from inside the `finally` block, i.e. on the closing line of the capture, after the segment work has already succeeded. It fires whenever `self._stream_ctx` is not None, and `capture_session()` ALWAYS passes the runner's capture stream to `BreakableCUDAGraphCapture(stream=...)`, so every breakable capture on every GPU dies there. Hit for real on the 2080 Ti; the one-line fix (pass the triple through) is verified on that host, where 42 prefill shapes capture afterwards. Two consequences, both fixed by the same line: * the successful path raises NameError instead of returning; * the unwinding path REPLACES the real diagnosis with the NameError -- the exact failure mode 769f0b9 was written to remove. The NameError also aborts `finally` before `_uninstall_wait_stream_hook()`, so the module-global hook state leaks to whatever runs next. Why the merge-window boots were GREEN. Not luck and not a stale tree: the breakable backend was never entered. Every boot on this rig serves Qwen3.6-27B (multimodal), and `_disable_breakable_cudagraph_if_incompatible` has a "multimodal model" rule, so prefill resolves to `backend='disabled'` while decode runs `backend='full'`: r3_p3_mergetip_armE.boot.log:5 Breakable CUDA graph is incompatible with multimodal model; disabling prefill CUDA graph. server_args: prefill=PhaseConfig(backend='disabled', ...), decode=PhaseConfig(backend='full', ...) TP0/1/2: Disable prefill CUDA graph because cuda_graph_config resolved prefill.backend='disabled' That is not one boot: across all 138 boot logs of the merge window the message appears 98 times and with no other reason. Zero boots on this rig ever constructed a `BreakableCUDAGraphCapture`, so no boot here could have observed the defect. The 2080 Ti host, serving a non-multimodal checkpoint, is the first place the backend actually ran. The test hole, closed. The sgl-project#164 suite constructs the context manager without `stream=`, so `_stream_ctx` stays None, the `if` is never taken, and the broken line is unreachable from any test -- while it is unavoidable in production. `TestCaptureOnADedicatedStream` constructs it the way `capture_session()` does, with a recording stub for the object `torch.cuda.stream()` returns, and asserts that the stub receives the triple: `(None, None, None)` on the plain path, the real exception on the unwinding path, plus a break-point capture (the shape real prefill capture has). The class also restores the leaked hook globals per test, so the leak cannot spread the red set beyond the tests that assert on it. Tests (CPU, no device; every CUDA object stubbed): test/registered/unit/model_executor/test_breakable_graph_error_masking.py before: 3 failed / 8 passed (all three: NameError at line 376) after: 11 passed

Conflicts in dt_candidate_p2p.py and test_dt_candidate_completion_channel.py resolved in favour of dev (native record observer, sgl-project#122/sgl-project#137/sgl-project#150). Kept from sgl-project#25: dt_completion_source.py and the additive transport_endpoint / regions / capabilities fields in TargetCandidateArena.registration(). The arena does not route its record polling through CompletionSource for now; re-wiring it over the native observer is the first step of sgl-project#164. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

- HostTcpServer reaps finished connection threads before retaining a new one, so _connection_threads tracks live concurrency instead of growing without bound over the one-connection-per-request fallback. - DraftFeaturePullTransport.close() joins remote pull workers for the full REMOTE_PULL_BUDGET_S (now the SSOT for both the request timeout and the close wait) instead of 5 s, records a quiescence error when a worker is still alive, and only then clears the worker bookkeeping. - Remote candidate pushes are staged, not synchronous: submit() enqueues an ordered D2H copy per remote row and a per-Target sender thread owns the blocking framed-TCP request, so batch latency no longer multiplies network RTTs on the scheduler thread. Cancellation tombstones use the same per-Target queues (FIFO) so a cancel cannot overtake a queued push for the same slot. Batch completion publishes through a CUDA event recorded on the sender stream; a sender failure is re-raised by the event's query()/synchronize() so the engine's existing quarantine paths handle it. close() quiesces the senders before CUDA teardown. - dt_tcp_port becomes Optional[int] = None (absent disables), a default-off server arg.

…mote push batch event finish_local() snapshotted batch.error into the event at publication time, but senders keep processing items after that (TCP is still in flight microseconds after the submit thread publishes the batch), so any failure of an in-flight item was silently swallowed: the batch recorded as complete, the wave retired with an OK result although its candidate never reached the Target, and the request parked forever server-side with no error anywhere (reproduced at full 598-request concurrency as a cohort of requests on one host-B target stopping mid-decode; reverting to the synchronous push made it disappear). The event now binds its batch and reads the failure under the batch lock at query()/synchronize() time, so late item_failed() results still surface through the engine's quarantine path. Verified inline: late and early failure surfacing, success path, and a 200-trial finalize-exactly-once race.

…essure) With the deferred-failure fix surfacing what the full-concurrency leg actually does, the async sender had a second defect: it removed the synchronous push's implicit pacing, so pushes arrived faster than the Target's serialized push handler (one lock, one H2D stream, a synchronize per row) can drain. The queue backlog grew past the 30 s request timeout and pushes failed en masse (~t=60s at 598 requests, cohorts across several host-B Targets), quarantining both Drafts. Each per-Target sender queue is now bounded to REMOTE_PUSH_QUEUE_MAX (256): the scheduler thread only waits when a Target has fallen seconds behind (256 items are ~seconds of backlog at the saturated drain rate, an order of magnitude under the timeout), while the common case stays non-blocking — the p2 win is preserved. Also harden the sender loop per review: a failure while recording the batch event is stored on the batch and the event is marked recorded, so the sender thread can never die with the event unpublished (which would have head-of-line-blocked every later push); item_added moved inside the staging try so a pinned-allocation failure cannot strand the batch count; close() puts the sentinel with a timeout on the now-bounded queue.

…fallback workers The async candidate push removed the synchronous push's implicit pacing and the full-concurrency leg then failed a third way: ~1000+ fresh TCP connections per second per (draft, target, port) five-tuple (one per pull and per push) exhausted the ephemeral-port space under TIME_WAIT within a minute, after which connects fail en masse (40/598 requests, drafts quarantined on push-completion failures, cutoff ~t=54s). A bounded sender queue alone could not help: the five-tuple churn rate, not the queue depth, was the exhausted resource. The wire protocol already supports sequential request/response rounds on one connection (the server handler loops per connection), so each long-lived worker — one hidden-pull worker and one candidate-push sender per Draft-Target pair — now keeps one persistent HostTcpConnection, reconnecting transparently on transport errors and retrying a request once with a fresh connection (replayed frames are idempotent on the server: same transfer nonce/generation and payload words; application rejections are never retried). This removes the per-request handshake entirely, which also serves the p2 finding. The one-shot HostTcpClient is removed; verified inline against the real HostTcpServer (50 sequential requests on one connection, reject-not-retried, reconnect after a hard close, two parallel connections with ordered responses).

Added GPT-NeoX

73845f5

aflah02 marked this pull request as draft February 7, 2024 21:08

Fix Error due to missing attribute access

b3373fd

aflah02 marked this pull request as ready for review February 7, 2024 21:26

aflah02 changed the title ~~Adding GPT-NeoX [WIP]~~ Adding GPT-NeoX Feb 7, 2024

merrymercy self-assigned this Mar 11, 2024

merrymercy force-pushed the main branch from 36daf8e to adc9742 Compare May 28, 2024 05:51

merrymercy closed this Jun 12, 2024

Garrybest pushed a commit to Garrybest/sglang that referenced this pull request Jan 9, 2026

update global_jit_compile.md and dynamic_continuous_batching.md (sgl-…

9b45025

…project#164)

Navigation Menu

Repository navigation

Adding GPT-NeoX - #164

Adding GPT-NeoX#164
aflah02 wants to merge 2 commits into
sgl-project:mainfrom
aflah02:main

aflah02 commented Feb 7, 2024 •

edited

Loading

Uh oh!

aflah02 commented Feb 7, 2024 •

edited

Loading

Uh oh!

aflah02 commented Feb 7, 2024

Uh oh!

aflah02 commented Feb 9, 2024

Uh oh!

merrymercy commented Feb 11, 2024 •

edited

Loading

Uh oh!

aflah02 commented Feb 20, 2024

Uh oh!

aflah02 commented Mar 1, 2024

Uh oh!

merrymercy commented Mar 11, 2024

Uh oh!

aflah02 commented Jun 12, 2024

Uh oh!

merrymercy commented Jun 12, 2024

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

aflah02 commented Feb 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aflah02 commented Feb 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aflah02 commented Feb 7, 2024

Uh oh!

aflah02 commented Feb 9, 2024

Uh oh!

merrymercy commented Feb 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aflah02 commented Feb 20, 2024

Uh oh!

aflah02 commented Mar 1, 2024

Uh oh!

merrymercy commented Mar 11, 2024

Uh oh!

aflah02 commented Jun 12, 2024

Uh oh!

merrymercy commented Jun 12, 2024

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

aflah02 commented Feb 7, 2024 •

edited

Loading

aflah02 commented Feb 7, 2024 •

edited

Loading

merrymercy commented Feb 11, 2024 •

edited

Loading