Fix Regression: Disable p2p for 4090 - #531
Merged
Merged
Conversation
…g ModelTpClient.step().
Contributor
|
As far as I know, RTX 40 series is the only modern Nvidia gpu that do not support P2P. Shame on Nvidia. Attempting to force use them will cause errors. |
ZX-ModelCloud
marked this pull request as ready for review
June 11, 2024 11:07
ZX-ModelCloud
changed the title
Contributor
|
My bad, didn't realize that I'm looking at a pull request instead of an issue. |
Contributor
|
Alternatively p2p can be enabled for 4090 GPUs with this fork of the gpu kernel modules (have not tried it yet): https://github.com/tinygrad/open-gpu-kernel-modules |
merrymercy
reviewed
Jun 11, 2024
Contributor
There was a problem hiding this comment.
should we directly return false here?
Contributor
There was a problem hiding this comment.
@merrymercy Tested using setattr(tgt, "gpu_p2p_access_check", lambda *arg, **kwargs: False) and the system locked up with 2x4090 tp=2. Have no idea why. So not doing the monkeypatch works but forcing it to return false also breaks.
Collaborator
There was a problem hiding this comment.
@Qubitium When I set this to False on A100 PCI-E, the system works well.
When not applying a monkey patch, it seems not to be locked up but keeps checking all the connections between any two peers like this
Original exception was:
Error in sys.excepthook:
Original exception was:
[rank0]:[W CudaIPCTypes.cpp:16] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
Error in sys.excepthook:
which costs a lot of time and gets a cache.
Qubitium
force-pushed
the
4090_gpu_p2p_access_check
branch
from
June 12, 2024 01:54
af9a47d to
cc7b740
Compare
ZX-ModelCloud
changed the title
Contributor
It works! Tested the tinycorp nvidia driver and nccl/p2p works for 4090 (albeit slow) . |
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
Co-authored-by: Qubitium <417764+Qubitium@users.noreply.github.com>
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 3, 2026
User standing order 2026-08-03: --reasoning-parser and --tool-call-parser are STANDARD serving-boot settings, not tuning knobs -- "damit man die modelle auch nutzen kann". A boot missing them answers HTTP 200 while degrading in three silent ways, all three observed on this rig's own FP8 boot: the chain-of-thought lands in `content` as raw </think> text, an Anthropic `thinking` block is refused outright, and a tool call comes back as a JSON-looking STRING instead of a structured `tool_calls` entry. Three cuts: * Runbook: the seven full launch recipes and the four boot scripts that actually SERVE (handover TP1/TP3, satellite prefill/decode) now carry the pair for their family. Scope is stated and deliberately narrow -- the measurement arms under gpu_battery/, dual_group/, probe*/, determinism/ and nordstern/ are NOT patched, because a reasoning parser moves text out of `content` and would shift the token accounting those arms exist to produce. * Planner: `flags.usability_parsers` resolves the pair from the checkpoint's `architectures` (path as fallback) over a SQUASHED identity string, so Qwen3.6-27B / qwen3_5 / Qwen3_5ForConditionalGeneration all resolve alike -- the first implementation used token-set membership and silently missed every dotted family, since _name_tokens does not split on '.'. The table is ordered specific-first because 'v3' is a substring of 'v32'. `validate_usability_parsers()` checks every emitted name against the live ReasoningParser.DetectorMap / FunctionCallParser.ToolCallParserEnum, so a registry rename turns the mapping red rather than shipping a flag value the server rejects. Both command generators emit it (feasibility._launch_flags, key_solver._usability_launch_flags); an unrecognised family emits a NAMED HINT instead of a bare command. * register_local_model.sh reads the live /server_info (authoritative ServerArgs) instead of guessing from the model listing, and writes "current boot lacks <what>; agentic tool use degraded" into the generated agent header plus stderr. Also records the sgl-project#530 coexistence rule: reserves for a shared card come from the co-tenant's DECLARED budget, never from a momentary observation. The translator held 4204 MiB on the 5090 while declaring 7500, so the INT8 boot reserved 13000,3800,3800 rather than 5500,3800,3800 -- ~135k KV tokens for a coexistence that survives the tenant growing into its own budget. Tests: test/registered/unit/test_usability_parsers_531.py, 11 hermetic + 13 subtests, green. Three independently EXECUTED can-fail arms: the pre-fix token-set match makes every dotted family resolve to None; moving the general deepseek v3 row ahead of v32 yields the wrong point-release parser; a bogus parser name makes the registry check fire. Helper warning proven both ways against a parser-less stub (fires, in file and on stderr) and the live INT8 boot (silent, reasoning=qwen3 toolcall=qwen3_coder). ruff + codespell clean on every touched file; the 7 remaining ruff findings under planner/ are pre-existing in files this change does not touch.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 3, 2026
…agent backend + sgl-project#531 usability trias as a standard boot setting Two commits: sgl-project#530: serves the local checkpoint as a Claude Code subagent backend. register_local_model.sh reads /server_info from the running server to register it rather than duplicating boot-time config; local_model_agent.sh wraps it as a launchable subagent. planner/feasibility.py, flags.py, and key_solver.py get the small hooks this registration path needs. sgl-project#531: makes the usability parser/template trio a standard boot setting across seven runbook recipes and four serving-boot scripts (scripts/handover/boot_a_tp1.sh, boot_b_tp3.sh, scripts/satellite/boot_main_decode.sh, boot_satellite_prefill.sh), plus a planner family-mapping with registry validation. Deliberate scope decision, already reviewed and approved, documented in the runbook: the measurement arms under gpu_battery/ etc. are NOT patched to carry the new parser/template setting, because doing so would shift the token accounting behind already-published numbers -- this merge does not "complete" that scope. Clean merge, one file needing 3-way combination (FEATURE_CATALOG.md, no conflict) -- this branch's base (e04aeae) is one merge behind this line's tip (0531280, which added the unrelated sgl-project#529 model_loader fixes) with no file overlap. All 17 sections and all 25 previously-preserved additive paragraphs verified present. Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree): - test_usability_parsers_531.py: 11 passed, 13 subtests passed, 0 failed -- matches exactly. - test/registered/unit/model_loader/, BOTH wheel states (default and SGLANG_GGUF_MXFP4_NATIVE=0), per the new 0-failed-absolute criterion established at the sgl-project#529 merge: IDENTICAL both times, 341 passed, 0 failed, 15 skipped, 67 subtests passed. - ruff check --select=F401,F821,UP037: 1 finding (planner/flags.py:1717 "Profile"), confirmed pre-existing (already verified multiple times this session) and outside this branch's diff hunk (which only appends new content at the end of the file). - codespell: clean on all 13 touched/added files.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 3, 2026
…ally reads it
User-caught defect. register_local_model.sh defaulted its target to
"$REPO_ROOT/.claude/agents", which for a WORKTREE checkout is a directory no
Claude Code session ever loads -- project agents come from the SESSION's own
project directory. The `local-model` type therefore appeared in no agent list
at all, while the script printed "wrote ..." and exited 0. The registration was
a no-op.
This is the success-claim-vs-state class, and the earlier "execution proof"
walked straight past it: the wrapper round-trip proved the WRAPPER, never the
REGISTRATION, because local_model_agent.sh reads
~/.config/htsglang/local_model_agent.env and never opens the agent file. No
amount of re-running that probe could have caught this.
Fix:
* Default target is the USER-GLOBAL ${HOME}/.claude/agents, explicitly not
repo-relative, so the next model switch cannot regenerate into nowhere.
* Closing probe re-reads the written file AT THE CANONICAL PATH, checks the
`name:` frontmatter, prints the absolute path and byte count, and exits 5/6
when the file is missing, empty or unusable. "Wrote the file" is no longer
the last word.
* An --agent-dir that is neither the user-global dir nor the current project
exits 7 with a named refusal, because the failure it guards against is a
SILENT no-op that a caller ignoring stderr would otherwise miss.
--allow-unread-agent-dir opts out for test harnesses.
* Output states the session-lifetime rule: agent lists load at session START,
so a re-registration reaches NEW sessions only.
Also hardens the /v1/messages probe, which the fix run exposed: a single 20 s
probe refused to update the config while the server was merely mid-prefill on a
55k-token request (~40 s of chunked prefill queued ahead of it), making a
transient load spike indistinguishable from a missing endpoint. Now 3 attempts
at 60 s, and on failure it asks /health so the message says whether the server
is absent or just loaded.
Runbook section 13 and the catalog record the defect and the session-lifetime
rule rather than only the corrected recipe.
Verified: default run prints
VERIFIED agent 'local-model' at /root/.claude/agents/local-model.md (2788 bytes)
exit 0 against the live INT8 boot. Can-fail arms executed: --agent-dir into an
unread path exits 7 (0 with the opt-out); a generator patched to emit broken
frontmatter exits 6 with "has no usable 'name:'". Stale worktree copy removed.
codespell clean; bash -n clean.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 3, 2026
…agent where a session actually reads it User-driven follow-up to sgl-project#531's local-model-agent registration. The first version wrote the registered agent to $REPO_ROOT/.claude/agents, which for a worktree is read by no session at all -- the agent type appeared in no agent list while the script reported a successful write, and the wrapper round-trip could not catch it because the wrapper reads ~/.config/htsglang/, never the agent file. That proved the wrapper, never the registration -- a §12 success-claims-are-not-evidence instance the fork's own convention exists to catch, caught against itself. Fix: register_local_model.sh now writes to the USER-GLOBAL ~/.claude/agents/local-model.md (read by every session, not just the worktree that ran the script), then VERIFIES the file at the path that is actually read and prints it -- the closing probe re-reads the canonical path rather than trusting its own write. An --agent-dir that is neither user-global nor the current project exits 7 (named exit codes 5/6/7 for existence/canonical-path/agent-dir-class failures). The endpoint probe now distinguishes "absent" from "merely busy" via /health rather than treating any non-200 the same way. Session-lifetime rule documented: an agent list loads at session START, so a re-registration reaches new sessions only. Runbook §13 and FEATURE_CATALOG.md updated; the old .gitignore entry for the repo-relative path stays as a backstop for a run that explicitly passes --agent-dir into the repo, with a comment explaining why the primary path moved. Clean merge, no conflict (this branch's true base was the previous sgl-project#530 tip 07118d7, one merge behind this line's current tip due to the intervening sgl-project#529 merge -- confirmed via diff against the correct base, not the raw diff against HEAD which spuriously looked like it reverted sgl-project#529's files). Four files touched (.gitignore, FEATURE_CATALOG.md, rig-runbook.md, register_local_model.sh) -- no Python, so no ruff check applicable. All 17 catalog sections and all 25 previously-preserved additive paragraphs verified present. Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree): - test_usability_parsers_531.py: 11 passed, 13 subtests, 0 failed -- unaffected by this shell/docs-only change. - test/registered/unit/model_loader/, BOTH wheel states (default and SGLANG_GGUF_MXFP4_NATIVE=0), per the 0-failed-absolute criterion: IDENTICAL both times, 341 passed, 0 failed, 15 skipped, 67 subtests passed. - bash -n scripts/dev/register_local_model.sh: syntax OK. - codespell: clean on all 4 touched files.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 4, 2026
… the local server (sgl-project#540) Two commits on top of the conformance merge (this branch's true base is exactly that commit -- zero drift beyond it), clean auto-merge including a second mid-paragraph FEATURE_CATALOG.md splice. Claude Code binds its endpoint per process (ANTHROPIC_BASE_URL and siblings read once at startup; the subagent frontmatter schema carries no baseUrl/provider/env key), so no in-client setting keeps a session's parent turns on api.anthropic.com while one subagent runs on the rig. local_model_agent.sh's whole-process move is the fallback, not the feature this branch builds. python/sglang/srt/entrypoints/anthropic/router.py listens on 127.0.0.1:30099 and forwards every request verbatim (path, query, method, all headers including the bearer, response bytes undecompressed, SSE streamed) to api.anthropic.com EXCEPT requests whose model is in --local-model, which go to 30030. Routing key exists because Claude Code passes --agents' "model" string to the wire unvalidated, so naming a local id in an agent definition is the whole binding; with no -m the id resolves from GET /v1/models, so the router follows a checkpoint switch rather than a hardcoded name. No header value is logged at any level. One body edit, framed correctly as a compatibility shim rather than a mechanism: "thinking":{"type":"disabled"} is filled in on locally-routed /v1/messages bodies that omit the field -- never rewriting an explicit value, never touching upstream traffic or count_tokens, and a NO-OP by construction once the serving process carries the absent-means-disabled front fix from the sibling conformance merge, since it only ever writes what the front now defaults to (--no-thinking-shim disables it). Own defect found by driving the router rather than reading it, fixed in the second commit: Accept-Encoding is now pinned to identity when the client omits it. The response body is forwarded undecompressed, so aiohttp adding its own "gzip, deflate" made the first version hand a gzipped body to a client that never advertised gzip -- a plain curl got binary garbage where the error envelope belonged. A client that DOES send the header keeps its own value. Two tests pin both directions; the absent-header test's first version failed because aiohttp's own test client adds the header, which is why it now uses skip_auto_headers. LIVE-BOOT PROOF, unlike the sibling conformance merge's hermetic-only verification: a real claude -p 2.1.221 was driven through the router against the live 30030 boot in a separate process, parent session and server untouched. The subagent returned a marker it could only get through a Read round trip, sglang:generation_tokens_total{priority="0"} on 30030 moved 14820 -> 14926 (+106), and the router's own decision log shows the split in order: two parent turns on claude-fable-5 upstream, two Qwen3.6-27B turns local (tool_use then tool_result), parent's closing turn upstream. Same run measured why the shim exists against the not-yet-restarted live boot: the identical body sent direct to 30030 without a thinking field spent its whole 40-token budget on a thinking block (stop_reason max_tokens, zero text) -- through the router it answered cleanly. That gap closes the moment 30030 restarts onto the conformance fix, which is exactly what makes the shim a no-op-by-construction bridge rather than a permanent mechanism. Tests: test_router.py, 19 hermetic tests (two mock aiohttp backends standing in for api.anthropic.com and the local front, nothing real touched), mutation-checked -- dropping the shim assignment fails 1, forcing everything upstream fails 10. FEATURE_CATALOG.md: second mid-paragraph splice into the existing "local checkpoint as a subagent backend" entry (sgl-project#530/sgl-project#531 area), applied cleanly since nothing on this line touched that text since the conformance merge landed. All 17 sections and every previously tracked additive paragraph verified intact, including the conformance merge's own new paragraph from the immediately preceding commit. Verification: test/registered/unit/entrypoints/anthropic/ together -- 95 passed, 9 subtests, 0 failed, matching the reported number exactly (20 conformance + 56 serving + 19 router). test/registered/unit/ model_loader/, BOTH wheel states (default and SGLANG_GGUF_MXFP4_NATIVE=0): IDENTICAL both times, 341 passed, 0 failed, 15 skipped, 67 subtests passed. ruff --select=F401,F821,UP037 and codespell clean on all touched files. bash -n on claude_local_router.sh: syntax OK. Live serving (30030, unchanged PID) and the router process (PID 37489) confirmed untouched throughout -- neither was restarted or reconfigured by this verification.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Sep 2, 2026
…e local checkpoint The generator asserted that Claude Code cannot bind a subagent's inference to a foreign endpoint. That is refuted: a subagent declaring a non-Claude model string sends that string in the `model` field of its own request bodies while the parent session's requests in the same process still carry `claude-*`. A model-routing proxy at ANTHROPIC_BASE_URL splits the two legs; no binary patch is involved. Generator changes: - frontmatter `model:` is now the live model id instead of `haiku`, so the agent's loop runs on the rig - probe the routing proxy (default http://127.0.0.1:8787) and record its state in the serving table; warn loudly when it is absent or does not carry this model id, because that is the failure mode that silently sends an unknown model id to Anthropic - rewrite the agent body: real mechanism, proxy precondition with the repair command, honest scope limits (text-only checkpoint, bounded tasks) - the local_model_agent.sh wrapper is kept and reframed as the no-proxy route for agents that are not themselves bound to the local model - the raw curl example now carries thinking:{"type":"disabled"}; without it the qwen3 reasoning parser can spend the whole max_tokens budget on a thinking block Validation on Claude Code 2.1.221 (sha256 60db8e88d4...): - wire probe: two model ids left one process, Qwen3.6-27B from the subagent and claude-fable-5 from the parent - end-to-end: a real subagent's agent loop was served by the live htsglang server (proxy log: POST /v1/messages?beta=true model=Qwen3.6-27B -> local 200) - generator re-run against the live server: 3724 bytes, proxy state `ready` - proxy-absent path exercised with --proxy-url pointing at a dead port: warning fires and the table records `absent` Proxy, falsifiers and evidence commands: /root/claude-code-qwen-patch/
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacktrace: