Repository navigation
Fix issue #367 – System message not supported for Anthropic (anthropic.BadRequestError) - #368
Merged
Merged
Conversation
This makes it possible to run tests under test/lang/ like so: python -m unittest test.lang.test_anthropic_backend
…dling
Using the system role with the Anthropic backend resulted in the following error:
anthropic.BadRequestError: Error code: 400 - {
'type': 'error',
'error': {
'type': 'invalid_request_error',
'message': 'messages: Unexpected role "system". The Messages API accepts a top-level `system` parameter, not "system" as an input message role.'
}
}
The fix converts system prompts passed as a message to
Anthropic's special `system` parameter while keeping the
interface compatible with other LLM backends.
Ying1123
requested changes
Apr 16, 2024
Ying1123
approved these changes
Apr 16, 2024
Contributor
|
@fronx Thanks for the contribution! It is merged. |
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
…c (anthropic.BadRequestError) (sgl-project#368) Co-authored-by: Ying Sheng <sqy1415@gmail.com>
shiyu7
pushed a commit
to shiyu7/sglang
that referenced
this pull request
Dec 26, 2025
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 1, 2026
…sgl-project#366) sgl-project#369 released the bar1 graph gate, so sgl-project#366 could finally measure. It still could not measure with CUDA graphs, and the reason is the result that matters most here. WHY EAGER. bar1 + CUDA graphs + NEXTN does not cold-capture in a usable time on the 27B FP8 vehicle. One graph boot ran >35 min without finishing capture, all three cards pinned at 100% util the whole time -- forward progress, not a wedge. The log says why, dozens of times: "Using default W8A8 Block FP8 kernel config ... Config file not found ... device_name=NVIDIA_GeForce_RTX_5090, dtype=fp8_w8a8, block_shape=[128,128]". The rig ships no tuned W8A8 block-fp8 configs for the 5090, so every GEMM shape falls into a cold triton autotune with ~3 min between shapes. That is the sgl-project#255/sgl-project#368 tuned-config gap, and warming those configs once would make every future bar1 graph boot fast -- a concrete, cheap follow-up. Running EAGER skips capture, costs ~34 s per boot, and puts both transports in the SAME regime, so the transport question is answered honestly. The table therefore does NOT quote sgl-project#354's graph-mode NCCL baseline; it measures NCCL itself, this window, eager. s17_bar1_eager_arm.sh boots one arm from the htsglang image on the PVE host and is parameterised by transport. Its gate is two-sided and mandatory per boot: a bar1 arm must log ACHIEVED=bar1 on every barlink group (world/tp/dcp x 3 ranks) and an NCCL arm must log ZERO barlink lines, so neither a silent fallback nor an accidental barlink run can enter the table. All 8 arms passed. Three environment defects were found and fixed inside it, each documented at the point of the fix: * TORCH_CUDA_ARCH_LIST in the image made the JIT extension build as _cuda_75_80_86_89_90_100_120, missing the warm _86_120 cache and cold-compiling seven arches. Pinned to '8.6;12.0' -> cache hit in 0.2 s. * The INT8 fork sgl-kernel wheel is built against CUDA 12.9 and the image is cu13, so installing it BROKE sgl_kernel outright (libcublas.so.12, then libcudart.so.12 missing -> common_ops unimportable -> every INT8 rank died with "needs sgl_kernel.int8_scaled_mm, which this build has no code for"). Fixed by mounting a directory of symlinks to every *.so.12* the CT999 venv ships and APPENDING it to the image's own LD_LIBRARY_PATH -- the SONAMEs differ from .so.13, so torch keeps its cu13 libs. * A single quote inside the wheel-inject string closed the docker -c '...' wrapper and killed INT8 boots in 2-3 s before the container existed. s17_bar1_eager_table.py renders the deliverable: both levers separately, never only the diagonal, with sgl-project#354's noise floors carried and any delta under its row floor printed as "within noise". Decode reads the client-side token rate for every arm: under eager + docker-logs buffering the server-log tick parser sees zero ticks, so one consistent source beats a mixed column. Test results (2026-08-01, Qwen3.6-27B TP=3 uneven 5090+2x3080, NEXTN 3/1/4, KV fp8_e4m3, ctx 32768, EAGER, 8 boots, all gates passed): point FP8 NCCL FP8 bar1 d(tr) INT8 NCCL INT8 bar1 d(tr) prefill_s1 1335.8 1395.8 +4.5% 1412.1 1687.5 +19.5% prefill_s8 1329.0 1454.6 +9.5% 1448.2 1674.5 +15.6% decode_bs1 25.4 36.0 +41.7% 35.4 33.8 -4.5% decode_bs8 217.8 219.7 within noise 230.2 260.3 +13.1% Transport lever: bar1 wins prefill in BOTH formats (FP8 +4.5/+9.5%, INT8 +19.5/+15.6%) and is mixed on decode. Format lever: INT8 > FP8 on prefill under both transports (NCCL +5.7/+9.0%, bar1 +20.9/+15.1%). Nothing under python/ was touched, so no repo test suite is affected. ruff, codespell and bash -n clean.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 5, 2026
… chain in one engine The fold is now a first-class arm, not a footnote. Folding w_q x w_scale into bf16/fp16 weights at BUILD time removes the constraint that shaped the INT8 design: with no runtime dequantize there is no activation quantization either, so TensorRT's inability to express a per-token dynamic scale stops mattering. The INT8 arms are per-stage engine islands with our quant kernel between them; folded, chain_decode_layer is ONE engine containing four GEMMs, the SiLU-gate multiply and the attention-core bridge, with nothing of ours inside it. 22 more engines built at the desk, CPU-only AOT, real checkpoint weights, 19 s for all 32. Variants: int8 (per-stage), fold_bf16, fold_fp16 (whole chain), plus one fp32_ref shape per rank as a labelled diagnostic. QUALITY, MEASURED RATHER THAN ASSERTED. The fold does not coarsen anything -- w_q x w_scale is the checkpoint's weight exactly, in a wider container -- and it removes the per-token activation quantization error the deployed path pays. So the harness computes an exact fp32 reference from the dequantized weights and gates each fold on being AT LEAST AS ACCURATE AS WHAT WE DEPLOY, rather than on a fixed tolerance. Gating a fold against the deployed OUTPUT would have measured the deployed path's own quantization error and called it a fold defect. Mock, random weights, directional: deployed 0.0112, fold bf16 0.0031, fold fp16 0.0024, max-abs vs exact. THE MOCK FOUND AN fp16 LIMIT. fp16 wins on mantissa (11 bits vs bf16's 8) and loses on range (5 exponent bits, max 65504; bf16 carries fp32's range). In the 4-GEMM chain_decode_layer the intermediates compound and the fp16 fold produced a NON-FINITE output where bf16 did not. Random weights exaggerate the magnitudes, so this is a flag for the card run and not a verdict -- but it qualifies the "2^-11 beats the INT8 grid" rationale exactly: the deep chains are where range, not mantissa, decides. Non-finite output is detected per arm and recorded as a result, never averaged into a number. THE STAIR IS THE POINT, and the memory cost is now priced per part. Every fold engine measures 1.999x the INT8 weight bytes; full-model fold does not fit the 3080s and was never the proposal. Per-part cost is in the manifest as weight_bytes_fold / weight_bytes_int8 / memory_multiplier_vs_int8 for every engine, and tabulated in TARGET_SELECTION, so the planner can price residency for any subset of the stair. Hot parts folded, everything else stays INT8 and stays offloadable. CROSSOVER, measured under graphs on both sides: the fold trades zero activation-quant kernels against 2x the weight bytes moved in a memory-bound GEMV. sgl-project#368's 78 % quant share was an EAGER number and graph replay already cuts it to ~11 %, so the fold is buying back a smaller slice than the eager figure suggests. New verdict cells fold_bf16_over_torch_graph, fold_fp16_over_torch_graph and fold_bf16_over_trt_int8 across M = 1,2,4,8; the last one separates "the fold helps" from "the engine helps", which are different wins and a per-part decision needs both. The fp32 arm is one shape, labelled DIAGNOSTIC_fp32_ref_over_torch_graph, and is disqualified as a deployment candidate by memory at 4 bytes per weight. It isolates TensorRT runtime quality from the quantization constraints and nothing else. Assumptions independently checked against the fork and the checkpoint before building: every language-model linear is bias=False with no .bias key in the checkpoint; every module in the target set is INT8-quantized (only lm_head, embed_tokens, in_proj_a/b, visual and mtp are ignored); and silu applies to the FIRST half times the second (activation.py:123-125), which is the epilogue the engines emit. Tests: mock_smoke.sh green with the fold arms in the rotation -- 10 points, 52 arm measurements, tolerance AND quality paths exercised on every point, fp16 chain overflow surfaced as a NOTE and not swallowed. 32 real engines built with checkpoint weights. ruff clean, codespell clean. NOTHING executed on a GPU: the cards were held by another window throughout and every step ran with CUDA_VISIBLE_DEVICES="".
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 17, 2026
… as data, and the embed-quant checks sgl-project#725. THE BRIEF'S PREMISE DOES NOT HOLD, and saying so is the finding. The task was to locate where the cost model prices quantized-linear lanes as M-independent or with a wrong crossover, and to stop bs=1-decode points being priced through the quant stage. Neither happens: the planner has NO activation-quant axis at all. decode is weight_bytes / effective_decode_bw (key_solver.decode_weight_time), prefill is 2*params / gemm_tflops (key_solver.prefill_compute_time), and quantization enters those two formulas only as a smaller weight-byte count and as which lane rate is picked (sgl-project#324). A T=2 decode point never passes through a quant stage, so there is no misprice to make red. Per the brief's own escape clause this reduces to data-with-provenance plus pins. TWO CORRECTIONS TO THE FRAMING, both load-bearing: 1. The real dispatch decision is not in the planner. It is layers/quantization/fp8_dequant_gemv.py:324 (fused_gemv_applicable, aspect test N >= K) and the sm120 NVFP4 buckets in nvfp4_scaled_mm_sm120.cuh: 192-201 -- per-call runtime branches. 2. sgl-project#368 measured the small-M quant penalty as largely a LAUNCH CONSTANT that CUDA-graph replay removes: int8_quant 0.0266 -> 0.0012 ms (~21x), quant share of the fused op 61% -> 11%. NInfer's thresholds are dispatch branches in a C++ header, i.e. taken per call in an eager engine, and a captured graph cannot branch on T at replay. Wiring an eager-measured crossover into our graph-captured decode path would import an artefact. The table is therefore recorded with NO consumer, deliberately (sgl-project#421's audit class: an instrument that looks wired and is not is worse than one honestly inert). WHAT THE HARVEST IS WORTH TODAY: corroboration, and it is real. Two codebases, two mechanisms, two GPU classes agree on which family must not take the small-M quant/fused path -- the K>N shapes. Our own sm86 aspect gate separates exactly those ([5120,6144] 0.86x LOSE, [5120,17408] 0.90x LOSE). Five of six shapes agree in direction. (d) sm86 stays EMPTY. NInfer measured sm120a only, and sgl-project#368 established sm86 is GEMM-slow rather than quant-dominant, so its crossovers are not merely unmeasured but expected to differ in KIND. An estimate would be a number with no measurement wearing the same type as five that have one. sgl-project#724, checks only, no port. Checkpoint read by headers only (120 and 2624 bytes of header JSON; no shard mmapped, no tensor data touched). (a) YES on all three counts, from two independent places. config.json: tie_word_embeddings False, and quantization_config.ignore names 'lm_head' and 're:.*embed_tokens.*'. Safetensors headers confirm it without trusting the config: lm_head.weight BF16 [248320, 5120], embed_tokens.weight BF16 [248320, 5120]. This CLOSES the one open divergence from sgl-project#725. That shape is exactly NInfer's lm_head row -- the single shape where their "A16 always" points opposite to our N>=K gate (N/K=48.5, selects fused). On this checkpoint lm_head is BF16, so no activation-quant gate ever sees it: moot. (b) NO gap in that form; the kernel exists and IS wired, just scoped. main model qwen3_5.py:1386-1405 builds embed_tokens quantized-resident via GGUFEmbeddingMethod (gguf.py:1532), gated explicitly to GGUF; every other quantization keeps quant_config=None and takes the dense path. NEXTN draft qwen3_5_mtp.py:185-204 set_embed_and_head_modules hands the draft the target's MODULE OBJECTS rather than tensors -- precisely because a packed qweight has no .weight to hand over -- so the draft inherits whatever the main model built and has no independent path to carry a gap. The only real delta is a capability question for its own decision: extending quantized-resident vocab beyond GGUF would require re-quantizing embed and lm_head, which this checkpoint's producer deliberately excluded, so the ~1.1 GiB/rank prize is not available without changing the checkpoint. Tests, hermetic (CUDA_VISIBLE_DEVICES="", no GPU, no model load, no serving): test_activation_quant_crossover_725.py 13 passed + 31 subtests MUTATION PROOF: estimating an sm86 row and making ABSENT answer False turns 3 pins red test/registered/unit/planner 2524 passed, 123 skipped, 188 subtests, 2 failed -- PROVEN PRE-EXISTING by re-running the failing file with both new files moved aside: same 2 fail (test_rejected_evidence_pins.py). ruff + codespell clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This fixes #367 by converting a system prompt provided as a message to Anthropic's special
systemparameter.It is assumed that a system prompt is only ever passed as the first message. After extracting it, the first message is removed from the message list.