pickle to msgpack migration

[rainj-me]

Motivation

#23206

Modifications

Followup PR 28688

Done

• Migrate pickle to mspack
• For custom object that could not directly migration to msgpack, will use PickleWrapperIpc to wrap
• For tensor and array like data type will convert to bytes
• Rely on SGLANG_LOG_PICKLE_IPC_OBJECTS=1 to log out all type that serialized by pickle
• Refactor the http2 granian server from MPServer to embedded Server.

Pending Migrate

• SamplingParams [Done]
• TraceNullContext [Done]
• TraceReqContext [Done]
• mm_inputs/image_inputs [Done]
• maybe more type that can not test them out for now

Accuracy Tests

SGLANG_LOG_PICKLE_IPC_OBJECTS=1  CUDA_VISIBLE_DEVICES=0,1,2,3 SGLANG_ENABLE_JIT_DEEPGEMM=1 TORCH_CUDA_ARCH_LIST=10.0 NVSHMEM_IB_ENABLE_IBGDA=0 NVSHMEM_ENABLE_NIC_PE_MAPPING=1 NVSHMEM_DISABLE_LOCAL_ONLY_PROXY=1 NVSHMEM_IB_GID_INDEX=0 NVSHMEM_HCA_LIST=mlx5_0:1,mlx5_3:1,mlx5_4:1,mlx5_5:1 NVSHMEM_BOOTSTRAP_UID_SOCK_IFNAME=eth0 NCCL_IB_GID_INDEX=0 NCCL_IB_HCA=mlx5_0:1,mlx5_3:1,mlx5_4:1,mlx5_5:1 NCCL_MNNVL_ENABLE=1 NCCL_CUMEM_ENABLE=1 NCCL_SOCKET_FAMILY=AF_INET6 GLOO_SOCKET_FAMILY=AF_INET6 NCCL_SOCKET_IFNAME=eth0 GLOO_SOCKET_IFNAME=eth0 SGLANG_DISABLE_TP_MEMORY_INBALANCE_CHECK=1 SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=1024 sglang serve --model-path  /data03/models/Qwen3-235B-A22B-FP8 --trust-remote-code --tp-size 4 --ep-size 4 --dp-size 4 --enable-dp-attention --enable-dp-lm-head    --device cuda  --host 0.0.0.0 --port 30000 --mem-fraction-static 0.8  --disable-radix-cache --attention-backend fa3 --moe-runner-backend deep_gemm --max-running-requests 512 --context-length 32768 --chunked-prefill 2048 --reasoning-parser qwen3 --disable-piecewise-cuda-graph --model-loader-extra-config '{"enable_multithread_load": true, "num_threads": 64}' --cuda-graph-max-bs 128

GSM8k

python3 benchmark/gsm8k/bench_sglang.py --num-shots 8 --num-questions 100 --parallel 8 --port 30000                        
100%|████████████████████████████████████████████████████████████████| 100/100 [01:32<00:00,  1.08it/s]
Accuracy: 0.970
Invalid: 0.000
Latency: 92.416 s
Output throughput: 153.231 token/s

Speed Tests and Profiling

Server command:

SGLANG_ENABLE_JIT_DEEPGEMM=0 sglang serve --model-path Qwen/Qwen3-0.6B-FP8 --disable-shared-experts-fusion --port 30000  --chunked-prefill-size 4096  --attention-backend fa3 --max-running-requests 4096  --page-size 64 --disable-radix-cache --disable-piecewise-cuda-graph --cuda-graph-max-bs 4096 

Concurrency = 1024

msgpack:

============ Serving Benchmark Result ============
Backend:                                 sglang-oai-chat
Traffic request rate:                    200.0     
Max request concurrency:                 1024      
Successful requests:                     4096      
Benchmark duration (s):                  83.67     
Total input tokens:                      4194304   
Total input text tokens:                 4194304   
Total generated tokens:                  819200    
Total generated tokens (retokenized):    818543    
Request throughput (req/s):              48.96     
Input token throughput (tok/s):          50130.64  
Output token throughput (tok/s):         9791.14   
Peak output token throughput (tok/s):    22598.00  
Peak concurrent requests:                1888      
Total token throughput (tok/s):          59921.78  
Concurrency:                             947.24    
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   19348.94  
Median E2E Latency (ms):                 20125.98  
P90 E2E Latency (ms):                    22493.05  
P95 E2E Latency (ms):                    22504.20  
P99 E2E Latency (ms):                    22518.92  
---------------Time to First Token----------------
Mean TTFT (ms):                          3504.18   
Median TTFT (ms):                        2934.70   
P90 TTFT (ms):                           6747.65   
P95 TTFT (ms):                           7489.32   
P99 TTFT (ms):                           8188.38   
---------------Inter-Token Latency----------------
Mean ITL (ms):                           79.67     
Median ITL (ms):                         47.68     
P90 ITL (ms):                            60.66     
P95 ITL (ms):                            72.28     
P99 ITL (ms):                            611.61    
Max ITL (ms):                            9043.99   
==================================================

pickle:

============ Serving Benchmark Result ============
Backend:                                 sglang-oai-chat
Traffic request rate:                    200.0     
Max request concurrency:                 1024      
Successful requests:                     4096      
Benchmark duration (s):                  95.19     
Total input tokens:                      4194304   
Total input text tokens:                 4194304   
Total generated tokens:                  819200    
Total generated tokens (retokenized):    818543    
Request throughput (req/s):              43.03     
Input token throughput (tok/s):          44063.20  
Output token throughput (tok/s):         8606.09   
Peak output token throughput (tok/s):    21701.00  
Peak concurrent requests:                1792      
Total token throughput (tok/s):          52669.29  
Concurrency:                             986.70    
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   22930.17  
Median E2E Latency (ms):                 23637.85  
P90 E2E Latency (ms):                    24233.78  
P95 E2E Latency (ms):                    24308.83  
P99 E2E Latency (ms):                    24316.35  
---------------Time to First Token----------------
Mean TTFT (ms):                          3986.54   
Median TTFT (ms):                        3207.19   
P90 TTFT (ms):                           8062.30   
P95 TTFT (ms):                           8733.72   
P99 TTFT (ms):                           9662.51    
---------------Inter-Token Latency----------------
Mean ITL (ms):                           95.24     
Median ITL (ms):                         50.97     
P90 ITL (ms):                            70.47     
P95 ITL (ms):                            81.47     
P99 ITL (ms):                            620.00    
Max ITL (ms):                            9296.85   
==================================================

Concurrency = 1

msgpack:

============ Serving Benchmark Result ============
Backend:                                 sglang-oai-chat
Traffic request rate:                    200.0     
Max request concurrency:                 1         
Successful requests:                     200       
Benchmark duration (s):                  66.69     
Total input tokens:                      204800    
Total input text tokens:                 204800    
Total generated tokens:                  40000     
Total generated tokens (retokenized):    39963     
Request throughput (req/s):              3.00      
Input token throughput (tok/s):          3070.82   
Output token throughput (tok/s):         599.77    
Peak output token throughput (tok/s):    604.00    
Peak concurrent requests:                5         
Total token throughput (tok/s):          3670.59   
Concurrency:                             1.00      
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   333.21    
Median E2E Latency (ms):                 332.98    
P90 E2E Latency (ms):                    335.49    
P95 E2E Latency (ms):                    336.64    
P99 E2E Latency (ms):                    337.73    
---------------Time to First Token----------------
Mean TTFT (ms):                          36.25     
Median TTFT (ms):                        36.12     
P90 TTFT (ms):                           38.50     
P95 TTFT (ms):                           38.90     
P99 TTFT (ms):                           39.55     
---------------Inter-Token Latency----------------
Mean ITL (ms):                           1.49      
Median ITL (ms):                         1.49      
P90 ITL (ms):                            1.51      
P95 ITL (ms):                            1.52      
P99 ITL (ms):                            1.58      
Max ITL (ms):                            3.14      
==================================================

pickle:

============ Serving Benchmark Result ============
Backend:                                 sglang-oai-chat
Traffic request rate:                    200.0     
Max request concurrency:                 1         
Successful requests:                     200       
Benchmark duration (s):                  66.85     
Total input tokens:                      204800    
Total input text tokens:                 204800    
Total generated tokens:                  40000     
Total generated tokens (retokenized):    39963     
Request throughput (req/s):              2.99      
Input token throughput (tok/s):          3063.40   
Output token throughput (tok/s):         598.32    
Peak output token throughput (tok/s):    601.00    
Peak concurrent requests:                4         
Total token throughput (tok/s):          3661.72   
Concurrency:                             1.00      
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   333.99    
Median E2E Latency (ms):                 333.93    
P90 E2E Latency (ms):                    334.76    
P95 E2E Latency (ms):                    335.10    
P99 E2E Latency (ms):                    335.31    
---------------Time to First Token----------------
Mean TTFT (ms):                          35.03     
Median TTFT (ms):                        34.95     
P90 TTFT (ms):                           35.92     
P95 TTFT (ms):                           36.09     
P99 TTFT (ms):                           36.35     
---------------Inter-Token Latency----------------
Mean ITL (ms):                           1.50      
Median ITL (ms):                         1.50      
P90 ITL (ms):                            1.52      
P95 ITL (ms):                            1.52      
P99 ITL (ms):                            1.54      
Max ITL (ms):                            3.18      
==================================================

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review and Merge Process

1. Ping Merge Oncalls to start the process. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

---

CI States

Latest PR Test (Base): 🚫 Run #27725799896
Latest PR Test (Extra): ❌ Run #27725799747

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[rainj-me]

/tag-run-ci-label

[rainj-me]

/rerun-failed-ci

[liujianyeey-oss]

Pulled the branch to read — three structural blockers, all probably contributing to the CI red. Inline. Skipping a handful of MAJOR observations on the codec layer (str primitive missing, dec_hook / pickle.loads sink, tensor device drop, _struct_types subclass discovery, rid type narrow) — happy to pile them on if you want; didn't want to drown out the merge-blockers.

[rainj-me]

Pulled the branch to read — three structural blockers, all probably contributing to the CI red. Inline. Skipping a handful of MAJOR observations on the codec layer (str primitive missing, dec_hook / pickle.loads sink, tensor device drop, _struct_types subclass discovery, rid type narrow) — happy to pile them on if you want; didn't want to drown out the merge-blockers.

I will fix all CI test before asking for merge approval.

[merrymercy]

can you just try Optional[List[List[Optional[float]]]]? also do the same for the other variables below this line input_token_logprobs_idx, output_token_logprobs_val, ..

Some example values are

 # no logprobs requested
  input_token_logprobs_val = None

  # one request, prompt logprobs available
  # first token has no previous-token probability, so it is None
  input_token_logprobs_val = [[None, -0.1832, -2.9417, -0.0048]]

  # two requests in a batch
  input_token_logprobs_val = [
      [None, -0.42, -1.87],
      [None, -3.12, -0.09, -5.44],
  ]

  # streaming follow-up chunk after input logprobs were already sent
  input_token_logprobs_val = [[]]

  # multi-item scoring path: no leading None; values correspond to delimiter positions
  input_token_logprobs_val = [[-0.77, -2.35, -0.18]]

so Optional[List[List[Optional[float]]]] should work

[merrymercy]

does the tracing still work after your changes?
@sufeng-buaa can you take a review?

[rainj-me]

/rerun-failed-ci