embedding: centralize capabilities and complete OpenAI compatibility

docs_new/docs/basic_usage/native_api.mdx

@@ -64,6 +64,7 @@ Get the information of the model.
 - `has_audio_understanding`: Whether the model has audio-understanding capability.
 - `model_type`: The model type from the HuggingFace config (e.g., "qwen2", "llama").
 - `architectures`: The model architectures from the HuggingFace config (e.g., ["Qwen2ForCausalLM"]).
+- `embedding`: The resolved embedding-serving plan. It includes pooling, normalization, execution and attention style, Matryoshka dimensions, cache policy, and effective BCG prefill settings. This field is available when the model configuration exposes an embedding capability contract.
 
 ```python Example
 url = f"http://localhost:{port}/get_model_info"
@@ -85,6 +86,7 @@ assert response_json.keys() == {
     "has_audio_understanding",
     "model_type",
     "architectures",
+    "embedding",
 }
 ```
 

docs_new/docs/basic_usage/openai_api_embeddings.mdx

@@ -12,7 +12,7 @@ This tutorial covers the embedding APIs for embedding models. For a list of the
 
 ## Launch A Server
 
-Launch the server in your terminal and wait for it to initialize. Remember to add `--is-embedding` to the command.
+Launch the server in your terminal and wait for it to initialize. Native encoder embedding architectures and `google/embeddinggemma-300m` are detected automatically. Decoder-style embedding models still require `--is-embedding`.
 
 
 
@@ -22,8 +22,8 @@ from sglang.utils import wait_for_server, print_highlight, terminate_process
 
 embedding_process, port = launch_server_cmd(
     """
-python3 -m sglang.launch_server --model-path Alibaba-NLP/gte-Qwen2-1.5B-instruct \
-    --host 0.0.0.0 --is-embedding --log-level warning
+sglang serve --model-path Alibaba-NLP/gte-Qwen2-1.5B-instruct \
+    --is-embedding --log-level warning
 """
 )
 
@@ -117,6 +117,24 @@ input_ids_embedding = json.loads(subprocess.check_output(curl_ids, shell=True))[
 print_highlight(f"Input IDs embedding (first 10): {input_ids_embedding[:10]}")
 ```
 
+## Compact Base64 Responses
+
+Set `encoding_format` to `base64` when JSON arrays would dominate response size. The encoded value contains little-endian FP32 values and can be decoded by OpenAI-compatible clients.
+
+```python Example
+response = requests.post(
+    f"http://localhost:{port}/v1/embeddings",
+    json={
+        "model": "Alibaba-NLP/gte-Qwen2-1.5B-instruct",
+        "input": text,
+        "encoding_format": "base64",
+    },
+)
+
+base64_embedding = response.json()["data"][0]["embedding"]
+print_highlight(f"Base64 embedding: {base64_embedding[:20]}...")
+```
+
 
 ```python Example
 terminate_process(embedding_process)

docs_new/docs/developer_guide/bench_serving.mdx

@@ -16,6 +16,7 @@ This guide explains how to benchmark online serving throughput and latency using
 - `sglang` / `sglang-native`: `POST /generate`
 - `sglang-oai`, `vllm`, `lmdeploy`: `POST /v1/completions`
 - `sglang-oai-chat`, `vllm-chat`, `lmdeploy-chat`: `POST /v1/chat/completions`
+- `sglang-embedding`, `vllm-embedding`: `POST /v1/embeddings`
 - `trt` (TensorRT-LLM): `POST /v2/models/ensemble/generate_stream`
 - `gserver`: Custom server (Not Implemented yet in this script)
 - `truss`: `POST /v1/models/model:predict`
@@ -55,6 +56,38 @@ python3 -m sglang.bench_serving \
   --model meta-llama/Llama-3.1-8B-Instruct
 ```
 
+### Fair embedding comparison
+
+Use the two embedding backends with the same model, tokenizer, input length, prompt count, and concurrency. The benchmark reports input-token throughput and end-to-end latency; embeddings have no decode-side TTFT or TPOT.
+
+```bash Command
+# Start either server on the same hardware and precision, then run one at a time.
+python3 -m sglang.bench_serving \
+  --backend sglang-embedding \
+  --model google/embeddinggemma-300m \
+  --dataset-name random \
+  --random-input-len 2048 \
+  --num-prompts 300 \
+  --max-concurrency 64 \
+  --warmup-requests 3 \
+  --flush-cache
+```
+
+```bash Command
+# vLLM's cache reset endpoint requires VLLM_SERVER_DEV_MODE=1 at server startup.
+python3 -m sglang.bench_serving \
+  --backend vllm-embedding \
+  --model google/embeddinggemma-300m \
+  --dataset-name random \
+  --random-input-len 2048 \
+  --num-prompts 300 \
+  --max-concurrency 64 \
+  --warmup-requests 3 \
+  --flush-cache
+```
+
+`--flush-cache` calls `/flush_cache` for SGLang and `/reset_prefix_cache` for vLLM after warmup. For vLLM, start the server with `VLLM_SERVER_DEV_MODE=1`; without it the benchmark fails loudly rather than accidentally measuring warm-cache performance.
+
 ### Datasets
 
 Select with `--dataset-name`:

docs_new/docs/supported-models/embedding_models.mdx

@@ -5,21 +5,29 @@ description: Dense and sparse embedding models with FlashInfer acceleration and
 SGLang provides robust support for embedding models by integrating efficient serving mechanisms with its flexible programming interface. This integration allows for streamlined handling of embedding tasks, facilitating faster and more accurate retrieval and semantic search operations. SGLang's architecture enables better resource utilization and reduced latency in embedding model deployment.
 
 <Warning>
-Embedding models are executed with `--is-embedding` flag and some may require `--trust-remote-code`
+Native encoder embedding architectures and `google/embeddinggemma-300m` are detected automatically. Decoder-style embedding models require `--is-embedding`; add `--trust-remote-code` when the model requires it.
 </Warning>
 
 ## Quick Start
 
 ### Launch Server
 
 ```bash
-python3 -m sglang.launch_server \
+sglang serve \
   --model-path Qwen/Qwen3-Embedding-4B \
-  --is-embedding \
-  --host 0.0.0.0 \
-  --port 30000
+  --is-embedding
+```
+
+### EmbeddingGemma
+
+EmbeddingGemma uses bidirectional attention and is auto-detected, so its best default command is simply:
+
+```bash
+sglang serve --model-path google/embeddinggemma-300m
 ```
 
+On CUDA, SGLang automatically uses breakable CUDA graph (BCG) for its full encoder prefill and disables incompatible radix-cache and chunked-prefill behavior. Do not add the deprecated piecewise CUDA graph knobs.
+
 ### Client Request
 
 ```python
@@ -30,7 +38,7 @@ url = "http://127.0.0.1:30000"
 payload = {
     "model": "Qwen/Qwen3-Embedding-4B",
     "input": "What is the capital of France?",
-    "encoding_format": "float"
+    "encoding_format": "float"  # or "base64" for compact FP32 responses
 }
 
 response = requests.post(url + "/v1/embeddings", json=payload).json()
@@ -43,12 +51,10 @@ print("Embedding:", response["data"][0]["embedding"])
 For multimodal models like GME that support both text and images:
 
 ```bash
-python3 -m sglang.launch_server \
+sglang serve \
   --model-path Alibaba-NLP/gme-Qwen2-VL-2B-Instruct \
   --is-embedding \
-  --chat-template gme-qwen2-vl \
-  --host 0.0.0.0 \
-  --port 30000
+  --chat-template gme-qwen2-vl
 ```
 
 ```python Example
@@ -85,11 +91,9 @@ print("Embeddings:", [x.get("embedding") for x in response.get("data", [])])
 If the model config already includes `matryoshka_dimensions` or `is_matryoshka` then no override is needed. Otherwise, you can use `--json-model-override-args` as below:
 
 ```bash Command
-python3 -m sglang.launch_server \
+sglang serve \
     --model-path Qwen/Qwen3-Embedding-0.6B \
     --is-embedding \
-    --host 0.0.0.0 \
-    --port 30000 \
     --json-model-override-args '{"matryoshka_dimensions": [128, 256, 512, 1024, 1536]}'
 ```
 
@@ -133,6 +137,12 @@ print("Embedding:", response["data"][0]["embedding"])
     </tr>
   </thead>
   <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>EmbeddingGemma</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`google/embeddinggemma-300m`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>N/A</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Bidirectional Gemma3 text encoder; auto-detected and served with BCG by default</td>
+    </tr>
     <tr>
       <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>E5 (Llama/Mistral based)</td>
       <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`intfloat/e5-mistral-7b-instruct`</td>

python/sglang/benchmark/serving.py

@@ -937,6 +937,7 @@ async def _call_profile_pd(profile_urls: List[Tuple[str, str]], mode: str) -> No
     "sglang-embedding": async_request_openai_embeddings,
     "vllm": async_request_openai_completions,
     "vllm-chat": async_request_openai_chat_completions,
+    "vllm-embedding": async_request_openai_embeddings,
     "lmdeploy": async_request_openai_completions,
     "lmdeploy-chat": async_request_openai_chat_completions,
     "trt": async_request_trt_llm,
@@ -954,12 +955,24 @@ async def _call_profile_pd(profile_urls: List[Tuple[str, str]], mode: str) -> No
     "sglang-embedding": "/v1/embeddings",
     "vllm": "/v1/completions",
     "vllm-chat": "/v1/chat/completions",
+    "vllm-embedding": "/v1/embeddings",
     "lmdeploy": "/v1/completions",
     "lmdeploy-chat": "/v1/chat/completions",
     "trt": "/v2/models/ensemble/generate_stream",
     "truss": "/v1/models/model:predict",
 }
 
+_EMBEDDING_BACKENDS = frozenset(("sglang-embedding", "vllm-embedding"))
+
+
+def flush_server_cache(base_url: str, backend: str) -> None:
+    """Flush an engine's prefix cache after benchmark warmup."""
+    cache_endpoint = (
+        "/reset_prefix_cache" if backend.startswith("vllm") else "/flush_cache"
+    )
+    response = requests.post(base_url + cache_endpoint, headers=get_auth_headers())
+    response.raise_for_status()
+
 
 @dataclass
 class BenchmarkMetrics:
@@ -1419,9 +1432,14 @@ async def limited_request_func(request_func_input, pbar):
             f"Warmup completed with {args.warmup_requests} sequences. Starting main benchmark run..."
         )
 
-    # Flush cache
-    if ("sglang" in backend and _get_bool_env_var("SGLANG_IS_IN_CI")) or flush_cache:
-        requests.post(base_url + "/flush_cache", headers=get_auth_headers())
+    # Flush cache after warmup so the measured run does not benefit from
+    # request-local prefix reuse. vLLM exposes a different, development-mode
+    # endpoint for the same purpose.
+    should_flush_cache = (
+        "sglang" in backend and _get_bool_env_var("SGLANG_IS_IN_CI")
+    ) or flush_cache
+    if should_flush_cache:
+        flush_server_cache(base_url, backend)
 
     time.sleep(1.0)
 
@@ -1595,7 +1613,7 @@ async def limited_request_func(request_func_input, pbar):
                 "Total input vision tokens:", metrics.total_input_vision
             )
         )
-    is_embedding = backend == "sglang-embedding"
+    is_embedding = backend in _EMBEDDING_BACKENDS
     if not is_embedding:
         print("{:<40} {:<10}".format("Total generated tokens:", metrics.total_output))
         print(
@@ -1963,6 +1981,7 @@ def run_benchmark(args_: argparse.Namespace):
             "sglang-oai": 30000,
             "lmdeploy": 23333,
             "vllm": 8000,
+            "vllm-embedding": 8000,
             "trt": 8000,
             "gserver": 9988,
             "truss": 8080,
@@ -2014,14 +2033,14 @@ def run_benchmark(args_: argparse.Namespace):
         print("No model specified or found. Please provide a model using `--model`.")
         sys.exit(1)
 
-    if args.backend != "sglang-embedding" and not check_chat_template(args.model):
+    if args.backend not in _EMBEDDING_BACKENDS and not check_chat_template(args.model):
         print(
             "\nWARNING It is recommended to use the `Chat` or `Instruct` model for benchmarking.\n"
             "Because when the tokenizer counts the output tokens, if there is gibberish, it might count incorrectly.\n"
         )
 
     if (
-        args.backend == "sglang-embedding"
+        args.backend in _EMBEDDING_BACKENDS
         and args.dataset_name in _EMBEDDING_UNSUPPORTED_DATASETS
     ):
         print(f"{args.dataset_name} dataset is unsupported for embeddings benchmark")