[None][feat] AutoDeploy re-onboard GPT_OSS

docs/source/models/supported-models.md

@@ -18,7 +18,7 @@ The following is a table of supported models for the PyTorch backend:
 | `Glm4MoeForCausalLM`                 | GLM-4.5, GLM-4.6, GLM-4.7          | `THUDM/GLM-4-100B-A10B`                      |
 | `Glm4MoeLiteForCausalLM` [^6]        | GLM-4.7-Flash                      | `zai-org/GLM-4.7-Flash`                      |
 | `GlmMoeDsaForCausalLM`               | GLM-5                              | `zai-org/GLM-5`                              |
-| `GptOssForCausalLM`                  | GPT-OSS                            | `openai/gpt-oss-120b`                        |
+| `GptOssForCausalLM` [^12]            | GPT-OSS                            | `openai/gpt-oss-20b`, `openai/gpt-oss-120b`  |
 | `KimiK25ForConditionalGeneration`    | Kimi-K2.5                          | `moonshotai/Kimi-K2.5`                       |
 | `LlamaForCausalLM`                   | Llama 3.1, Llama 3, Llama 2, LLaMA | `meta-llama/Meta-Llama-3.1-70B`              |
 | `Llama4ForConditionalGeneration`     | Llama 4                            | `meta-llama/Llama-4-Scout-17B-16E-Instruct`  |
@@ -68,6 +68,7 @@ Note: Support for other models may vary. Features marked "N/A" are not applicabl
 [^9]: Supported via the [AutoDeploy](../features/auto_deploy/auto-deploy.md) backend. See [AD config](../../../examples/auto_deploy/model_registry/configs/minimax_m2.7.yaml).
 [^10]: Requires manually upgrading transformers to 5.5.3 for Gemma 4.
 [^11]: Audio modality only supported on E2B/E4B variants.
+[^12]: Also supported via the [AutoDeploy](../features/auto_deploy/auto-deploy.md) backend (native PyTorch backend support is unchanged). See AD configs for [20B](../../../examples/auto_deploy/model_registry/configs/gpt_oss_20b.yaml) and [120B](../../../examples/auto_deploy/model_registry/configs/gpt_oss_120b.yaml).
 
 
 # Multimodal Feature Support Matrix (PyTorch Backend)

examples/auto_deploy/cookbooks/gpt_oss_trtllm_cookbook.ipynb

@@ -0,0 +1,281 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Deploying GPT-OSS with TensorRT-LLM (AutoDeploy)\n",
+    "\n",
+    "This notebook walks you through deploying OpenAI's `openai/gpt-oss-20b` and `openai/gpt-oss-120b` models using TensorRT-LLM's AutoDeploy backend.\n",
+    "\n",
+    "[TensorRT-LLM](https://nvidia.github.io/TensorRT-LLM/) is NVIDIA's open-source library for accelerating and optimizing LLM inference on NVIDIA GPUs. AutoDeploy is a one-shot graph-transform pipeline that translates a HuggingFace model into a deploy-ready engine — see the [AutoDeploy guide](https://nvidia.github.io/TensorRT-LLM/torch/auto_deploy/auto-deploy.html) for details.\n",
+    "\n",
+    "**Model Resources:**\n",
+    "- [HuggingFace — gpt-oss-20b](https://huggingface.co/openai/gpt-oss-20b)\n",
+    "- [HuggingFace — gpt-oss-120b](https://huggingface.co/openai/gpt-oss-120b)\n",
+    "- [OpenAI — gpt-oss announcement](https://openai.com/index/introducing-gpt-oss/)\n",
+    "- [OpenAI — gpt-oss model card](https://openai.com/index/gpt-oss-model-card/)\n",
+    "\n",
+    "**Model Highlights:**\n",
+    "- Mixture-of-Experts (MoE) architecture released by OpenAI in 2025 under Apache 2.0\n",
+    "- `gpt-oss-20b`: 21B parameters total, 3.6B active, 24 layers, 32 experts, top-4 routing, ~16 GB MXFP4 weights\n",
+    "- `gpt-oss-120b`: 117B parameters total, 5.1B active, 36 layers, 128 experts, top-4 routing, ~63 GB MXFP4 weights\n",
+    "- Native MXFP4 quantization for MoE weights (handled by AutoDeploy's `quantize_mxfp4_moe` transform)\n",
+    "- 64 query heads / 8 key-value heads (GQA), `head_dim=64`, `hidden_size=2880`\n",
+    "- Per-head learnable attention sinks; alternating sliding (window=128) and full-attention layers\n",
+    "- 131,072 token context length (YaRN-scaled RoPE)\n",
+    "- Channel-based reasoning protocol (`analysis` → `final`)\n",
+    "\n",
+    "**Prerequisites:**\n",
+    "- NVIDIA GPU with CUDA 12.x and recent drivers\n",
+    "  - `gpt-oss-20b`: ≥ 1×80 GB GPU (the cookbook below pairs it with 2 GPUs for parallelism)\n",
+    "  - `gpt-oss-120b`: ≥ 4×80 GB or 8×80 GB GPUs (the cookbook below uses 8)\n",
+    "- Python 3.10+\n",
+    "- TensorRT-LLM ([container](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tensorrt-llm/containers/release) or pip install)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Prerequisites & Environment\n",
+    "\n",
+    "Set up a containerized environment for TensorRT-LLM by running the following command in a terminal:\n",
+    "\n",
+    "```shell\n",
+    "docker run --rm -it --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 --gpus=all -p 8000:8000 nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc1\n",
+    "```\n",
+    "\n",
+    "You now have TensorRT-LLM set up!"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# If pip not found\n",
+    "!python -m ensurepip --default-pip"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%pip install torch openai"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Verify GPU\n",
+    "\n",
+    "Check that CUDA is available and the GPU is detected correctly."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Environment check\n",
+    "import sys\n",
+    "\n",
+    "import torch\n",
+    "\n",
+    "print(f\"Python: {sys.version}\")\n",
+    "print(f\"CUDA available: {torch.cuda.is_available()}\")\n",
+    "print(f\"Num GPUs: {torch.cuda.device_count()}\")\n",
+    "\n",
+    "if torch.cuda.is_available():\n",
+    "    for i in range(torch.cuda.device_count()):\n",
+    "        print(f\"GPU[{i}]: {torch.cuda.get_device_name(i)}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## OpenAI-Compatible Server\n",
+    "\n",
+    "Start a local OpenAI-compatible server with TensorRT-LLM via the terminal, within the running docker container.\n",
+    "\n",
+    "Each gpt-oss size has its own AutoDeploy YAML under `examples/auto_deploy/model_registry/configs/`:\n",
+    "- `gpt_oss_20b.yaml` (world_size=2)\n",
+    "- `gpt_oss_120b.yaml` (world_size=8)\n",
+    "\n",
+    "Pick the YAML that matches the model size you want to deploy."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Load `gpt-oss-20b`\n",
+    "\n",
+    "Launch the TensorRT-LLM server on 2 GPUs:\n",
+    "\n",
+    "```shell\n",
+    "trtllm-serve \"openai/gpt-oss-20b\" \\\n",
+    "  --host 0.0.0.0 \\\n",
+    "  --port 8000 \\\n",
+    "  --backend _autodeploy \\\n",
+    "  --extra_llm_api_options examples/auto_deploy/model_registry/configs/gpt_oss_20b.yaml\n",
+    "```\n",
+    "\n",
+    "### Load `gpt-oss-120b`\n",
+    "\n",
+    "Launch the TensorRT-LLM server on 8 GPUs:\n",
+    "\n",
+    "```shell\n",
+    "trtllm-serve \"openai/gpt-oss-120b\" \\\n",
+    "  --host 0.0.0.0 \\\n",
+    "  --port 8000 \\\n",
+    "  --backend _autodeploy \\\n",
+    "  --extra_llm_api_options examples/auto_deploy/model_registry/configs/gpt_oss_120b.yaml\n",
+    "```\n",
+    "\n",
+    "Both YAMLs are self-contained — they include the compile backend, attention backend, world size, KV-cache settings and the CUDA-graph batch-size buckets needed for serving."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Your server is now running!"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Use the API\n",
+    "\n",
+    "Use the OpenAI-compatible client to send requests to the TensorRT-LLM server."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from openai import OpenAI\n",
+    "\n",
+    "# Setup client\n",
+    "BASE_URL = \"http://0.0.0.0:8000/v1\"\n",
+    "API_KEY = \"null\"\n",
+    "client = OpenAI(base_url=BASE_URL, api_key=API_KEY)\n",
+    "\n",
+    "# Set this to whichever model you started the server with.\n",
+    "MODEL_ID = \"openai/gpt-oss-20b\"  # or \"openai/gpt-oss-120b\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Basic chat completion\n",
+    "print(\"Chat Completion Example\")\n",
+    "print(\"=\" * 50)\n",
+    "\n",
+    "response = client.chat.completions.create(\n",
+    "    model=MODEL_ID,\n",
+    "    messages=[\n",
+    "        {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n",
+    "        {\"role\": \"user\", \"content\": \"Where is the capital of Iceland?\"},\n",
+    "    ],\n",
+    "    temperature=1.0,\n",
+    "    top_p=1.0,\n",
+    "    max_tokens=128,\n",
+    ")\n",
+    "\n",
+    "print(\"Response:\")\n",
+    "print(response.choices[0].message.content)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Streaming chat completion\n",
+    "print(\"Streaming response:\")\n",
+    "print(\"=\" * 50)\n",
+    "\n",
+    "stream = client.chat.completions.create(\n",
+    "    model=MODEL_ID,\n",
+    "    messages=[\n",
+    "        {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n",
+    "        {\"role\": \"user\", \"content\": \"Write a Python function that checks if a number is prime.\"},\n",
+    "    ],\n",
+    "    temperature=1.0,\n",
+    "    max_tokens=512,\n",
+    "    stream=True,\n",
+    ")\n",
+    "\n",
+    "for chunk in stream:\n",
+    "    if chunk.choices[0].delta.content:\n",
+    "        print(chunk.choices[0].delta.content, end=\"\", flush=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Evaluation Parameters\n",
+    "\n",
+    "OpenAI's gpt-oss model card recommends the following defaults:\n",
+    "\n",
+    "- `temperature`: 1.0\n",
+    "- `top_p`: 1.0\n",
+    "- `max_tokens`: 131072 (model's context limit; trim for serving SLOs)\n",
+    "\n",
+    "The model uses a channel-based reasoning protocol (`analysis` → `final`); the `analysis` segment is the model's chain-of-thought and the `final` segment is the user-facing answer. Both are returned via the OpenAI-compatible API as a single response — the chat template handles the formatting."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Additional Resources\n",
+    "\n",
+    "- [TensorRT-LLM Documentation](https://nvidia.github.io/TensorRT-LLM/)\n",
+    "- [AutoDeploy Guide](https://nvidia.github.io/TensorRT-LLM/torch/auto_deploy/auto-deploy.html)\n",
+    "- [gpt-oss-20b on HuggingFace](https://huggingface.co/openai/gpt-oss-20b)\n",
+    "- [gpt-oss-120b on HuggingFace](https://huggingface.co/openai/gpt-oss-120b)\n",
+    "- [OpenAI gpt-oss announcement blog](https://openai.com/index/introducing-gpt-oss/)\n",
+    "- [OpenAI gpt-oss model card](https://openai.com/index/gpt-oss-model-card/)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

examples/auto_deploy/model_registry/configs/gpt_oss_120b.yaml

@@ -0,0 +1,21 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+# OpenAI GPT-OSS-120B (128 experts, top-4, MXFP4 quantized) — standalone AD serving config.
+# 36 layers (alternating sliding/full), GQA (64 Q / 8 KV heads), head_dim=64, hidden=2880.
+# Weights are stored in MXFP4 on HF; AD's quantize_mxfp4_moe transform handles it.
+runtime: trtllm
+model_factory: AutoModelForCausalLM
+attn_backend: trtllm
+compile_backend: torch-cudagraph
+skip_loading_weights: false
+world_size: 4
+max_batch_size: 128
+max_seq_len: 4096
+max_num_tokens: 8192
+enable_chunked_prefill: true
+cuda_graph_config:
+  batch_sizes: [1, 2, 4, 8, 16, 32, 64, 128]
+kv_cache_config:
+  enable_block_reuse: false
+  free_gpu_memory_fraction: 0.8

examples/auto_deploy/model_registry/configs/gpt_oss_20b.yaml

@@ -0,0 +1,21 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+# OpenAI GPT-OSS-20B (32 experts, top-4, MXFP4 quantized) — standalone AD serving config.
+# 24 layers (alternating sliding/full), GQA (64 Q / 8 KV heads), head_dim=64, hidden=2880.
+# Weights are stored in MXFP4 on HF; AD's quantize_mxfp4_moe transform handles it.
+runtime: trtllm
+model_factory: AutoModelForCausalLM
+attn_backend: trtllm
+compile_backend: torch-cudagraph
+skip_loading_weights: false
+world_size: 1
+max_batch_size: 128
+max_seq_len: 4096
+max_num_tokens: 8192
+enable_chunked_prefill: true
+cuda_graph_config:
+  batch_sizes: [1, 2, 4, 8, 16, 32, 64, 128]
+kv_cache_config:
+  enable_block_reuse: false
+  free_gpu_memory_fraction: 0.8

examples/auto_deploy/model_registry/models.yaml

@@ -155,8 +155,8 @@ models:
   config_id: default_ws_2
   yaml_extra: ['dashboard_default.yaml', 'world_size_2.yaml']
 - name: openai/gpt-oss-20b
-  config_id: default_ws_2
-  yaml_extra: ['dashboard_default.yaml', 'world_size_2.yaml']
+  config_id: gpt_oss_20b
+  yaml_extra: ['gpt_oss_20b.yaml']
 - name: ibm-granite/granite-3.0-8b-instruct
   config_id: default_ws_2
   yaml_extra: ['dashboard_default.yaml', 'world_size_2.yaml']
@@ -293,8 +293,8 @@ models:
   config_id: multimodal
   yaml_extra: ['dashboard_default.yaml', 'world_size_8.yaml', 'multimodal.yaml']
 - name: openai/gpt-oss-120b
-  config_id: num_hidden_layers_5
-  yaml_extra: ['dashboard_default.yaml', 'world_size_8.yaml', 'num_hidden_layers_5.yaml']
+  config_id: gpt_oss_120b
+  yaml_extra: ['gpt_oss_120b.yaml']
 - name: meta-llama/Llama-4-Scout-17B-16E-Instruct
   config_id: multimodal__llama4_scout
   yaml_extra: ['dashboard_default.yaml', 'world_size_8.yaml', 'multimodal.yaml', 'llama4_scout.yaml']