Tutorial: Running Gemma4 on Foundry Local

[justinchuby]

Running Gemma4 on Foundry Local — Complete Guide

This tutorial walks through exporting a Gemma4 model with mobius, building ORT + GenAI from source, and serving the model through Foundry Local's OpenAI-compatible API.

Tested on: 8x NVIDIA H200 GPUs, CUDA 13.0, Ubuntu, Python 3.12

Prerequisites

• Linux machine with NVIDIA GPU
• conda (or any Python 3.10+ environment)
• ~20 GB disk space for model weights

Step 1: Environment Setup

# Create conda environment
conda create -n onnx python=3.12 -y
conda activate onnx

# Verify CUDA
nvidia-smi  # Should show GPU info
nvcc --version  # Need CUDA toolkit (12.8+ or 13.0)

If CUDA toolkit is not installed:

# CUDA 13.0 (if available)
# Or download from https://developer.nvidia.com/cuda-downloads
export CUDA_HOME=/usr/local/cuda-13.0  # adjust to your installation
export PATH=$CUDA_HOME/bin:$PATH

Set up cuDNN (if not system-installed):

# cuDNN is often available via pip
pip install nvidia-cudnn-cu12

# Create a cuDNN home directory for ORT build
mkdir -p ~/cudnn9/{lib,include}
CUDNN_PKG=$(python -c "import nvidia.cudnn; import pathlib; print(pathlib.Path(nvidia.cudnn.__file__).parent)")
ln -sf $CUDNN_PKG/lib/* ~/cudnn9/lib/
ln -sf $CUDNN_PKG/include/* ~/cudnn9/include/
export CUDNN_HOME=~/cudnn9

Step 2: Build ORT from Source

ORT 1.27+ is required for Gemma4 support (head_dim=512 in GroupQueryAttention).

git clone https://github.com/microsoft/onnxruntime.git ~/dev/onnxruntime
cd ~/dev/onnxruntime

./build.sh \
  --config Release \
  --use_cuda \
  --cuda_home $CUDA_HOME \
  --cudnn_home $CUDNN_HOME \
  --cmake_extra_defines CMAKE_CUDA_ARCHITECTURES=native onnxruntime_USE_FLASH_ATTENTION=ON \
  --build_wheel \
  --enable_pybind \
  --parallel \
  --skip_tests

Note: The build may fail on test runner targets (onnxruntime_perf_test) due to abseil linking.
The core library and pybind module still build. Build the wheel manually:

cd build/Linux/Release
python ~/dev/onnxruntime/setup.py bdist_wheel
pip install dist/onnxruntime-*.whl --force-reinstall --no-deps

Verify:

import onnxruntime as ort
print(ort.__version__)  # 1.27.0
print(ort.get_available_providers())  # Should include CUDAExecutionProvider

Step 3: Create ORT Install Layout for GenAI

GenAI needs headers + libraries in a specific layout:

mkdir -p ~/ort-install/{include,lib}

# Headers
cp ~/dev/onnxruntime/include/onnxruntime/core/session/*.h ~/ort-install/include/

# Libraries
cp ~/dev/onnxruntime/build/Linux/Release/libonnxruntime.so ~/ort-install/lib/
cp ~/dev/onnxruntime/build/Linux/Release/libonnxruntime_providers_cuda.so ~/ort-install/lib/
cp ~/dev/onnxruntime/build/Linux/Release/libonnxruntime_providers_shared.so ~/ort-install/lib/

Step 4: Build GenAI from Source

GenAI 0.14+ (with PR #2103) is required for Gemma4 multimodal support.

git clone https://github.com/microsoft/onnxruntime-genai.git ~/dev/onnxruntime-genai
cd ~/dev/onnxruntime-genai

python build.py \
  --config Release \
  --use_cuda \
  --cuda_home $CUDA_HOME \
  --ort_home ~/ort-install \
  --parallel \
  --skip_tests \
  --skip_examples \
  --cmake_extra_defines CMAKE_CUDA_ARCHITECTURES=native \
  --update --build

pip install build/Linux/Release/wheel/onnxruntime_genai_cuda-*.whl --no-deps

Verify (should show NO API version warnings):

import onnxruntime_genai as og
print(og.__version__)  # 0.14.0-dev
print(og.is_cuda_available())  # True

Step 5: Export Gemma4 with mobius

pip install mobius-ai[transformers]

# Export with ONNX external data (required for proper CUDA alignment)
mobius build \
  --model google/gemma-4-e2b-it \
  --dtype f16 \
  --optimize \
  --ep default \
  --runtime ort-genai \
  --external-data onnx \
  ~/gemma4-e2b-it-onnx/

This produces:

~/gemma4-e2b-it-onnx/
├── decoder/model.onnx + model.onnx.data
├── embedding/model.onnx + model.onnx.data
├── vision_encoder/model.onnx + model.onnx.data
├── audio_encoder/model.onnx + model.onnx.data
├── genai_config.json
├── tokenizer.json
├── tokenizer_config.json
├── image_processor.json
└── audio_feature_extraction.json

Quick test (without Foundry):

import onnxruntime_genai as og

model = og.Model("~/gemma4-e2b-it-onnx")
tokenizer = og.Tokenizer(model)

prompt = "<bos><start_of_turn>user\nWhat is the capital of France?<end_of_turn>\n<start_of_turn>model\n"
ids = tokenizer.encode(prompt)

params = og.GeneratorParams(model)
params.set_search_options(max_length=80, do_sample=False, repetition_penalty=1.2)

gen = og.Generator(model, params)
gen.append_tokens(ids)
while not gen.is_done():
    gen.generate_next_token()

output = gen.get_sequence(0)
print(tokenizer.decode(output))
# Expected: "The capital of France is Paris."
# Speed: ~12-15 tok/s on CPU

Step 6: Install Foundry Local

pip install foundry-local-sdk

# IMPORTANT: Foundry installs its own onnxruntime + onnxruntime-genai.
# Re-install our builds to override:
pip install ~/dev/onnxruntime/build/Linux/Release/dist/onnxruntime-*.whl --force-reinstall --no-deps
pip install ~/dev/onnxruntime-genai/build/Linux/Release/wheel/onnxruntime_genai_cuda-*.whl --force-reinstall --no-deps

Why this works: Foundry's native core creates symlinks from its binary directory to the Python-installed ORT/GenAI packages. When we install our versions, Foundry automatically picks them up.

Fix .so.dbg bug (if present):

# Foundry may ship a .so.dbg file that confuses the file finder
CORE_DIR=$(python -c "import foundry_local_core; import pathlib; print(pathlib.Path(foundry_local_core.__file__).parent / bin)")
rm -f "$CORE_DIR/Microsoft.AI.Foundry.Local.Core.so.dbg"

Step 7: Register Custom Model in Foundry Local

from foundry_local_sdk import Configuration, FoundryLocalManager

config = Configuration(app_name="test", model_cache_dir="/tmp/foundry-cache")
manager = FoundryLocalManager(config)

Find the cache directory and place your model:

CACHE_DIR="/tmp/foundry-cache"  # or ~/.foundry/cache on default installs
MODEL_DIR="$CACHE_DIR/Custom/gemma-4-e2b-it"
mkdir -p "$MODEL_DIR"

# Copy all exported model files
cp -r ~/gemma4-e2b-it-onnx/* "$MODEL_DIR/"

Create inference_model.json in the model directory:

{
  "Name": "gemma-4-e2b-it",
  "PromptTemplate": {
    "user": "<start_of_turn>user\n{Content}<end_of_turn>",
    "assistant": "<start_of_turn>model\n{Content}<end_of_turn>",
    "prompt": "<start_of_turn>user\n{Content}<end_of_turn>\n<start_of_turn>model"
  }
}

Step 8: Serve and Test

from foundry_local_sdk import Configuration, FoundryLocalManager
import requests, json

config = Configuration(app_name="gemma4", model_cache_dir="/tmp/foundry-cache")
manager = FoundryLocalManager(config)
manager.download_and_register_eps()
manager.start_web_service()

base = manager.urls[0]  # e.g., http://127.0.0.1:42545

# Load model
m = manager.catalog.get_model("gemma-4-e2b-it")
m.load()

# Chat completion (OpenAI-compatible API)
r = requests.post(f"{base}/v1/chat/completions", json={
    "model": "gemma-4-e2b-it",
    "messages": [{"role": "user", "content": "What is the capital of France?"}],
    "max_tokens": 50,
    "temperature": 0,
})
print(r.json()["choices"][0]["message"]["content"])
# Output: "Paris."

manager.stop_web_service()

Using OpenAI Python client:

from openai import OpenAI

client = OpenAI(base_url=f"{base}/v1", api_key="none")
response = client.chat.completions.create(
    model="gemma-4-e2b-it",
    messages=[{"role": "user", "content": "Write a short poem about AI."}],
    max_tokens=200,
)
print(response.choices[0].message.content)

Expected Results

Test	Result
Model registration	✅ Appears in Foundry catalog
Model loading	✅ m.load() succeeds
Chat completion	✅ Correct responses
Speed (CPU)	~11-15 tok/s
108-token poem generation	9.3s

Known Issues

1. CUDA EP segfault during GenAI prefill (issue CUDA EP: CUBLAS misaligned address crash during prefill with Gemma4 model microsoft/onnxruntime-genai#2120). Use CPU EP for now (set provider_options: [] in genai_config.json).

2. Safetensors external data has 16-byte alignment — insufficient for cuBLAS FP16 GEMM. Always use --external-data onnx which provides 4096-byte alignment.

3. Foundry SDK .so.dbg bug: get_native_binary_paths() may resolve to .so.dbg instead of .so. Remove the .dbg file as workaround.

4. GenAI provider name: genai_config.json should use "cuda" (lowercase) not "CUDAExecutionProvider" for the provider key. The NormalizeProviderName() function in GenAI doesn't normalize CUDA.

5. Foundry SDK model_cache_dir: The Python SDK's get_model() works for custom models in the Custom/ directory, but list_models() may not show them in the full catalog listing.

Versions Used

Component	Version
ORT	1.27.0 (built from source)
GenAI	0.14.0-dev (main, includes PR #2103)
Foundry Local SDK	1.0.0
mobius	latest (fun-asr-support branch)
CUDA	13.0
GPUs	8x NVIDIA H200
Python	3.12