From b24dda9455f711f3aa4dc86db397e7938f3c8290 Mon Sep 17 00:00:00 2001 From: Justin Chu Date: Thu, 23 Apr 2026 20:20:52 +0000 Subject: [PATCH 1/3] docs: clarify --ep vs --optimize in getting-started guide Add comprehensive documentation for --ep (recommended) and --optimize (manual) CLI flags. Explain when to use each, with examples for all supported EPs. Note that --ep affects graph construction while --optimize is post-hoc only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu --- docs/getting-started.md | 64 +++++++++++++++++++++++++++++++++++++---- 1 file changed, 58 insertions(+), 6 deletions(-) diff --git a/docs/getting-started.md b/docs/getting-started.md index 605b255a..1dd5abb4 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -177,7 +177,7 @@ rewrite(model, pattern_rewrite_rules=skip_norm_rules()) Or via CLI: ```bash -mobius build --model Qwen/Qwen2.5-0.5B output/ --optimize +mobius build --model Qwen/Qwen2.5-0.5B output/ --ep cuda --dtype f16 ``` ## CLI Reference @@ -193,20 +193,71 @@ mobius build --model meta-llama/Llama-3.2-1B output/ # From a local config directory (with safetensors weights) mobius build --config /path/to/model/ output/ -# With options +# Target a specific execution provider (recommended) mobius build --model Qwen/Qwen2.5-0.5B output/ \ - --dtype f16 \ - --external-data safetensors \ - --optimize + --ep cuda --dtype f16 + +# Export for ORT GenAI runtime +mobius build --model Qwen/Qwen2.5-0.5B output/ \ + --ep cuda --dtype f16 --runtime ort-genai ``` Key flags: - `--dtype` — Target dtype (`f32`, `f16`, `bf16`) +- `--ep` — Target execution provider (see below) +- `--runtime` — Target runtime format (`ort-genai`) - `--no-weights` — Build graph only (no weight download) - `--external-data` — `onnx` (default) or `safetensors` -- `--optimize` — Apply rewrite rules (e.g. fused attention) +- `--optimize` — Apply individual rewrite rules (see below) - `--component` — Build only one component from a diffusers pipeline +#### `--ep` vs `--optimize`: when to use which + +**Use `--ep` (recommended)** when building for a known runtime/hardware target. +It drives the entire build pipeline — graph construction, operator fusion, +dead input removal, and KV cache sizing are all tailored for the target EP: + +```bash +# CUDA GPU (uses GQA, SkipNorm, PackQKV fusions) +mobius build --model meta-llama/Llama-3.2-1B output/ --ep cuda --dtype f16 + +# DirectML (uses GQA without fused RoPE) +mobius build --model meta-llama/Llama-3.2-1B output/ --ep dml --dtype f16 + +# TensorRT-RTX (uses GQA, no SkipLayerNorm) +mobius build --model meta-llama/Llama-3.2-1B output/ --ep trt-rtx --dtype f16 + +# WebGPU +mobius build --model meta-llama/Llama-3.2-1B output/ --ep webgpu --dtype f16 + +# Portable ONNX (no vendor fusions, runs on any ONNX runtime) +mobius build --model meta-llama/Llama-3.2-1B output/ --ep onnx-standard +``` + +Run `mobius list eps` to see all available execution providers. + +**Use `--optimize` only** for manual, targeted rewrite rule application on +a pre-built model. Rules are applied post-hoc and do not affect graph +construction. This is useful for experimentation or when `--ep` doesn't +cover a specific optimization: + +```bash +# Apply specific rules +mobius build --model meta-llama/Llama-3.2-1B output/ \ + --optimize=group_query_attention,skip_norm + +# Apply all available rules +mobius build --model meta-llama/Llama-3.2-1B output/ --optimize +``` + +Available rules: `group_query_attention`, `skip_norm`, `skip_layer_norm`, +`packed_attention`, `bias_gelu`. + +> **Note**: `--ep` and `--optimize` cannot be used together. `--ep` is +> strictly more capable — it affects both graph construction and +> optimization, while `--optimize` only applies rewrite rules after the +> graph is built. + ### `build-gguf` — Build from a GGUF file ```bash @@ -219,6 +270,7 @@ mobius build-gguf model.gguf --output output/ mobius list models # All supported architectures mobius list tasks # Available task types mobius list dtypes # Supported dtypes +mobius list eps # Available execution providers ``` ### `info` — Inspect a model From 7a8489ae6c28b9bd22485d5f794aae4ed6a84f02 Mon Sep 17 00:00:00 2001 From: Justin Chu Date: Thu, 23 Apr 2026 20:24:25 +0000 Subject: [PATCH 2/3] docs: fix --ep/--optimize note, they can be combined Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu --- docs/getting-started.md | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/docs/getting-started.md b/docs/getting-started.md index 1dd5abb4..b24f0614 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -248,15 +248,20 @@ mobius build --model meta-llama/Llama-3.2-1B output/ \ # Apply all available rules mobius build --model meta-llama/Llama-3.2-1B output/ --optimize + +# Combine EP-aware building with additional post-hoc rules +mobius build --model meta-llama/Llama-3.2-1B output/ \ + --ep cuda --dtype f16 --optimize=bias_gelu ``` Available rules: `group_query_attention`, `skip_norm`, `skip_layer_norm`, `packed_attention`, `bias_gelu`. -> **Note**: `--ep` and `--optimize` cannot be used together. `--ep` is -> strictly more capable — it affects both graph construction and -> optimization, while `--optimize` only applies rewrite rules after the -> graph is built. +> **Tip**: Prefer `--ep` over `--optimize` for production builds. `--ep` +> affects both graph construction and optimization (EP-aware KV cache +> sizing, dead input removal, operator fusion), while `--optimize` only +> applies rewrite rules after the graph is built. They can be combined +> when you need both EP-aware construction and additional post-hoc rules. ### `build-gguf` — Build from a GGUF file From bcd911ee29a7d361ec10e0bfd3a1562660ac747e Mon Sep 17 00:00:00 2001 From: Justin Chu Date: Thu, 23 Apr 2026 20:28:24 +0000 Subject: [PATCH 3/3] docs: add cpu and default EP examples, clarify default vs onnx-standard Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu --- docs/getting-started.md | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/docs/getting-started.md b/docs/getting-started.md index b24f0614..d7ab8986 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -218,19 +218,25 @@ It drives the entire build pipeline — graph construction, operator fusion, dead input removal, and KV cache sizing are all tailored for the target EP: ```bash -# CUDA GPU (uses GQA, SkipNorm, PackQKV fusions) +# Default (portable ONNX with standard fusions, no vendor-specific ops) +mobius build --model meta-llama/Llama-3.2-1B output/ + +# CPU (GQA fusion for f32) +mobius build --model meta-llama/Llama-3.2-1B output/ --ep cpu + +# CUDA GPU (GQA, SkipNorm, PackQKV fusions for f16/bf16) mobius build --model meta-llama/Llama-3.2-1B output/ --ep cuda --dtype f16 -# DirectML (uses GQA without fused RoPE) +# DirectML (GQA without fused RoPE) mobius build --model meta-llama/Llama-3.2-1B output/ --ep dml --dtype f16 -# TensorRT-RTX (uses GQA, no SkipLayerNorm) +# TensorRT-RTX (GQA, no SkipLayerNorm) mobius build --model meta-llama/Llama-3.2-1B output/ --ep trt-rtx --dtype f16 # WebGPU mobius build --model meta-llama/Llama-3.2-1B output/ --ep webgpu --dtype f16 -# Portable ONNX (no vendor fusions, runs on any ONNX runtime) +# Strict ONNX standard (zero custom ops, runs on any ONNX runtime) mobius build --model meta-llama/Llama-3.2-1B output/ --ep onnx-standard ```