Skip to content

Commit 2e91111

Browse files
justinchubyCopilot
andauthored
docs: clarify --ep vs --optimize in getting-started guide (#201)
Improve the CLI documentation in `docs/getting-started.md`: - **`--ep` (recommended)**: Full EP-aware pipeline — graph construction, operator fusion, dead input removal, KV cache sizing. Examples for all 6 EPs (cpu, cuda, dml, webgpu, trt-rtx, onnx-standard). - **`--optimize` (manual)**: Post-hoc rewrite rule application. Useful for experimentation. Does not affect graph construction. - Clear note that the two cannot be combined and `--ep` is strictly more capable. - Added `mobius list eps` to the CLI reference. - Updated the quick-start example to use `--ep cuda --dtype f16` instead of `--optimize`. --------- Signed-off-by: Justin Chu <justinchu@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent e845a3d commit 2e91111

1 file changed

Lines changed: 69 additions & 6 deletions

File tree

docs/getting-started.md

Lines changed: 69 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -177,7 +177,7 @@ rewrite(model, pattern_rewrite_rules=skip_norm_rules())
177177
Or via CLI:
178178

179179
```bash
180-
mobius build --model Qwen/Qwen2.5-0.5B output/ --optimize
180+
mobius build --model Qwen/Qwen2.5-0.5B output/ --ep cuda --dtype f16
181181
```
182182

183183
## CLI Reference
@@ -193,20 +193,82 @@ mobius build --model meta-llama/Llama-3.2-1B output/
193193
# From a local config directory (with safetensors weights)
194194
mobius build --config /path/to/model/ output/
195195

196-
# With options
196+
# Target a specific execution provider (recommended)
197197
mobius build --model Qwen/Qwen2.5-0.5B output/ \
198-
--dtype f16 \
199-
--external-data safetensors \
200-
--optimize
198+
--ep cuda --dtype f16
199+
200+
# Export for ORT GenAI runtime
201+
mobius build --model Qwen/Qwen2.5-0.5B output/ \
202+
--ep cuda --dtype f16 --runtime ort-genai
201203
```
202204

203205
Key flags:
204206
- `--dtype` — Target dtype (`f32`, `f16`, `bf16`)
207+
- `--ep` — Target execution provider (see below)
208+
- `--runtime` — Target runtime format (`ort-genai`)
205209
- `--no-weights` — Build graph only (no weight download)
206210
- `--external-data``onnx` (default) or `safetensors`
207-
- `--optimize` — Apply rewrite rules (e.g. fused attention)
211+
- `--optimize` — Apply individual rewrite rules (see below)
208212
- `--component` — Build only one component from a diffusers pipeline
209213

214+
#### `--ep` vs `--optimize`: when to use which
215+
216+
**Use `--ep` (recommended)** when building for a known runtime/hardware target.
217+
It drives the entire build pipeline — graph construction, operator fusion,
218+
dead input removal, and KV cache sizing are all tailored for the target EP:
219+
220+
```bash
221+
# Default (portable ONNX with standard fusions, no vendor-specific ops)
222+
mobius build --model meta-llama/Llama-3.2-1B output/
223+
224+
# CPU (GQA fusion for f32)
225+
mobius build --model meta-llama/Llama-3.2-1B output/ --ep cpu
226+
227+
# CUDA GPU (GQA, SkipNorm, PackQKV fusions for f16/bf16)
228+
mobius build --model meta-llama/Llama-3.2-1B output/ --ep cuda --dtype f16
229+
230+
# DirectML (GQA without fused RoPE)
231+
mobius build --model meta-llama/Llama-3.2-1B output/ --ep dml --dtype f16
232+
233+
# TensorRT-RTX (GQA, no SkipLayerNorm)
234+
mobius build --model meta-llama/Llama-3.2-1B output/ --ep trt-rtx --dtype f16
235+
236+
# WebGPU
237+
mobius build --model meta-llama/Llama-3.2-1B output/ --ep webgpu --dtype f16
238+
239+
# Strict ONNX standard (zero custom ops, runs on any ONNX runtime)
240+
mobius build --model meta-llama/Llama-3.2-1B output/ --ep onnx-standard
241+
```
242+
243+
Run `mobius list eps` to see all available execution providers.
244+
245+
**Use `--optimize` only** for manual, targeted rewrite rule application on
246+
a pre-built model. Rules are applied post-hoc and do not affect graph
247+
construction. This is useful for experimentation or when `--ep` doesn't
248+
cover a specific optimization:
249+
250+
```bash
251+
# Apply specific rules
252+
mobius build --model meta-llama/Llama-3.2-1B output/ \
253+
--optimize=group_query_attention,skip_norm
254+
255+
# Apply all available rules
256+
mobius build --model meta-llama/Llama-3.2-1B output/ --optimize
257+
258+
# Combine EP-aware building with additional post-hoc rules
259+
mobius build --model meta-llama/Llama-3.2-1B output/ \
260+
--ep cuda --dtype f16 --optimize=bias_gelu
261+
```
262+
263+
Available rules: `group_query_attention`, `skip_norm`, `skip_layer_norm`,
264+
`packed_attention`, `bias_gelu`.
265+
266+
> **Tip**: Prefer `--ep` over `--optimize` for production builds. `--ep`
267+
> affects both graph construction and optimization (EP-aware KV cache
268+
> sizing, dead input removal, operator fusion), while `--optimize` only
269+
> applies rewrite rules after the graph is built. They can be combined
270+
> when you need both EP-aware construction and additional post-hoc rules.
271+
210272
### `build-gguf` — Build from a GGUF file
211273

212274
```bash
@@ -219,6 +281,7 @@ mobius build-gguf model.gguf --output output/
219281
mobius list models # All supported architectures
220282
mobius list tasks # Available task types
221283
mobius list dtypes # Supported dtypes
284+
mobius list eps # Available execution providers
222285
```
223286

224287
### `info` — Inspect a model

0 commit comments

Comments
 (0)