Skip to content
Merged
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 63 additions & 6 deletions docs/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -177,7 +177,7 @@ rewrite(model, pattern_rewrite_rules=skip_norm_rules())
Or via CLI:

```bash
mobius build --model Qwen/Qwen2.5-0.5B output/ --optimize
mobius build --model Qwen/Qwen2.5-0.5B output/ --ep cuda --dtype f16
```

## CLI Reference
Expand All @@ -193,20 +193,76 @@ mobius build --model meta-llama/Llama-3.2-1B output/
# From a local config directory (with safetensors weights)
mobius build --config /path/to/model/ output/

# With options
# Target a specific execution provider (recommended)
mobius build --model Qwen/Qwen2.5-0.5B output/ \
--dtype f16 \
--external-data safetensors \
--optimize
--ep cuda --dtype f16

# Export for ORT GenAI runtime
mobius build --model Qwen/Qwen2.5-0.5B output/ \
--ep cuda --dtype f16 --runtime ort-genai
```

Key flags:
- `--dtype` — Target dtype (`f32`, `f16`, `bf16`)
- `--ep` — Target execution provider (see below)
- `--runtime` — Target runtime format (`ort-genai`)
- `--no-weights` — Build graph only (no weight download)
- `--external-data` — `onnx` (default) or `safetensors`
- `--optimize` — Apply rewrite rules (e.g. fused attention)
- `--optimize` — Apply individual rewrite rules (see below)
- `--component` — Build only one component from a diffusers pipeline

#### `--ep` vs `--optimize`: when to use which

**Use `--ep` (recommended)** when building for a known runtime/hardware target.
It drives the entire build pipeline — graph construction, operator fusion,
dead input removal, and KV cache sizing are all tailored for the target EP:
Comment thread
justinchuby marked this conversation as resolved.

```bash
# CUDA GPU (uses GQA, SkipNorm, PackQKV fusions)
mobius build --model meta-llama/Llama-3.2-1B output/ --ep cuda --dtype f16

# DirectML (uses GQA without fused RoPE)
mobius build --model meta-llama/Llama-3.2-1B output/ --ep dml --dtype f16

# TensorRT-RTX (uses GQA, no SkipLayerNorm)
mobius build --model meta-llama/Llama-3.2-1B output/ --ep trt-rtx --dtype f16

# WebGPU
mobius build --model meta-llama/Llama-3.2-1B output/ --ep webgpu --dtype f16

# Portable ONNX (no vendor fusions, runs on any ONNX runtime)
mobius build --model meta-llama/Llama-3.2-1B output/ --ep onnx-standard
```
Comment thread
justinchuby marked this conversation as resolved.

Run `mobius list eps` to see all available execution providers.

**Use `--optimize` only** for manual, targeted rewrite rule application on
a pre-built model. Rules are applied post-hoc and do not affect graph
construction. This is useful for experimentation or when `--ep` doesn't
cover a specific optimization:

```bash
# Apply specific rules
mobius build --model meta-llama/Llama-3.2-1B output/ \
--optimize=group_query_attention,skip_norm

# Apply all available rules
mobius build --model meta-llama/Llama-3.2-1B output/ --optimize

# Combine EP-aware building with additional post-hoc rules
mobius build --model meta-llama/Llama-3.2-1B output/ \
--ep cuda --dtype f16 --optimize=bias_gelu
```

Available rules: `group_query_attention`, `skip_norm`, `skip_layer_norm`,
`packed_attention`, `bias_gelu`.

Comment thread
justinchuby marked this conversation as resolved.
> **Tip**: Prefer `--ep` over `--optimize` for production builds. `--ep`
> affects both graph construction and optimization (EP-aware KV cache
> sizing, dead input removal, operator fusion), while `--optimize` only
> applies rewrite rules after the graph is built. They can be combined
> when you need both EP-aware construction and additional post-hoc rules.

### `build-gguf` — Build from a GGUF file

```bash
Expand All @@ -219,6 +275,7 @@ mobius build-gguf model.gguf --output output/
mobius list models # All supported architectures
mobius list tasks # Available task types
mobius list dtypes # Supported dtypes
mobius list eps # Available execution providers
```

### `info` — Inspect a model
Expand Down
Loading