Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 48 additions & 15 deletions .github/workflows/linux-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -32,11 +32,11 @@ env:
# ORT is still built from source either way). Part of the ort-install cache
# key. Remove a PR once its fix lands in the pinned ONNX Runtime release.
ONNXRUNTIME_PR_PATCHES: ""
OGA_VERSION: "0.14.0"
OGA_BRANCH: "rel-0.15.1"
# Space-separated microsoft/onnxruntime-genai PR numbers fetched from
# pull/<n>.patch and `git apply` on top of v<OGA_VERSION>. Part of the OGA
# cache key. Remove a PR once it lands in the pinned OGA release.
OGA_PR_PATCHES: "2194"
# pull/<n>.patch and `git am` on top of OGA_BRANCH. Part of the OGA cache key.
# Remove a PR once it lands in OGA_BRANCH.
OGA_PR_PATCHES: "2376"
# The GPU arch the artifact targets. CI runners have no GPU, so it is passed
# explicitly (build.py auto-detects from /sys on a real GPU host).
HIP_ARCHITECTURES: gfx1151
Expand Down Expand Up @@ -284,12 +284,49 @@ jobs:
# build.py (whose target is the project .so) because OGA is a
# CI/artifact concern.
# -----------------------------------------------------------------------
# Both the release branch and every patch PR move under a fixed name, so
# neither identifies its content. Resolve their heads once: the cache key
# folds in the short SHAs and the checkout below uses the branch SHA this
# step resolved, so a mid-run push cannot desync key from source.
- name: Resolve OGA source SHAs
id: oga-shas
env:
GH_TOKEN: ${{ github.token }}
OGA_PRS: ${{ env.OGA_PR_PATCHES }}
run: |
set -euo pipefail
branch="$(gh api "repos/microsoft/onnxruntime-genai/commits/${OGA_BRANCH}" --jq .sha)"
if [ -z "$branch" ]; then
echo "Failed to resolve head SHA for OGA branch ${OGA_BRANCH}"
exit 1
fi
echo "OGA branch head: ${OGA_BRANCH} ${branch}"
echo "branch-sha=${branch}" >> "$GITHUB_OUTPUT"
echo "branch-sha12=${branch:0:12}" >> "$GITHUB_OUTPUT"

prs="$(echo "$OGA_PRS" | xargs || true)"
if [ -z "$prs" ]; then
echo "shas=none" >> "$GITHUB_OUTPUT"
exit 0
fi
shas=""
for pr in $prs; do
sha="$(gh api "repos/microsoft/onnxruntime-genai/pulls/${pr}" --jq .head.sha)"
if [ -z "$sha" ]; then
echo "Failed to resolve head SHA for OGA PR #${pr}"
exit 1
fi
shas="${shas:+${shas}_}${pr}-${sha:0:12}"
done
echo "OGA PR heads: $shas"
echo "shas=$shas" >> "$GITHUB_OUTPUT"

- name: Cache OGA artifacts
id: cache-oga
uses: actions/cache@cdf6c1fa76f9f475f3d7449005a359c84ca0f306 # v5.0.3
with:
path: ${{ runner.workspace }}/oga-artifacts
key: linux-oga-${{ env.OGA_VERSION }}-pr${{ env.OGA_PR_PATCHES }}-am-whl-ort${{ env.ONNXRUNTIME_VERSION }}-ortpr${{ env.ONNXRUNTIME_PR_PATCHES }}
key: linux-oga-${{ env.OGA_BRANCH }}-${{ steps.oga-shas.outputs.branch-sha12 }}-pr${{ steps.oga-shas.outputs.shas }}-am-whl-ort${{ env.ONNXRUNTIME_VERSION }}-ortpr${{ env.ONNXRUNTIME_PR_PATCHES }}

# Checkout + build are skipped on an OGA cache hit (step-level gate); the
# install-copy below always runs so the
Expand All @@ -299,19 +336,18 @@ jobs:
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: microsoft/onnxruntime-genai
ref: v${{ env.OGA_VERSION }}
ref: ${{ steps.oga-shas.outputs.branch-sha }}
submodules: recursive
path: onnxruntime-genai
fetch-depth: 1
show-progress: false

# Fetch and apply microsoft/onnxruntime-genai PR patches listed in
# OGA_PR_PATCHES (env). pull/<n>.patch is a multi-commit mbox; `git am`
# replays each commit in order, so a PR that adds -> modifies -> renames
# the same path within its own series (e.g. #2194: morphizen_ep/* ->
# amdgpu/*) applies cleanly. `git apply` pre-checks every hunk against the
# base tree at once and self-conflicts on such a series. The OGA cache key
# includes the PR list, so editing it forces a rebuild.
# replays each commit in order, so a PR whose series adds -> modifies ->
# renames the same path applies cleanly. `git apply` pre-checks every hunk
# against the base tree at once and self-conflicts on such a series. The
# OGA cache key includes the PR heads, so editing the list forces a rebuild.
- name: Apply OGA PR patches
if: steps.cache-oga.outputs.cache-hit != 'true'
working-directory: onnxruntime-genai
Expand All @@ -324,10 +360,6 @@ jobs:
echo "OGA_PR_PATCHES is empty; nothing to apply"
exit 0
fi
# pull/<n>.patch is a format-patch series; apply with `git am` so that
# intra-series file renames (morphizen_ep -> amdgpu) replay against the
# correct intermediate tree. `git apply` flattens the series and chokes
# on the rename whose pre-image only exists after an earlier commit.
# git am needs a committer identity on a bare runner.
git config user.email "ci@onnx-hipdnn-ep.local"
git config user.name "onnx-hipdnn-ep CI"
Expand Down Expand Up @@ -367,6 +399,7 @@ jobs:
--cmake_generator Ninja \
--ort_home "$ORT_HOME" \
--skip_tests --skip_examples \
--no_telemetry \
--parallel \
--build_dir "${{ runner.workspace }}/build-oga" \
--cmake_extra_defines \
Expand Down
71 changes: 53 additions & 18 deletions .github/workflows/windows-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -46,11 +46,11 @@ jobs:
# Remove a PR number once its fix is in the pinned ONNXRUNTIME_VERSION.
ONNXRUNTIME_PR_PATCHES: ""
SCCACHE_GHA_ENABLED: "true"
OGA_VERSION: "0.14.0"
OGA_BRANCH: "rel-0.15.1"
# Space-separated microsoft/onnxruntime-genai PR numbers fetched from
# pull/<n>.patch and `git apply` on top of v<OGA_VERSION>. Part of the OGA
# cache key. Remove a PR once it lands in the pinned OGA release.
OGA_PR_PATCHES: "2194"
# pull/<n>.patch and `git am` on top of OGA_BRANCH. Part of the OGA cache
# key. Remove a PR once it lands in OGA_BRANCH.
OGA_PR_PATCHES: "2376"
# ilammy/msvc-dev-cmd and mozilla-actions/sccache-action have no
# Node.js 24 releases yet; allow Node.js 20 until they are updated.
ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: "true"
Expand Down Expand Up @@ -360,28 +360,62 @@ jobs:
# Cache only the two artifacts we need: model_benchmark.exe and
# onnxruntime-genai.dll, keyed on OGA commit + ORT version.
# -----------------------------------------------------------------------
# Both the release branch and every patch PR move under a fixed name, so
# neither identifies its content. Resolve their heads once: the cache key
# folds in the short SHAs and the checkout below uses the branch SHA this
# step resolved, so a mid-run push cannot desync key from source.
- name: Resolve OGA source SHAs
id: oga-shas
shell: pwsh
env:
GH_TOKEN: ${{ github.token }}
OGA_PRS: ${{ env.OGA_PR_PATCHES }}
run: |
$branch = gh api "repos/microsoft/onnxruntime-genai/commits/$env:OGA_BRANCH" --jq .sha
if ($LASTEXITCODE -ne 0 -or -not $branch) {
throw "Failed to resolve head SHA for OGA branch $env:OGA_BRANCH"
}
Write-Host "OGA branch head: $env:OGA_BRANCH $branch"
Add-Content -Path $env:GITHUB_OUTPUT -Value "branch-sha=$branch"
Add-Content -Path $env:GITHUB_OUTPUT -Value "branch-sha12=$($branch.Substring(0, 12))"

$prs = $env:OGA_PRS.Trim()
if (-not $prs) {
Add-Content -Path $env:GITHUB_OUTPUT -Value "shas=none"
exit 0
}
$parts = foreach ($pr in $prs -split '\s+') {
if (-not $pr) { continue }
$sha = gh api "repos/microsoft/onnxruntime-genai/pulls/$pr" --jq .head.sha
if ($LASTEXITCODE -ne 0 -or -not $sha) {
throw "Failed to resolve head SHA for OGA PR #$pr"
}
"$pr-$($sha.Substring(0, 12))"
}
$shas = $parts -join '_'
Write-Host "OGA PR heads: $shas"
Add-Content -Path $env:GITHUB_OUTPUT -Value "shas=$shas"

- name: Cache OGA artifacts
id: cache-oga
uses: actions/cache@cdf6c1fa76f9f475f3d7449005a359c84ca0f306 # v5.0.3
with:
path: ${{ runner.workspace }}/oga-artifacts
# OGA is built against the patched ort-install, so the key folds in
# both OGA's own PR list and ONNXRUNTIME_PR_PATCHES; editing either
# forces a rebuild. The mm<N> token bumps when the OGA artifact set
# changes shape, or when a PR's branch content changes under the same
# number (mm2: PR 2194 now carries the AMDGPU OGA integration).
key: oga-dml-mm2-${{ env.OGA_VERSION }}-pr${{ env.OGA_PR_PATCHES }}-ort${{ env.ONNXRUNTIME_VERSION }}-ortpr${{ env.ONNXRUNTIME_PR_PATCHES }}
# both OGA's own sources and ONNXRUNTIME_PR_PATCHES; a new commit on
# either side forces a rebuild.
key: oga-dml-${{ env.OGA_BRANCH }}-${{ steps.oga-shas.outputs.branch-sha12 }}-pr${{ steps.oga-shas.outputs.shas }}-ort${{ env.ONNXRUNTIME_VERSION }}-ortpr${{ env.ONNXRUNTIME_PR_PATCHES }}

# OGA is built from upstream microsoft/onnxruntime-genai + PR patches
# (OGA_PR_PATCHES). PR 2194 (the dynamic-shape-morphizen branch) now also
# carries the AMDGPU umbrella integration, so the patched upstream build
# is the AMDGPU-targeted OGA -- no fork checkout needed.
# (OGA_PR_PATCHES). PR 2376 carries the AMDGPU umbrella integration, so
# the patched upstream build is the AMDGPU-targeted OGA -- no fork
# checkout needed.
- name: Checkout OGA
if: steps.cache-oga.outputs.cache-hit != 'true'
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: microsoft/onnxruntime-genai
ref: v${{ env.OGA_VERSION }}
ref: ${{ steps.oga-shas.outputs.branch-sha }}
path: source-oga
submodules: true
fetch-depth: 1
Expand All @@ -403,10 +437,10 @@ jobs:
Write-Host "OGA_PR_PATCHES is empty; nothing to apply"
exit 0
}
# pull/<n>.patch is a format-patch series; apply with `git am` so that
# intra-series file renames (morphizen_ep -> amdgpu) replay against the
# correct intermediate tree. `git apply` flattens the series and chokes
# on the rename whose pre-image only exists after an earlier commit.
# pull/<n>.patch is a format-patch series; apply with `git am` so each
# commit replays against the intermediate tree it was authored against.
# `git apply` flattens the series and chokes on anything that depends on
# an earlier commit in it (a file rename, for example).
git config user.email "ci@rocm.amd.com"
git config user.name "ROCm CI"
foreach ($pr in $prs -split '\s+') {
Expand Down Expand Up @@ -446,6 +480,7 @@ jobs:
--use_dml \
--ort_home "$ORT_HOME" \
--skip_tests --skip_examples \
--no_telemetry \
--parallel \
--build_dir "${{ runner.workspace }}/build-oga" \
--cmake_extra_defines \
Expand Down Expand Up @@ -1288,7 +1323,7 @@ jobs:
$content = Get-Content $f.FullName -Raw
$ttftMatch = [regex]::Match($content, 'Prompt processing \(time to first token\):\s*\n\s*avg \(us\):\s+([\d.]+)')
$tpsMatch = [regex]::Match($content, 'Token generation:\s*\n\s*avg \(us\):\s+([\d.]+)\s*\n\s*avg \(tokens/s\):\s+([\d.]+)')
$memMatch = [regex]::Match($content, 'Peak working set size \(bytes\):\s+(\d+)')
$memMatch = [regex]::Match($content, 'Peak working set size:\s+(\d+)\s+bytes')
$ttft = if ($ttftMatch.Success) { "{0:F1}" -f ([double]$ttftMatch.Groups[1].Value / 1000) } else { "-" }
$tps = if ($tpsMatch.Success) { "{0:F1}" -f [double]$tpsMatch.Groups[2].Value } else { "-" }
$mem = if ($memMatch.Success) { "{0:F2}" -f ([double]$memMatch.Groups[1].Value / 1GB) } else { "-" }
Expand Down
23 changes: 12 additions & 11 deletions docs/quick_start.md
Original file line number Diff line number Diff line change
Expand Up @@ -248,19 +248,19 @@ cp "$LOCAL_DIR/bin/onnxruntime_providers_shared.dll" "$ORT_HOME/lib/"

```bash
cd .. # Go to workspace directory (sibling of hip-ep)
git clone -b v0.14.0 https://github.com/microsoft/onnxruntime-genai.git
# Upstream published no v0.15.1 tag; the release lives on the rel-0.15.1 branch.
git clone -b rel-0.15.1 https://github.com/microsoft/onnxruntime-genai.git
cd onnxruntime-genai
git submodule update --init --recursive

# Apply the AMDGPU integration PR on top of the upstream tag. pull/<n>.patch is
# a format-patch series (it renames src/morphizen_ep -> src/amdgpu mid-series),
# so apply it with `git am` -- `git apply` flattens the series and fails on the
# rename whose pre-image only exists after an earlier commit.
curl -fsSL https://github.com/microsoft/onnxruntime-genai/pull/2194.patch -o /tmp/oga-2194.patch
git am --3way --whitespace=nowarn /tmp/oga-2194.patch
# Apply the AMDGPU integration PR on top of the branch. pull/<n>.patch is a
# format-patch series, so apply it with `git am` -- `git apply` flattens the
# series and fails on anything that depends on an earlier commit in it.
curl -fsSL https://github.com/microsoft/onnxruntime-genai/pull/2376.patch -o /tmp/oga-2376.patch
git am --3way --whitespace=nowarn /tmp/oga-2376.patch
```

> **Note**: the upstream tag + PR list are pinned in CI via `OGA_VERSION` and
> **Note**: the upstream branch + PR list are pinned in CI via `OGA_BRANCH` and
> `OGA_PR_PATCHES` in
> [`.github/workflows/windows-build.yml`](../.github/workflows/windows-build.yml);
> match those for byte-for-byte reproducibility.
Expand All @@ -281,6 +281,7 @@ python build.py \
--use_dml \
--ort_home "$ORT_HOME" \
--skip_tests --skip_examples \
--no_telemetry \
--parallel \
--build_dir ../build/onnxruntime-genai \
--cmake_extra_defines \
Expand Down Expand Up @@ -371,7 +372,7 @@ python onnxruntime-genai/benchmark/python/benchmark_e2e.py \

`benchmark_e2e.py` runs with the default `-e follow_config`, so the model's
`genai_config.json` selects the EP via `provider_options`. With the upstream OGA
(v0.14.0 + PR2194, DeviceType AMDGPU) this is the AMD GPU umbrella
(0.15.1 + PR2376, DeviceType AMDGPU) this is the AMD GPU umbrella
(`provider_options [{ "AMDGPU": {"profile": "hip"} }]`), which loads
`amdgpu-ep.dll` and needs the umbrella DLLs colocated (see
`.github/workflows/windows-build.yml`); the default wheel ships only the hipgpu
Expand Down Expand Up @@ -488,8 +489,8 @@ present there too.

The EP is selected by the model's `genai_config.json` `provider_options` and
auto-discovered next to `onnxruntime-genai.dll` -- do NOT pass `--ep_library`
(upstream `model_benchmark` rejects it). With the upstream OGA (v0.14.0 +
PR2194) the EP is the AMD GPU umbrella (`provider_options [{ "AMDGPU":
(upstream `model_benchmark` rejects it). With the upstream OGA (0.15.1 +
PR2376) the EP is the AMD GPU umbrella (`provider_options [{ "AMDGPU":
{"profile": "hip"} }]`), so `amdgpu-ep.dll` must sit next to the OGA DLLs (see
`.github/workflows/windows-build.yml`).

Expand Down
2 changes: 1 addition & 1 deletion docs/quick_start_linux.md
Original file line number Diff line number Diff line change
Expand Up @@ -262,7 +262,7 @@ prebuilt package to get it.

The EP is selected by the model's `genai_config.json` `provider_options` and
auto-discovered next to the OGA runtime lib -- do NOT pass `--ep_library`
(upstream `model_benchmark` rejects it). With the upstream OGA (v0.14.0 + PR2194)
(upstream `model_benchmark` rejects it). With the upstream OGA (0.15.1 + PR2376)
the EP is the AMD GPU umbrella (`provider_options [{ "AMDGPU": {"profile": "hip"} }]`);
the prebuilt package bundles the umbrella libs.

Expand Down
2 changes: 1 addition & 1 deletion tools/perf-report/format_perf_report.py
Original file line number Diff line number Diff line change
Expand Up @@ -176,7 +176,7 @@ def parse(cls, lines: list[str]) -> "OgaHeadline":
cur_block = None
continue

m = re.match(r"Peak working set size \(bytes\):\s*(\d+)", line)
m = re.match(r"Peak working set size:\s*(\d+)\s+bytes", line)
if m:
out.peak_ws_bytes = int(m.group(1))
cur_block = None
Expand Down
2 changes: 1 addition & 1 deletion tools/perf-report/perf_multimodal_report.py
Original file line number Diff line number Diff line change
Expand Up @@ -178,7 +178,7 @@ def parse(cls, lines: list[str]) -> "OgaHeadline":
cur_block = None
continue

m = re.match(r"Peak working set size \(bytes\):\s*(\d+)", line)
m = re.match(r"Peak working set size:\s*(\d+)\s+bytes", line)
if m:
out.peak_ws_bytes = int(m.group(1))
cur_block = None
Expand Down
Loading