Skip to content

Add real-model integration test pipeline - #2198

Merged
baijumeswani merged 4 commits into
mainfrom
baijumeswani/ci
Jun 9, 2026
Merged

Add real-model integration test pipeline#2198
baijumeswani merged 4 commits into
mainfrom
baijumeswani/ci

Conversation

@baijumeswani

Copy link
Copy Markdown
Collaborator

Adds an ADO pipeline that validates the ORT GenAI native layer against real models pulled from the foundrylocalmodels storage account.

What it does

For every (model, execution provider) pair, the pipeline:

  1. Builds an ORT GenAI wheel from source (one build per OS / arch).
  2. Spawns a separate ADO test job that downloads only its model from blob, installs the wheel, and runs pytest.

Coverage

OS / arch Providers tested
Windows x64 CPU, CUDA, WebGPU
Linux x64 CPU, CUDA
macOS arm64 CPU, WebGPU

Suites

The hardwired model catalog lives in test/python/integration/models.py:

  • pr (6 small models, one per architecture family): runs on every PR.
  • all (22 models, broader coverage): runs on every merge to main.

See test/python/integration/README.md for the local workflow and the decision table for when to add a new model to pr vs all.

Copilot AI review requested due to automatic review settings June 8, 2026 21:41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an Azure DevOps pipeline and accompanying Python integration test suite to validate the ONNX Runtime GenAI native runtime end-to-end against real models (pulled per-job from the foundrylocalmodels storage account), across a matrix of OS/arch and execution providers.

Changes:

  • Introduces a real-model pytest integration suite (test/python/integration/) with model cataloging + model-path resolution.
  • Adds helper tooling (tools/python/integration/suite_paths.py) to compute per-(model, EP) blob subpaths for targeted azcopy downloads.
  • Adds a new multi-stage ADO pipeline (.pipelines/integration-tests.yml + templates) to build wheels and fan out one test job per (model, EP).

Reviewed changes

Copilot reviewed 13 out of 14 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
tools/python/integration/suite_paths.py CLI helper to print the blob subpath for a given (model, device) pair.
test/python/integration/test_integration_text.py New text-generation integration test exercising model load + greedy generation for a chosen EP.
test/python/integration/resolver.py Resolves logical model id + device to the newest on-disk vN/ model directory under ORTGENAI_MODEL_ROOT.
test/python/integration/README.md Documents local + CI workflow, model layout, and suite membership guidelines.
test/python/integration/models.py Defines the hardwired Foundry model catalog and the pr/all suites.
test/python/integration/conftest.py Adds pytest CLI options and parametrization for --model, --execution-provider, and model root.
test/python/integration/init.py Marks the integration directory as a Python package.
test/python/conftest.py Makes --test_models optional so integration tests can run without providing unit-test model assets.
.pipelines/stages/jobs/steps/integration-pytest-step.yml Template step to install the built wheel + deps and run the integration pytest suite.
.pipelines/stages/jobs/steps/integration-fetch-models-step.yml Template step to azcopy a single model’s blob prefix onto the agent.
.pipelines/stages/jobs/integration-test-job.yml Defines the per-(model, EP) test job (downloads wheel + model, runs pytest).
.pipelines/stages/jobs/integration-build-job.yml Defines the per-OS/arch wheel build job used by the integration pipeline.
.pipelines/stages/integration-stage.yml Orchestrates build stages and fans out per-model test jobs per OS/EP.
.pipelines/integration-tests.yml Top-level ADO pipeline wiring: triggers, parameters, suite selection, and stage composition.

Comment thread test/python/integration/test_integration_text.py
Comment thread .pipelines/stages/jobs/integration-build-job.yml
Comment thread .pipelines/integration-tests.yml
Comment thread test/python/integration/models.py Outdated
Comment thread test/python/integration/models.py
Comment thread tools/python/integration/suite_paths.py Outdated
Comment thread test/python/integration/check_models_in_sync.py Outdated
@baijumeswani
baijumeswani enabled auto-merge (squash) June 9, 2026 21:57
@baijumeswani
baijumeswani merged commit b12919b into main Jun 9, 2026
63 of 66 checks passed
@baijumeswani
baijumeswani deleted the baijumeswani/ci branch June 9, 2026 23:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants