Add real-model integration test pipeline - #2198
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an Azure DevOps pipeline and accompanying Python integration test suite to validate the ONNX Runtime GenAI native runtime end-to-end against real models (pulled per-job from the foundrylocalmodels storage account), across a matrix of OS/arch and execution providers.
Changes:
- Introduces a real-model pytest integration suite (
test/python/integration/) with model cataloging + model-path resolution. - Adds helper tooling (
tools/python/integration/suite_paths.py) to compute per-(model, EP) blob subpaths for targeted azcopy downloads. - Adds a new multi-stage ADO pipeline (
.pipelines/integration-tests.yml+ templates) to build wheels and fan out one test job per (model, EP).
Reviewed changes
Copilot reviewed 13 out of 14 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tools/python/integration/suite_paths.py | CLI helper to print the blob subpath for a given (model, device) pair. |
| test/python/integration/test_integration_text.py | New text-generation integration test exercising model load + greedy generation for a chosen EP. |
| test/python/integration/resolver.py | Resolves logical model id + device to the newest on-disk vN/ model directory under ORTGENAI_MODEL_ROOT. |
| test/python/integration/README.md | Documents local + CI workflow, model layout, and suite membership guidelines. |
| test/python/integration/models.py | Defines the hardwired Foundry model catalog and the pr/all suites. |
| test/python/integration/conftest.py | Adds pytest CLI options and parametrization for --model, --execution-provider, and model root. |
| test/python/integration/init.py | Marks the integration directory as a Python package. |
| test/python/conftest.py | Makes --test_models optional so integration tests can run without providing unit-test model assets. |
| .pipelines/stages/jobs/steps/integration-pytest-step.yml | Template step to install the built wheel + deps and run the integration pytest suite. |
| .pipelines/stages/jobs/steps/integration-fetch-models-step.yml | Template step to azcopy a single model’s blob prefix onto the agent. |
| .pipelines/stages/jobs/integration-test-job.yml | Defines the per-(model, EP) test job (downloads wheel + model, runs pytest). |
| .pipelines/stages/jobs/integration-build-job.yml | Defines the per-OS/arch wheel build job used by the integration pipeline. |
| .pipelines/stages/integration-stage.yml | Orchestrates build stages and fans out per-model test jobs per OS/EP. |
| .pipelines/integration-tests.yml | Top-level ADO pipeline wiring: triggers, parameters, suite selection, and stage composition. |
baijumeswani
force-pushed
the
baijumeswani/ci
branch
from
June 9, 2026 04:04
a244ba4 to
be81c9e
Compare
baijumeswani
force-pushed
the
baijumeswani/ci
branch
from
June 9, 2026 07:00
be81c9e to
8657097
Compare
kunal-vaishnavi
approved these changes
Jun 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an ADO pipeline that validates the ORT GenAI native layer against real models pulled from the
foundrylocalmodelsstorage account.What it does
For every (model, execution provider) pair, the pipeline:
Coverage
Suites
The hardwired model catalog lives in
test/python/integration/models.py:pr(6 small models, one per architecture family): runs on every PR.all(22 models, broader coverage): runs on every merge to main.See
test/python/integration/README.mdfor the local workflow and the decision table for when to add a new model toprvsall.