Python: Flaky test report by giles17 · Pull Request #5342 · microsoft/agent-framework

giles17 · 2026-04-17T22:17:47Z

Motivation and Context

As part of CI/CD hardening, we need better visibility into flaky integration tests to reduce noise and improve signal quality. Currently, there's no easy way to see which tests are intermittently failing across CI runs, which makes it hard to prioritize fixes or identify regressions.

This PR adds automated flaky test trend reporting and enables additional integration tests in CI.

Description

Flaky Test Trend Report

Adds a new CI job (python-flaky-test-report) that runs after all integration test jobs complete. It follows the same artifact-upload → aggregate → Job Summary pattern used by scripts/sample_validation/.

How it works:

Each of the 6 integration test jobs (OpenAI, Azure OpenAI, Misc, Functions, Foundry, Cosmos) now produces a JUnit XML file (pytest.xml) and uploads it as an artifact
The report job downloads all artifacts, parses the XML, and merges results into a single run
Results are combined with cached history (up to 5 previous runs) to generate a markdown trend table
The trend table is posted to the GitHub Actions Job Summary

Report features:

Overall pass/fail/skip counts per run
Per-test status grid with emoji indicators (✅ passed, ❌ failed, ⏭️ skipped, ⚠️ xfail)
File column showing the test module for easy identification
Provider column showing which integration suite the test belongs to
Robust XML parsing — corrupt/truncated files are skipped with a warning, not a crash

CI Workflow Changes

Added --junitxml=pytest.xml to all integration test jobs in both workflow files
Added actions/upload-artifact steps to upload JUnit XML from each job
Added python-flaky-test-report aggregation job to both python-merge-tests.yml and python-integration-tests.yml
Fixed Cosmos job --junitxml path (pre-existing bug: uv run --directory wrote XML to the wrong location)
Added Foundry embedding env vars (FOUNDRY_MODELS_ENDPOINT, FOUNDRY_MODELS_API_KEY, FOUNDRY_EMBEDDING_MODEL, FOUNDRY_EMBEDDING_DIMENSIONS) to python-merge-tests.yml to enable the Foundry embedding integration test in CI

Files Changed

python/scripts/flaky_report/__init__.py — New package
python/scripts/flaky_report/__main__.py — CLI entry point
python/scripts/flaky_report/aggregate.py — JUnit XML parser, history management, trend report generator
.github/workflows/python-merge-tests.yml — Artifact uploads, report job, Cosmos path fix, Foundry embedding env vars
.github/workflows/python-integration-tests.yml — Artifact uploads, report job, --junitxml flag

Note: The report job is intentionally NOT added to the merge gate's needs list — it runs as an informational side-job and cannot block merges.

Contribution Checklist

The code builds clean without any errors or warnings
The PR follows the Contribution Guidelines
All unit tests pass, and I have added new tests where possible
Is this a breaking change? If yes, add "[BREAKING]" prefix to the title of the PR.

Parse JUnit XML (pytest.xml) from each integration test job and aggregate results into a markdown trend report showing per-test pass/fail/skip status across the last 5 runs. Changes: - Add python/scripts/flaky_report/ package (JUnit XML parser + trend report generator following the sample_validation pattern) - Add upload-artifact steps to all 6 integration test jobs in both python-merge-tests.yml and python-integration-tests.yml - Add python-flaky-test-report aggregation job with history caching - Add --junitxml=pytest.xml to integration-tests.yml jobs (already present in merge-tests.yml) - Fix Cosmos job --junitxml path (use absolute path since uv run --directory changes cwd) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Guard against missing reports directory in load_current_run() - Only run report job when at least one integration test job completed (skip when all jobs are skipped, e.g. on pull_request events) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Use explicit provider name mapping in _derive_provider() so OpenAI renders correctly instead of 'Openai' - Fix operator precedence in workflow if-expressions by wrapping success/failure checks in parentheses Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Adds a Python-based “flaky test trend” report that aggregates per-job JUnit XML outputs from CI, persists a short history via GitHub Actions cache, and posts a consolidated markdown report to the workflow job summary.

Changes:

Introduces python/scripts/flaky_report to parse multiple pytest.xml artifacts, merge them, and generate a markdown trend report.
Updates Python CI workflows to produce --junitxml=pytest.xml per provider job and upload those XML files as artifacts.
Adds a new “Flaky Test Report” job to download artifacts, restore/save history cache, and publish the unified report.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
python/scripts/flaky_report/aggregate.py	Implements JUnit XML aggregation, history persistence, and markdown trend report generation.
python/scripts/flaky_report/main.py	Adds `python -m scripts.flaky_report ...` entry point that dispatches to the aggregator CLI.
python/scripts/flaky_report/init.py	Documents the flaky report package purpose and usage.
.github/workflows/python-merge-tests.yml	Uploads per-job `pytest.xml` artifacts and adds a downstream aggregation/report job with caching.
.github/workflows/python-integration-tests.yml	Uploads per-job `pytest.xml` artifacts and adds the same downstream aggregation/report job with caching.

- Add File column showing module name (e.g., test_openai_chat_client) to disambiguate tests with the same function name across files - Detect pytest xfail tests in JUnit XML (type=pytest.xfail) and show them with a distinct warning emoji instead of skip emoji - Update legend to include xfail explanation Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…o flaky-test-report

When a test is inside a class, pytest writes the classname as e.g. 'pkg.test_file.TestClass'. The previous rsplit logic extracted 'TestClass' instead of 'test_file'. Now detect uppercase-starting segments as class names and use the preceding segment instead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ocstring - Use datetime.now(timezone.utc) for accurate UTC timestamps - Catch ET.ParseError per-file so corrupt XML doesn't crash the report - Remove separate 'error' key from summary (errors folded into 'failed') - Fix _short_name docstring to show actual dotted classname::name format Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

giles17 and others added 3 commits April 17, 2026 12:33

Copilot AI review requested due to automatic review settings April 17, 2026 22:17

moonbox3 added the python label Apr 17, 2026

giles17 marked this pull request as draft April 17, 2026 22:18

github-actions Bot changed the title ~~Flaky test report~~ Python: Flaky test report Apr 17, 2026

Copilot started reviewing on behalf of giles17 April 17, 2026 22:18 View session

Copilot AI reviewed Apr 17, 2026

View reviewed changes

Comment thread python/scripts/flaky_report/aggregate.py

Comment thread python/scripts/flaky_report/aggregate.py Outdated

Comment thread python/scripts/flaky_report/aggregate.py Outdated

Comment thread python/scripts/flaky_report/aggregate.py Outdated

giles17 temporarily deployed to integration April 17, 2026 22:42 — with GitHub Actions Inactive

giles17 temporarily deployed to integration April 17, 2026 23:42 — with GitHub Actions Inactive

giles17 temporarily deployed to integration April 18, 2026 00:05 — with GitHub Actions Inactive

giles17 temporarily deployed to integration April 21, 2026 04:19 — with GitHub Actions Inactive

giles17 had a problem deploying to integration April 21, 2026 04:19 — with GitHub Actions Failure

giles17 temporarily deployed to integration April 21, 2026 04:29 — with GitHub Actions Inactive

giles17 and others added 2 commits April 21, 2026 10:39

Merge branch 'main' of https://github.com/giles17/agent-framework int…

f8b0caa

…o flaky-test-report

giles17 had a problem deploying to integration April 21, 2026 21:02 — with GitHub Actions Failure

giles17 temporarily deployed to integration April 21, 2026 21:02 — with GitHub Actions Inactive

giles17 marked this pull request as ready for review April 21, 2026 21:21

TaoChenOSU approved these changes Apr 21, 2026

View reviewed changes

chetantoshniwal approved these changes Apr 22, 2026

View reviewed changes

giles17 added this pull request to the merge queue Apr 22, 2026

Merged via the queue into main with commit 3f23e1d Apr 22, 2026
61 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python: Flaky test report#5342

Python: Flaky test report#5342
giles17 merged 9 commits intomainfrom
flaky-test-report

giles17 commented Apr 17, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

giles17 commented Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation and Context

Description

Flaky Test Trend Report

CI Workflow Changes

Files Changed

Contribution Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

giles17 commented Apr 17, 2026 •

edited

Loading