Implement weighted test batching using live ADO Analytics API

[m-nash]

Design doc

Full background and experiment handoff: https://gist.github.com/m-nash/b2429de2a9f6abcce018abd9c23aac49

Problem

The current PR test batching splits packages alphabetically into fixed-size buckets with no awareness of test runtime. This creates too many jobs, wastes agent time, and can place several expensive test suites in the same batch.

Example from build 6102688 (BatchSize=20):

• Slowest batch: 61 min → timeout
• Fastest batch: 14 min
• Imbalance ratio: 4.4x

Reducing the fixed package count avoids some timeouts, but increases job count and setup overhead instead of balancing the work.

At larger scales, expanding every package directly into the test matrix can also exceed Azure Pipelines YAML limits. The 100-package baseline hit this limit and abandoned 123 test jobs.

Solution

Replace fixed-size splitting with LPT (Longest Processing Time) bin packing balanced by test runtimes queried live from the Azure DevOps Analytics API.

How it works

1. Get-TestAssemblyWeights.ps1 resolves the test assemblies required by the current PR
2. It queries successful runs from the preceding 24 hours, followed by a scoped 30-day fallback
3. Assembly runtimes are aggregated into package weights
4. Apply-WeightedBatching.ps1 uses deterministic LPT bin packing to distribute expensive packages across balanced buckets
5. macOS receives smaller effective batches, while Linux and Windows consolidate more work per job
6. Packages are consolidated before matrix expansion, reducing job count and avoiding YAML expansion limits

Failure handling

• Retries transient Analytics failures such as throttling, connection errors, and server errors
• Rejects empty or catastrophically sparse Analytics responses
• Assigns a nonzero one-second fallback weight to new or unknown assemblies
• Falls back to existing count-based batching when runtime weighting cannot be applied
• Skips weighting when the package set is too small to benefit
• Verifies that every expected package appears exactly once after consolidation
• Logs query coverage, resolved weights, fallback assignments, and final bucket contents

Test results

20-package comparison

Metric	Current	Weighted	Improvement
Test jobs	31	17	45% fewer
Test agent time	447.0 min	302.7 min	32% lower
Slowest test job	26.2 min	23.5 min	10% faster
Pipeline wall-clock	39m 27s	38m 24s	1m 3s faster

50-package comparison

Metric	Current	Weighted	Improvement
Test jobs	65	34	48% fewer
Test agent time	1,008.7 min	664.2 min	34% lower
Slowest test job	39.9 min	36.3 min	9% faster
Pipeline wall-clock	50m 7s	47m 9s	2m 58s faster

100-package comparison

The weighted run completed with:

• 88 test jobs
• Slowest test job: 33.3 minutes
• No batching timeout
• No YAML expansion failure

The current-batching baseline exceeded Azure Pipelines YAML limits during test package expansion, causing 123 test jobs to be abandoned. Consolidating weighted packages before matrix expansion resolves this scaling failure.

Query performance

Scale	Assemblies	Coverage	Query time
20 packages	110	97.3%	7.9s
50 packages	143	97.2%	7.5s
100 packages	454	96.9%	43.9s

Missing assemblies used the one-second fallback.

Conclusion

Adopt runtime-weighted test batching.

The complete comparisons show:

• 45-48% fewer test jobs
• 32-34% lower test agent time
• Faster overall pipeline wall-clock
• Slowest test job below 37 minutes
• Successful execution at the 100-package scale without exceeding YAML limits

This reduces CI cost and agent-pool pressure while preserving substantial timeout headroom and improving scalability.

Comparison builds

Scale	Weighted PR	Current PR	Weighted build	Current build
20 packages	#58079	#59401	Build 6653343	Build 6653349
50 packages	#58080	#59402	Build 6653344	Build 6653348
100 packages	#57904	#59403	Build 6653346	Build 6653358

Follow-up: Azure.Storage.Blobs 30-minute bottleneck

After weighted batching, Azure.Storage.Blobs remains a roughly 30-minute macOS critical-path job because it is already isolated in its own bucket and cannot be subdivided by package-level scheduling.

In the isolated net10.0 run, the complete job took 33.2 minutes: 20.9 minutes in Build & Test—approximately 10.3 minutes restoring/building and 10.5 minutes executing/discovering tests—plus 12.3 minutes of pipeline setup and result-handling overhead. A follow-up analysis documents the measurements and possible options for addressing work within this package.

Fixes: #58028
Related: #57679

[Copilot]

Pull request overview

This PR improves PR test batching for the .NET SDK repo by replacing count/alphabetical bucketing with weighted LPT bin-packing, using live test duration estimates from the Azure DevOps Analytics OData API to better balance CI batch runtimes (especially for macOS).

Changes:

• Added a new PowerShell script to query ADO Analytics for per-test-assembly wall-clock runtimes and emit per-package weights.
• Added Split-ArrayByWeight helper implementing LPT (Longest Processing Time) bin-packing based on weights and a target bucket time.
• Wired the weights file + target time through PR matrix generation templates and CI job parameters (PR builds only for weight collection).

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
eng/scripts/Get-TestAssemblyWeights.ps1	New script to scan sdk/ test projects, query ADO Analytics OData, and write a package→seconds weights JSON.
eng/common/scripts/Helpers/Package-Helpers.ps1	Adds Split-ArrayByWeight LPT batching helper.
eng/common/scripts/job-matrix/Create-PrJobMatrix.ps1	Loads optional weights file and switches batching strategy to weighted LPT when available.
eng/pipelines/templates/steps/pr-matrix-presteps.yml	Adds PR-only pre-step to generate the weights file using System.AccessToken.
eng/common/pipelines/templates/jobs/generate-job-matrix.yml	Passes -WeightsFile and -TargetBatchTimeSeconds through to PR matrix generation.
eng/pipelines/templates/jobs/ci.yml	Configures PR matrix generation to use the weights file and a 30-minute target bucket time (1800s).

[github-actions]

Hi m-nash. Thank you for your interest in helping to improve the Azure SDK experience and for your contribution. We've noticed that there hasn't been recent engagement on this pull request. If this is still an active work stream, please let us know by pushing some changes or leaving a comment. Otherwise, we'll close this out in 7 days.

[Copilot]

Review details

Suppressed comments (4)

eng/scripts/Get-TestAssemblyWeights.ps1:1

• Get-TestAssemblyWeights.ps1 uses ForEach-Object -Parallel, which requires PowerShell 7+. Without a #Requires -Version 7.0 guard, running this script under Windows PowerShell 5.1 will fail immediately with a parameter-binding error.

Consider adding a version requirement at the top of the script to fail fast with a clear message.

<#

eng/pipelines/templates/steps/pr-matrix-presteps.yml:106

• The fallback warning message hard-codes "(10 packages per job)", but the actual count-based batching size is controlled elsewhere (e.g., PRJobBatchSize) and may not be 10 in all pipelines that reuse this template. This can make the warning misleading when someone changes the batch size.

Consider removing the hard-coded number (or wiring it through as a parameter) so the log remains accurate.

          $reason = $_.Exception.Message -replace "`r?`n", " "
          Write-Host "##vso[task.logissue type=warning]Runtime-weighted test batching could not be applied: $reason Falling back to count-based test batching (10 packages per job)."
          Remove-Item $weightsFile -Force -ErrorAction SilentlyContinue

eng/pipelines/templates/steps/pr-matrix-presteps.yml:120

• The runtime-weighted batching step applies Apply-WeightedBatching.ps1 -Target 240, but Apply-WeightedBatching.ps1 documents a default Target of 1800 (seconds) and the PR description states batches target ~30 minutes. With weights expressed in seconds (see Get-TestAssemblyWeights.ps1), 240 is a 4-minute bucket target, which seems inconsistent with the stated 30-minute goal unless this is intentionally a smaller "sub-bucket" size.

Can you clarify the intended unit/target here and align the value (or update the docs/description) so the configured target matches the desired per-job runtime behavior?

          eng/scripts/Apply-WeightedBatching.ps1 `
            -PackageInfoFolder "$(Build.ArtifactStagingDirectory)/PackageInfo" `
            -WeightsFile $weightsFile `
            -Target 240

eng/pipelines/templates/jobs/ci.yml:230

• PR metadata/description says the implementation adds Split-ArrayByWeight and new Create-PrJobMatrix.ps1 parameters like -WeightsFile / -TargetBatchTimeSeconds, but the actual flow in this branch uses eng/scripts/Apply-WeightedBatching.ps1 + PRWeightedBucketsPerJob/PRWeightedBatchingApplied in generate-job-matrix.yml instead. This mismatch makes it hard to validate the PR against the description/design.

Please update the PR description (and/or the "Changes" table) to reflect the current approach and the files/parameters that were actually modified.

          # Runtime weighting first creates small balanced buckets. Linux and Windows
          # combine three buckets while slower macOS keeps one bucket per job. If
          # runtime weighting fails, the default preserves count-based batching.
          PRJobBatchSize: 10
          PRWeightedBucketsPerJob:
            Linux: 3
            Windows: 3
            Mac: 1
          PRWeightedBatchingApplied: $(TestRuntimeWeightingApplied)

• Files reviewed: 7/7 changed files
• Comments generated: 0 new
• Review effort level: Lite

[azure-pipelines]

Azure Pipelines:
Successfully started running 2 pipeline(s).
9 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[azure-pipelines]

Azure Pipelines:
Successfully started running 2 pipeline(s).
10 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.