Faster test-recording restore via shared clone

[m-nash]

Summary

Replaces the per-package sequential test-proxy restore loop in eng/pipelines/templates/steps/set-artifact-packages.yml with a single shared bare git clone that fans out to all packages via local hardlinks / object alternates.

The work and CI measurements behind this change are documented in detail in the design note that drove it (CI runs through three pipeline generations across 20-, 50-, and 100-package test batches).

What changes

• New: eng/scripts/Restore-RecordingsShared.ps1
• Modified: eng/pipelines/templates/steps/set-artifact-packages.yml — the per-artifact foreach block becomes a single call to the new script.

For each unique assets repo, the new script:

1. Maintains one shared bare clone at $(Agent.TempDirectory)/.assets-shared/<owner__repo> (git init --bare + git remote add origin https://github.com/<repo>.git).
2. Fetches all required tags in chunks of 30 in one git fetch per chunk, using explicit +refs/tags/<tag>:refs/tags/<tag> refspecs (no --depth=1 — shallow clones force git clone --local to fall back to upload-pack, defeating the optimization).
3. Materializes each package via:

   git clone --local --shared --no-checkout <shared> <package>/.assets
   git -C <package>/.assets checkout <tag> -- .
   

   With a fully-fetched local repo, --local --shared reuses objects from the shared clone via alternates instead of re-running upload-pack against GitHub.

The script validates the shared clone on startup (recreating if the remote URL is wrong or the directory is corrupt), pre-deletes any stale per-package .assets, and fails loudly (throws + non-zero exit) on any failure — there is no silent fallback to producing potentially-wrong recordings. It also prepends -c safe.bareRepository=all so the script works on hosts where safe.bareRepository=explicit is set.

Performance

CI: per-job Restore Recordings duration (from the design note)

PR size	Pipeline	Mean	Max	Total
20 pkg	Baseline	65.2 s	189.2 s	40.2 min
20 pkg	Shared (this change)	13.7 s	33.3 s	11.0 min
50 pkg	Baseline	104.1 s	322.8 s	119.7 min
50 pkg	Shared (this change)	14.6 s	42.6 s	17.5 min
100 pkg	Baseline	136.9 s	370.3 s	198.5 min
100 pkg	Shared (this change)	14.7 s	59.9 s	24.8 min

≈ −89% mean / −84% max / −87% total at 100 packages.

Local apples-to-apples benchmark (8 packages, this change vs. spec PR)

Sequential test-proxy restore vs. one call to the new script, both starting from a clean temp folder, against the same 8 real packages (Identity, KeyVault×4, AppConfiguration, Tables, ServiceBus):

Run	Time	vs baseline
Baseline (sequential test-proxy restore)	111.9 s	1.0×
New (shared clone, cold temp dir)	9.5 s	11.8× faster
New (shared clone, warm — tags cached)	6.8 s	16.4× faster

Correctness

Per the design note: equivalence was verified by SHA-256-hashing every file in every restored .assets directory across both strategies — 1885 / 1885 files matched. A full ib job run on the 100-package PR with the new restore produced test counts within 0.2% of the baseline (single transient NuGet error, not test-related).

This PR's local benchmark also confirms .assets directories are populated with the expected layout (the assets-repo content at the specified tag).

Why this is a clear win

1. Independent of other pipeline work — touches only the restore step.
2. Saves 10×+ on a step that runs in every ib job, every CI run — not just the test PRs.
3. Smaller variance on Mac (currently the worst max/min spread in the matrix).
4. Verified output equivalence — every restored file hashes the same as the baseline restore.
5. Fails loudly rather than silently producing wrong recordings.

Rollout

1. Merge this PR.
2. Benefit is realized immediately on every ib job, including service-directory PRs that have no relation to the broader weighted-batching work this came out of.

[Copilot]

Pull request overview

This PR optimizes CI test-recording restore by replacing the per-package sequential test-proxy restore loop with a single PowerShell script that uses a shared bare git clone per assets repo and fans out per-package .assets directories via git clone --local --shared.

Changes:

• Added eng/scripts/Restore-RecordingsShared.ps1 to restore recordings using a shared bare clone plus batched tag fetches.
• Updated eng/pipelines/templates/steps/set-artifact-packages.yml to call the new shared-restore script instead of looping over packages and invoking test-proxy restore.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
eng/scripts/Restore-RecordingsShared.ps1	New shared-clone restore implementation (batched tag fetch + local shared clones).
eng/pipelines/templates/steps/set-artifact-packages.yml	Swaps per-package restore loop for a single call to the new shared restore script.

[Jesse Squire (jsquire)]

I think copilot hit a couple of the things that I'd call out. Because the assets.json is part of the local working set and this will run in CI, it would be good to think through any validation and sanitization of values used from it.

We'll want to constrain to reasonable efforts and pragmatism. Even in the worst case, an attacker could only compromise an ephemeral sandbox with no access to company resources, tokens, etc. So, defense-in-depth rather than vulnerability.