Make the helix "Files" API able to list only the most recent iteration

[dotnet-bot]

Migrated from https://github.com/dotnet/core-eng/issues/13983

@ChadNedzlek wrote:

Right now, helix might upload lots of copies of files, with test retries and infra-retries involved, there could be many "copies" of the same file.

We made a change to the upload code to make sure these got unique URL's, so that we didn't lose any, but it sounds like there are some JIT optimization scenarios that were depending on the physical layout of the container, which changed.

We should get those scenarios to use the "Files" API, and then we should make it so that there is an option to get the the files only from the iteration that completed, (since most scenarios won't care about, like, the previous infra-retry).

Release Note Description

Add optional 'latestOnly' parameter to Files APIs, to allow fetching the final local iteration of a given work item file.

[missymessa]

Should go into a Helix epic.

[missymessa]

Not work for DevWF. Kicking out to triage. Legit customer request. Needs a good home.

[BruceForstall]

Relevant task that assumes a particular file naming: DownloadFilesFromResults (created with #2103)

[MattGal]

I had some spare time to look at this and I put together the API side version of this here: https://dnceng.visualstudio.com/internal/_git/dotnet-helix-service/pullrequest/28431

Once this is rolled out, we can switch the Helix SDK in Arcade to use the Files API with this flag turned on, and 404s should stop occurring when the execution attempt is > 1.

[MattGal]

Taking into FROPs / tracking/blocked. Once this is on helix service prod I will make the above changes to the arcade SDK.

[MattGal]

API changes rolled out, will work on this when I can

[MattGal]

Created a draft PR of this change, will promote once I get some time to test it #12251

[MattGal]

@BruceForstall this change has been in the runtime repo since dotnet/runtime#80916 and there have been successful SuperPMI downloads logged, e.g. https://dnceng.visualstudio.com/internal/_build/results?buildId=2099532&view=logs&j=913ce52b-8ad3-5a93-c0fe-123235d07c67&t=4d3e1c8b-b1b9-5177-6235-437fac00bffb. Let me know if you see any issues... this should no longer be broken if work items run more than one attempt.

[BruceForstall]

Thanks @MattGal !