Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make the helix "Files" API able to list only the most recent iteration #8850

Closed
dotnet-bot opened this issue Apr 7, 2022 · 9 comments
Closed
Assignees
Labels
Ops - Compliance First-responder-style issues handled by the Operations V-Team due to prioritization or urgency level

Comments

@dotnet-bot
Copy link
Contributor

dotnet-bot commented Apr 7, 2022

Migrated from https://github.com/dotnet/core-eng/issues/13983

@ChadNedzlek wrote:

Right now, helix might upload lots of copies of files, with test retries and infra-retries involved, there could be many "copies" of the same file.

We made a change to the upload code to make sure these got unique URL's, so that we didn't lose any, but it sounds like there are some JIT optimization scenarios that were depending on the physical layout of the container, which changed.

We should get those scenarios to use the "Files" API, and then we should make it so that there is an option to get the the files only from the iteration that completed, (since most scenarios won't care about, like, the previous infra-retry).

Release Note Description

Add optional 'latestOnly' parameter to Files APIs, to allow fetching the final local iteration of a given work item file.

@missymessa
Copy link
Member

Should go into a Helix epic.

@missymessa
Copy link
Member

Not work for DevWF. Kicking out to triage. Legit customer request. Needs a good home.

@michellemcdaniel michellemcdaniel added Needs Triage A new issue that needs to be associated with an epic. and removed Needs Triage A new issue that needs to be associated with an epic. labels Apr 28, 2022
@BruceForstall
Copy link
Member

Relevant task that assumes a particular file naming: DownloadFilesFromResults (created with #2103)

@MattGal
Copy link
Member

MattGal commented Dec 15, 2022

I had some spare time to look at this and I put together the API side version of this here: https://dnceng.visualstudio.com/internal/_git/dotnet-helix-service/pullrequest/28431

Once this is rolled out, we can switch the Helix SDK in Arcade to use the Files API with this flag turned on, and 404s should stop occurring when the execution attempt is > 1.

@MattGal MattGal self-assigned this Jan 5, 2023
@MattGal MattGal added the Ops - Compliance First-responder-style issues handled by the Operations V-Team due to prioritization or urgency level label Jan 5, 2023
@MattGal
Copy link
Member

MattGal commented Jan 5, 2023

Taking into FROPs / tracking/blocked. Once this is on helix service prod I will make the above changes to the arcade SDK.

@MattGal
Copy link
Member

MattGal commented Jan 12, 2023

API changes rolled out, will work on this when I can

@MattGal
Copy link
Member

MattGal commented Jan 18, 2023

Created a draft PR of this change, will promote once I get some time to test it #12251

@MattGal
Copy link
Member

MattGal commented Jan 31, 2023

@BruceForstall this change has been in the runtime repo since dotnet/runtime#80916 and there have been successful SuperPMI downloads logged, e.g. https://dnceng.visualstudio.com/internal/_build/results?buildId=2099532&view=logs&j=913ce52b-8ad3-5a93-c0fe-123235d07c67&t=4d3e1c8b-b1b9-5177-6235-437fac00bffb. Let me know if you see any issues... this should no longer be broken if work items run more than one attempt.

@MattGal MattGal closed this as completed Jan 31, 2023
@BruceForstall
Copy link
Member

Thanks @MattGal !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Ops - Compliance First-responder-style issues handled by the Operations V-Team due to prioritization or urgency level
Projects
None yet
Development

No branches or pull requests

5 participants