Skip to content

Conversation

@StephanTLavavej
Copy link
Member

Our Azure Pipelines checks were unable to run for most of Tuesday (2021-02-16), briefly from (Pacific Time) 3:39 PM to 3:49 PM and then continuously from 5:09 PM until now (~2 AM Wednesday), with the error message:

Failed to increase capacity with error: The requested size for resource '/subscriptions/ddddd02b-5e77-473c-afde-c40253b5343e/resourceGroups/STLBUILD-2021-02-09/providers/Microsoft.Compute/virtualMachineScaleSets/StlBuild-2021-02-09-Vmss' is currently not available in location 'westus2' zones '' for subscription 'ddddd02b-5e77-473c-afde-c40253b5343e'. Please try another size or deploy to a different location or zones. See https://aka.ms/azureskunotavailable for details.

It doesn't appear that we're over budget, and vcpkg was experiencing the same error. Attempting to create VMs of many different SKUs in West US 2 failed, so it's not specific to our current SKU. Attempting to create VMs in West US and South Central US also failed. However, @BillyONeal had the idea to try a different continent (suspecting that recent extreme weather in the US was related), and the first region I tried, North Europe, succeeded.

If this Virtual Machine Scale Set can scale up to handle a build/test run, I think we should switch to it temporarily. We don't perform very much network access (just cloning repos, and we have gone to considerable lengths to minimize how much of llvm-project we download), and it isn't latency-sensitive at all.

And on the topic of network access, I recently noticed that we're uploading and storing a zillion bytes. This PR attempts to fix #1653 in a minimal way, leaving the machinery in place if it needs to be refined in the future.

@StephanTLavavej StephanTLavavej added high priority Important! infrastructure Related to repository automation labels Feb 17, 2021
@StephanTLavavej StephanTLavavej requested a review from a team as a code owner February 17, 2021 10:06
@StephanTLavavej

This comment has been minimized.

@StephanTLavavej

This comment has been minimized.

@azure-pipelines

This comment has been minimized.

@CaseyCarter
Copy link
Contributor

CaseyCarter commented Feb 17, 2021

I'm going to merge this now and port to the feature branches, rather than waiting for @StephanTLavavej to do so as we usually would, to unblock us. I will delay merging to MSVC for the next batch to avoid wasting CI cycles for what is effectively a non-functional change as far as the internal repo is concerned.

@CaseyCarter CaseyCarter merged commit 8f79acf into microsoft:main Feb 17, 2021
@CaseyCarter
Copy link
Contributor

Note that I did not Changelog this PR. We typically changelog the VMSS updates because they include changes to our required tool versions, but this PR has no such changes. I don't think it's of great interest to our contributors where the CI runs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

high priority Important! infrastructure Related to repository automation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Azure Pipelines: 11 TiB of Pipeline Artifacts is excessive

3 participants