Deployment fails randomly: Deployment Failure Reason: Failure during content distribution.

[meziantou]

Describe the bug

The deployment job sometimes fails with the error Deployment Failure Reason: Failure during content distribution. If I retry the GitHub job a few minutes / hours later, it succeeds. It happened a few times this week. The following job succeeds at the 4th attempt.

Run Azure/static-web-apps-deploy@v1
  with:
    azure_static_web_apps_api_token: ***
    repo_token: ***
    action: upload
    app_location: azure-swa
    skip_app_build: true
  env:
    DOTNET_SKIP_FIRST_TIME_EXPERIENCE: 1
    DOTNET_NOLOGO: true
    DOTNET_TieredPGO: 1
    ArtifactStagingDirectory: /home/runner/work/project/project/artifacts
    NodeJSVersion: 16
/usr/bin/docker run --name e2849036e93e424f3b45859a9548680c5990b7_f254ff --label e28490 --workdir /github/workspace --rm -e DOTNET_SKIP_FIRST_TIME_EXPERIENCE -e DOTNET_NOLOGO -e DOTNET_TieredPGO -e ArtifactStagingDirectory -e NodeJSVersion -e INPUT_AZURE_STATIC_WEB_APPS_API_TOKEN -e INPUT_REPO_TOKEN -e INPUT_ACTION -e INPUT_APP_LOCATION -e INPUT_OUTPUT_LOCATION -e INPUT_SKIP_APP_BUILD -e INPUT_API_BUILD_COMMAND -e INPUT_API_LOCATION -e INPUT_APP_ARTIFACT_LOCATION -e INPUT_APP_BUILD_COMMAND -e INPUT_ROUTES_LOCATION -e INPUT_CONFIG_FILE_LOCATION -e HOME -e GITHUB_JOB -e GITHUB_REF -e GITHUB_SHA -e GITHUB_REPOSITORY -e GITHUB_REPOSITORY_OWNER -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RETENTION_DAYS -e GITHUB_RUN_ATTEMPT -e GITHUB_ACTOR -e GITHUB_WORKFLOW -e GITHUB_HEAD_REF -e GITHUB_BASE_REF -e GITHUB_EVENT_NAME -e GITHUB_SERVER_URL -e GITHUB_API_URL -e GITHUB_GRAPHQL_URL -e GITHUB_REF_NAME -e GITHUB_REF_PROTECTED -e GITHUB_REF_TYPE -e GITHUB_WORKSPACE -e GITHUB_ACTION -e GITHUB_EVENT_PATH -e GITHUB_ACTION_REPOSITORY -e GITHUB_ACTION_REF -e GITHUB_PATH -e GITHUB_ENV -e RUNNER_OS -e RUNNER_ARCH -e RUNNER_NAME -e RUNNER_TOOL_CACHE -e RUNNER_TEMP -e RUNNER_WORKSPACE -e ACTIONS_RUNTIME_URL -e ACTIONS_RUNTIME_TOKEN -e ACTIONS_CACHE_URL -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/_temp/_runner_file_commands":"/github/file_commands" -v "/home/runner/work/project/project":"/github/workspace" e28490:36e93e424f3b45859a9548680c5990b7
DeploymentId: 18676749-2e13-4bf7-a959-a36c86618d04

App Directory Location: 'azure-swa' was found.
No Api directory specified. Azure Functions will not be created.
Looking for event info
Failed to parse GitHub Event Json (/github/workflow/event.json). Proceeding
Could not get event info. Proceeding
Skipping step to build /github/workspace/azure-swa with Oryx
Using 'staticwebapp.config.json' file for configuration information, 'routes.json' will be ignored.
Either no Api directory was specified, or the specified directory was not found. Azure Functions will not be created.
Zipping App Artifacts
Done Zipping App Artifacts
Uploading build artifacts.
Finished Upload. Polling on deployment.
Status: InProgress. Time: 0.0989484(s)
Status: InProgress. Time: 15.2882766(s)
Status: InProgress. Time: 30.4238533(s)
Status: InProgress. Time: 47.2068267(s)
Status: InProgress. Time: 62.6592872(s)
Status: InProgress. Time: 77.91025(s)
Status: InProgress. Time: 93.2929964(s)
Status: InProgress. Time: 108.4562858(s)
Status: InProgress. Time: 123.6240022(s)
Status: InProgress. Time: 138.8958045(s)
Status: InProgress. Time: 154.0757933(s)
Status: InProgress. Time: 169.1921353(s)
Status: InProgress. Time: 184.3383472(s)
Status: InProgress. Time: 199.4665996(s)
Status: Failed. Time: 214.5293545(s)
Deployment Failed :(
Deployment Failure Reason: Failure during content distribution.

For further information, please visit the Azure Static Web Apps documentation at https://docs.microsoft.com/en-us/azure/static-web-apps/
If you believe this behavior is unexpected, please raise a GitHub issue at https://github.com/azure/static-web-apps/issues/
Exiting

Expected behavior
The deployment works every time

Screenshots

[miwebst]

Can you provide any details on the default hostname of the app (i.e. happy-dog-123)? If you do not feel comfortable providing that information here, I'd recommend filing a support request.

[meziantou]

Default hostname: https://zealous-river-050505f0f.azurestaticapps.net

[jpalo]

Getting this same error intermittently as well: icy-sea-009cf3103. Last issue today 3 hours ago. Retrying later or immediately after the error has so far worked, but NOT this time, so unable to deploy updates. Filed support request, will update here of the outcome.

UPDATE: Oh well, 3rd attempt, and it deployed successfully. Will discuss with support anyway...

---End of Oryx build logs---
Finished building app with Oryx
Using 'staticwebapp.config.json' file for configuration information, 'routes.json' will be ignored.
Either no Api directory was specified, or the specified directory was not found. Azure Functions will not be created.
Zipping App Artifacts
Done Zipping App Artifacts
Uploading build artifacts.
Finished Upload. Polling on deployment.
Status: InProgress. Time: 0.3286494(s)
Status: InProgress. Time: 15.6519709(s)
Status: InProgress. Time: 31.0321979(s)
Status: InProgress. Time: 46.4328469(s)
Status: InProgress. Time: 61.7466375(s)
Status: InProgress. Time: 76.9399129(s)
Status: InProgress. Time: 95.337563(s)
Status: InProgress. Time: 110.6155604(s)
Status: InProgress. Time: 125.8977842(s)
Status: InProgress. Time: 141.1042243(s)
Status: InProgress. Time: 156.4833813(s)
Status: InProgress. Time: 171.7872681(s)
Status: InProgress. Time: 187.1270068(s)
Status: Failed. Time: 203.5911316(s)
Deployment Failed :(
Deployment Failure Reason: Failure during content distribution.

[whoruyh]

I am also experiencing the same symptoms.
Randomly throws an error.
Any solution?

[meziantou]

Any solution?

Switch to another product... That's what I did as I was unable to reliably deploy my project 😟
But I kept the deploy job on SWA just to see how flaky it is. Maybe I'll switch back to SWA when the job reliably succeeds.

[miwebst]

We are working on a solution, will likely have an update in mid-January. Some things that you can do to improve this reliability in the short term (these may or may not be possible depending on how you build / operate your site):

• Reduce the number of unique files changing between uploads. If you are generating unique file paths on each build this can negatively affect our upload performance
• Reduce the total app size
• Reduce total number of files you are uploading

As I mentioned, these may not be suitable / possible options in which case you will need to wait until mid-January for the fixes we are working on. We apologize for the inconvenience.

[vxphuong]

@miwebst have any update? I met the same issue.

[meziantou]

@miwebst Any update on the issue?

On my side, the deployment continues to fail almost every day :(

Domain: mango-dune-05cf06d10.azurestaticapps.net
Deployment Id:

• f27ee0f1-7097-4097-935e-c013b28c29ba (2022-07-14T12:26:21Z)
• 9736ca3b-bef0-401b-b9a3-203e061894af (2022-07-13T12:21:13Z)

[LevYas]

Also happened to me, two times in a row,
DeploymentId: 2f3ba0d0-9f82-4b45-a04d-7396d97dcd2e and DeploymentId: 6f98cdb2-f37e-430f-9db7-bf8f6f311994
Any updates?

[jpalo]

I've had ticket open at Microsoft for over a year. They've promised me several times that "now the issue has been fixed" and that "now your instance has been moved somewhere else and consider yourself privileged as now YOU will no longer see the issue". Yet it still occurs every few weeks. I've always reported the issue to them, but in the last 2 months or so, they haven't even bothered to ackowledge my reports so I'm very confused how come a paid service (I have Standard tier) behaves like this. Bad service, bad support.

[akib1997]

I am still facing the same issue!

[rohanbeemup]

same here :(

[leprechaun19]

yes, the same

[aherreragrubbrr]

Same here, it started to fail out of the blue.

[karimch27]

same here

[Aduen]

yes

[akhatri-PXL]

I am aslo facing the same issue starting today, yesterday it was working.

[tge-fyayc]

Same here! Started this afternoon.

[ronit113]

same here.

[ursaciuc-adrian]

same here.

[Uliura]

I just recreated the static app, and everything deployed to the fresh app with no errors.

[kddsultan]

The same here. Microsoft has not provided in any Alerts on this

[capacit-antonkrogh]

Same also this afternoon...

[karimch27]

From Azure :
Impact Statement: Starting at 10:02 UTC on 26 Apr 2024, you have been identified as a customer using App Service who may experience issues when attempting to update content for Static Web Apps.Current Status: We are aware of this issue and are actively investigating. The next update will be provided within 60 minutes, or as events warrant.

[jgconti]

Same issue here!

[duhseekoh]

From Azure : Impact Statement: Starting at 10:02 UTC on 26 Apr 2024, you have been identified as a customer using App Service who may experience issues when attempting to update content for Static Web Apps.Current Status: We are aware of this issue and are actively investigating. The next update will be provided within 60 minutes, or as events warrant.

There is clearly an Azure incident taking place -- why is that not reflected here? https://azure.status.microsoft/en-us/status
All green?

[Ronkiro]

Also having this problem right now... Certainly an incident

[sijojlouis]

Also having this problem right now... Certainly an incident

+1

[kyjovskym]

+1

[btcpbordeaux]

experiencing the issue as well. have heard diddly-squat from azure about the incident

[hewseph]

+1

[Romilkajaria]

+1

[sabupa94]

+1

[davidpeterko]

+1

[jkuczek15]

+1

[Alex-atTimec]

+1

[massej]

+1 here

[usapath-mitren]

+1 has any one heard back from Azure?

[twitthoeft-gls]

+ 1

[Geeky2023]

Same here for the production env, working fine in other static pages

[zrelja]

+1. Not randomly, every time. Any news from Azure?

[jonasacc]

+1

[oscarmendezzavaleta]

mismo error +1

[mikereman]

+1

[Ronkiro]

From my service integrity's page

Current Status: We are looking into a potential configuration issue at this time. We are continuing to investigate backlogs and telemetry to better understand the issue. Engineers are engaged working on identifying a mitigation workstream at this time. The next update will be provided within 2 hours, or as events warrant.

[musaib-js]

Facing the same error with my deployment. Deployed during the day and it worked fine, but tried deploying new changes now and it has, surprisingly, started giving this error.

[jonnydeitos]

+1

[rweatherly]

+1 same issue for the last couple hours now

[mauricenijkamp]

It deployed for me, finally! 😀
Waited all day for this...

[rweatherly]

It deployed for me, finally! 😀 Waited all day for this...

yea, mine started working too finally!!

[Alex-atTimec]

Still not working for me although it did sit in the 'Status: InProgress.' state for some time, '574.9181287(s)' to be exact before it failed again.

Done Zipping Api Artifacts
Zipping App Artifacts
Done Zipping App Artifacts
Uploading build artifacts.
Finished Upload. Polling on deployment.
Status: InProgress. Time: 0.0626579(s)
Status: InProgress. Time: 15.1157295(s)
Status: InProgress. Time: 30.174356(s)
Status: InProgress. Time: 45.2320053(s)
Status: InProgress. Time: 60.2885684(s)
Status: InProgress. Time: 75.3468862(s)
Status: InProgress. Time: 90.4018077(s)
Status: InProgress. Time: 105.4837089(s)
Status: InProgress. Time: 120.5356472(s)
Status: InProgress. Time: 135.6266597(s)
Status: InProgress. Time: 150.7075347(s)
Status: InProgress. Time: 165.7611455(s)
Status: InProgress. Time: 180.8220229(s)
Status: InProgress. Time: 195.8943762(s)
Status: InProgress. Time: 210.9485943(s)
Status: InProgress. Time: 226.007419(s)
Status: InProgress. Time: 241.0774695(s)
Status: InProgress. Time: 256.2482617(s)
Status: InProgress. Time: 271.301779(s)
Status: InProgress. Time: 286.3802545(s)
Status: InProgress. Time: 301.4465208(s)
Status: InProgress. Time: 316.5004288(s)
Status: InProgress. Time: 331.57111(s)
Status: InProgress. Time: 346.6272549(s)
Status: InProgress. Time: 361.6839332(s)
Status: InProgress. Time: 376.7591777(s)
Status: InProgress. Time: 391.8209117(s)
Status: InProgress. Time: 409.1715017(s)
Status: InProgress. Time: 424.2517913(s)
Status: InProgress. Time: 439.3135553(s)
Status: InProgress. Time: 454.3813247(s)
Status: InProgress. Time: 469.4409497(s)
Status: InProgress. Time: 484.5113868(s)
Status: InProgress. Time: 499.5844211(s)
Status: InProgress. Time: 514.6386885(s)
Status: InProgress. Time: 529.7146974(s)
Status: InProgress. Time: 544.8071528(s)
Status: InProgress. Time: 559.861878(s)
Status: Failed. Time: 574.9181287(s)
Deployment Failed :(
Deployment Failure Reason: Web app warm up timed out. Please try again later.

For further information, please visit the Azure Static Web Apps documentation at https://docs.microsoft.com/en-us/azure/static-web-apps/
If you believe this behavior is unexpected, please raise a GitHub issue at https://github.com/azure/static-web-apps/issues/
Exiting```

[Ronkiro]

@Alex-atTimec it seems that your issue is unrelated, i recommend checking other issues or opening a new one

Actually, it seems to be the same from #1457

[Alex-atTimec]

@Alex-atTimec it seems that your issue is unrelated, i recommend checking other issues or opening a new one

Actually, it seems to be the same from #1457

Thanks! It actually worked after a couple more attempts.

[rohanbeemup]

Worked for me.. thank you!

[asbjornu]

In #632 (comment), @miwebst wrote almost 3 years ago that this should be fixed "mid-january 2022". It is apparently not fixed, as I'm struggling with this issue right now. I've tried to re-run the deployment dozens of times, but it just keeps failing. Needless to say, the error message Deployment Failure Reason: Failure during content distribution is wholly unsatisfactory.

What's strange is if I attempt to re-deploy an old build in the same repository, it succeeds. Even resetting main to the old successful commit and deploying that, succeeds.

The changes I want to have deployed also works in a development environment, whose only difference to the production environment is its AZURE_STATIC_WEB_APPS_API_TOKEN. Having the same code-change deployed to production fails with Failure during content distribution every single time. I've tried to clear the GitHub Actions cache, and it did not affect this at all.

The problem seems to be isolated to GitHub “release published" builds. All other builds succeed. I've tried to change the deployment key and even created an entirely new Static Web App, but the behavior is consistently the same. I suppose this is due to production deployments actually doing "content distribution", while preview environments don't distribute the content.

After further investigation, I've managed to successfully deploy to production by reducing the number of files to deploy from 6307 to 759. So it seems like the problem might be that waiting for content distribution simply times out after 5 minutes. I haven't figured out the exact file count threshold is, but as 759 files consistently deploys successfully in around 45 seconds, swa deploy pushes ~16 files per second, which linearly scales to somewhere around 5000 files before the timeout limit is reached after 300 seconds.

Seeing how SWA is touted as a replacement for Blob Storage Static Websites and a competitor to Netlify, Cloud Cannon, etc., I find this limitation laughable. I've worked on projects which hosted tens of thousands of files on Blob Storage Static Websites without a problem. Needless to say, SWA should be able to do that too.

[asbjornu]

I've been in contact with Microsoft Support regarding this issue and have documented my interaction and how the issue was resolved in Azure/static-web-apps-cli#853. TL;DR:

1. The issue is a combination of a high amount of folders and files.
2. The issue is acknowledged as a defect in Azure Static Web Apps.
3. The only short-term solution is to keep the number of files and folders to less than 3000.
4. The long-term solution is that Microsoft is creating a new SKU that will take this issue into account.