Run-From-Package creates an invalid deployment

[flagbug]

Since yesterday, after every deployment we make to our App Service, the application fails to start. We're using Run-From-Package as deployment method and are seeing the following error in the App Service event log:

Application '/LM/W3SVC/61535351/ROOT' with physical root 'D:\home\site\wwwroot\' failed to start process with commandline 'dotnet .\Alexandria.dll' with multiple retries. Failed to bind to port '7356'. First 30KB characters of captured stdout and stderr logs from multiple retries:
Unhandled Exception: System.IO.FileLoadException: Could not load file or assembly 'D:\home\site\wwwroot\Alexandria.dll'. An assertion failure has occurred. (Exception from HRESULT: 0x8007029C)
Process Id: 11020.
File Version: 13.0.19218.0. Description: IIS ASP.NET Core Module V2 Request Handler. Commit: 4a42afc5aea63750638e118560d43db04bd9ccc2

In this case, Alexandria.dll is our application entry-point. It's an ASP.NET Core 2.2 application.

Or deployment setup consists of the default production and a custom staging slot, with 2 App Service instances running at minimum. We're running theRun-From-Package deployment on the staging slot, with auto swap enabled, so the application can be deployed with zero downtime. This issue is impacting all of our customers after a deployment.

A simple restart of App Service fixes this issue.

After searching around a bit, the issue we're experiencing here seems to be very similar to Azure/app-service-announcements-discussions#32 (comment)

Basically two questions are open for me here:

1. Why is App Service unable to load our .dll?
2. Why would App Service auto-swap an application that isn't even able to start? That seems completely crazy to me and should be prevented by all kinds of health checks on the side of App Service.

Please let me know what useful information I can provide in addition.

[suwatch]

Before we auto swap, we ensure 1) deployment successful 2) app is responsive and return 200 on root path access. It seems to contradict with your 2nd question. Best is for us to check the log and confirm. Do share the UTC time of the most recent deployment (auto-swap) as well as the webapp name (here or indirectly) that left your production in bad stage. We can investigate.

[flagbug]

@suwatch WebApp name is mimo-api-production, happened the first time after deployemtn on Thursday ~15:15 UTC and then again after deployment on Friday ~09:45UTC

FYI, since then we've changed the deployment strategy back to WebDeploy where this problem doesn't occur, so you'll only see the staging slot having Run-From-Package enabled right now.

[suwatch]

Thanks for the info .. we are taking a look and will update what we find.

[suwatch]

@flagbug I was wrong on swap would happen if only 200. Swap will happen regardless of the status by default. However, recently we made an improvement to allow the app to specify condition (expected statuses) before swapping. Please checkout WEBSITE_SWAP_WARMUP_PING_STATUSES.

[flagbug]

@suwatch Any update on the problem with the App Service being unable to load our .dll? We've switched back to WebDeploy for now, but this seems like a major flaw in the Run-From-Package, it means we can't trust the platform to deploy your application properly

[suwatch]

Can you create and repro on one of your test site and share us that? Give info about what DLL was not loaded etc.

[jvano]

Hi

If the problem persists and is related to running it on Azure App Service, please open a support incident in Azure:
https://learn.microsoft.com/en-us/azure/azure-portal/supportability/how-to-create-azure-support-request

This way we can better track and assist you on this case

Thanks,

Joaquin Vano
Azure App Service