Skip to content
This repository has been archived by the owner on Jul 5, 2020. It is now read-only.

Verify that App Services slot swaps are being detected in Web SDK #872

Closed
d3r3kk opened this issue Apr 3, 2018 · 5 comments
Closed

Verify that App Services slot swaps are being detected in Web SDK #872

d3r3kk opened this issue Apr 3, 2018 · 5 comments

Comments

@d3r3kk
Copy link
Contributor

d3r3kk commented Apr 3, 2018

There is some concern that slot swap detection is not reliable in the Web SDK for App Services. We have seen evidence that the slot swap was not being detected properly in Azure Functions, and we must ensure that other App Services (Web Apps, for instance) are not affected in the same way.

See comments in PR #870 regarding the use of WEBSITE_HOSTNAME for further insight & details.

Repro Steps

  1. Create test web application in Azure (use the New Devops Project workflow)
  2. Add a deployment slot to the application's CD pipeline
  3. Instrument the application with the latest beta Web SDK
  4. Perform a slot swap and watch App Insights Analytics to ensure WEBSITE_HOSTNAME is being recorded properly.

Actual Behavior

TO BE VERIFIED

Expected Behavior

The nodeName property of application insights telemetry will update to reflect the correct slot name within 30 seconds of the slot change.

Version Info

SDK Version : latest beta.
.NET Version : 4.5 (.NET full framework) or 1.3 (.NET Standard)
Hosting Info (IIS/Azure WebApps/ etc) : WebApps

@d3r3kk d3r3kk added this to the 2.6 milestone Apr 3, 2018
@d3r3kk d3r3kk self-assigned this Apr 3, 2018
@d3r3kk
Copy link
Contributor Author

d3r3kk commented Apr 3, 2018

See PR Azure/azure-webjobs-sdk#1382 for further detail.

d3r3kk added a commit that referenced this issue Apr 5, 2018
Add heartbeat provider for Azure App Services (in particular, Web Apps) that will add some infrastructure-identifying properties.
- Solves issue #648, #868, and #872.
- Both `AzureWebAppRoleEnvironment` and `AppServiceHeartbeatTelemetryModule` need to update their telemetry cache whenever the environment chances (ie, slot swap). `AppServiceEnvironmentVariableMonitor` solves this.
- Keep the value of WEBSITE_SITE_NAME and other App Services values current during runtime
- Merge PR #870 and #869, regarding heartbeat providers for App Services
- Ensure adequate testing is added, update current Env Var tests to be testable in a parallel environment
- `EnvironmentVariableMonitor` only updates the environment variables at set intervals, its subscribers only update when they need to.
- Disable monitor if we don't detect that we are in Azure App Service environment, and on SecurityExceptions, disable the monitor altogether
@TimothyMothra TimothyMothra modified the milestones: 2.6, 2.7 Apr 16, 2018
@TimothyMothra
Copy link
Member

@d3r3kk will you be able to test this next week?
Please follow up with @Dmitry-Matveev

@TimothyMothra TimothyMothra modified the milestones: 2.7, 2.7-Beta1 Apr 19, 2018
@d3r3kk
Copy link
Contributor Author

d3r3kk commented Apr 19, 2018

Will investigate by EOD tomorrow.

@d3r3kk
Copy link
Contributor Author

d3r3kk commented Apr 20, 2018

Environment values investigation

Can we use the environment to detect slot-swaps in Application Insights?

tl;dr No, we cannot. The environment variables on the host are not propagated to the process after a slot swap.

Findings at Runtime

What does the environment say in a demo app?

The environment variables are not updating how we need them to, in order to detect slot - swaps.
I wrote a demo ASP.NET app to show the current values of the environment variables before/after a slot swap.

Here's the variables in the 'dev slot' when I deploy my app into the DEV environment in my release definition in VSTS:

Environment Variable Name Value
WEBSITE_SITE_NAME dek-app2infra
WEBSITE_HOME_STAMPNAME waws-prod-sn1-115
WEBSITE_HOSTNAME dek-app2infra-dev.azurewebsites.net
WEBSITE_OWNER_NAME aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee+dekapp2infra9700-SouthCentralUSwebspace
WEBSITE_SLOT_NAME

And here's the values again after swapping the slots, my demo app is now in the 'production' slot:

Environment Variable Name Value
WEBSITE_SITE_NAME dek-app2infra
WEBSITE_HOME_STAMPNAME waws-prod-sn1-115
WEBSITE_HOSTNAME dek-app2infra-dev.azurewebsites.net
WEBSITE_OWNER_NAME aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee+dekapp2infra9700-SouthCentralUSwebspace
WEBSITE_SLOT_NAME

Notice that the WEBSITE_HOSTNAME is the url to the dev slot of the site I created, and that no value changed at all.

Peek Under the Hood

What is very odd to me, is that the underlying system seems to have different values for these variables! Here's the values from both of the slot's kudu site, from the Environment Variables section:

Environment Variable Dev Slot Value Production Slot Value
WEBSITE_SITE_NAME dek-app2infra dek-app2infra
WEBSITE_HOME_STAMPNAME waws-prod-sn1-115 waws-prod-sn1-115
WEBSITE_HOSTNAME dek-app2infra-dev.azurewebsites.net dek-app2infra.azurewebsites.net
WEBSITE_OWNER_NAME aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee+dekapp2infra9700-SouthCentralUSwebspace aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee+dekapp2infra9700-SouthCentralUSwebspace
WEBSITE_SLOT_NAME

Summary

So our current attempt to fix up this issue, inspired in part by the Azure Functions team's use of a special environment variable 'WEBSITE_SLOT_NAME' didn't pan out. No environment variable is currently giving us any indication of the slot the app is running in, and we will have to use some other method to detect when this happens.

TODO

Ask the Antares team if they plan on supporting the update of the environment variables we want to rely on for this.

If environment variable updates won't get updated: Remove the Environment variable monitor & subscription model, and investigate further to find a better way to detect slot swaps. Perhaps using request headers or some other more definitive way exists (we could repurpose the environment variable monitor to be a request-header-value monitor).

@d3r3kk d3r3kk removed their assignment Apr 24, 2018
@d3r3kk
Copy link
Contributor Author

d3r3kk commented Apr 24, 2018

Note that this does not impede the app2infra scenario for Web Apps.

@d3r3kk d3r3kk modified the milestones: 2.7-Beta1, Future Apr 24, 2018
@d3r3kk d3r3kk closed this as completed Jul 13, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

2 participants