Migrations: Run unattended upgrades in background, add liveness/readiness health probes (closes #21987)

[AndyButland]

Description

This PR looks to address the proposal outlined in #21987.

When UpgradeUnattended: true is configured and database migrations are needed, migrations previously ran synchronously inside BootUmbracoAsync() — before the HTTP server bound any port. This could cause the hosting platform to consider the web application down and kill the process. IIS for example has a default 120-second start-up timeout. For most migrations this is plenty, but there are some, in particular going to some majors, that will take longer than this on large websites.

This PR moves migration execution into a BackgroundService so BootUmbracoAsync() returns quickly, The web server will start accepting connections immediately, and the application can serve health probe responses and maintenance pages while migrations run in the background.

Changes

New runtime states

A new value added to RuntimeLevel: Upgrading — unattended upgrade is running in the background; HTTP server is up

Background migration service (UnattendedUpgradeBackgroundService)

This is a new BackgroundService that takes over the migration sequence previously handled inline during start up when UpgradeUnattended: true. It replicates the full notification sequence (premigrations → post-premigrations → unattended upgrade) that we had before, then initialises components and fires the application-started lifetime events.

Health probes

Two new endpoints, available in every hosting configuration (registered in Umbraco.Web.Common):

Endpoint	Behaviour
GET /umbraco/api/health/live	Always 200 if the process is responding — no checks run
GET /umbraco/api/health/ready	200 when RuntimeLevel.Run; 503 Degraded otherwise

Both are anonymous and bypass the maintenance-page reroute.

HTTP behaviour during Upgrading

Surface	Behaviour
Website (front-end)	Returns HTTP 503 with the new Upgrading.cshtml view (configurable via Umbraco:CMS:Global:UpgradingViewPath)
Surface controllers	Returns 503 with the new Upgrading.cshtml view
Backoffice	Returns screen indicating that an upgrade is in progress
Management API	Returns JSON ProblemDetails HTTP 503
Delivery API	Returns JSON ProblemDetails HTTP 503

Testing

Automated

Various unit tests have been added to verify both the new behaviour and fix coverage gaps for files changed in existing behaviour.

Manual

Preparation

A slow migration is useful for holding the application in Upgrading long enough to verify each surface. Create a temporary migration:

internal sealed class SlowTestMigration : AsyncMigrationBase
{
    public SlowTestMigration(IMigrationContext context) : base(context) { }

    protected override async Task MigrateAsync()
    {
        Logger.LogInformation("SlowTestMigration: starting 60-second delay...");
        await Task.Delay(TimeSpan.FromSeconds(60));
        Logger.LogInformation("SlowTestMigration: delay complete.");
    }
}

Register it in UmbracoPlan.cs after the last migration:

To<SlowTestMigration>("{A1B2C3D4-E5F6-7890-ABCD-EF1234567890}");

Set UpgradeUnattended: true in appsettings.json:

"Umbraco": {
  "CMS": {
    "Unattended": {
      "UpgradeUnattended": true
    }
  }
}

Test: Unattended upgrade

Start the application. Within the first 60 seconds (while the migration is running):

• GET /umbraco/api/health/live → 200 Healthy
• GET /umbraco/api/health/ready → 503 Degraded (body contains "Umbraco is not yet ready")
• GET / (or any front-end page) → 503 with Upgrading.cshtml content (see screenshot below)
• GET /umbraco (backoffice) → custom backoffice page for maintenance (see screenshot below)
• GET /umbraco/management/api/v1/manifest/manifest/public → 503 JSON ProblemDetails ("The application is currently being upgraded")
• GET /umbraco/delivery/api/v1/content → 503 JSON ProblemDetails
• Any surface controller endpoint → 503 with Upgrading.cshtml content

After the 60-second delay completes:

• GET /umbraco/api/health/ready → 200 Healthy
• GET / → normal front-end content
• GET /umbraco → backoffice loads normally
• GET /umbraco/management/api/v1/manifest/manifest/public → normal JSON response

Screenshots

Test: Attended upgrade path unchanged

Set UpgradeUnattended: false. Roll back the migration using the following SQL:

update umbracoKeyValue
set value = '{B8C9D0E1-F2A3-4B5C-8D7E-9F0A1B2C3D4E}'
where [key] = 'Umbraco.Core.Upgrader.State+Umbraco.Core'

Start the application.

• Backoffice shows the upgrade screen at /umbraco as before
• No regressions in attended upgrade behaviour

Test: Custom upgrading view

Add to appsettings.json:

"Umbraco": {
  "CMS": {
    "Global": {
      "UpgradingViewPath": "~/Views/MyUpgrading.cshtml"
    }
  }
}

Create Views/MyUpgrading.cshtml with custom content.

Again set UpgradeUnattended: true and roll back the migration.

Start the application.

Documentation

We should document the new healthchecks.

[Copilot]

Pull request overview

This PR moves unattended Umbraco upgrade migrations from synchronous startup execution (which caused IIS ANCM/Kubernetes timeout issues) to a BackgroundService, allowing the HTTP server to start immediately and respond to health probes during migration.

Changes:

• Adds two new RuntimeLevel values (Upgrading, UpgradeFailed) and a new UnattendedUpgradeBackgroundService that runs migrations after HTTP server startup
• Adds liveness (/umbraco/api/health/live) and readiness (/umbraco/api/health/ready) endpoints via ASP.NET Core health checks
• Updates MaintenanceModeActionFilterAttribute, BootFailedMiddleware, routing, and controller base classes to handle the new Upgrading/UpgradeFailed states

Reviewed changes

Copilot reviewed 33 out of 33 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/Umbraco.Core/RuntimeLevel.cs	Adds Upgrading=4 and UpgradeFailed=5 enum values
src/Umbraco.Infrastructure/Runtime/RuntimeState.cs	Sets Upgrading instead of Run when unattended upgrades are pending
src/Umbraco.Infrastructure/Runtime/CoreRuntime.cs	Returns early for Upgrading state (background service takes over)
src/Umbraco.Infrastructure/Install/UnattendedUpgradeBackgroundService.cs	New background service running migration sequence
src/Umbraco.Core/Extensions/RuntimeStateExtensions.cs	Allows Upgrading level in RunUnattendedBootLogic()
src/Umbraco.Core/Configuration/Models/GlobalSettings.cs	Adds UpgradingViewPath setting
src/Umbraco.Web.Common/ActionsResults/MaintenanceResult.cs	Accepts optional view path parameter
src/Umbraco.Web.Common/Controllers/MaintenanceModeActionFilterAttribute.cs	Handles Upgrading; made public; returns JSON ProblemDetails for API controllers
src/Umbraco.Web.Common/Middleware/BootFailedMiddleware.cs	Handles UpgradeFailed with static 503 page
src/Umbraco.Web.Common/HealthChecks/UmbracoReadinessHealthCheck.cs	New ASP.NET Core readiness check
src/Umbraco.Web.Common/Extensions/ApplicationBuilderExtensions.cs	Registers health probe endpoints; adds BootFailedMiddleware to normal boot path
src/Umbraco.Web.Common/DependencyInjection/UmbracoBuilderExtensions.cs	Registers UmbracoReadinessHealthCheck
src/Umbraco.Web.Website/Routing/EagerMatcherPolicy.cs	Routes Upgrading to maintenance page
src/Umbraco.Web.Website/Routing/UmbracoRouteValueTransformer.cs	Early return for Upgrading state
src/Umbraco.Web.Website/Routing/FrontEndRoutes.cs	Includes Upgrading in route creation check
src/Umbraco.Web.Website/Controllers/SurfaceController.cs	Adds [MaintenanceModeActionFilter]
src/Umbraco.Cms.Api.Management/Controllers/ManagementApiControllerBase.cs	Adds [MaintenanceModeActionFilter]
src/Umbraco.Cms.Api.Delivery/Controllers/DeliveryApiControllerBase.cs	Adds [MaintenanceModeActionFilter]
src/Umbraco.Cms.Api.Management/Routing/BackOfficeAreaRoutes.cs	Includes Upgrading in route creation
src/Umbraco.Cms.Api.Management/Routing/PreviewRoutes.cs	Includes Upgrading in route creation
src/Umbraco.Infrastructure/Persistence/Repositories/Implement/UserRepository.cs	Includes Upgrading in upgrade-state checks
src/Umbraco.Infrastructure/Persistence/Repositories/Implement/AuditEntryRepository.cs	Includes Upgrading in upgrade-state check
src/Umbraco.Infrastructure/Security/BackOfficeUserStore.cs	Includes Upgrading in upgrade-state check
src/Umbraco.Infrastructure/DependencyInjection/UmbracoBuilder.CoreServices.cs	Registers UnattendedUpgradeBackgroundService as hosted service
src/Umbraco.Infrastructure/Runtime/RuntimeState.cs	Sets Upgrading for unattended upgrade/package migration cases
src/Umbraco.Web.UI.Client/src/packages/core/backend-api/types.gen.ts	Adds UPGRADING/UPGRADE_FAILED to frontend enum
src/Umbraco.Web.UI.Client/src/apps/app/app.element.ts	Handles Upgrading state in frontend app
src/Umbraco.Cms.Api.Management/OpenApi.json	Adds new enum values to OpenAPI spec
src/Umbraco.Cms.StaticAssets/umbraco/UmbracoWebsite/Upgrading.cshtml	New maintenance view for upgrading state
Test files	New unit tests for new components and behavior

[Migaroez]

Backend LGTM, added small comment on the tests.

[Migaroez]

LGTM, fine to merge in as soon as we get a front end look at the small change and the backoffice tests pass

[iOvergaard]

From a FE perspective, this looks good so far, just two minor things.

[iOvergaard]

FE - LGTM

[AndyButland]

Docs PR is here: umbraco/UmbracoDocs#7880