Fix target scaler fallback race condition via shared storage

[alrod]

Problem

On Elastic Premium plans, when a target scaler throws NotSupportedException (e.g., ServiceBus without Manage claim), the fallback to incremental scale monitoring fails if the Scale Controller and primary host run on different workers.

_targetScalersInError was a static in-process HashSet<string>. Only the admin/host/scale/status code path (called by the Scale Controller) invokes target scalers and populates this set. The ScaleMonitorService.TakeMetricsSamplesAsync() path (running on the primary host) reads the set but never populates it. When these run on different workers, the primary host's _targetScalersInError stays empty — it never discovers the failure, excludes the incremental monitor, collects no metrics, and produces no scale-in votes. The app stays stuck at max workers indefinitely.

Root Cause

The fundamental issue is that _targetScalersInError is process-local shared state that needs to be visible across multiple workers — the same class of problem solved by IScaleMetricsRepository (for metrics) and IConcurrencyStatusRepository (for concurrency snapshots), both of which use blob storage.

Fix

Introduce a new internal ITargetScalerErrorRepository abstraction backed by blob storage, following the established BlobStorageConcurrencyStatusRepository pattern.

New Files

File	Description
ITargetScalerErrorRepository	Internal interface: AddAsync, GetAsync
NullTargetScalerErrorRepository	No-op default (used when storage isn't configured)
BlobStorageTargetScalerErrorRepository	Blob-backed impl persisting to scale/{hostId}/targetScalersInError.json

Modified Files

File	Description
ScaleManager.cs	Removed static _targetScalersInError. Injected ITargetScalerErrorRepository. Writes errors via AddAsync on NotSupportedException. GetScalersToSample is now async and accepts the repository directly (reads inside IsTargetScalingEnabled block).
ScaleMonitorService.cs	Injected ITargetScalerErrorRepository. Passes repository to GetScalersToSample.
WebJobsServiceCollectionExtensions.cs	Registers NullTargetScalerErrorRepository as default
StorageServiceCollectionExtensions.cs	Registers BlobStorageTargetScalerErrorRepository when Azure Storage is configured

Key Design Decisions

ETag-based optimistic concurrency: AddAsync uses a read-modify-write loop with ETag conditions to prevent lost updates when multiple instances write concurrently. Uses IfMatch for existing blobs and IfNoneMatch = * for new blobs. Retries up to 3 times on 412/409.

TTL-based expiry (no ClearAsync): Instead of explicitly clearing errors on startup (which races with other instances), the blob stores a lastUpdated timestamp alongside the scaler set. GetAsync returns empty if the data is older than 10 minutes. This makes recovery automatic — once the error stops occurring (e.g., customer grants Manage claim and restarts), no new AddAsync calls refresh the timestamp, and the data ages out.

Internal interface: ITargetScalerErrorRepository is internal — it's an implementation detail, not public API surface.

How It Fixes the Race Condition

1. Worker A (running Scale Controller) calls GetScaleStatusAsync → catches NotSupportedException → writes scaler ID to blob via AddAsync
2. Worker B (primary host) calls TakeMetricsSamplesAsync → GetScalersToSample reads blob via repository → sees the error → falls back to incremental monitor → collects metrics → produces scale-in votes

Recovery Flow

1. TBS call fails → NotSupportedException → AddAsync() → blob created/updated with current timestamp
2. As long as the error persists, each Scale Controller poll refreshes the timestamp
3. Customer grants Manage claim → restarts app → TBS succeeds → no AddAsync calls → blob ages out after 10 minutes → TBS used again

Testing

• Updated existing GetScalersToSample_FallsBackToMonitor_OnTargetScalerError test to use repository
• Updated all ScaleManager constructor calls for new parameter
• Added GetScalersToSample_WithErrorSet_FiltersCorrectly test
• Added GetScalersToSample_WithNullErrorSet_NoFiltering test (uses NullTargetScalerErrorRepository)
• Added GetScaleStatusAsync_SecondCall_ReadsPersistedError test
• Added OnTimer_CrossWorkerFallback_SharedRepository test
• Added GetAsync_StaleData_ReturnsEmpty TTL expiry test
• Added GetAsync_FreshData_ReturnsScalers TTL expiry test
• Updated E2E blob tests to verify new TargetScalerErrorState format
• All 162 scale + public surface tests pass (0 errors, 0 warnings)