Target Based Scaling for Storage Queues

[chiangvincent]

Overview

This PR introduces target based scaling for Storage Queues (and Blobs). To determine the target number of workers for a queue trigger in a function application, we divide the queueLength by the BatchSize defined in customer's host.json. This new scaling model is being implemented across all extensions.

Target Based Scaling PRs for Other Extensions

• Service Bus
• Event Hubs
• Cosmos DB

Checklist for Target Based Scaling Changes

• create QueueMetricsProvider
• remove duplicate logic between QueueSaleMonitor, QueueMetricsProvider
• QueueListener implement ITargetScalerProvider, IScaleMonitorProvider
• fix unit tests
• add QueueTargetScaler
• add unit tests for QueueTargetScaler
• E2E tests with Queue trigger with locally running function app

Contributing to the Azure SDK

Logs from E2E Verification With Local Function App

Steps:

1. Generate local nuget packages with Storage changes from this PR (Storage, Queue, Blob packages)
2. Create function app referencing local nuget packages
3. Publish and run the application locally with azure-functions-host repo
4. Enable runtime driven scaling, target based scaling, and add ScmHostingConfigurations.txt to enable Queue and Blob TBS
5. Send continuous Blob/Queue messages
6. Ping admin/host/scale/status API using Postman, observe target worker count and local azure-functions-host logs emitted

Verification Logs

Target Based Scaling Enabled, Queue Trigger

Target Based Scaling Disabled, Queue Trigger

Target Based Scaling Enabled, Blob Trigger

Target Based Scaling Disabled, Blob Trigger

Please see our CONTRIBUTING.md if you are not familiar with contributing to this repository or have questions.

For specific information about pull request etiquette and best practices, see this section.

[amnguye]

looks like the GetMonitor_ReturnsSharedMonitor test is giving CI trouble.

System.NullReferenceException : Object reference not set to an instance of an object.

   at Microsoft.Azure.WebJobs.Extensions.Storage.Common.Listeners.QueueListener.GetMonitor() in /Users/runner/work/1/s/sdk/storage/Microsoft.Azure.WebJobs.Extensions.Storage.Common/src/Shared/Queues/QueueListener.cs:line 461
   at Microsoft.Azure.WebJobs.Extensions.Storage.Blobs.Listeners.SharedBlobQueueListener.GetMonitor() in /Users/runner/work/1/s/sdk/storage/Microsoft.Azure.WebJobs.Extensions.Storage.Blobs/src/Listeners/SharedBlobQueueListener.cs:line 76
   at Microsoft.Azure.WebJobs.Extensions.Storage.Blobs.Listeners.BlobListener.GetMonitor() in /Users/runner/work/1/s/sdk/storage/Microsoft.Azure.WebJobs.Extensions.Storage.Blobs/src/Listeners/BlobListener.cs:line 114
   at Microsoft.Azure.WebJobs.Extensions.Storage.Blobs.Listeners.BlobListenerTests.GetMonitor_ReturnsSharedMonitor() in /Users/runner/work/1/s/sdk/storage/Microsoft.Azure.WebJobs.Extensions.Storage.Blobs/tests/Listeners/BlobListenerTests.cs:line 24

[amnguye]

Could we start an email thread in regards to this PR, with information of the purpose and the context? It would help in reviewing. Thanks.

[chiangvincent]

looks like the GetMonitor_ReturnsSharedMonitor test is giving CI trouble.

System.NullReferenceException : Object reference not set to an instance of an object.

   at Microsoft.Azure.WebJobs.Extensions.Storage.Common.Listeners.QueueListener.GetMonitor() in /Users/runner/work/1/s/sdk/storage/Microsoft.Azure.WebJobs.Extensions.Storage.Common/src/Shared/Queues/QueueListener.cs:line 461
   at Microsoft.Azure.WebJobs.Extensions.Storage.Blobs.Listeners.SharedBlobQueueListener.GetMonitor() in /Users/runner/work/1/s/sdk/storage/Microsoft.Azure.WebJobs.Extensions.Storage.Blobs/src/Listeners/SharedBlobQueueListener.cs:line 76
   at Microsoft.Azure.WebJobs.Extensions.Storage.Blobs.Listeners.BlobListener.GetMonitor() in /Users/runner/work/1/s/sdk/storage/Microsoft.Azure.WebJobs.Extensions.Storage.Blobs/src/Listeners/BlobListener.cs:line 114
   at Microsoft.Azure.WebJobs.Extensions.Storage.Blobs.Listeners.BlobListenerTests.GetMonitor_ReturnsSharedMonitor() in /Users/runner/work/1/s/sdk/storage/Microsoft.Azure.WebJobs.Extensions.Storage.Blobs/tests/Listeners/BlobListenerTests.cs:line 24

[chiangvincent]

looks like the GetMonitor_ReturnsSharedMonitor test is giving CI trouble.

System.NullReferenceException : Object reference not set to an instance of an object.

   at Microsoft.Azure.WebJobs.Extensions.Storage.Common.Listeners.QueueListener.GetMonitor() in /Users/runner/work/1/s/sdk/storage/Microsoft.Azure.WebJobs.Extensions.Storage.Common/src/Shared/Queues/QueueListener.cs:line 461
   at Microsoft.Azure.WebJobs.Extensions.Storage.Blobs.Listeners.SharedBlobQueueListener.GetMonitor() in /Users/runner/work/1/s/sdk/storage/Microsoft.Azure.WebJobs.Extensions.Storage.Blobs/src/Listeners/SharedBlobQueueListener.cs:line 76
   at Microsoft.Azure.WebJobs.Extensions.Storage.Blobs.Listeners.BlobListener.GetMonitor() in /Users/runner/work/1/s/sdk/storage/Microsoft.Azure.WebJobs.Extensions.Storage.Blobs/src/Listeners/BlobListener.cs:line 114
   at Microsoft.Azure.WebJobs.Extensions.Storage.Blobs.Listeners.BlobListenerTests.GetMonitor_ReturnsSharedMonitor() in /Users/runner/work/1/s/sdk/storage/Microsoft.Azure.WebJobs.Extensions.Storage.Blobs/tests/Listeners/BlobListenerTests.cs:line 24

I added mock constructors to QueueScaleMonitor and QueueTargetScaler, and instantiated them in the mock ctor for the QueueListener. Let me know if there's a better way around this -- I tried mocking the QueueListener class but wasn't able to because it's sealed.

[amnguye]

You might have to merge main after this PR goes through, but it's only a small change so you might not have to.
#33682

Nevermind probably not.

[chiangvincent]

Synced with @amnguye offline, soft approval until webjobs sdk 3.0.36 and azure-sdk-for-net changes are pushed

[chiangvincent]

3.0.36 has been released, but there is an issue affecting azure-sdk-for-net extensions, @alrod for context. We expect this to be fixed by the end of this week, and I will update the PR and ping when ready for final merge. (The blocker is not related to Storage TBS code)

[azure-sdk]

API change check

API changes are not detected in this pull request.