Add ScaleController V3 integration by davidmrdavid · Pull Request #2462 · Azure/azure-functions-durable-extension

davidmrdavid · 2023-05-02T18:37:50Z

This PR

This PR allows Scale Controller V3 to receive TBS worker requests for DF. This PR builds upon the changes in #2452 so please review that PR first.

Since this PR builds upon the changes in #2452, that' is also the branch we're targeting.

Main Changes

The main objective of this PR is the implementation of DurableTaskTriggersScaleProvider, which the Scale Controller Host will utilize to make scaling decisions.

This class implements IScaleMonitorProvider and ITargetScalerProvider by delegating to the GetTargetScaler and GetScaleMonitor implementations of #2452.

When constructing DurableTaskTriggersScaleProvider, we receive the DI services provider from the Scale Controller Host and the Sync Triggers payload for the function app. We read the user's config (hubName, backend type, etc) from the Sync Triggers payload.

Scale Controller V3 will obtain an instance of this class by invoking private static IWebJobsBuilder AddDurableScaleForTrigger(this IWebJobsBuilder builder, TriggerMetadata triggerMetadata) during initialization.

Finally, note that DurableTaskTriggersScaleProvider will not be constructed in the Functions Host. It is constructed only by the ScaleController Host through a call to AddDurableScaleForTrigger.

Secondary Changes

For code re-use, this PR removes a several scaling utility methods defined in DurableTaskExtension.cs and puts them inside a new file: Scale/ScaleUtils.cs. That way, we're able to re-use these utilites in the
DurableTaskTriggersScaleProvider implementation.

We also add #nullable enable in all new files.

References:

Equivalent ServiceBus PR: [WebJobs.Extensions.ServiceBus] Scaler APIs azure-sdk-for-net#34557
Equivalent EventHubs PR: alrod/azure-sdk-for-net@78e99c2

Issue describing the changes in this PR

resolves #issue_for_this_pr

Pull request checklist

davidmrdavid · 2023-05-08T18:55:44Z

src/WebJobs.Extensions.DurableTask/Scale/ScaleUtils.cs

+                // the durability provider does not support target-based scaling.
+                // Create an empty target scaler to avoid exceptions (unless target-based scaling is actually turned on).
+                return new NoOpTargetScaler(functionId);
+            }


unknown: should we "fail fast" on the scalehost if TBS is not supported?

At the very least, I think we should have a log that says that the durability provider doesn't support TBS.

Yeah I don't disagree, although I don't have a ScaleController-provided ILogger instance handy in this class. Let me try to get some clarity on this edge case from the Sc team to figure out how we want to deal with this.

…s-durable-extension into dajusto/scv3

davidmrdavid · 2023-05-11T23:54:39Z

src/WebJobs.Extensions.DurableTask/DurableTaskExtension.cs

            this.PlatformInformationService = platformInformationService ?? throw new ArgumentNullException(nameof(platformInformationService));
-            this.ResolveAppSettingOptions();
+            DurableTaskOptions.ResolveAppSettingOptions(this.Options, this.nameResolver);



made this method static so it may be re-used in DurableTaskTriggersScaleProvider

davidmrdavid · 2023-05-11T23:55:30Z

src/WebJobs.Extensions.DurableTask/DurableTaskExtension.cs

            this.TraceHelper = new EndToEndTraceHelper(logger, this.Options.Tracing.TraceReplayEvents);
            this.LifeCycleNotificationHelper = lifeCycleNotificationHelper ?? this.CreateLifeCycleNotificationHelper();
-            this.durabilityProviderFactory = this.GetDurabilityProviderFactory(this.Options, logger, orchestrationServiceFactories);
+            this.durabilityProviderFactory = GetDurabilityProviderFactory(this.Options, logger, orchestrationServiceFactories);


Also made this static so it may be re-used in GetDurabilityProviderFactory.

src/WebJobs.Extensions.DurableTask/DurableTaskJobHostConfigurationExtensions.cs

…what gets recognized in ScV3's logging pipeline

…s-durable-extension into dajusto/scv3

bachuv · 2023-09-21T23:31:28Z

src/WebJobs.Extensions.DurableTask/Scale/ScaleUtils.cs

+                // the durability provider does not support target-based scaling.
+                // Create an empty target scaler to avoid exceptions (unless target-based scaling is actually turned on).
+                return new NoOpTargetScaler(functionId);
+            }


At the very least, I think we should have a log that says that the durability provider doesn't support TBS.

src/WebJobs.Extensions.DurableTask/Scale/DurableTaskTriggersScaleProvider.cs

cgillum · 2023-09-21T23:39:58Z

src/WebJobs.Extensions.DurableTask/Options/DurableTaskOptions.cs

+        {
+            if (options == null)
+            {
+                throw new InvalidOperationException($"{nameof(options)} must be set before resolving app settings.");


Souldn't these be simple ArgumentNullException?

I now see that you copied this from DurableTaskExtension.cs. It made sense for it to be InvalidOperationException in that case since these were not parameters. Now that you've made them into parameters, the correct thing to do is make this ArgumentNullException.

I don't mind making this change but I want to clarify that this method isn't just copied from DurableTaskExtension.cs, it's been refactored out of DurableTaskExtension.cs so that it can be used here (for TBS purposes) and in the DurableTaskExtension.cs. With that in mind, do you still want me to change InvalidOperationException for ArgumentNullException, or keep it as is?

I could also just duplicate the code in case we want different exception types in different locations.

cgillum · 2023-09-21T23:42:23Z

src/WebJobs.Extensions.DurableTask/DurableTaskJobHostConfigurationExtensions.cs

+        /// </summary>
+        /// <param name="builder">The <see cref="IWebJobsBuilder"/> to configure.</param>
+        /// <returns>Returns the provided <see cref="IWebJobsBuilder"/>.</returns>
+        public static IWebJobsBuilder AddDurableScaleForTrigger(this IWebJobsBuilder builder, TriggerMetadata triggerMetadata)


naming nit: I wonder if a method name like AddScalerForDurableTriggers() would be a bit clearer. We're not really adding "Durable scale" to anything.

I agree the name is weird (and there's no a "durable scale" operation, whatever that means :-) ), I'm just adhering to the naming conventions in the ScV3 repo.

Other builder methods include:

AddTimerScaleForTrigger

AddServiceBusScaleForTrigger

AddCosmosDbScaleForTrigger

AddAzureStorageQueuesScaleForTrigger
etc.

I could definitely rename this to AddDurableFunctionsScaleForTrigger if that helps. However, I think the convention of Add<Extension>ScaleForTrigger means that the naming will be strange no matter what.

src/WebJobs.Extensions.DurableTask/DurableTaskJobHostConfigurationExtensions.cs

src/WebJobs.Extensions.DurableTask/Scale/ScaleUtils.cs

src/WebJobs.Extensions.DurableTask/Scale/DurableTaskTriggersScaleProvider.cs

src/WebJobs.Extensions.DurableTask/Listener/DurableTaskTargetScaler.cs

cgillum · 2023-09-22T00:12:11Z

src/WebJobs.Extensions.DurableTask/Listener/DurableTaskTargetScaler.cs

+                    scaleControllerLog = $"Error: target worker count for '{this.functionId}' was negative: '{this.functionId}'." +
+                        "An exception was thrown." + metricsLog;
+                    this.logger.LogError(scaleControllerLog);
+                    throw new Exception(scaleControllerLog);


I'm just now noticing this, but it's bad practice to throw Exception. Consider using InvalidOperationException instead.

Actually, doesn't this result in double-exception and double logging? If I'm reading the code correctly, we catch this exception a few lines down and make another call to this.logger.LogError. It seems like we need to reconsider the design here, because using exceptions as control flow like this is also bad practice.

Yeah you're right, it's getting double logged; good call out.

A part of me wants a try-catch surrounding this whole operation so we can log any exceptions that arise, and to include in those logs the contextual information needed to debug this. That's what I do in the outer catch block.

Do you think it would suffice to remove the LogError when the target worker count is negative, and simply throw an exception? Alternatively, I could refactor the outer try-catch so that it ends right when the target worker count is calculated (i.e line 71)

At a minimum we should remove the first LogError and rely on the one inside the catch-block. But I wonder if we even need that one since we're throwing. Wouldn't it be expected that some other code catches these exceptions and logs them as errors?

@alrod, do you know what the expected behavior is for logging and throwing exceptions? Do we log the error or will it be handled in the scale controller?

Still looking for an update on this.

numWorkersToRequest is calcuated as:
int numWorkersToRequest = (int)Math.Max(activityWorkers, orchestratorWorkers);

How numWorkersToRequest can be negative?

If you want to throw an exception from GetScaleResultAsync I would just thorwing InvalidOperationException/ArgumentOutOfRangeException without writing the error log - Scale Controller will take care about the error logging.

Something like:
https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/servicebus/Microsoft.Azure.WebJobs.Extensions.ServiceBus/src/Listeners/ServiceBusTargetScaler.cs#L94

Thanks @alrod. I agree that the current code cannot provide negative numbers, my check is just ensuring that remains true if the code is ever refactored. It may a bit too defensive, so I don't mind removing it if others think it's overkill :).

@cgillum: I've removed the double logging on this commit: 0533264

yeah, I think it's an overkill. I vote to remove the code. Or at least change the exception to InvalidOperationException.

Changed the exception type to InvalidOperationException: 7f47657

…tionExtensions.cs Co-authored-by: Chris Gillum <cgillum@microsoft.com>

src/WebJobs.Extensions.DurableTask/DurableTaskJobHostConfigurationExtensions.cs

cgillum

Added a few more comments. I think the only one I consider blocking is the one about double (or maybe even triple) logging errors.

src/WebJobs.Extensions.DurableTask/DurableTaskJobHostConfigurationExtensions.cs

cgillum · 2023-09-25T21:26:37Z

src/WebJobs.Extensions.DurableTask/Listener/DurableTaskTargetScaler.cs

+                    scaleControllerLog = $"Error: target worker count for '{this.functionId}' was negative: '{this.functionId}'." +
+                        "An exception was thrown." + metricsLog;
+                    this.logger.LogError(scaleControllerLog);
+                    throw new Exception(scaleControllerLog);


At a minimum we should remove the first LogError and rely on the one inside the catch-block. But I wonder if we even need that one since we're throwing. Wouldn't it be expected that some other code catches these exceptions and logs them as errors?

src/WebJobs.Extensions.DurableTask/Scale/DurableTaskTriggersScaleProvider.cs

alrod · 2023-09-26T17:15:48Z

src/WebJobs.Extensions.DurableTask/Listener/DurableTaskTargetScaler.cs

-
-            // target worker count should never be negative
-            if (numWorkersToRequest < 0)
+            DurableTaskTriggerMetrics? metrics = null;


To fall back to regular scale for Netherite and MSSQL you need to throw NonSupportedException:

azure-sdk-for-net/sdk/servicebus/Microsoft.Azure.WebJobs.Extensions.ServiceBus/src/Listeners/ServiceBusTargetScaler.cs at main · Azure/azure-sdk-for-net (github.com)

Do you have this implemented for here?
Can we add a test for this?

I updated the exception here.

Yes, we can add a test.

this was incorporated in my latest commit

cgillum · 2023-10-03T17:22:29Z

@bachuv I'm looking for resolution on two open comment threads in DurableTaskTargetScaler.cs. I'll sign off when those are resolved.

alrod · 2023-10-05T15:54:38Z

src/WebJobs.Extensions.DurableTask/Listener/DurableTaskListener.cs

+                    this.config.Options.HubName));

 #endif
 #if FUNCTIONS_V3_OR_GREATER


In SC we have "FUNCTIONS_V3_OR_GREATER", right?

I believe so. SC has TFM .net6.0 and FUNCTIONS_V3_OR_GREATER targets netcoreapp3.1, which is "older" than .net6.0. So yes, I believe this code triggers in SC.

alrod · 2023-10-05T15:57:41Z

src/WebJobs.Extensions.DurableTask/DurableTaskJobHostConfigurationExtensions.cs

+        internal static IWebJobsBuilder AddDurableScaleForTrigger(this IWebJobsBuilder builder, TriggerMetadata triggerMetadata)
+        {
+            // this segment adheres to the followings pattern: https://github.com/Azure/azure-sdk-for-net/pull/38673/files
+            builder.Services.AddSingleton(serviceProvider => new DurableTaskTriggersScaleProvider(serviceProvider.GetService<IOptions<DurableTaskOptions>>(), serviceProvider.GetService<INameResolver>(), serviceProvider.GetService<ILoggerFactory>(), serviceProvider.GetService<IEnumerable<IDurabilityProviderFactory>>(), triggerMetadata));


We can pass serviceProvider as parameter to the DurableTaskTriggersScaleProvider ctor and resolve all required services inside the ctor

Thanks @alrod. I think the code did this originally, but as per this comment we decided not to resolve services in the constructor: #2462 (comment)

alrod · 2023-10-05T16:00:58Z

src/WebJobs.Extensions.DurableTask/DurableTaskJobHostConfigurationExtensions.cs

+        {
+            // this segment adheres to the followings pattern: https://github.com/Azure/azure-sdk-for-net/pull/38673/files
+            builder.Services.AddSingleton(serviceProvider => new DurableTaskTriggersScaleProvider(serviceProvider.GetService<IOptions<DurableTaskOptions>>(), serviceProvider.GetService<INameResolver>(), serviceProvider.GetService<ILoggerFactory>(), serviceProvider.GetService<IEnumerable<IDurabilityProviderFactory>>(), triggerMetadata));
+            builder.Services.AddSingleton<IScaleMonitorProvider>(serviceProvider => serviceProvider.GetRequiredService<DurableTaskTriggersScaleProvider>());


There was a bug in this pattern, please change according to:
Azure/azure-sdk-for-net#38756

Thanks, addressed here: 1975b41

src/WebJobs.Extensions.DurableTask/Listener/DurableTaskTargetScaler.cs

cgillum · 2023-10-05T16:34:36Z

src/WebJobs.Extensions.DurableTask/Listener/DurableTaskTargetScaler.cs

+                var metricsLog = $"Metrics: workItemQueueLength={metrics?.WorkItemQueueLength}. controlQueueLengths={metrics?.ControlQueueLengths}. " +
+                    $"maxConcurrentOrchestrators={this.MaxConcurrentOrchestrators}. maxConcurrentActivities={this.MaxConcurrentActivities}";
+                var errorLog = $"Error: target worker count for '{this.functionId}' resulted in exception." + metricsLog + $"Exception: {ex}";
+                throw new Exception(errorLog, ex);


nit: we're effectively double-logging the exception here my including the same exception information in both errorLog and as an inner exception ex. This can be confusing for someone trying to debug scale controller logs.

You're right, I think I was being overly verbose here. I've updated the code to remove the string representation of the error and instead just rely on the "innerException" parameter: f5c6d3f

…caler.cs Co-authored-by: Chris Gillum <cgillum@microsoft.com>

alrod · 2023-10-05T16:59:39Z

src/WebJobs.Extensions.DurableTask/Scale/ScaleUtils.cs

+    internal static class ScaleUtils
+    {
+#if !FUNCTIONS_V1
+        internal static IScaleMonitor GetScaleMonitor(DurabilityProvider durabilityProvider, string functionId, FunctionName functionName, string? connectionName, string hubName)


Can we also add unit tests for ScaleUtils.GetTargetScaler and ScaleUtils.GetScaleMonitor

Just added them: 635aa75

and expanded them here: 3ad0f28

…kTargetScaler

davidmrdavid · 2023-10-06T02:00:24Z

test/FunctionsV2/DurableTaskTargetScalerTests.cs

+        [Theory]
+        [Trait("Category", PlatformSpecificHelpers.TestCategory)]
+        [InlineData(true)]
+        [InlineData(false)]
+        public async void ScaleHostE2ETest(bool isTbsEnabled)
+        {
+            Action<ScaleOptions> configureScaleOptions = (scaleOptions) =>
+            {
+                scaleOptions.IsTargetScalingEnabled = isTbsEnabled;
+                scaleOptions.MetricsPurgeEnabled = false;
+                scaleOptions.ScaleMetricsMaxAge = TimeSpan.FromMinutes(4);
+                scaleOptions.IsRuntimeScalingEnabled = true;
+                scaleOptions.ScaleMetricsSampleInterval = TimeSpan.FromSeconds(1);
+            };
+            using (FunctionsV2HostWrapper host = (FunctionsV2HostWrapper)TestHelpers.GetJobHost(this.loggerProvider, nameof(this.ScaleHostE2ETest), enableExtendedSessions: false, configureScaleOptions: configureScaleOptions))
+            {
+                await host.StartAsync();
+
+                IScaleStatusProvider scaleManager = host.InnerHost.Services.GetService<IScaleStatusProvider>();
+                var client = await host.StartOrchestratorAsync(nameof(TestOrchestrations.FanOutFanIn), 50, this.output);
+                var status = await client.WaitForCompletionAsync(this.output, timeout: TimeSpan.FromSeconds(400));
+                var scaleStatus = await scaleManager.GetScaleStatusAsync(new ScaleStatusContext());
+                await host.StopAsync();
+                Assert.Equal(OrchestrationRuntimeStatus.Completed, status?.RuntimeStatus);
+
+                // We inspect the Host's logs for evidence that the Host is correctly sampling our scaling requests.
+                // the expected logs depend on whether TBS is enabled or not
+                var expectedSubString = "scale monitors to sample";
+                if (isTbsEnabled)
+                {
+                    expectedSubString = "target scalers to sample";
+                }
+
+                bool containsExpectedLog = this.loggerProvider.GetAllLogMessages().Select(p => p.FormattedMessage ?? "").Any(p => p.Contains(expectedSubString));
+                Assert.True(containsExpectedLog);
+            }
+        }


@alrod: an E2E test, using the pre-existing DF testing infra, has been added here ^

I realize this is not exactly like the E2E tests you suggested (doing so would have taken more time due to how this codebase is structured and the DF initialization boilerplate), but I think this, in addition to the unit tests we already have, should have sufficient testing coverage.

davidmrdavid added 2 commits May 2, 2023 11:36

draft implementation

13b8405

only instantiate necessary services

62e988d

davidmrdavid commented May 8, 2023

View reviewed changes

davidmrdavid added 4 commits May 10, 2023 16:37

refactor

aac6b0b

remove connectionName changes

c97113d

add storage provider payload

35d1fc8

Merge branch 'dajusto/tbs' of https://github.com/Azure/azure-function…

694af1c

…s-durable-extension into dajusto/scv3

davidmrdavid marked this pull request as ready for review May 11, 2023 23:51

davidmrdavid changed the title ~~[WIP] Add ScaleController V3 integration~~ Add ScaleController V3 integration May 11, 2023

remove unused import

4580fb7

davidmrdavid commented May 11, 2023

View reviewed changes

src/WebJobs.Extensions.DurableTask/DurableTaskJobHostConfigurationExtensions.cs Outdated Show resolved Hide resolved

davidmrdavid added 10 commits May 19, 2023 14:27

add version suffix

34cf4dc

update version suffix

c69eb6e

hack: use CreateLogger<T> over CreateLogger(<string>) because that's …

d2b963c

…what gets recognized in ScV3's logging pipeline

reorder statements

f5be838

remove logging hack

e8a9f16

Remove logging message

59e73e0

remove empty constructor

ab423bc

remove unused imports

847c3c6

incorporate some PR feedback

a755660

Merge branch 'dajusto/tbs' of https://github.com/Azure/azure-function…

4c4bc83

…s-durable-extension into dajusto/scv3

bachuv reviewed Sep 22, 2023

View reviewed changes

cgillum reviewed Sep 22, 2023

View reviewed changes

davidmrdavid and others added 4 commits September 22, 2023 10:38

Update src/WebJobs.Extensions.DurableTask/DurableTaskJobHostConfigura…

b6cf5d8

…tionExtensions.cs Co-authored-by: Chris Gillum <cgillum@microsoft.com>

incorporate some PR feedback

1c82e28

add comments back

4b02438

refactor all scaling utils to the static ScaleUtils class

5e7fc18

alrod reviewed Sep 25, 2023

View reviewed changes

src/WebJobs.Extensions.DurableTask/DurableTaskJobHostConfigurationExtensions.cs Outdated Show resolved Hide resolved

davidmrdavid requested review from bachuv and cgillum September 25, 2023 21:06

cgillum reviewed Sep 25, 2023

View reviewed changes

alrod reviewed Sep 26, 2023

View reviewed changes

addressed some PR feedback

becd05e

bachuv requested a review from cgillum October 3, 2023 17:15

alrod reviewed Oct 5, 2023

View reviewed changes

remove double error loggging

0533264

cgillum reviewed Oct 5, 2023

View reviewed changes

src/WebJobs.Extensions.DurableTask/Listener/DurableTaskTargetScaler.cs Outdated Show resolved Hide resolved

Address DI bug

1975b41

cgillum reviewed Oct 5, 2023

View reviewed changes

davidmrdavid and others added 2 commits October 5, 2023 09:37

Update src/WebJobs.Extensions.DurableTask/Listener/DurableTaskTargetS…

4a061d4

…caler.cs Co-authored-by: Chris Gillum <cgillum@microsoft.com>

remove double logging of exception

f5c6d3f

cgillum approved these changes Oct 5, 2023

View reviewed changes

alrod reviewed Oct 5, 2023

View reviewed changes

davidmrdavid added 7 commits October 5, 2023 10:53

add ScaleUtils unit tests

635aa75

expand scaleUtils tests

3ad0f28

InvalidOperationException changed for NotSupportedException

9380a70

change ArithmeticException to InvalidOperationException in DurableTas…

7f47657

…kTargetScaler

patch unit tests

f29c14d

add E2E scaler test

e15344b

removing unused imports

e2595bf

davidmrdavid commented Oct 6, 2023

View reviewed changes

alrod approved these changes Oct 6, 2023

View reviewed changes

pass build errors

c06b641

davidmrdavid merged commit 731d809 into dajusto/tbs Oct 6, 2023

davidmrdavid deleted the dajusto/scv3 branch October 6, 2023 20:40

Conversation

davidmrdavid commented May 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

This PR

Main Changes

Secondary Changes

References:

Issue describing the changes in this PR

Pull request checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davidmrdavid Sep 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davidmrdavid Sep 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alrod Oct 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davidmrdavid Oct 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alrod Oct 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cgillum left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

davidmrdavid commented May 2, 2023 •

edited

Loading

davidmrdavid Sep 22, 2023 •

edited

Loading

davidmrdavid Sep 22, 2023 •

edited

Loading

alrod Oct 5, 2023 •

edited

Loading

davidmrdavid Oct 5, 2023 •

edited

Loading

alrod Oct 5, 2023 •

edited

Loading

alrod Sep 26, 2023 •

edited

Loading

alrod Oct 5, 2023 •

edited

Loading