Do not signal cancellation when in drain mode

[liliankasem]

The Drain Mode feature in Azure Functions is intended to allow ongoing invocations to finish executing during a graceful shut down, therefore when drain mode is enabled for any given function, we should not be signaling cancellation.

A request for all of the WebJobs extensions to address this went out a couple years ago, but this extension was flagged as one that does not currently use the DrainModeManager and may require changes.

Tasks

• Investigate the Kafka WebJobs extension and determine if ongoing invocations are cancelled when drain mode is requested
• If an ongoing invocation is being cancelled, implement changes to ensure that when drain mode is enabled, the invocation continues executing and the CT is not signaled.

How to test the behaviour

• Create a new .NET (isolated) Function
• Write a function that is a) using the KafkaTrigger binding and, b) handling the cancellation token
• Start the function and trigger an invocation
• Request drain mode via /admin/host/drain

Expected behaviour:

• The function invocation should continue executing as normal

Example function:

This is an example of a function I used to test the TimerTrigger. If drain mode is enabled, we should not be hitting the catch-block:

[Function("MyTimer")]
public async Task Run([TimerTrigger("*/30 * * * * *")] TimerInfo myTimer, CancellationToken cancellationToken)
{
    _logger.LogWarning($"Timer trigger function executed at: {DateTime.Now}");

    if (myTimer.ScheduleStatus is not null)
    {
        _logger.LogWarning($"Next timer schedule at: {myTimer.ScheduleStatus.Next}");
    }

    try
    {
        cancellationToken.ThrowIfCancellationRequested();
        await Task.Delay(5000, cancellationToken);
        _logger.LogWarning($"Timer trigger completed at: {DateTime.Now}");
    }
    catch (OperationCanceledException)
    {
        _logger.LogWarning("Timer trigger function was cancelled.");
    }
}

Examples of this issue being addressed in other extensions

Do not pass cancelled token to ExecuteAsync when draining

• EventHub: Event Hubs - Don't pass cancelled token to TryExecute when draining azure-sdk-for-net#38067
• ServiceBus: Don't pass cancelled token to TryExecute when draining azure-sdk-for-net#38063
• Storage: [Storage] [WebJobs] Do not pass cancelled token to ExecuteAsync when draining  azure-sdk-for-net#40792
• Timer: Update TimerListener to not signal cancellation when in drain mode azure-webjobs-sdk-extensions#917

[nzthiago]

@raorugan can you please help proiritize this one?