ServiceBusConnection has already been closed. Please create a new instance

[ThomasArdal]

I have recently added Microsoft.Extensions.Logging to my Azure Functions. This reveals a range of different errors logged from the runtime as well as bindings. I'm creating GitHub issues when I see a new error, to try and understand if each error is something I should take seriously or not. I generally think that Azure Functions log way too many errors internally. The general answer here on GitHub is that "you can ignore that error". Logging errors that can simply be ignored is a bad practice IMO since important errors disappear in a large amount of logged errors.

For today's error, I'm seeing this:

System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'ServiceBusConnection has already been closed. Please create a new instance'.
   at Microsoft.Azure.ServiceBus.ServiceBusConnection.ThrowIfClosed()
   at Microsoft.Azure.ServiceBus.ClientEntity.ThrowIfClosed()
   at Microsoft.Azure.ServiceBus.Core.MessageReceiver.AbandonAsync(String lockToken, IDictionary`2 propertiesToModify)
   at Microsoft.Azure.ServiceBus.MessageReceivePump.AbandonMessageIfNeededAsync(Message message)

For this function, I have a ServiceBus input binding like this:

public async Task Run([ServiceBusTrigger("%topicName%", "%subscriptionName%", Connection = "connection")]string mySbMsg)

The mySbMsg variable is parsed like this:

JsonConvert.DeserializeObject<UptimeResultMessage>(mySbMsg, new JsonSerializerSettings())

No further requests are made to Azure ServiceBus in the function.

This error seems to be something happening internally in either the runtime or the binding. Can this error be ignored? Should it even be logged?

Sorry for the semi-rant here, but when we are drowning in errors from the runtime, getting a good overview of the log is hard.

I'm running on .NET Core 3.1 on Azure Functions v3 and have the following NuGet packages installed:

<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.ServiceBus" Version="4.1.1" />
<PackageReference Include="Microsoft.NET.Sdk.Functions" Version="3.0.7" />
<PackageReference Include="Microsoft.Azure.Functions.Extensions" Version="1.0.0" />

[VamseeInala]

Even I am seeing this error sometimes on calling the "CompleteAsync" on messageReceiver object.

[paulbatum]

@sidkri I have a feeling that this error is due to:
#95

Does that seem likely?

[sidkri]

@paulbatum, the #95 is needed to drain messages during graceful shutdown via "IHost.StopAsync()". It would be related if the errors reported here are happening while the function host is shutting down.
@ThomasArdal / @VamseeInala - could you please check the logs and let us know if this error happens just before the function host stops?

[ThomasArdal]

I'm afraid I put an ignore rule on this error a long time ago. Happens over and over again and taking focus from real errors.

[thelgevold]

@sidkri @ThomasArdal @paulbatum I am seeing this error as well on a daily basis.

Based on your comments above I have gone through my logs to correlate the Cannot access a disposed object. Object name: 'ServiceBusConnection has already been closed error to a prior Stopping JobHost event.

In my case these events seem to always occur in tandem, following a pattern where Stopping JobHost is called < 1-2 seconds prior to seeing the ServicebusConnection error. In one case the difference in time was 11 seconds, but in most cases I see the ServiceBus error < 1 second after the JobHost stop event.

If this is the expected behaviour, would it be possible to mute this error since it seems to happen quite frequently?

[sidkri]

@thelgevold, thanks for checking your logs and confirming the behavior. It appears to be a timing/sequence issue during shutdown of the Functions host. I'll investigate to understand what our options are to avoid this or suppress and report back.

[chrisseroka]

I can observe the behavior while running app in Service Fabric

[sidkri]

This issue will be resolved when #95 is implemented.

[sidkri]

@ThomasArdal, @VamseeInala, @thelgevold the fix has been rolled out via 4.2.0 of the Service Bus extension:
https://www.nuget.org/packages/Microsoft.Azure.WebJobs.Extensions.ServiceBus/4.2.0

We are initially requiring users to opt-in to the new behavior so please contact us via support to provide your function app names as we need to enable them in the system to activate the new behavior. It will only apply to situations where the function app is scaled-in. If you restart/stop the app or the worker is upgraded for maintenance, the issue can still occur though the chances are reduced with the new code. We are exploring options to expand support for draining messages in more scenarios.