Skip to content

fixing race in Binder that can cause NullReferenceExceptions#3167

Merged
brettsam merged 5 commits into
devfrom
brettsam/binder
Nov 20, 2025
Merged

fixing race in Binder that can cause NullReferenceExceptions#3167
brettsam merged 5 commits into
devfrom
brettsam/binder

Conversation

@brettsam

@brettsam brettsam commented Nov 19, 2025

Copy link
Copy Markdown
Member

For a long time, we've seen a decent number of errors coming from this code with the stack:

System.InvalidOperationException : Error while handling parameter _binder after function returned: ---> System.NullReferenceException : Object reference not set to an instance of an object.
   at async Microsoft.Azure.WebJobs.Binder.Complete(CancellationToken cancellationToken) at D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Bindings\Runtime\Binder.cs : 156
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Bindings.Runtime.RuntimeValueProvider.SetValueAsync(Object value,CancellationToken cancellationToken) at D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Bindings\Runtime\RuntimeValueProvider.cs : 34
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ParameterHelper.ProcessOutputParameters(CancellationToken cancellationToken) at D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 959

It's unclear how this is even possible given that the code here can seemingly never be null. But we're going to do some null-checks here to see if we can prevent these exceptions while we continue to investigate.

Edit:
We figured out the underlying issue. In Functions if you have multiple input bindings we call BindAsync() all in parallel. This adds them to the underlying _binders list.

But list.Add(item) isn't thread-safe -- it can cause internal resizing, etc, which means that internally things can be in a bad state. Then later enumerating it can see null items when there should be none. If you use one of these null items, you get a NullRef.

I was confused why the typical "add while enumerating" exceptions weren't being thrown, and it seemed like we would never even be in this state. But now it all makes sense with how Functions calls BindAsync() in parallel.

So the issue isn't "adding in one thread while enumerating in another" but it's "adding from multiple threads and then later enumerating". That explains why it never throws "collection was modified".

Comment thread src/Microsoft.Azure.WebJobs.Host/Bindings/Runtime/Binder.cs Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants