JsonNode is not thread safe

[gregsdennis]

Description

I have an issue reported where there is a failure in a multithreaded environment. (I have taken many precautions in my code to ensure that it is threadsafe.)

In isolating the issue, I found that JsonNode itself doesn't seem to be thread safe, and so my code can ultimately never be.

Reproduction Steps

[Test]
public void Issue337_ParallelComparisons()
{
	string? failed = null;

	var arrayText = "[\"DENIED\",\"GRANTED\"]";
	var valueText = "\"GRANTED\"";

	var array = JsonNode.Parse(arrayText);
	var value = JsonNode.Parse(valueText);

	try
	{
		Parallel.ForEach(Enumerable.Range(1, 1000000).ToList().AsParallel(), i =>
		{
			var array0 = array[0];
			var array1 = array[1];
			var array0Hash = JsonNodeEqualityComparer.Instance.GetHashCode(array0);
			var array1Hash = JsonNodeEqualityComparer.Instance.GetHashCode(array1);
			var valueHash = JsonNodeEqualityComparer.Instance.GetHashCode(value);

			if (array0Hash != valueHash && array1Hash != valueHash)
			{
				failed ??= $@"Hashcode failed on iteration {i}

value: {valueHash} - {value.AsJsonString()}
array[0]: {array0Hash} - {array0.AsJsonString()}
array[1]: {array1Hash} - {array1.AsJsonString()}";
				return;
			}

			//if (!JsonNodeEqualityComparer.Instance.Equals(array0, value) &&
			//	!JsonNodeEqualityComparer.Instance.Equals(array1, value))
			//{
			//	failed ??= "Equals failed";
			//}

			//if (!array.Contains(value, JsonNodeEqualityComparer.Instance))
			//{
			//	failed ??= "Contains failed";
			//}
		});
	}
	finally
	{
		if (failed != null)
		{
			Console.WriteLine(failed);
			Assert.Fail();
		}
	}
}

The test was distilled from an issue attempting to identify a single JSON value within an array (see linked issue above).

Expected behavior

The test should succeed, proving consistency in hash code generation and int comparisons.

Actual behavior

.Net Core 3.1

The test completes but fails. Usually, I find that the failure occurs in the 20K-30Kth iteration.

It seems that the hash codes are generated properly, but for some reason the int comparison fails. The VS debugger screenshot below clearly shows the values are the same, and yet they're being evaluated as unequal.

I also noticed behavior where the hashcode checks succeed but the equality or contains checks fail. However, I haven't thoroughly checked to make sure my code (JsonNodeEqualityComparer) isn't at fault here. This is why they're commented out.

.Net 5, 6, & 7(rc2)

The test fails with InvalidOperationException when attempting to access array[0] stating "Nullable object must have a value." This generally occurs on subsequent iterations, not the first one, and usually on multiple threads.

Stack trace

System.AggregateException : One or more errors occurred. (Nullable object must have a value.) (Nullable object must have a value.) (Nullable object must have a value.) (Nullable object must have a value.)
  ----> System.InvalidOperationException : Nullable object must have a value.
  ----> System.InvalidOperationException : Nullable object must have a value.
  ----> System.InvalidOperationException : Nullable object must have a value.
  ----> System.InvalidOperationException : Nullable object must have a value.
   at System.Threading.Tasks.TaskReplicator.Run[TState](ReplicatableUserAction`1 action, ParallelOptions options, Boolean stopOnFirstFailure)
   at System.Threading.Tasks.Parallel.PartitionerForEachWorker[TSource,TLocal](Partitioner`1 source, ParallelOptions parallelOptions, Action`1 simpleBody, Action`2 bodyWithState, Action`3 bodyWithStateAndIndex, Func`4 bodyWithStateAndLocal, Func`5 bodyWithEverything, Func`1 localInit, Action`1 localFinally)
--- End of stack trace from previous location ---
   at System.Threading.Tasks.Parallel.ThrowSingleCancellationExceptionOrOtherException(ICollection exceptions, CancellationToken cancelToken, Exception otherException)
   at System.Threading.Tasks.Parallel.PartitionerForEachWorker[TSource,TLocal](Partitioner`1 source, ParallelOptions parallelOptions, Action`1 simpleBody, Action`2 bodyWithState, Action`3 bodyWithStateAndIndex, Func`4 bodyWithStateAndLocal, Func`5 bodyWithEverything, Func`1 localInit, Action`1 localFinally)
   at System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal](IEnumerable`1 source, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Action`3 bodyWithStateAndIndex, Func`4 bodyWithStateAndLocal, Func`5 bodyWithEverything, Func`1 localInit, Action`1 localFinally)
   at System.Threading.Tasks.Parallel.ForEach[TSource](IEnumerable`1 source, Action`1 body)
   at Json.More.Tests.GithubTests.Issue337_ParallelComparisons() in C:\projects\json-everything\Json.More.Tests\GithubTests.cs:line 27
--InvalidOperationException
   at System.Nullable`1.get_Value()
   at System.Text.Json.Nodes.JsonArray.CreateNodes()
   at System.Text.Json.Nodes.JsonNode.get_Item(Int32 index)
   at Json.More.Tests.GithubTests.<>c__DisplayClass0_0.<Issue337_ParallelComparisons>b__0(Int32 i) in C:\projects\json-everything\Json.More.Tests\GithubTests.cs:line 29
   at System.Threading.Tasks.Parallel.<>c__DisplayClass44_0`2.<PartitionerForEachWorker>b__1(IEnumerator& partitionState, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion)
--- End of stack trace from previous location ---
   at System.Threading.Tasks.Parallel.<>c__DisplayClass44_0`2.<PartitionerForEachWorker>b__1(IEnumerator& partitionState, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion)
   at System.Threading.Tasks.TaskReplicator.Replica`1.ExecuteAction(Boolean& yieldedBeforeCompletion)
   at System.Threading.Tasks.TaskReplicator.Replica.Execute()
--InvalidOperationException
   at System.Nullable`1.get_Value()
   at System.Text.Json.Nodes.JsonArray.CreateNodes()
   at System.Text.Json.Nodes.JsonNode.get_Item(Int32 index)
   at Json.More.Tests.GithubTests.<>c__DisplayClass0_0.<Issue337_ParallelComparisons>b__0(Int32 i) in C:\projects\json-everything\Json.More.Tests\GithubTests.cs:line 29
   at System.Threading.Tasks.Parallel.<>c__DisplayClass44_0`2.<PartitionerForEachWorker>b__1(IEnumerator& partitionState, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion)
--- End of stack trace from previous location ---
   at System.Threading.Tasks.Parallel.<>c__DisplayClass44_0`2.<PartitionerForEachWorker>b__1(IEnumerator& partitionState, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion)
   at System.Threading.Tasks.TaskReplicator.Replica`1.ExecuteAction(Boolean& yieldedBeforeCompletion)
   at System.Threading.Tasks.TaskReplicator.Replica.Execute()
--InvalidOperationException
   at System.Nullable`1.get_Value()
   at System.Text.Json.Nodes.JsonArray.CreateNodes()
   at System.Text.Json.Nodes.JsonNode.get_Item(Int32 index)
   at Json.More.Tests.GithubTests.<>c__DisplayClass0_0.<Issue337_ParallelComparisons>b__0(Int32 i) in C:\projects\json-everything\Json.More.Tests\GithubTests.cs:line 29
   at System.Threading.Tasks.Parallel.<>c__DisplayClass44_0`2.<PartitionerForEachWorker>b__1(IEnumerator& partitionState, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion)
--- End of stack trace from previous location ---
   at System.Threading.Tasks.Parallel.<>c__DisplayClass44_0`2.<PartitionerForEachWorker>b__1(IEnumerator& partitionState, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion)
   at System.Threading.Tasks.TaskReplicator.Replica`1.ExecuteAction(Boolean& yieldedBeforeCompletion)
   at System.Threading.Tasks.TaskReplicator.Replica.Execute()
--InvalidOperationException
   at System.Nullable`1.get_Value()
   at System.Text.Json.Nodes.JsonArray.CreateNodes()
   at System.Text.Json.Nodes.JsonNode.get_Item(Int32 index)
   at Json.More.Tests.GithubTests.<>c__DisplayClass0_0.<Issue337_ParallelComparisons>b__0(Int32 i) in C:\projects\json-everything\Json.More.Tests\GithubTests.cs:line 29
   at System.Threading.Tasks.Parallel.<>c__DisplayClass44_0`2.<PartitionerForEachWorker>b__1(IEnumerator& partitionState, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion)
--- End of stack trace from previous location ---
   at System.Threading.Tasks.Parallel.<>c__DisplayClass44_0`2.<PartitionerForEachWorker>b__1(IEnumerator& partitionState, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion)
   at System.Threading.Tasks.TaskReplicator.Replica`1.ExecuteAction(Boolean& yieldedBeforeCompletion)
   at System.Threading.Tasks.TaskReplicator.Replica.Execute()

Regression?

I can't tell if this ever worked in any runtime, but given that it's not working as expected in .Net Core 3.1, I'd say it's always been an issue. I don't know what the issue may be in later runtimes. The exception is at least indicative that something went wrong, although it's not clear what that was.

Known Workarounds

No response

Configuration

This is all run on Windows 11, x64. I doubt it's related to the configuration, though.

Other information

No response

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[gregsdennis]

Area is System.Text.Json

Tagging subscribers to this area: @dotnet/area-system-text-json, @gregsdennis
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

I have an issue reported where there is a failure in a multithreaded environment. (I have taken many precautions in my code to ensure that it is threadsafe.)

In isolating the issue, I found that JsonNode itself doesn't seem to be thread safe, and so my code can ultimately never be.

Reproduction Steps

[Test]
public void Issue337_ParallelComparisons()
{
	string? failed = null;

	var arrayText = "[\"DENIED\",\"GRANTED\"]";
	var valueText = "\"GRANTED\"";

	var array = JsonNode.Parse(arrayText);
	var value = JsonNode.Parse(valueText);

	try
	{
		Parallel.ForEach(Enumerable.Range(1, 1000000).ToList().AsParallel(), i =>
		{
			var array0 = array[0];
			var array1 = array[1];
			var array0Hash = JsonNodeEqualityComparer.Instance.GetHashCode(array0);
			var array1Hash = JsonNodeEqualityComparer.Instance.GetHashCode(array1);
			var valueHash = JsonNodeEqualityComparer.Instance.GetHashCode(value);

			if (array0Hash != valueHash && array1Hash != valueHash)
			{
				failed ??= $@"Hashcode failed on iteration {i}

value: {valueHash} - {value.AsJsonString()}
array[0]: {array0Hash} - {array0.AsJsonString()}
array[1]: {array1Hash} - {array1.AsJsonString()}";
				return;
			}

			//if (!JsonNodeEqualityComparer.Instance.Equals(array0, value) &&
			//	!JsonNodeEqualityComparer.Instance.Equals(array1, value))
			//{
			//	failed ??= "Equals failed";
			//}

			//if (!array.Contains(value, JsonNodeEqualityComparer.Instance))
			//{
			//	failed ??= "Contains failed";
			//}
		});
	}
	finally
	{
		if (failed != null)
		{
			Console.WriteLine(failed);
			Assert.Fail();
		}
	}
}

The test was distilled from an issue attempting to identify a single JSON value within an array (see linked issue above).

Expected behavior

The test should succeed, proving consistency in hash code generation and int comparisons.

Actual behavior

.Net Core 3.1

The test completes but fails. Usually, I find that the failure occurs in the 20K-30Kth iteration.

It seems that the hash codes are generated properly, but for some reason the int comparison fails. The VS debugger screenshot below clearly shows the values are the same, and yet they're being evaluated as unequal.

I also noticed behavior where the hashcode checks succeed but the equality or contains checks fail. However, I haven't thoroughly checked to make sure my code (JsonNodeEqualityComparer) isn't at fault here. This is why they're commented out.

.Net 5, 6, & 7(rc2)

The test fails with InvalidOperationException when attempting to access array[0] stating "Nullable object must have a value." This generally occurs on subsequent iterations, not the first one, and usually on multiple threads.

Stack trace

System.AggregateException : One or more errors occurred. (Nullable object must have a value.) (Nullable object must have a value.) (Nullable object must have a value.) (Nullable object must have a value.)
  ----> System.InvalidOperationException : Nullable object must have a value.
  ----> System.InvalidOperationException : Nullable object must have a value.
  ----> System.InvalidOperationException : Nullable object must have a value.
  ----> System.InvalidOperationException : Nullable object must have a value.
   at System.Threading.Tasks.TaskReplicator.Run[TState](ReplicatableUserAction`1 action, ParallelOptions options, Boolean stopOnFirstFailure)
   at System.Threading.Tasks.Parallel.PartitionerForEachWorker[TSource,TLocal](Partitioner`1 source, ParallelOptions parallelOptions, Action`1 simpleBody, Action`2 bodyWithState, Action`3 bodyWithStateAndIndex, Func`4 bodyWithStateAndLocal, Func`5 bodyWithEverything, Func`1 localInit, Action`1 localFinally)
--- End of stack trace from previous location ---
   at System.Threading.Tasks.Parallel.ThrowSingleCancellationExceptionOrOtherException(ICollection exceptions, CancellationToken cancelToken, Exception otherException)
   at System.Threading.Tasks.Parallel.PartitionerForEachWorker[TSource,TLocal](Partitioner`1 source, ParallelOptions parallelOptions, Action`1 simpleBody, Action`2 bodyWithState, Action`3 bodyWithStateAndIndex, Func`4 bodyWithStateAndLocal, Func`5 bodyWithEverything, Func`1 localInit, Action`1 localFinally)
   at System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal](IEnumerable`1 source, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Action`3 bodyWithStateAndIndex, Func`4 bodyWithStateAndLocal, Func`5 bodyWithEverything, Func`1 localInit, Action`1 localFinally)
   at System.Threading.Tasks.Parallel.ForEach[TSource](IEnumerable`1 source, Action`1 body)
   at Json.More.Tests.GithubTests.Issue337_ParallelComparisons() in C:\projects\json-everything\Json.More.Tests\GithubTests.cs:line 27
--InvalidOperationException
   at System.Nullable`1.get_Value()
   at System.Text.Json.Nodes.JsonArray.CreateNodes()
   at System.Text.Json.Nodes.JsonNode.get_Item(Int32 index)
   at Json.More.Tests.GithubTests.<>c__DisplayClass0_0.<Issue337_ParallelComparisons>b__0(Int32 i) in C:\projects\json-everything\Json.More.Tests\GithubTests.cs:line 29
   at System.Threading.Tasks.Parallel.<>c__DisplayClass44_0`2.<PartitionerForEachWorker>b__1(IEnumerator& partitionState, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion)
--- End of stack trace from previous location ---
   at System.Threading.Tasks.Parallel.<>c__DisplayClass44_0`2.<PartitionerForEachWorker>b__1(IEnumerator& partitionState, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion)
   at System.Threading.Tasks.TaskReplicator.Replica`1.ExecuteAction(Boolean& yieldedBeforeCompletion)
   at System.Threading.Tasks.TaskReplicator.Replica.Execute()
--InvalidOperationException
   at System.Nullable`1.get_Value()
   at System.Text.Json.Nodes.JsonArray.CreateNodes()
   at System.Text.Json.Nodes.JsonNode.get_Item(Int32 index)
   at Json.More.Tests.GithubTests.<>c__DisplayClass0_0.<Issue337_ParallelComparisons>b__0(Int32 i) in C:\projects\json-everything\Json.More.Tests\GithubTests.cs:line 29
   at System.Threading.Tasks.Parallel.<>c__DisplayClass44_0`2.<PartitionerForEachWorker>b__1(IEnumerator& partitionState, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion)
--- End of stack trace from previous location ---
   at System.Threading.Tasks.Parallel.<>c__DisplayClass44_0`2.<PartitionerForEachWorker>b__1(IEnumerator& partitionState, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion)
   at System.Threading.Tasks.TaskReplicator.Replica`1.ExecuteAction(Boolean& yieldedBeforeCompletion)
   at System.Threading.Tasks.TaskReplicator.Replica.Execute()
--InvalidOperationException
   at System.Nullable`1.get_Value()
   at System.Text.Json.Nodes.JsonArray.CreateNodes()
   at System.Text.Json.Nodes.JsonNode.get_Item(Int32 index)
   at Json.More.Tests.GithubTests.<>c__DisplayClass0_0.<Issue337_ParallelComparisons>b__0(Int32 i) in C:\projects\json-everything\Json.More.Tests\GithubTests.cs:line 29
   at System.Threading.Tasks.Parallel.<>c__DisplayClass44_0`2.<PartitionerForEachWorker>b__1(IEnumerator& partitionState, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion)
--- End of stack trace from previous location ---
   at System.Threading.Tasks.Parallel.<>c__DisplayClass44_0`2.<PartitionerForEachWorker>b__1(IEnumerator& partitionState, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion)
   at System.Threading.Tasks.TaskReplicator.Replica`1.ExecuteAction(Boolean& yieldedBeforeCompletion)
   at System.Threading.Tasks.TaskReplicator.Replica.Execute()
--InvalidOperationException
   at System.Nullable`1.get_Value()
   at System.Text.Json.Nodes.JsonArray.CreateNodes()
   at System.Text.Json.Nodes.JsonNode.get_Item(Int32 index)
   at Json.More.Tests.GithubTests.<>c__DisplayClass0_0.<Issue337_ParallelComparisons>b__0(Int32 i) in C:\projects\json-everything\Json.More.Tests\GithubTests.cs:line 29
   at System.Threading.Tasks.Parallel.<>c__DisplayClass44_0`2.<PartitionerForEachWorker>b__1(IEnumerator& partitionState, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion)
--- End of stack trace from previous location ---
   at System.Threading.Tasks.Parallel.<>c__DisplayClass44_0`2.<PartitionerForEachWorker>b__1(IEnumerator& partitionState, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion)
   at System.Threading.Tasks.TaskReplicator.Replica`1.ExecuteAction(Boolean& yieldedBeforeCompletion)
   at System.Threading.Tasks.TaskReplicator.Replica.Execute()

Regression?

I can't tell if this ever worked in any runtime, but given that it's not working as expected in .Net Core 3.1, I'd say it's always been an issue. I don't know what the issue may be in later runtimes. The exception is at least indicative that something went wrong, although it's not clear what that was.

Known Workarounds

No response

Configuration

This is all run on Windows 11, x64. I doubt it's related to the configuration, though.

Other information

No response

Author:	gregsdennis
Assignees:	-
Labels:	area-System.Text.Json, untriaged
Milestone:	-

[eiriktsarpalis]

It would help if you could share the full implementation of the equality comparer, although my guess would be that you have hit #76440. TL;DR JsonNode.Parse wraps JsonDocument which has had issues when calling JsonElement.GetString() concurrently. Assuming that your equality comparer does do that either directly or indirectly, then it is conceivable that it's a dupe of that issue.

Could you try your repro with your latest nightly build of the nuget package to verify?

This issue has been marked needs-author-action and may be missing some important information.

[stephentoub]

@eiriktsarpalis, I don't believe it's the same issue, or at least not solely. The stack trace shows the error stemming from JsonArray.CreateNodes; JsonArray has a private JsonElement? _jsonElement field, which the implementation uses in a non-thread-safe manner.

If we actually intend for the whole JsonDocument/Element/Node/etc. type set to be thread-safe, someone needs to audit all of the code... it appears to be doing a lot of lazy initialization without much mind paid to thread-safety. It doesn't actually look like it was developed with the intent of being thread-safe.

[eiriktsarpalis]

It doesn't actually look like it was developed with the intent of being thread-safe.

Agreed that this is not the case. JsonNode is meant to provide a mutable DOM implementation and we don't explicitly provide any thread-safety guarantees. It might be that we could improve thread-safety when it comes to lazy initializations as long as no modifications are being made on the instance.

[vcsjones]

JsonNode is meant to provide a mutable DOM implementation and we don't explicitly provide any thread-safety guarantees

JsonArray and JsonObject, types that derive from JsonNode, are documented as thread-safe when reading.

[gregsdennis]

JsonNode.Parse wraps JsonDocument which has had issues when calling JsonElement.GetString() concurrently. Assuming that your equality comparer does do that either directly or indirectly, then it is conceivable that it's a dupe of that issue.

Yes, it does call that, but only if the values are strings (which is true for this test). I can try the test with numbers to see if the problem persists. (EDIT The test passes for numbers instead of strings.)

---

The code that it's calling is here and here

The important bits are toward the bottom of each method where it's comparing string values. The rest (i.e. the code for objects and arrays) isn't invoked because the test is only ever getting the hash code for nodes containing strings.

---

I don't know if it's a quirk of the debugger, but I'm also very concerned about the screenshot I posted of the immediate window that shows a comparison between two obviously equal integers resulting as not equal. As you can see from the test, I'm storing the calculated hash codes into vars, then comparing the vars. The hash code calculation shouldn't factor into the test, but I don't know how this is lowered by the compiler, so I can't say for sure.