System.Text.Json Reference Loop Handling

[MoienTajik]

One of the key features of JSON.NET serialization was the ReferenceLoopHandling which gives the ability to ignore the reference loops likes this :

public class Employee
{
    public string Name { get; set; }

    public Employee Manager { get; set; }
}

private static void Main()
{
    var joe = new Employee { Name = "Joe - User" };
    var mike = new Employee { Name = "Mike - Manager" };
    joe.Manager = mike;
    mike.Manager = mike;

    var json = JsonConvert.SerializeObject(joe, new JsonSerializerSettings
    {
        Formatting = Formatting.Indented,
        ReferenceLoopHandling = ReferenceLoopHandling.Ignore
    });
	
    Console.WriteLine(json);
}

And it produces the appropriate result :

{
  "Name": "Joe User",
  "Manager": {
    "Name": "Mike Manager"
  }
}

However, I couldn't find such a feature in System.Text.JSON, And when I've tried to Serialize the same object with JsonSerializer :

private static void Main()
{
    var joe = new Employee { Name = "Joe User" };
    var mike = new Employee { Name = "Mike Manager" };
    joe.Manager = mike;
    mike.Manager = mike;

    var json = JsonSerializer.ToString(joe, new JsonSerializerOptions
    {
        WriterOptions = new JsonWriterOptions
        {
            Indented = true,
        }
    });

    Console.WriteLine(json);
}

I've got this exception :
System.InvalidOperationException: 'CurrentDepth (1000) is equal to or larger than the maximum allowed depth of 1000. Cannot write the next JSON object or array.'

So, Is this feature exist in System.Text.Json now that I couldn't find it?
And if not, Are there any plans to support this?

[xsoheilalizadeh]

This could make a lot of bugs, I have these self-reference models in my APIs and since now I never had Reference Loop issue with Json.NET. If we decided to replace the Json.NET with the new System.Text.Json we should consider this feature as soon as it is possible.

[ahsonkhan]

So, Is this feature exist in System.Text.Json now that I couldn't find it?
And if not, Are there any plans to support this?

This is a known limitation of System.Text.Json (we do not have this feature yet) and we have little runway left to design and implement this for 3.0. We plan to add this for vNext, for sure (there are quite a few features/capabilities in Json.NET that wouldn't work in System.Text.Json and given the constraints, we had to prioritize what made it in).

This could make a lot of bugs, I have these self-reference models in my APIs and since now I never had Reference Loop issue with Json.NET

Given we detect and throw for self-reference models, how could it cause bugs?

[ahsonkhan]

cc @steveharter

[AmrAlSayed0]

Would this API be good?
This new property should be added to the JsonWriterOptions struct.

public struct JsonWriterOptions
{
    public ReferenceLoopHandlingOption ReferenceLoopHandling { get; set; }
}

public struct ReferenceLoopHandlingOption
{
    //A static member for easy access to pre-defined options.
   // This causes it to ignore all looped refrences.
    public static ReferenceLoopHandlingIgnore = new ReferenceLoopHandlingOption ( ReferenceLoopHandling.Ignore );
   //This causes the writer to output the looped reference only once after the first time.
   // {
   //   "Name": "Joe User",
   //   "Manager": {
   //     "Name": "Mike Manager",
   //     "Manager": {
   //       "Name":"Mike Manager"
   //     }
   //   }
   // }
    public static ReferenceLoopHandlingOnce = new ReferenceLoopHandlingOption ( ReferenceLoopHandling.Once );
    // Current (and default?) behavior.
    public static ReferenceLoopHandlingAll = new ReferenceLoopHandlingOption ( ReferenceLoopHandling.All );
    public ReferenceLoopHandlingOption ( ReferenceLoopHandling loophandling );
    // Also there is the option to specify how many times do you want the writer to write looped refrences (after the first one)
    // new ReferenceLoopHandlingOption ( ReferenceLoopHandling.Many , 3 );
    public ReferenceLoopHandlingOption ( ReferenceLoopHandling loophandling , int loopOutputTimes );
}

public enum ReferenceLoopHandling
{
    Ignore ,
    Once ,
    Many ,
    All
}

[ahsonkhan]

From @josundt (https://github.com/dotnet/corefx/issues/40045#issue-477089090):

Object/collection graphs containing circular references (or multiple references to the same entity), can be handled in a few different ways when serializing/deserializing to/from JSON.

Best practice is naturally to avoid graphs with circular references when using JSON serialization, but techniques to deal with it exist, and some times it may be needed. We use it a few places in our software.

NewtonSoft.Json supports this, and I guess it is a goal for System.Text.Json to make the feature gap with NewtonSoft as small possible.

During serialization, when an object/collection that has already been serialized is referenced for the second time or more, the common technique I've seen is to replace the repeated objects/collection in the JSON with a

{ "$ref": "some-sort-of-pointer" }

Some use JSONPath to reference the object/array serialized previously in the JSON document.

NewtonSoft.Json uses a different techique with specific serialization settings: It adds an extra "$id" property to all serialized objects in the JSON document, and the "$ref" value is the "$id" property of the object/array serialized previously.

Example using NewtonSoft.Json:

using System;
using System.Collections.Generic;
using Newtonsoft.Json;

namespace CircularSerialization
{
    public class Bike
    {
        public List<Tire> Tires { get; set; } = new List<Tire>();
    }

    public class Tire
    {
        public Bike Bike { get; set; }
    }

    static class Program
    {
        static void Main()
        {
            var bike = new Bike();
            bike.Tires.Add(new Tire { Bike = bike });
            bike.Tires.Add(new Tire { Bike = bike });

            var settings = new JsonSerializerSettings
            {
                PreserveReferencesHandling = PreserveReferencesHandling.Objects,
                ReferenceLoopHandling = ReferenceLoopHandling.Ignore
            };
            var serialized = JsonConvert.SerializeObject(bike, settings);

            Console.Write(serialized);
            // {"$id":"1","Tires":[{"$id":"2","Bike":{"$ref":"1"}},{"$id":"3","Bike":{"$ref":"1"}}]}
        }
    }
}

I guess that if System.Text.Json.JsonSerializer will support circular/multiple references, some more memory allocation will be required during serialization/deserialization when the setting is switched on (so such a setting should naturally be off by default).

Is there a plan to support options for handling circular references in JsonSerializerSettings?
And if so, will you support the "$id" technique or the JSONPath techique, or maybe support both?

Reference:
https://github.com/douglascrockford/JSON-js/blob/master/cycle.js

[euclid47]

Will the enhancement be released in 3.1?

[davidnmbond]

Really need a fix for this one. It's not currently possible to serialize EF Core queries with populated navigation properties.

I'd suggest an approach similar to NewtonSoft's ReferenceLoopHandling:

https://www.newtonsoft.com/json/help/html/SerializationSettings.htm#ReferenceLoopHandling

[MagicAndre1981]

Will the enhancement be released in 3.1?

Milestone is set to 5, so no it will be not part of 3.1 and so this is useless at all.

[julienGrd]

Really need a fix for this one. It's not currently possible to serialize EF Core queries with populated navigation properties.

I'd suggest an approach similar to NewtonSoft's ReferenceLoopHandling:

https://www.newtonsoft.com/json/help/html/SerializationSettings.htm#ReferenceLoopHandling

You can for the moment switch to newtonsoft, im facing the same problem and i use this class extensions (adapt the settings to your needs)

public static class HttpClientExtension
    {
        public static async Task<T> PostJsonCustomAsync<T>(this HttpClient sender, string requestUrl, object postData)
        {
            sender.DefaultRequestHeaders.Add("Accept", "application/json");

            string stringPostData = JsonConvert.SerializeObject(postData, Settings);

            HttpContent body = new StringContent(stringPostData, Encoding.UTF8, "application/json");
            var response = await sender.PostAsync(requestUrl, body);

            string text = await response.Content.ReadAsStringAsync();

            T data = JsonConvert.DeserializeObject<T>(text, Settings);

            return data;
        }


        public static async Task<T> GetJsonCustomAsync<T>(this HttpClient sender, string requestUrl)
        {
            sender.DefaultRequestHeaders.Add("Accept", "application/json");

            var response = await sender.GetAsync(requestUrl);

            string text = await response.Content.ReadAsStringAsync();

            T data = JsonConvert.DeserializeObject<T>(text, Settings);

            return data;
        }

        public static async Task<T> PutJsonCustomAsync<T>(this HttpClient sender, string requestUrl, object putData)
            where T : new()
        {
            sender.DefaultRequestHeaders.Add("Accept", "application/json");

            string stringPutData = JsonConvert.SerializeObject(putData, Settings);

            HttpContent body = new StringContent(stringPutData, Encoding.UTF8, "application/json");
            var response = await sender.PutAsync(requestUrl, body);

            string text = await response.Content.ReadAsStringAsync();

            T data = JsonConvert.DeserializeObject<T>(text, Settings);

            return data;
        }

        private static JsonSerializerSettings Settings = new JsonSerializerSettings()
        {
            DateParseHandling = DateParseHandling.None,
            DateTimeZoneHandling = DateTimeZoneHandling.Unspecified
        };
    }

[ahsonkhan]

Folks on this thread who are requesting this feature, do you need it for deserialization as well as serialization? Most of the examples here are about serialization.

Also, how heavily (if at all) do you rely on Newtonsoft.Json's MetadataPropertyHandling.ReadAhead feature (to enable out-of-order metadata property support):
https://www.newtonsoft.com/json/help/html/DeserializeMetadataPropertyHandling.htm

I'd be interested in seeing current usages.

Also cc @ajcvickers, @Andrzej-W

[Andrzej-W]

@ahsonkhan, people serialize objects to deserialize them later - we need support for loops in both of them. Now when we have Blazor Wasm (officially it will be released in May 2020) this is a typical scenario:

• we have ASP.NET Web API service which is used to read some data from database and serialize it,
• we read this response in Blazor and have to deserialize it.

Databases are usually designed in such a way that we can read child items when we have parent item and we can read parent item when we have one of its children. As a direct consequence we have bidirectional links between POCO classes used in Entity Framework. Simple example: in InvoiceHeader class we have a property with list of InvoiceLines and in every InvoiceLine we have a reference to InvoiceHeader. This is similar to Bike and Tire example in one of the posts above.

In my opinion this is such a basic requirement that it have to be implemented as soon as possible. Without support for reference loops it will be very hard to write any real world Blazor application or any other application which have to serialize and deserialize object graphs used in Entity Framework.

[ajcvickers]

@ahsonkhan Handling of cycles is required for serializing the results from EF (or any other OR/M) for the reasons stated by @Andrzej-W. However, I don't know which specific flags from Newtonsoft.Json map to this.

[BreyerW]

Handling loops on both serialize and deserialize is required to be useful on many complex scenarios (EF was good example, i also use it privately). Best if you also provide something akin to ReferenceResolver like json.net do, because not everyone needs full ref loops and sometimes they want to restrict it to types that have guid as an example.

[jozkee]

people serialize objects to deserialize them later - we need support for loops in both of them

I agree, having a way to support loops and the ability to round-trip them is a must; to address that, System.Text.Json should implement something similar to Json.Net's PreserveReferencesHandling.All and MetadataPropertyHandling.Default.

With that said, is there any value left in including a feature similar to ReferenceLoopHandling.Ignore?

[Darlingtone]

I don't think System.Text.Json should have be released nor made the default handler for ASP.Net without any form of Cycles (Reference Loop) handling, given how deeply dependent ASP.Net or any modern .Net application for that matter depends on EF.

Using EF for any real world application you are most likely (99.99% likely) to require Reference Loop Handling with serialization/deserialization, so am wondering how useful System.Text.Json is really is at this stage.

Secondly, for a truly drop in place replacement for Json.Net I would have expected that the default JsonSerializationOptions match as closely as possible to Json.Net defaults