Deserialize multiple objects from stream

[schauerte]

Hi there,
in my scenario, I want to retrieve objects from a streamed server response (over HTTP). Since this can contain many objects and take some time, I want to start deserializing and working with the objects as they arrive. This works with the SupportMultipleContent property of the Newtonsoft.Json.JsonTextReader class, but deserializing seems somewhat slow - so we decided to give Jil a try.
Unfortumately, I was not able to find a similar setting in Jil. Trying to just deserialize the first object by providing the stream leads to a Jil.DeserializationException "Expected end of stream" (as expected).

I just grabed the source and tried to comment out the Check ExpectEndOfStream in InlineDeserializer.BuildWithNew (just for testing) but this leaves me with an InvalidProgramException:

System.InvalidProgramException: Common Language Runtime detected an invalid program.
at _DynamicMethod0(TextReader , Int32 )
at Jil.JSON.Deserialize[T](TextReader reader, Options options) in C:\Projekte\Jil\Jil\JSON.cs:line 4688

Is there an easy way to allow reading multiple objects?

kind rergards
Andreas

[kevin-montrose]

There's no easy way to do this with Jil at this time. I'll think on what it'd take to support this, but I can't promise anything; Jil's hyper-focused so adding new config options isn't trivial.

[schauerte]

That's more I can expect. Let me know, if I can help in any way (unfortunately I'm not so used to emit).

Thanks in advance

[schauerte]

I have taken a moment to review this issue and created a pull request (#174).
One thing, I don't understand is the InvalidProgramException mentioned above.

The InlineDeserializer.BuildWithNew method looks very clear:

    void BuildWithNew(Type forType)
    {
        AddGlobalVariables();

        ConsumeWhiteSpace();

        Build(null, forType, allowRecursion: false);

        // we have to consume this, otherwise we might succeed with invalid JSON
        ConsumeWhiteSpace();

        // We also must confirm that we read everything, again otherwise we might accept garbage as valid
        ExpectEndOfStream();

        Emit.Return();
    }

But trying to avoid the exception in order to support streamed processing yields strange results:

Variant A

Just skipping ExpectEndOfstream yield the InvalidProgramException:

        // we have to consume this, otherwise we might succeed with invalid JSON
        ConsumeWhiteSpace();

        // We also must confirm that we read everything, again otherwise we might accept garbage as valid
        //ExpectEndOfStream();

Variant B

Skipping (the second) ConsumeWhiteSpace and ExpectEndOfStream works without exception (but complicates End-Of-Stream detection):

        // we have to consume this, otherwise we might succeed with invalid JSON
        //ConsumeWhiteSpace();

        // We also must confirm that we read everything, again otherwise we might accept garbage as valid
        //ExpectEndOfStream();

Variant C:

Patching ExpectEndOfStream body (as done in #174) works also without exception but seems somewhat odd.

Can anybody explain what is happening here, has an idea or at least some hint on how to debug this?

Many thanks
Andreas