Performance and functional comparison

[big-andy-coates]

Hi,

I've recently needed to compare the performance and functionality of this and other JVM based JSON validation libraries, and I thought you might like to see how this implementation compares. I've published to the code and the results here: https://github.com/creek-service/json-schema-validation-comparison.

While I've spent time trying to get the best out of each library under test, it's possible that the results for this implementation could be improved if someone with more in-depth knowledge were to look into it. If you feel I'm not showing in its best light, please feel free to raise PRs to fix issues and improve your score.

Please feel free to link to the code and results.

I hope this is of some use to you.

Thanks,

Andy

[erosb]

Hello Andy, thanks a lot. Great work. Do you plan to test with bigger json schemas as well?

…

On Sat, Oct 21, 2023, 13:56 Andy Coates ***@***.***> wrote: Hi, I've recently needed to compare the performance and functionality of this and other JVM based JSON validation libraries, and I thought you might like to see how this implementation compares. I've published to the code and the results here: https://github.com/creek-service/json-schema-validation-comparison. While I've spent time trying to get the best out of each library under test, it's possible that the results for this implementation could be improved if someone with more in-depth knowledge were to look into it. If you feel I'm not showing in its best light, please feel free to raise PRs to fix issues and improve your score. Please feel free to link to the code and results. I hope this is of some use to you. Thanks, Andy — Reply to this email directly, view it on GitHub <#29>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACCWBSWVA7XGKN5FP43KM3YAOZ7ZAVCNFSM6AAAAAA6KC7DLGVHI2DSMVQWIX3LMV43ASLTON2WKOZRHE2TKNBWGA4TCMI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[big-andy-coates]

At the moment no, it's just using the standard set of JSON schema test cases. Do you think it would be useful to add a large JSON schema?

[erosb]

Hello @big-andy-coates yes, I think it would make sense, some users were reporting performance problems when using very big schemas. The FHIR project maintains a huge schema you can use for perf tests.

Also I'm not sure if you are aware of the Bowtie project, which is not specific to Java, but has a very similar goal to your evauation - you may want to keep an eye on it.

[big-andy-coates]

Thanks @erosb,

I'm talking with the people behind Bowtie. (I wasn't aware, but they reached out). They're covering the functional side, but not the perf side, which is much more language specific. I'll be helping them a bit adding coverage for implementations they don't have yet.

I'll see if I can find time to add a large schema perf test. Though, if I'm honest, that doesn't really fit the use case I was doing the comparison for.

If you have time, would you mind raising an issue with an example large schema and document(s). Of course, if you were to raise a PR as well, that would be super useful :)

[big-andy-coates]

When I last looked at this repo it seemed to still be in initial development, as a fair number of required functional tests failed. Was that / is that assumption correct?

[erosb]

Hello @big-andy-coates , the support for $dynamicRef and $dynamicAnchor is partial, the rest should be (almost) stable.

[erosb]

Hello @big-andy-coates I wanted to raise an issue for your project but couldn't (the "Submit Issue" button is disabled for me) so I'm putting this here:

would you mind raising an issue with an example large schema and document(s)

I think it would make sense to test with some reasonably big schemas and instances, since the JSON Schema Test Suite works with very small jsons (both schemas and instances) which are very specific to the actual testcases.

So I suggest gathering some more realistic testcases. One potential way would be testing with the FHIR schema, but not sure.

I remember @octavianN you faced performance issues with OxygenXML (which uses everit-org/json-schema) in the past, maybe you can help out here, do you have a good testsuite for performance testing the schema validators?

An other matter regarding the comparisons (we may track it in a separate issue): most java libraries implement:

• a separate schema loading phase (when the schema json is mapped to a java model)
• and a validation phase (where a json instance is validated against the java model).

Regarding that, for most applications, the validation phase has much higher importance regarding the 2nd phase (validation), since very often the schema gets loaded only once per application lifecycle, then the validation runs several times. I'm not sure if/how the current benchmark handles this separation.

[octavianN]

Hi,

We ran the JSON Schema Test suite and as part of the performance evaluation we used larger schemas such as:

• FHIR schema
• UBL 2.1 JSON Schema
• ECV-JsonSchema

[erosb]

Thank you for your input @octavianN !

[big-andy-coates]

the "Submit Issue" button is disabled for me

Summit button will only be disabled if you didn't add a title to the issue. (Standard for Github).

Will you please do me a favour and confirm. I've tested this end with a different login, but I'd like to confirm I don't have an issue with access :)

[big-andy-coates]

An other matter regarding the comparisons (we may track it in a separate issue): most java libraries implement:

• a separate schema loading phase (when the schema json is mapped to a java model)
• and a validation phase (where a json instance is validated against the java model).

Regarding that, for most applications, the validation phase has much higher importance regarding the 2nd phase (validation), since very often the schema gets loaded only once per application lifecycle, then the validation runs several times. I'm not sure if/how the current benchmark handles this separation.

Agreed, for this reason the performance tests only cover the second part, i.e. the validation, not the loading / parsing of schemas.

[big-andy-coates]

@octavianN any chance I could trouble you to create an issue under https://github.com/creek-service/json-schema-validation-comparison with a sample schema and JSON document?

[big-andy-coates]

FYI, microsite with performance and functionality comparison is now live: https://www.creekservice.org/json-schema-validation-comparison/