Strict mode for JSON parsing #2323

marten-voorberg · 2023-02-24T14:03:19Z

Note that this PR is still a work in progress.

Aims to resolve #2295.

In a non-lenient JSON parser, certain non-spec compliant values are accepted. Thie code in this PR allows you to set a JSONReader to strict mode in which non-spec compliant values are not allowed.

The following non-spec compliant values are accepted by the default JSONReader but not a JSONReader in strict mode:

Capitalization in keywords. In normal mode, any combination of lower and uppercase letters in the keywords false, true, and null is allowed (e.g. nUlL or FALSE). In strict mode, only the fully lower-case values are allowed.
JsonReader allows unescaped control characters (U+0000 through U+001F) in normal mode but not in strict mode.
JsonReader supports the escape sequence \', representing a ' in normal mode. In strict mode, this raises an exception.
JsonReader supports the escape sequence \LF (with LF being the Unicode character U+000A), resulting in a LF within the read JSON string. In strict mode, this raises an exception.

… strict mode is used

…specific escape sequence representing in strict mode

…in strict mode

…strictness

# Conflicts: # gson/src/test/java/com/google/gson/stream/JsonReaderTest.java

marten-voorberg · 2023-02-24T14:04:13Z

The API for setStrict is not final and will definitely be changed to one of the options discussed in #2295.

MaicolAntali

Thanks! 😄

gson/src/main/java/com/google/gson/GsonBuilder.java

gson/src/test/java/com/google/gson/stream/JsonReaderTest.java

eamonnmcmanus

Thanks, this is a good start!

We do require new code to follow the Google Java Style Guide, even if not all existing code does. You may be able to configure your IDE to use google-java-format.

eamonnmcmanus · 2023-02-24T17:39:45Z

gson/src/main/java/com/google/gson/stream/JsonReader.java

-    if (c == 't' || c == 'T') {
+
+    // Uppercase letters are not recognized if strict mode is used.
+    if (c == 't' || (!strict && c == 'T')) {


Rather than checking strict in all these ifs, I think it would be simpler to start the for loop below at 0 instead of 1. Also, instead of checking strict in the loop, before the loop you could set keywordUpper to be the same as keyword if strict is true. (Maybe renaming it to something like alternativeKeyword.)

I've rewritten this in the most recent version. Personally, I'm not convinced its cleaner than what was there before so let me know what you think.

@eamonnmcmanus What do you think of this bit of logic has been rewritten?

gson/src/main/java/com/google/gson/stream/JsonReader.java

gson/src/test/java/com/google/gson/stream/JsonReaderTest.java

gson/src/main/java/com/google/gson/GsonBuilder.java

… and GsonBuilder

gson/src/main/java/com/google/gson/Gson.java

… Google java style

* Feat #5: add new lines to make JsonReader able to detect unescaped control characters (U+0000 through U+001F) and throw exceptions. * Feat #5: add new lines to make JsonReader able to detect unescaped control characters (U+0000 through U+001F) and throw exceptions. * Test #11: Added two tests for testing implementation of control character handling in strict mode and moved the implementation to nextQuotedValue * Test #11: Added two tests for testing implementation of control character handling in strict mode and moved the implementation to nextQuotedValue --------- Co-authored-by: LMC117 <[email protected]> Co-authored-by: Marten Voorberg <[email protected]>

* Doc #17: Update JavaDoc in JsonReader and Strictness * Doc #17: Update JavaDoc in Gson and GsonBuilder

marten-voorberg · 2023-04-08T21:24:26Z

Thanks for the response. I won't be able to work on this in the coming week but I'll be sure to take look when I'm back.

…

On Sat, 8 Apr 2023, 21:26 Éamonn McManus, ***@***.***> wrote: I was going to say the same thing as @Marcono1234 <https://github.com/Marcono1234>: if someone calls GsonBuilder.setStrictness, a new method, then we should do the right thing and apply that to toJson and fromJson. (I might have implied otherwise earlier but I have come around to this way of thinking.) — Reply to this email directly, view it on GitHub <#2323 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADMQBHAY46OR6GWW6AUMSQDXAG3XZANCNFSM6AAAAAAVG56EYI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

marten-voorberg · 2023-04-18T18:38:56Z

I comitted some changes where the strictness now works as follows:

By default, a GSON object has a strictness == null. When this is the case, the GSON instance should behave exactly as a GSON instance does in the current release to maintain backwards compatibility.
If the GSON strictness is set, then this strictness value will be imposed on any writers and readers that are used through the instance, but there old values will be restored when the method returns.

I think if we chose to go this way, we should indeed rename Strictness.DEFAULT to something along the lines of Strictness.LEGACY_STRICT.

I have not updated any of the JavaDoc but will do so once we agree on this approach (or decide to go some other way).

eamonnmcmanus · 2023-04-18T18:44:40Z

That approach sounds good to me.

marten-voorberg · 2023-04-19T13:16:52Z

A couple things which we might want to consider:

Should it be possible to set the strictness to null manually by calling GsonBuilder.setStrictness(null)? I am inclined to disallow this as calling this method means you are actively thinking about stricntess and you should just chose which one you prefer.
What should setLenient(boolean) do for the JsonWriter and JsonReader? The functionality of passing true is obvious but what should the value be if false is passed? I guess it should be LEGACY_STRICT for backwards compatability reasons although I also see the case for making this method parameterless for consistency with the GsonBuilder or deprecating it as it now seems to be preferable to use setStrictness(LENIENT) instead.

marten-voorberg · 2023-05-01T15:25:18Z

A couple things which we might want to consider:

* Should it be possible to set the strictness to `null` manually by calling `GsonBuilder.setStrictness(null)`? I am inclined to disallow this as calling this method means you are actively thinking about stricntess and you should just chose which one you prefer.

* What should `setLenient(boolean)` do for the `JsonWriter` and `JsonReader`? The functionality of passing true is obvious but what should the value be if false is passed? I guess it should be `LEGACY_STRICT` for backwards compatability reasons although I also see the case for making this method parameterless for consistency with the `GsonBuilder` or deprecating it as it now seems to be preferable to use `setStrictness(LENIENT)` instead.

@Marcono1234 @eamonnmcmanus Could you comment on the above? Once this is sorted, we can finalize that JavaDoc and then merge it in :).

Marcono1234 · 2023-05-04T20:53:25Z

Sorry for the late response; I don't know what Éamonn thinks about this, but personally I agree with you:

GsonBuilder.setStrictness(null) should not be possible but instead throw a NullPointerException
For JsonReader / JsonWriter setLenient(false) use LEGACY_STRICT to remain backward compatible, but deprecate these setLenient methods and in the @deprecated Javadoc tag refer to setStrictness

Also, as side note, this question is still unanswered I think:

One open question was whether you missed marten-voorberg@4f6c4e0 by accident in your pull request. (Might be good to check if any other commit is missing as well.)

Marcono1234 · 2023-05-11T18:14:35Z

@eamonnmcmanus, do you agree with that?

gson/src/main/java/com/google/gson/stream/JsonReader.java

eamonnmcmanus · 2023-05-11T20:28:20Z

Yes, I agree your earlier comment.

Sorry I haven't been following this more closely. Is it ready to be merged into the strictness branch?

marten-voorberg · 2023-05-13T09:42:13Z

Sorry I haven't been following this more closely. Is it ready to be merged into the strictness branch?

No problem. I think this can be merged into the branch! @eamonnmcmanus

One open question was whether you missed marten-voorberg@4f6c4e0 by accident in your pull request. (Might be good to check if any other commit is missing as well.)

@Marcono1234 Good point. I've not directly added the commit back in but I have added an equivalent test case that uses the more modern assertThrows(). As far as I can tell, there are no other missing commits.

I've now deprecated setLenient in JsonReader, JsonWriter, and GsonBuilder. I think the next step is to merge it into the strictness branch. After that, we should probably make sure that the JavaDoc is up-to-date everywhere as we changed the functionality quite a bit througout this PR so there might be some lingering, outdated documentation.

Marcono1234

Thanks for the changes! Sorry, I hope you weren't waiting for my response. Just to make sure there is no misunderstanding, to me the changes look good.

There might be a few small documentation changes I want to make, and maybe extending the U+2028 and U+2029 tests, but this is definitely not blocking this PR.

As you mentioned, it would be good to merge this to the strictness branch already and then perform any additional changes in separate PRs. The first thing would be merging master back into the strictness branch.

@eamonnmcmanus, what do you think?

* Feat #6: Add strict flag to Gson and GsonBuilder * Test #2: Add failing tests for capitalized keywords * Feat #2: JsonReader does not read (partially) capitalized keywords if strict mode is used * Feat #3: Added implementation and tests for JSONReader not accepting specific escape sequence representing in strict mode * Test #3: Simplify test cases by removing unnecessary array * Feat #3: Improve error by including the illegal character * Feat #5: JsonReader does not allow unespaced control flow characters in strict mode * Test #5: Test unespaced control flow characters in strict mode * Feat #4: Disallow espaced newline character in strict mode * Test #4: Add tests for (dis)allowing newline character depensding on strictness * Test #5: Test case for unescaped control char in non-strict mode * Test #2: Simplify test cases * Feat #13: Change leniency API to Strictness enum in JsonReader, Gson, and GsonBuilder * Feat #15: Change JsonWriter API to also use Strictness * Test #15: Test Strictness in JsonWriter API * Doc #15: Add and update documentation for Strictness in JsonWriter API * refactor #12: Fixed typos and empty catch brackets in tests * refactor #12: Resolved importing wildcards, made some lines adhere to Google java style * #5 Add test case for unescaped control characters * Feat #5: add new lines to make JsonReader able to detect unescaped control characters (U+0000 through U+001F) and throw exceptions. * Feat #5: add new lines to make JsonReader able to detect unescaped control characters (U+0000 through U+001F) and throw exceptions. * Test #11: Added two tests for testing implementation of control character handling in strict mode and moved the implementation to nextQuotedValue * Test #11: Added two tests for testing implementation of control character handling in strict mode and moved the implementation to nextQuotedValue --------- Co-authored-by: LMC117 <[email protected]> Co-authored-by: Marten Voorberg <[email protected]> * Doc #17: Add and change javadoc of public methods * Doc #17: Update JavaDoc in JsonReader and Strictness * Doc #17: Update JavaDoc in Gson and GsonBuilder * Test #34: Add tests for setting strictness through GsonBuilder * Fix: Add Fix broken test * Fix: Invalid JavaDoc in Gson.java * Doc #17: update outdated javadoc * #37: Resolve more PR feedback * Fix #37: Resolve various PR comments * Fix #37: Resolve various PR comments * Refactor #35: Refactor JsonReader#peekKeyword to reduce the amount of strictness checks (#39) * Doc #40: Update JavaDoc based on PR feedback * Doc #40: Update old RFC in GsonBuilder documentation * Doc #40: Fix formatting error in JavaDoc * Doc #40: Add tests for setting strictness and lenient to JsonReaderTest * Test #43: Changed tests to make use of assertThrows * test #43: Changed tests to make use of assertThrows as per feedback * Test #43: Update JsonWriterTest#testStrictnessNull to use assertThrows * Test #43: Update JsonWriterTest#testStrictnessNull to use assertThrows * test #43: Resolve PR recommendations * Test #43: Mini change to TC * Test #43: Mini change to TC --------- Co-authored-by: Marten Voorberg <[email protected]> * doc #46: Resolved comments in main PR * Feat #45: Change Gson.fromJson and Gson.toJson to be strict when the provided writer/reader is strict * Fix #45: Small type * Update gson/src/test/java/com/google/gson/stream/JsonReaderTest.java Co-authored-by: Marcono1234 <[email protected]> * Fix #45: Resolve various comments by Marcono1234 * Update gson/src/main/java/com/google/gson/GsonBuilder.java Co-authored-by: Marcono1234 <[email protected]> * Fix #45: Resolve various comments by Marcono1234 * Fix #45: Resolve various comments by eamonmcmanus * Strictness mode follow-up * Update Troubleshooting.md and Gson default lenient mode documentation * Always use GSON strictness when set. * Rename Strictness.DEFAULT to Strictness.LEGACY_STRICT * Update JavaDoc with new strictness functionality * Replace default with legacy strict for JsonReader javadoc * Add JSONReader test cases for U2028 and U2029 * Refactor JSONReader#peekKeyWord() based on @eamonmcmanus's suggestion * Deprecate setLenient in favor of setStrictness --------- Co-authored-by: Carl Peterson <unknown> Co-authored-by: Gustaf Johansson <[email protected]> Co-authored-by: gustajoh <[email protected]> Co-authored-by: LMC117 <[email protected]> Co-authored-by: Marcono1234 <[email protected]>

eamonnmcmanus · 2023-05-31T00:44:39Z

OK, my attempts to merge the strictness branch seem to have left it in a not-great state. I will see if I can fix it tomorrow.

eamonnmcmanus · 2023-05-31T14:09:20Z

I believe the new strictness2 branch now reflects what I should have done with strictness. Please confirm. For now I have removed the deprecation tags from setLenient since they cause compilation failures now that we compile with -Werror. It would make sense to restore the deprecation and fix those failures, either by calling the replacement methods or by using @SuppressWarnings("deprecation").

Marcono1234 · 2023-06-03T23:23:15Z

Thanks a lot for resolving the merge conflicts! Comparing the original state of strictness after the merge with strictness2 using https://github.com/google/gson/compare/9ce668e10bb2aed284cb8b5ad0345f0dbea8670b..strictness2 (I hope that URL is correct), it looks to me like nothing was lost (except the setLenient changes you mentioned).

In the end it would be good to compare strictness2 and master again before merging, just to make sure it does not accidentally revert something from master.

* Strict mode for JSON parsing (#2323) * Feat #6: Add strict flag to Gson and GsonBuilder * Test #2: Add failing tests for capitalized keywords * Feat #2: JsonReader does not read (partially) capitalized keywords if strict mode is used * Feat #3: Added implementation and tests for JSONReader not accepting specific escape sequence representing in strict mode * Test #3: Simplify test cases by removing unnecessary array * Feat #3: Improve error by including the illegal character * Feat #5: JsonReader does not allow unespaced control flow characters in strict mode * Test #5: Test unespaced control flow characters in strict mode * Feat #4: Disallow espaced newline character in strict mode * Test #4: Add tests for (dis)allowing newline character depensding on strictness * Test #5: Test case for unescaped control char in non-strict mode * Test #2: Simplify test cases * Feat #13: Change leniency API to Strictness enum in JsonReader, Gson, and GsonBuilder * Feat #15: Change JsonWriter API to also use Strictness * Test #15: Test Strictness in JsonWriter API * Doc #15: Add and update documentation for Strictness in JsonWriter API * refactor #12: Fixed typos and empty catch brackets in tests * refactor #12: Resolved importing wildcards, made some lines adhere to Google java style * #5 Add test case for unescaped control characters * Feat #5: add new lines to make JsonReader able to detect unescaped control characters (U+0000 through U+001F) and throw exceptions. * Feat #5: add new lines to make JsonReader able to detect unescaped control characters (U+0000 through U+001F) and throw exceptions. * Test #11: Added two tests for testing implementation of control character handling in strict mode and moved the implementation to nextQuotedValue * Test #11: Added two tests for testing implementation of control character handling in strict mode and moved the implementation to nextQuotedValue --------- Co-authored-by: LMC117 <[email protected]> Co-authored-by: Marten Voorberg <[email protected]> * Doc #17: Add and change javadoc of public methods * Doc #17: Update JavaDoc in JsonReader and Strictness * Doc #17: Update JavaDoc in Gson and GsonBuilder * Test #34: Add tests for setting strictness through GsonBuilder * Fix: Add Fix broken test * Fix: Invalid JavaDoc in Gson.java * Doc #17: update outdated javadoc * #37: Resolve more PR feedback * Fix #37: Resolve various PR comments * Fix #37: Resolve various PR comments * Refactor #35: Refactor JsonReader#peekKeyword to reduce the amount of strictness checks (#39) * Doc #40: Update JavaDoc based on PR feedback * Doc #40: Update old RFC in GsonBuilder documentation * Doc #40: Fix formatting error in JavaDoc * Doc #40: Add tests for setting strictness and lenient to JsonReaderTest * Test #43: Changed tests to make use of assertThrows * test #43: Changed tests to make use of assertThrows as per feedback * Test #43: Update JsonWriterTest#testStrictnessNull to use assertThrows * Test #43: Update JsonWriterTest#testStrictnessNull to use assertThrows * test #43: Resolve PR recommendations * Test #43: Mini change to TC * Test #43: Mini change to TC --------- Co-authored-by: Marten Voorberg <[email protected]> * doc #46: Resolved comments in main PR * Feat #45: Change Gson.fromJson and Gson.toJson to be strict when the provided writer/reader is strict * Fix #45: Small type * Update gson/src/test/java/com/google/gson/stream/JsonReaderTest.java Co-authored-by: Marcono1234 <[email protected]> * Fix #45: Resolve various comments by Marcono1234 * Update gson/src/main/java/com/google/gson/GsonBuilder.java Co-authored-by: Marcono1234 <[email protected]> * Fix #45: Resolve various comments by Marcono1234 * Fix #45: Resolve various comments by eamonmcmanus * Strictness mode follow-up * Update Troubleshooting.md and Gson default lenient mode documentation * Always use GSON strictness when set. * Rename Strictness.DEFAULT to Strictness.LEGACY_STRICT * Update JavaDoc with new strictness functionality * Replace default with legacy strict for JsonReader javadoc * Add JSONReader test cases for U2028 and U2029 * Refactor JSONReader#peekKeyWord() based on @eamonmcmanus's suggestion * Deprecate setLenient in favor of setStrictness --------- Co-authored-by: Carl Peterson <unknown> Co-authored-by: Gustaf Johansson <[email protected]> Co-authored-by: gustajoh <[email protected]> Co-authored-by: LMC117 <[email protected]> Co-authored-by: Marcono1234 <[email protected]> * Strictness follow-up (#2408) * Strictness mode follow-up - Remove mentions of `null` Gson strictness; this is an implementation detail - Fix incorrect / outdated documentation - Reduce links to RFC; if there is already a link to it in a previous sentence don't link to it again - Extend and update tests - Minor punctuation changes in documentation for consistency * Deprecate `setLenient` methods * `strictness2` fixes & improvements (#2456) * Adjust ProGuard default rules and shrinking tests (#2420) * Adjust ProGuard default rules and shrinking tests * Adjust comment * Add shrinking test for class without no-args constructor; improve docs * Improve Unsafe mention in Troubleshooting Guide * Improve comment for `-if class *` * Bump com.google.guava:guava from 32.0.1-jre to 32.1.1-jre (#2444) Bumps [com.google.guava:guava](https://github.com/google/guava) from 32.0.1-jre to 32.1.1-jre. - [Release notes](https://github.com/google/guava/releases) - [Commits](https://github.com/google/guava/commits) --- updated-dependencies: - dependency-name: com.google.guava:guava dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump com.google.guava:guava-testlib from 32.0.1-jre to 32.1.1-jre (#2443) Bumps [com.google.guava:guava-testlib](https://github.com/google/guava) from 32.0.1-jre to 32.1.1-jre. - [Release notes](https://github.com/google/guava/releases) - [Commits](https://github.com/google/guava/commits) --- updated-dependencies: - dependency-name: com.google.guava:guava-testlib dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Support non-generic type for `TypeToken.getParameterized` for legacy reasons (#2447) This partially restores the behavior before a589ef2, except that back then for a non-generic type a bogus `TypeToken(ParameterizedType)` was created, whereas now a `TypeToken(Class)` is created instead. * Fixed Typo in GsonBuilder.java (#2449) * Make date-formatting tests less fragile with regular expressions. (#2450) * Make date-formatting tests less fragile with regular expressions. This is not great. We should really ensure that formatted dates are the same regardless of JDK version. There is code that attempts to do that but it is not really effective. So for now we fudge around the differences by using regular expressions to paper over the differences. * Temporarily add test-debugging code. * Another attempt at debugging a test failure. * Fix pattern in assertion. * Modification in test cases (#2454) * Fixed Typo in GsonBuilder.java * Suggestions on Test cases * Modified test cases using assertThrows method (JUnit) * Update gson/src/test/java/com/google/gson/JsonArrayAsListTest.java Co-authored-by: Marcono1234 <[email protected]> * Update gson/src/test/java/com/google/gson/GsonTest.java Co-authored-by: Marcono1234 <[email protected]> * Update gson/src/test/java/com/google/gson/JsonArrayAsListTest.java Co-authored-by: Marcono1234 <[email protected]> * Update gson/src/test/java/com/google/gson/JsonStreamParserTest.java Co-authored-by: Marcono1234 <[email protected]> * Update gson/src/test/java/com/google/gson/JsonStreamParserTest.java Co-authored-by: Marcono1234 <[email protected]> * Update gson/src/test/java/com/google/gson/JsonStreamParserTest.java Co-authored-by: Marcono1234 <[email protected]> * Update gson/src/test/java/com/google/gson/ToNumberPolicyTest.java Co-authored-by: Marcono1234 <[email protected]> * Update gson/src/test/java/com/google/gson/TypeAdapterTest.java Co-authored-by: Marcono1234 <[email protected]> * Update gson/src/test/java/com/google/gson/TypeAdapterTest.java Co-authored-by: Marcono1234 <[email protected]> * Update gson/src/test/java/com/google/gson/ToNumberPolicyTest.java Co-authored-by: Marcono1234 <[email protected]> * Update gson/src/test/java/com/google/gson/ToNumberPolicyTest.java Co-authored-by: Marcono1234 <[email protected]> --------- Co-authored-by: Marcono1234 <[email protected]> * Minor follow-up changes --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: elevne <[email protected]> Co-authored-by: Éamonn McManus <[email protected]> Co-authored-by: Wonil <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Marten <[email protected]> Co-authored-by: Gustaf Johansson <[email protected]> Co-authored-by: gustajoh <[email protected]> Co-authored-by: LMC117 <[email protected]> Co-authored-by: Marcono1234 <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: elevne <[email protected]> Co-authored-by: Wonil <[email protected]>

…e#2437) * Strict mode for JSON parsing (google#2323) * Feat #6: Add strict flag to Gson and GsonBuilder * Test #2: Add failing tests for capitalized keywords * Feat #2: JsonReader does not read (partially) capitalized keywords if strict mode is used * Feat #3: Added implementation and tests for JSONReader not accepting specific escape sequence representing in strict mode * Test #3: Simplify test cases by removing unnecessary array * Feat #3: Improve error by including the illegal character * Feat #5: JsonReader does not allow unespaced control flow characters in strict mode * Test #5: Test unespaced control flow characters in strict mode * Feat #4: Disallow espaced newline character in strict mode * Test #4: Add tests for (dis)allowing newline character depensding on strictness * Test #5: Test case for unescaped control char in non-strict mode * Test #2: Simplify test cases * Feat #13: Change leniency API to Strictness enum in JsonReader, Gson, and GsonBuilder * Feat #15: Change JsonWriter API to also use Strictness * Test #15: Test Strictness in JsonWriter API * Doc #15: Add and update documentation for Strictness in JsonWriter API * refactor #12: Fixed typos and empty catch brackets in tests * refactor #12: Resolved importing wildcards, made some lines adhere to Google java style * #5 Add test case for unescaped control characters * Feat #5: add new lines to make JsonReader able to detect unescaped control characters (U+0000 through U+001F) and throw exceptions. * Feat #5: add new lines to make JsonReader able to detect unescaped control characters (U+0000 through U+001F) and throw exceptions. * Test #11: Added two tests for testing implementation of control character handling in strict mode and moved the implementation to nextQuotedValue * Test #11: Added two tests for testing implementation of control character handling in strict mode and moved the implementation to nextQuotedValue --------- Co-authored-by: LMC117 <[email protected]> Co-authored-by: Marten Voorberg <[email protected]> * Doc #17: Add and change javadoc of public methods * Doc #17: Update JavaDoc in JsonReader and Strictness * Doc #17: Update JavaDoc in Gson and GsonBuilder * Test google#34: Add tests for setting strictness through GsonBuilder * Fix: Add Fix broken test * Fix: Invalid JavaDoc in Gson.java * Doc #17: update outdated javadoc * google#37: Resolve more PR feedback * Fix google#37: Resolve various PR comments * Fix google#37: Resolve various PR comments * Refactor google#35: Refactor JsonReader#peekKeyword to reduce the amount of strictness checks (google#39) * Doc google#40: Update JavaDoc based on PR feedback * Doc google#40: Update old RFC in GsonBuilder documentation * Doc google#40: Fix formatting error in JavaDoc * Doc google#40: Add tests for setting strictness and lenient to JsonReaderTest * Test google#43: Changed tests to make use of assertThrows * test google#43: Changed tests to make use of assertThrows as per feedback * Test google#43: Update JsonWriterTest#testStrictnessNull to use assertThrows * Test google#43: Update JsonWriterTest#testStrictnessNull to use assertThrows * test google#43: Resolve PR recommendations * Test google#43: Mini change to TC * Test google#43: Mini change to TC --------- Co-authored-by: Marten Voorberg <[email protected]> * doc google#46: Resolved comments in main PR * Feat google#45: Change Gson.fromJson and Gson.toJson to be strict when the provided writer/reader is strict * Fix google#45: Small type * Update gson/src/test/java/com/google/gson/stream/JsonReaderTest.java Co-authored-by: Marcono1234 <[email protected]> * Fix google#45: Resolve various comments by Marcono1234 * Update gson/src/main/java/com/google/gson/GsonBuilder.java Co-authored-by: Marcono1234 <[email protected]> * Fix google#45: Resolve various comments by Marcono1234 * Fix google#45: Resolve various comments by eamonmcmanus * Strictness mode follow-up * Update Troubleshooting.md and Gson default lenient mode documentation * Always use GSON strictness when set. * Rename Strictness.DEFAULT to Strictness.LEGACY_STRICT * Update JavaDoc with new strictness functionality * Replace default with legacy strict for JsonReader javadoc * Add JSONReader test cases for U2028 and U2029 * Refactor JSONReader#peekKeyWord() based on @eamonmcmanus's suggestion * Deprecate setLenient in favor of setStrictness --------- Co-authored-by: Carl Peterson <unknown> Co-authored-by: Gustaf Johansson <[email protected]> Co-authored-by: gustajoh <[email protected]> Co-authored-by: LMC117 <[email protected]> Co-authored-by: Marcono1234 <[email protected]> * Strictness follow-up (google#2408) * Strictness mode follow-up - Remove mentions of `null` Gson strictness; this is an implementation detail - Fix incorrect / outdated documentation - Reduce links to RFC; if there is already a link to it in a previous sentence don't link to it again - Extend and update tests - Minor punctuation changes in documentation for consistency * Deprecate `setLenient` methods * `strictness2` fixes & improvements (google#2456) * Adjust ProGuard default rules and shrinking tests (google#2420) * Adjust ProGuard default rules and shrinking tests * Adjust comment * Add shrinking test for class without no-args constructor; improve docs * Improve Unsafe mention in Troubleshooting Guide * Improve comment for `-if class *` * Bump com.google.guava:guava from 32.0.1-jre to 32.1.1-jre (google#2444) Bumps [com.google.guava:guava](https://github.com/google/guava) from 32.0.1-jre to 32.1.1-jre. - [Release notes](https://github.com/google/guava/releases) - [Commits](https://github.com/google/guava/commits) --- updated-dependencies: - dependency-name: com.google.guava:guava dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump com.google.guava:guava-testlib from 32.0.1-jre to 32.1.1-jre (google#2443) Bumps [com.google.guava:guava-testlib](https://github.com/google/guava) from 32.0.1-jre to 32.1.1-jre. - [Release notes](https://github.com/google/guava/releases) - [Commits](https://github.com/google/guava/commits) --- updated-dependencies: - dependency-name: com.google.guava:guava-testlib dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Support non-generic type for `TypeToken.getParameterized` for legacy reasons (google#2447) This partially restores the behavior before a589ef2, except that back then for a non-generic type a bogus `TypeToken(ParameterizedType)` was created, whereas now a `TypeToken(Class)` is created instead. * Fixed Typo in GsonBuilder.java (google#2449) * Make date-formatting tests less fragile with regular expressions. (google#2450) * Make date-formatting tests less fragile with regular expressions. This is not great. We should really ensure that formatted dates are the same regardless of JDK version. There is code that attempts to do that but it is not really effective. So for now we fudge around the differences by using regular expressions to paper over the differences. * Temporarily add test-debugging code. * Another attempt at debugging a test failure. * Fix pattern in assertion. * Modification in test cases (google#2454) * Fixed Typo in GsonBuilder.java * Suggestions on Test cases * Modified test cases using assertThrows method (JUnit) * Update gson/src/test/java/com/google/gson/JsonArrayAsListTest.java Co-authored-by: Marcono1234 <[email protected]> * Update gson/src/test/java/com/google/gson/GsonTest.java Co-authored-by: Marcono1234 <[email protected]> * Update gson/src/test/java/com/google/gson/JsonArrayAsListTest.java Co-authored-by: Marcono1234 <[email protected]> * Update gson/src/test/java/com/google/gson/JsonStreamParserTest.java Co-authored-by: Marcono1234 <[email protected]> * Update gson/src/test/java/com/google/gson/JsonStreamParserTest.java Co-authored-by: Marcono1234 <[email protected]> * Update gson/src/test/java/com/google/gson/JsonStreamParserTest.java Co-authored-by: Marcono1234 <[email protected]> * Update gson/src/test/java/com/google/gson/ToNumberPolicyTest.java Co-authored-by: Marcono1234 <[email protected]> * Update gson/src/test/java/com/google/gson/TypeAdapterTest.java Co-authored-by: Marcono1234 <[email protected]> * Update gson/src/test/java/com/google/gson/TypeAdapterTest.java Co-authored-by: Marcono1234 <[email protected]> * Update gson/src/test/java/com/google/gson/ToNumberPolicyTest.java Co-authored-by: Marcono1234 <[email protected]> * Update gson/src/test/java/com/google/gson/ToNumberPolicyTest.java Co-authored-by: Marcono1234 <[email protected]> --------- Co-authored-by: Marcono1234 <[email protected]> * Minor follow-up changes --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: elevne <[email protected]> Co-authored-by: Éamonn McManus <[email protected]> Co-authored-by: Wonil <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Marten <[email protected]> Co-authored-by: Gustaf Johansson <[email protected]> Co-authored-by: gustajoh <[email protected]> Co-authored-by: LMC117 <[email protected]> Co-authored-by: Marcono1234 <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: elevne <[email protected]> Co-authored-by: Wonil <[email protected]>

marten-voorberg and others added 13 commits February 23, 2023 10:33

Feat #6: Add strict flag to Gson and GsonBuilder

9b2d343

Test #2: Add failing tests for capitalized keywords

fe6dd37

Feat #2: JsonReader does not read (partially) capitalized keywords if…

a78764a

… strict mode is used

Feat #3: Added implementation and tests for JSONReader not accepting …

4895e8f

…specific escape sequence representing in strict mode

Test #3: Simplify test cases by removing unnecessary array

fd3f757

Feat #3: Improve error by including the illegal character

0c28e0b

Feat #5: JsonReader does not allow unespaced control flow characters …

947a312

…in strict mode

Test #5: Test unespaced control flow characters in strict mode

0f57cc9

Feat #4: Disallow espaced newline character in strict mode

695159b

Test #4: Add tests for (dis)allowing newline character depensding on …

bda7321

…strictness

Test #5: Test case for unescaped control char in non-strict mode

d438cbd

Test #2: Simplify test cases

23897d2

Merge remote-tracking branch 'origin/master'

26463e7

# Conflicts: # gson/src/test/java/com/google/gson/stream/JsonReaderTest.java

marten-voorberg marked this pull request as draft February 24, 2023 14:06

MaicolAntali reviewed Feb 24, 2023

View reviewed changes

gson/src/main/java/com/google/gson/GsonBuilder.java Outdated Show resolved Hide resolved

gson/src/test/java/com/google/gson/stream/JsonReaderTest.java Show resolved Hide resolved

eamonnmcmanus requested changes Feb 24, 2023

View reviewed changes

marten-voorberg mentioned this pull request Feb 25, 2023

Resolve feedback in PR 2323 marten-voorberg/DD2480-gson#12

Closed

Feat #13: Change leniency API to Strictness enum in JsonReader, Gson,…

d3950fa

… and GsonBuilder

MaicolAntali reviewed Feb 26, 2023

View reviewed changes

gson/src/main/java/com/google/gson/Gson.java Outdated Show resolved Hide resolved

marten-voorberg added 3 commits February 27, 2023 10:19

Feat #15: Change JsonWriter API to also use Strictness

c303c61

Test #15: Test Strictness in JsonWriter API

2b55804

Doc #15: Add and update documentation for Strictness in JsonWriter API

d2a792b

MaicolAntali mentioned this pull request Feb 27, 2023

14 control flow characters not in 0x0000 0x001f #2329

Closed

gustajoh and others added 6 commits February 27, 2023 11:11

refactor #12: Fixed typos and empty catch brackets in tests

0cd64da

refactor #12: Resolved importing wildcards, made some lines adhere to…

f2abd89

… Google java style

Doc #17: Add and change javadoc of public methods

7319777

* Doc #17: Update JavaDoc in JsonReader and Strictness * Doc #17: Update JavaDoc in Gson and GsonBuilder

Test #34: Add tests for setting strictness through GsonBuilder

b29baa0

Fix: Add Fix broken test

0af517b

Marcono1234 and others added 3 commits April 18, 2023 13:06

Strictness mode follow-up

8453738

Update Troubleshooting.md and Gson default lenient mode documentation

8593b57

Always use GSON strictness when set.

9a7f33a

marten-voorberg added 3 commits April 19, 2023 14:48

Rename Strictness.DEFAULT to Strictness.LEGACY_STRICT

4d0ea89

Update JavaDoc with new strictness functionality

be59dd6

Replace default with legacy strict for JsonReader javadoc

48bf90c

eamonnmcmanus reviewed May 11, 2023

View reviewed changes

gson/src/main/java/com/google/gson/stream/JsonReader.java Outdated Show resolved Hide resolved

marten-voorberg added 3 commits May 13, 2023 11:29

Add JSONReader test cases for U2028 and U2029

caaad5b

Refactor JSONReader#peekKeyWord() based on @eamonmcmanus's suggestion

39cd9a0

Deprecate setLenient in favor of setStrictness

4342ef8

Marcono1234 approved these changes May 28, 2023

View reviewed changes

eamonnmcmanus approved these changes May 30, 2023

View reviewed changes

eamonnmcmanus merged commit 9ce668e into google:strictness May 30, 2023

Marcono1234 mentioned this pull request Jun 4, 2023

Strictness follow-up #2408

Merged

9 tasks

Marcono1234 mentioned this pull request Mar 17, 2024

Support strict mode for JsonParser #2648

Merged

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strict mode for JSON parsing #2323

Strict mode for JSON parsing #2323

marten-voorberg commented Feb 24, 2023 •

edited by eamonnmcmanus

Loading

marten-voorberg commented Feb 24, 2023

MaicolAntali left a comment

eamonnmcmanus left a comment

eamonnmcmanus Feb 24, 2023

marten-voorberg Feb 28, 2023

marten-voorberg Mar 1, 2023

marten-voorberg commented Apr 8, 2023 via email

marten-voorberg commented Apr 18, 2023

eamonnmcmanus commented Apr 18, 2023

marten-voorberg commented Apr 19, 2023

marten-voorberg commented May 1, 2023

Marcono1234 commented May 4, 2023

Marcono1234 commented May 11, 2023

eamonnmcmanus commented May 11, 2023

marten-voorberg commented May 13, 2023

Marcono1234 left a comment

eamonnmcmanus commented May 31, 2023

eamonnmcmanus commented May 31, 2023

Marcono1234 commented Jun 3, 2023

Strict mode for JSON parsing #2323

Strict mode for JSON parsing #2323

Conversation

marten-voorberg commented Feb 24, 2023 • edited by eamonnmcmanus Loading

marten-voorberg commented Feb 24, 2023

MaicolAntali left a comment

Choose a reason for hiding this comment

eamonnmcmanus left a comment

Choose a reason for hiding this comment

eamonnmcmanus Feb 24, 2023

Choose a reason for hiding this comment

marten-voorberg Feb 28, 2023

Choose a reason for hiding this comment

marten-voorberg Mar 1, 2023

Choose a reason for hiding this comment

marten-voorberg commented Apr 8, 2023 via email

marten-voorberg commented Apr 18, 2023

eamonnmcmanus commented Apr 18, 2023

marten-voorberg commented Apr 19, 2023

marten-voorberg commented May 1, 2023

Marcono1234 commented May 4, 2023

Marcono1234 commented May 11, 2023

eamonnmcmanus commented May 11, 2023

marten-voorberg commented May 13, 2023

Marcono1234 left a comment

Choose a reason for hiding this comment

eamonnmcmanus commented May 31, 2023

eamonnmcmanus commented May 31, 2023

Marcono1234 commented Jun 3, 2023

marten-voorberg commented Feb 24, 2023 •

edited by eamonnmcmanus

Loading