Add a strict parsing mode to the JsonReader and improve its javadoc

[marcus-h]

By default, the JsonReader accepts unescaped control characters that
are embedded in a string or name (which is technically just a string).
According to RFC 8259, RFC 7159, and RFC 4627, this is forbidden.
From RFC 8259 Section 7 "Strings":

"All Unicode characters may be placed within the
quotation marks, except for the characters that MUST be escaped:
quotation mark, reverse solidus, and the control characters (U+0000
through U+001F)."

Accepting unescaped control characters can at least cause some confusion
as the following naive program demonstrates:

marcus@linux:~> cat NaiveJsonProcessor.java
import java.io.IOException;
import java.io.StringReader;

import com.google.gson.stream.JsonReader;

public class NaiveJsonProcessor {
  public static void parseAndLog(String jsonInput) throws IOException {
    JsonReader reader = new JsonReader(new StringReader(jsonInput));
    String parsed = reader.nextString();
    if (parsed.equals("foo")) {
        throw new IllegalStateException("foo is forbidden");
    }
    /*
     * According to the JsonReader's documentation "[...] this parser
     * is strict and only accepts JSON as specified by RFC 4627" (see
     * documentation of setLenient). Hence, we can safely log the
     * raw jsonInput to stdout because it contains no unescaped control
     * characters, which could be interpreted by a terminal.
     * Oops... wrong assumption:)
     */
    System.out.println("Processed: " + jsonInput);
  }

  public static void main(String[] args) {
    String jsonInput = "\"foobar\u001b[3D\u001b[K\"";
    try {
        // the log entry might confuse the user...
        parseAndLog(jsonInput);
    } catch (IOException e) {
        e.printStackTrace();
    }
  }
}
marcus@linux:~> javac -cp /path/to/gson/classes NaiveJsonProcessor.java
marcus@linux:~> java -cp /path/to/gson/classes:. NaiveJsonProcessor
Processed: "foo"
marcus@linux:~>

Since the unescaped control characters of the raw jsonInput are interpreted
by the terminal, it looks as if we processed the JSON text "foo" even
though this string should result in an IllegalStateException (of course in
reality we did not process "foo").

This PR fixes this by adding an optional strict mode to the JsonReader.

If preferred, I can also split commit 473a9b3
("Make JsonReader tests reusable in the com.google.gson.stream package") into two commits.

[googlebot]

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

---

What to do if you already signed the CLA

Individual signers

• It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

• Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
• The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
• The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

[marcus-h]

@googlebot I signed it!

[googlebot]

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

[Marcono1234]

Suggested change

	* strings (and names). For the latter, consider the two java strings
	* strings (and names). For the latter, consider the two Java strings

[marcus-h]

Fixed. (I just did a forced push.)

[Marcono1234]

Why has this been removed? Is this incorrect and current non-lenient mode also allows non-object or array top-level values?

[marcus-h]

Yes, other top-level elements like a number, boolean etc. were already supported (which
conforms to RFC 8259) in the non-lenient mode.

[Marcono1234]

This is slightly misleading because it is more lenient (="semi-strict") than RFC 8259, though this might not matter here much

[marcus-h]

Ugh. Good catch! I removed the "only" part. Now, it says that it "accepts RFC 8259
(including some slight variations)".
I just did a forced push.

Thanks a lot for your (initial) review! Much appreciated!

[Marcono1234]

You might have to adjust JsonReader.readEscapeCharacter() for strict mode as well. Currently it accepts:

• \'
• \LF

Both are not allowed by RFC 8259.

Also maybe it would be useful to cross-link in the javadoc between the corresponding methods, e.g.

• setStrict has @see #isStrict()
• isStrict has @see #setStrict(boolean)

Though the existing documentation sadly does not really do that either.

[Marcono1234]

#2437 was been merged which adds a new Strictness mode API. This should hopefully address the issues here, and also in the meantime the documentation was adjusted to be more explicit about Gson not being strict by default and how to enable strict parsing.

I will therefore close this pull request, but thanks nonetheless for your work on this! If you notice anything which is missing in the current implementation or think the documentation can be improved, then any pull request for this is appreciated.