encoding/json compatibility issues with malformed unicode strings

[stevenjohnstone]

Example program:

package main

import (
        "encoding/json"
        "fmt"
        "reflect"
        "strings"

        jsoniter "github.com/json-iterator/go"
)

type jsoner struct {
        Name      string
        Unmarshal func([]byte, interface{}) error
        Marshal   func(interface{}) ([]byte, error)
}

var json2 = jsoniter.ConfigCompatibleWithStandardLibrary

func printStringArray(label string, s []string) {
        output := []string{}
        for _, v := range s {
                output = append(output, fmt.Sprintf("%v", []byte(v)))
        }

        fmt.Printf("%s: %s ([%s])\n", label, s, strings.Join(output, ","))
}

func main() {

        data := []byte("[\"\xa8\"]")

        jsoners := []jsoner{
                {
                        Name:      "encoding/json",
                        Unmarshal: json.Unmarshal,
                        Marshal:   json.Marshal,
                },
                {
                        Name:      "jsoniter",
                        Unmarshal: json2.Unmarshal,
                        Marshal:   json2.Marshal,
                },
        }

        for _, j := range jsoners {

                var s []string

                if err := j.Unmarshal(data, &s); err != nil {
                        panic(err)
                }

                printStringArray(j.Name, s)

                data2, err := j.Marshal(s)
                if err != nil {
                        panic(err)
                }

                fmt.Printf("%s: data  = %s (%v)\n", j.Name, string(data), data)
                fmt.Printf("%s: data2 = %s (%v)\n", j.Name, string(data2), data2)

                var s2 []string

                if err := j.Unmarshal(data2, &s2); err != nil {
                        panic(err)
                }

                printStringArray(j.Name, s2)

                if !reflect.DeepEqual(s, s2) {
                        fmt.Printf("%s: Error, %v != %v\n", j.Name, s, s2)
                }
        }

}

Program output:

$ go run main.go 
encoding/json: [�] ([[239 191 189]])
encoding/json: data  = ["�"] ([91 34 168 34 93])
encoding/json: data2 = ["�"] ([91 34 239 191 189 34 93])
encoding/json: [�] ([[239 191 189]])
jsoniter: [�] ([[168]])
jsoniter: data  = ["�"] ([91 34 168 34 93])
jsoniter: data2 = ["\ufffd"] ([91 34 92 117 102 102 102 100 34 93])
jsoniter: [�] ([[239 191 189]])
jsoniter: Error, [�] != [�]

In the example above, jsoniter stores the string "\xa8" without interpretation: the value it stores is equivalent to string([]byte{0xa8}). In Marshal, however, the string "\xa8" is interpreted as unicode with the malformed "\xa8" replaced with the replacement character \ufffd (seen above as [239 191 189]).
In contrast, encoding/json interprets the contents of the string as malformed unicode and replaces 0xa8 with the unicode replacement character \uffd in both the unmarshal and marshal steps.

This causes Unmarshal, Marshal, Unmarshal to be different from a single Unmarshal in these cases which is unexpected.

From https://golang.org/pkg/encoding/json/#Unmarshal:

When unmarshaling quoted strings, invalid UTF-8 or invalid UTF-16 surrogate pairs are not treated as an error. Instead, they are replaced by the Unicode replacement character U+FFFD.

[liggitt]

Just saw this as well, running the go-fuzz json corpus on json-iterator