Deserializing Structs containing flattened RawValues always fails

[fegies]

use serde_json::value::{Value,RawValue};
use serde::Deserialize;
use std::collections::HashMap;

#[derive(Deserialize, Debug)]
struct TestA<'a> {
    a: i32,
    #[serde(borrow)]
    #[serde(flatten)]
    rest: HashMap::<&'a str, &'a RawValue>,
}

#[derive(Deserialize, Debug)]
struct TestB<'a> {
    #[serde(borrow)]
    #[serde(flatten)]
    rest: HashMap::<&'a str, &'a RawValue>,
}

#[derive(Deserialize, Debug)]
struct TestC<'a> {
    #[serde(borrow)]
    #[serde(flatten)]
    rest: HashMap::<&'a str, Value>,
}

fn main() {
    let s = r#"{
        "a": 42,
        "b": "foo"
    }"#;
    let r = serde_json::from_str::<TestA>(s);
    println!("{:?}", r);
    let r = serde_json::from_str::<TestB>(s);
    println!("{:?}", r);
    let r = serde_json::from_str::<TestC>(s);
    println!("{:?}", r);
}

Output

Err(Error("invalid type: newtype struct, expected any valid JSON value", line: 4, column: 5))
Err(Error("invalid type: newtype struct, expected any valid JSON value", line: 4, column: 5))
Ok(TestC { rest: {"a": Number(42), "b": String("foo")} })

Expected behaviour

s is deserialized to all three structs successfully

Relevant Rust Playground

https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=1e2131c60a6b23e4a01c316786cbd7a6

Versions in use:

• latest stable rustc 1.40.0
• serde_json: 1.0.44
• serde: 1.0.104

[krisselden]

@fegies you will get that error when you are deserializing borrowed keys that require unescaping backslash, use String or Cow for the keys in your HashMap

[fegies]

@krisselden

Deserializing to a Hashmap of with key type &str might lead to errors in cases with unesaping.

This example is not such a case though. (As shown by the fact that TestC deserializes correctly)

I think this error might be because of an Interaction with #[serde(flatten)] and #[serde(borrow)], and not related to the hashmap.

Please observe the following example

#[derive(Deserialize, Debug)]
struct DInner<'a> {
    #[serde(borrow)]
    b: &'a RawValue,
}

#[derive(Deserialize, Debug)]
struct TestD<'a> {
    a : i32,
    #[serde(borrow)]
    #[serde(flatten)]
    inner: DInner<'a>,
}

which fails to deserialize with the same error, while the modified examples with owned HashMap key values still fails to deserialize.

relevant rust playground:
https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=1e1d61ada0881ae66fd16f699fefc0f8

[krisselden]

@fegies sorry, you are correct, seems to be an issue the code flatten generates.

Here is fairly simple workaround https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=e8e922add8a58d097d33cd98a3019560

[krisselden]

@fegies a better version little bit more code (but not much) and does it in one pass https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=3c2227417c489db7e833b8d90d2ba81b

I wanted the same thing because I only want to process part of a json and pass through the rest.

[arnauorriols]

Just wondering, which level of difficulty does fixing this issue have? We are affected by it and we are willing to dedicate some time to work on a PR if feasible.

[clchiou]

I encountered the same issue, and it appears that the reason flatten-generated code cannot work with RawValue is as follows:

• The flatten-generated code performs a two-pass deserialization. First, it deserializes the input into the serde-internal Content type, and then it deserializes Content into the output type (RawValue in this case).
• RawValue's magic relies on passing a magic token to the deserializer, which is lost during the two-pass deserialization.

So, it seems that this bug cannot be easily fixed unless the way flatten generates code is changed.

[Thomasdezeeuw]

I think this is a duplicate of #1051.