String copy policy behavior?

[samvik]

Describe the issue
First of all thanks for a grate library.

We are finding that it is a bit unintuitive and error prone when it comes to string handling. More specifically where const char* is not copied but char* is. Since we try to keep as much as possible const and this has nothing to do with lifetime we often end up having to const_cast pointers in order to force arduinojson to copy the string (easy to forget and leads to strange runtime errors). We have mitigated the issue a bit by implementing custom converters for our string classes (mainly etl::string and etl::string_view), but this has it's limitations see question 2 below.

So to my questions.

1. Any way to disable zerocopy entirely for a single document or for the entire library? Or is it possible to use another setter that will always copy?

2. The custom converters only seems to work on the value and not the keys, is this by design? See example below.

etl::string_view str = ...;
doc[str] = 1234; // Does not compile
doc["name"] = str; // Compiles and work

3. Is there a better way of handling/controlling the behavior? In other words have we misunderstood how to use arduinojson efficiently?

Thanks
/Björn

Troubleshooter report

1. The issue happens at run time
2. The issue concerns serialization
3. Output contains garbage
4. serializeJson() produces garbage
5. Program doesn't call deserializeJson()
6. Program calls serializeJson(const JsonDocument&, ...)
7. Program uses DynamicJsonDocument
8. Program doesn't use String

Environment
Here is the environment that I'm using':

• NXP iMX RT1061
• Core/runtime: FreeRTOS
• IDE: MCU Xpresso

[bblanchon]

Hi Björn,

The "zero-copy mode" is a feature of the deserializer, and it's independent of the "string copy policy" you are referring to.
I don't claim ArduinoJson to be perfect, and I admit that this policy can be a bit confusing because it's not explicit.
However, it's a good compromise and does the right thing most of the time.

If you see yourself calling const_cast<char*> frequently (which is rather strange, honestly), you can create a free function that does it for you, for example:

char* mustBeCopied(const char* s) { 
  return const_cast<char*>(s);
}

doc["key"] = mustBeCopied("value");

ArduinoJson doesn't allow altering the copy policy for const char*; I currently have no intention of adding such a feature.

Regarding your second question, custom converters, as you saw, allow conversions between a user-defined type and a JsonVariant; they are therefore limited to values and cannot be used for keys.

There is an undocumented API to add support for new string classes: you must add an overload of ARDUINOJSON_NAMESPACE::adaptString() and a specialization of ARDUINOJSON_NAMESPACE::IsString; see Strings/Adapters/StdString.hpp for example.

I can probably modify the library so that every iterable of char (i.e., every class that has begin() and end() that return a char iterator) is automatically treated as a string, which would make etl::string work out of the box.
I'm pretty sure some metaprogramming voodoo will make this possible; in the meantime, use the undocumented API mentioned above.

Best regards,
Benoit

[samvik]

Hello Benoit,

Thanks for the answers. Sorry for the confusion between "zero-copy mode" and "string copy policy". It is "string copy policy" i'm referring to.

I agree it's both ugly and strange to have to do a const cast to control if the value is copied or not, that is the reason for these questions. As I said the problem is that const is not an indication of lifetime which makes it a bit tricky to have "long lived" JsonDocuments. I understand that ArduinoJson might be designed for short lived documents where the copy policy might make sense (be a pure optimization) but as soon as you store the document or start passing it around you run into strange runtime errors if you are not very careful. I think some kind of functionality to make the copy policy explicit would prevent a lot of errors. If this is something you are open to include in some form lets discuss, perhaps I can help implementing it.

I have implemented the free function mitigation, the problem is that it does not force a compile time error if you forget it.

Will have a look at the adaptString will that also make it possible to use them as keys?

+1 for the general string support. :-)

Thanks again for the support
/Björn

[bblanchon]

Hi Björn,

You're looking at this in the wrong way: the JsonDocument behaves like a regular struct with only one exception.

It stores std::string and const char* exactly as the following struct would:

struct SomeStruct {
  std::string aString;  // stores a copy
  const char* aPointer;  // stores a pointer
};

The exception concerns char* (and const __FlashStringHelper*) which is stored by copy and not by pointer.
Why? To allow the following use case:

StaticJsonDocument<128> doc;
for (int i=0; i<4;  i++) {
   char buf[32];
   sprintf(bug, "value-%d", i);
   doc.add(buf);
}

If char* were stored by pointer, the resulting document would be:

["value-3", "value-3", "value-3", "value-3"]

This policy was added in version 5.13.0 and greatly simplified library usage.

Why don't we copy all strings, including const char*?
Because it would mean storing string literals by copy, which is unacceptable.

If you don't want to use char*, then pass the etl::string directly to the JsonDocument.
As soon as you override adaptString() and specialize IsString, etl::string will be supported for keys and values.

Lastly, remember that ArduinoJson is a serialization library, not a container library.
There are a few caveats to using a JsonDocument as a container; please read Why must I create a separate config object?.

Best regards,
Benoit

[samvik]

Hi Benoit,

Thanks for the explanation, as I kind of suspected, we are using the library in a not "supported" way. Our application need the ability to handle unstructured data and ArduinoJson does a surprisingly good job at this, with the exception of the const char* case. We will have to evaluate and see if the adaptString will be sufficient for us or if we need to find another solution.

Thanks again
/Björn

[bblanchon]

I'm pretty sure some metaprogramming voodoo will make this possible; in the meantime, use the undocumented API mentioned above.

I did that in the generic-strings, but unfortunately, the member function detection idiom doesn't work with inherited functions, so it fails with StringSumHelper.
To fix this, we need decltype, so we need to wait until ArduinoJson requires C++11.

[bblanchon]

Generic string support is available in ArduinoJson 6.21.0