Extra newline at end of "dag get" result

[ianopolous]

Version information:

go-ipfs version: 0.4.5-dev-4cb236c
Repo version: 4
System version: amd64/linux
Golang version: go1.7.1

Type:

Bug

Priority:

P1 (for me)

Description:

Using either the cli or http api the dag get command seems to add an extra newline 0x0A on the end of the raw data.

To reproduce using cli:

echo '{"data":1234}' | ipfs dag put -f cbor

then

ipfs dag get zdpuAs3whHmb9T1NkHSLGF45ykcKrEBxSLiEx6YpLzmKbQLEB | wc -c

should give 13, but gives 14, and the extra byte on the end is a newline

The same JSON through the http api results in the same extra byte. The JSON is not special, similar JSON with more/different elements does the same.

[Kubuxu]

JSON isn't a stable format, we have no way of knowing was was the initial format of the json you have passed in.

Also you are passing it in with a new line, but doing printf "%s" '{"data":1234}' won't change anything.
The formatting of the output JSON format is arbitrary if you store it as pure data.

[ianopolous]

ipfs dag put --help says that the default input format is JSON.

[djdv]

I think Kubuxu is saying that echo is outputting your string but also a newline, so what you meant to pass in and what is actually being passing in are different. printf will print the string and not add a newline.

[Kubuxu]

ipfs dag put --help says that the default input format is JSON.

Yes, input format is JSON but storage format is CBOR, which is stable format.

JSON isn't a stable format, we have no way of knowing was was the initial format of the json you have passed in.

There are infinitely many ways you can store same data in JSON ({"data":1234}, {"data":1234 }, {"data": 1234}, and so on) but from the usability standpoint it is more important that the same information has the same address.

If you want to store the white-space information, use plain ipfs add which doesn't interpret data your are passing it.

[ianopolous]

@djdv I tried the printf suggestion, but, as Jakub predicted, it gives the same result.

@Kubuxu I get that JSON -> CBOR is many to one, and you need some canonical format. What I don't get is why that canonical format has a newline outside a minimal form of the json. As a consumer of the api, even one who is well versed in other aspects of IPFS this is surprising to me. (Also, I originally submitted this after discussing it with @whyrusleeping)

[whyrusleeping]

Yeah, the primary issue here isnt that the input object differs from the output object, the problem is that theres a newline after the json blob for some reason.

[jacobheun]

Verified this is still an issue in 0.5

[lidel]

For the record, this is still an issue in 0.9.0.
Adding to maintenance board.

[aschmahmann]

This may get resolved in the refactoring of dag get to support go-ipld-prime (e.g. #8171). For now let's just add a sharness test that expects to fail and we can flip the switch to expecting success in the rewriting PR to see if the issue persists.

[masih]

This may get resolved in the refactoring of dag get to support go-ipld-prime (e.g. #8171). For now let's just add a sharness test that expects to fail and we can flip the switch to expecting success in the rewriting PR to see if the issue persists.

Implemented here: #8280

[masih]

@aschmahmann I can confirm #7995 fixes this issue.

[aschmahmann]

Great, we'll close this issue out once it's been incorporated into master.

[BigLep]

Closing since it's in RC