Implementation of deterministic serialization.

[cheako]

I know, from basic reaserch, that deterministic serialization is out of spec for protobuf. That doesn't mean it's somehow impossible... Just that it's not default. I also accept that implementing some form of deterministic serialization on top of protobuf will introduce a performance hit.

What I'm thinking of is writing two functions. One that takes any protobuf and dose something like a spell correction where it'll arrange and encode to produce a canonical form. The second would as *quickly as possible determine if a give protobuf is in this canonical form.

• No reason it needs to be quick, just that in my use case there is allot of time where a protobuf should be in the canonical form and this needs to be confirmed in a pass/fail situation.

I could do this as its own project, but I feel this belongs as part of the rust-protobuf project.

Where I got the idea for this was the response to the MtGox hacks. In that case message signatures where malleable and this lead to ppl being able to move Bitcoin out of MtGox holdings without MtGox seeing the transactions(that are then under a new/altered ID). The solution was to take the signatures and only except a subset of the specification, leading to deterministic serialization of something that was previously non-deterministic.

If there is a more applicable encoder I should look at, I'm open to looking into it. Here is my project details. I'm wanting to sign(ecdsa) messages and be able to validate the signature, after transmission across a WebSocket. I'm also looking to alter and re-sign the same message. I don't want ppl to be able to dictate what is signed, that's obviously open to abuse. So I'm looking to load a message, alter a timestamp and a nonce then store the timestamp and nonce with the signature to be store that with the original/unaltered message. I need to be able to recreate the message that was signed without storing a full copy. I plan to have something like 100 or 1000 signatures of the same message, used to judge the usefulness of a given message. Here is what I'm currently thinking about for a message:

message InsertMsg {
    uint32 magic = 1; /* Use to describe the version/format of the decrypted part */
    U256 key = 2; /* Used to retrieve the message. */
    Seq seq = 3; /* This is for "x of y" message combining */
    uint64 time = 4;
    uint64 nonce = 5;
    bytes inner = 6; /* This would be encrypted */
    uint32 check_sum = 7; /* This would be used to validate correct decryption, mainly useful in cases of key collisions. */
}

The goal is for these messages to be distributed using a centralized WebSocket server. Where the server just sees how popular each message is, but otherwise has little idea of what's going on.

Thanks.

[stepancheg]

Serialization in rust-protobuf is deterministic, except for:

• it is not guaranteed to not change in the future version, still standard compliant, but maybe slightly different
• map fields and unknown fields are serialized without any particular order

Can you give an example, when serialization is not deterministic?

[cheako]

That's just what my reaserch showed. I did see the implementation looked mostly deterministic. I don't currently have any map or unknown fields... Though adding functions to output those in a sorted way would be welcomed. Still wanting for an "is in deterministic form" test function.

[stepancheg]

Closing due to old age.

If the issue persists, please reopen!