Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implementation of deterministic serialization. #496

Closed
cheako opened this issue Jun 10, 2020 · 3 comments
Closed

Implementation of deterministic serialization. #496

cheako opened this issue Jun 10, 2020 · 3 comments

Comments

@cheako
Copy link

cheako commented Jun 10, 2020

I know, from basic reaserch, that deterministic serialization is out of spec for protobuf. That doesn't mean it's somehow impossible... Just that it's not default. I also accept that implementing some form of deterministic serialization on top of protobuf will introduce a performance hit.

What I'm thinking of is writing two functions. One that takes any protobuf and dose something like a spell correction where it'll arrange and encode to produce a canonical form. The second would as *quickly as possible determine if a give protobuf is in this canonical form.

  • No reason it needs to be quick, just that in my use case there is allot of time where a protobuf should be in the canonical form and this needs to be confirmed in a pass/fail situation.

I could do this as its own project, but I feel this belongs as part of the rust-protobuf project.

Where I got the idea for this was the response to the MtGox hacks. In that case message signatures where malleable and this lead to ppl being able to move Bitcoin out of MtGox holdings without MtGox seeing the transactions(that are then under a new/altered ID). The solution was to take the signatures and only except a subset of the specification, leading to deterministic serialization of something that was previously non-deterministic.

If there is a more applicable encoder I should look at, I'm open to looking into it. Here is my project details. I'm wanting to sign(ecdsa) messages and be able to validate the signature, after transmission across a WebSocket. I'm also looking to alter and re-sign the same message. I don't want ppl to be able to dictate what is signed, that's obviously open to abuse. So I'm looking to load a message, alter a timestamp and a nonce then store the timestamp and nonce with the signature to be store that with the original/unaltered message. I need to be able to recreate the message that was signed without storing a full copy. I plan to have something like 100 or 1000 signatures of the same message, used to judge the usefulness of a given message. Here is what I'm currently thinking about for a message:

message InsertMsg {
    uint32 magic = 1; /* Use to describe the version/format of the decrypted part */
    U256 key = 2; /* Used to retrieve the message. */
    Seq seq = 3; /* This is for "x of y" message combining */
    uint64 time = 4;
    uint64 nonce = 5;
    bytes inner = 6; /* This would be encrypted */
    uint32 check_sum = 7; /* This would be used to validate correct decryption, mainly useful in cases of key collisions. */
}

The goal is for these messages to be distributed using a centralized WebSocket server. Where the server just sees how popular each message is, but otherwise has little idea of what's going on.

Thanks.

@stepancheg
Copy link
Owner

Serialization in rust-protobuf is deterministic, except for:

  • it is not guaranteed to not change in the future version, still standard compliant, but maybe slightly different
  • map fields and unknown fields are serialized without any particular order

Can you give an example, when serialization is not deterministic?

@cheako
Copy link
Author

cheako commented Jun 21, 2020

That's just what my reaserch showed. I did see the implementation looked mostly deterministic. I don't currently have any map or unknown fields... Though adding functions to output those in a sorted way would be welcomed. Still wanting for an "is in deterministic form" test function.

@stepancheg
Copy link
Owner

Closing due to old age.

If the issue persists, please reopen!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants