Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DNM] experimental UTXO commitments #1916

Closed

Conversation

antiochp
Copy link
Member

@antiochp antiochp commented Nov 3, 2018

Playing around with possible approaches/implementation for "UTXO commitments". Not sure if this would be useful in its current state.

This PR allows a parallel UTXO PMMR to be built alongside the existing output (aka TXO) PMMR.
Too expensive to do this in realtime so hooked it up to run on demand.

Also too complex to maintain in realtime to handle rewind, sync etc.

We can trigger a UTXO PMMR rebuild via the /compact API endpoint (easiest place to hook it into) -

curl -X POST 127.0.0.1:13413/v1/chain/compact
...
2018-11-03T13:25:56.755685+00:00 DEBUG grin_chain::chain - Rebuilding UTXO set (experimental).
2018-11-03T13:25:56.755747+00:00 DEBUG grin_chain::txhashset::utxo_set - *** rebuild: rebuilding the utxo_set (rewinding aka truncating to pos 0 first).
2018-11-03T13:25:56.755758+00:00 DEBUG grin_chain::txhashset::utxo_set - Truncating utxo_set extension.
2018-11-03T13:25:56.794249+00:00 WARN grin_chain::chain - ********** UTXO root: 21025ec4 (output root: f0e58c30)
...

Approx 40ms to rebuild the UTXO MMR in testnet4 based on the existing output MMR (can be pruned/compacted). This is rebuilding from scratch with no thought put into optimizing this.

The existing output MMR is "append only". spending an output does not change the data in the MMR and will not affect the root of the MMR. Only by appending new outputs to the MMR will the root change.
We can prune and compact the output MMR and leave the root unaffected by removing spent outputs.

To construct the UTXO MMR we simply replace spent outputs with a "zero" sentinel value.
Each leaf is then hashed via hash_with_index("zero", pos).
The interesting thing is these can roll-up to parent nodes in a deterministic way.
For example, if pos 1 and pos 2 are both spent we can hash them as -

  • hash1 = hash_with_index("zero", 1)
  • hash2 = hash_with_index("zero", 2)

and the parent node at pos 3 can be generated as before -

  • hash3 = hash_with_index((hash1, hash2), 3)

And we can recreate the hash at pos 3 without needing to know anything about the underlying data beyond them both having been removed/spent.

Related - #1883 (@tromp suggested "zeroing out" spent commitments to fix the spent hash "malleability").
Related - #1733 (proof of "unspentness")

Inspiration from - https://petertodd.org/2016/delayed-txo-commitments (Delayed TXO commitments).

Thinking out loud -
Maybe we don't need to do this in realtime - maybe there are ways we can do this in a "delayed" fashion. We commit to the UTXO MMR every 60 blocks (hourly) or 1,440 blocks (daily) say.
This would allow us to generate a Merkle proof for any unspent output older than a day, for example.
Nodes could generate proofs for their own outputs by maintaining some subset of the UTXO MMR.
Nodes could verify this without needing to maintain the full UTXO set etc.

Thoughts? @tromp @ignopeverell

@tromp
Copy link
Contributor

tromp commented Nov 3, 2018

Besides the malleability of pruned hashes, isn't there also the malleability of reindexing the whole UTXO set? E.g. all spent outputs could be completely eliminated, and we basically pretend that a tree build on only utxo leaves (i.e. UTXO MMR) is the new TXO MMR. What's the advantage of committing to all the zeroed spent outputs as well?

@antiochp
Copy link
Member Author

antiochp commented Nov 4, 2018

@tromp I don't think I fully understand your comment.

The benefit of keeping the zero'd out spent outputs is we maintain the same overall structure between the output and UTXO MMRs. The output at pos n is the same output in both MMRs, they are just different views on the same data.

The intent here was to keep the output MMR in its existing state - its append only and very efficient for what we want it to do.
And we can see that it is relatively easy to translate the output MMR into a UTXO MMR (using a zero value for spent outputs).

Pruning/compacting outputs in this structure would not involve reindexing the whole UTXO set, which would be necessary if we used a UTXO MMR that only included unspent leaves.

@tromp
Copy link
Contributor

tromp commented Nov 4, 2018

My comment was pointing out that an attacker able to rewrite horizon blocks can go as far as making it look like no tx was ever spent.

I see that preserving txo MMR structure minimizes computation, but It's not append only if every node has to make the spent txo hash changes to verify a UTXO commitment.

Or is this only an optional feature, that a full node is not required to support?

@antiochp
Copy link
Member Author

Closing this for now - not something we're going to be implementing prior to mainnet.

@antiochp antiochp closed this Nov 13, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants