p2p/enr: initial implementation

[nonsense]

initial implementation of ENR according to ethereum/EIPs#778

TODO:

• compare RLP encoding/decoding with Python impl at https://github.com/fjl/p2p-drafts.
• reject records longer than 300 bytes.
• set r.raw on RLP decode, so that record is ready for RLP encoding after RLP decoding.
• sort pairs on Set, use binary search on Load
• ensure that setting the same key multiple times will update it instead of including it twice.
• get rid of double-RLP encoding of pair.value - once in Set() and once is signAndEncode()
• add documentation
• use binary search on Set
• add more tests to check decode/encode and sorting

[nonsense]

This feels like a hack. I am looking to see if there is a nicer way to wrap a list.

[nonsense]

The signature returned from the python impl has a length of 64, and recovering of the public key fails.

[fjl]

I think the API for keys/values should be different because there can be arbitrary keys used by different parts of the codebase. We need a way to define new keys outside of package enr.

Another thing to keep in mind is that record validity is defined to be signature validity only. So a record containing the "ip4" key with RLP value [1, 2, 3] is still a valid (but useless) record. The valid vs. useless distinction is important.

Here's my API proposal:

package enr

// Key is implemented by known node record key types.
//
// To define a new key that is to be included in a node record,
// create a Go type that satisfies this interface. The type should
// also implement rlp.Decoder if additional checks are needed on the value.
type Key interface{
	ENRKey() string
}

// IP4 represents a 4-byte IPv4 address in a node record.
type IP4 net.IP

// ENRKey returns the node record key for an IPv4 address.
func (IP4) ENRKey() string {
	return "ip4"
}

func (v IP4) EncodeRLP(w io.Writer) error {
	ip4 := net.IP(v).To4()
	if ip4 == nil {
		return fmt.Errorf("invalid IPv4 address: %v", v)
	}
	return rlp.Encode(w, ip4)
}

func (v *IP4) DecodeRLP(s *rlp.Stream) error {
	if err := s.Decode((*net.IP)(v)); err != nil {
		return err
	}
	if len(*v) != 4 {
		return fmt.Errorf("invalid IPv4 address, want 4 bytes: %v", *v)
	}
	return nil
}

To construct a node record add Keys to the zero value, then sign it.

func settingAndEncoding() {
	// r is an empty record.
	var r enr.Record
	
	// Add basic metadata.
	r.Set(enr.IP4(net.ParseIP("127.0.0.1")))
	r.SetSeq(1)

	// At this point r is non-empty but not signed, so EncodeRLP() returns an error.
	_, err := rlp.EncodeToBytes(r)
	
	// Sign with secp256k1 key. This adds the "id" and "secp256k1" k/v pairs
	// and the signature.
	if err := r.Sign(privkey); err != nil {
		panic(err)
	}
	
	// This works now:
	_, err := rlp.EncodeToBytes(r)

	// Update metadata. This invalidates the signature.
	r.Set(enr.IP4(net.ParseIP("127.0.0.2")))
	r.SetSeq(2)

	// This will return an error again until the record is re-signed.
	_, err := rlp.EncodeToBytes(r)
}

To get a value out of a node record, pass the Key you want.
This will decode the value if present.

func decodingAndAccessing(input []byte) {
	// Decoding into enr.Record validates the signature.
	var r enr.Record
	err := rlp.DecodeBytes(input, &r)
	if err != nil {
		panic(err)
	}

	// You can get any k/v value like this:
	var ip4 enr.IP4
	ok, err := r.Load(&ip4)
	// ip4 contains the address if it was present in the record.
        // ok says whether the key was present, err says whether the
        // value is valid.
	
	// Accessing basic information:
	seq := r.Seq()
	addr := r.NodeAddr()
}

[fjl]

You can use btcec.Signature.Verify instead: https://godoc.org/github.com/btcsuite/btcd/btcec#example-package--VerifySignature

[fjl]

No need to export this method.

[fjl]

You can avoid most of the RLP hackery by using higher-level types instead. To encode a record, put elements into []interface{} then call rlp.EncodeToBytes:

list := []interface{}{nil, e.seq}
// add k/v pairs
for _, r := range e.records {
    list = append(list, k, v)
}
// sign the tail of the list
sigcontent, err := rlp.EncodeToBytes(list[1:])
...
// add signature to front
list[0] = sig
record, err := rlp.EncodeToBytes(list)

[fjl]

Something I forgot in the review summary earlier: the terms are slightly off in your implementation.
A "record" is the whole thing. key/value "pairs" are the elements of a "record". This should be reflected in identifiers.

[nonsense]

@fjl thanks for the comments. The proposed API is indeed better. I think I addressed your comments, however I still haven't confirmed the RLP encoding/decoding vs the Python impl. - there might be some differences there. Will ping you when this is ready for review.

[fjl]

This is starting to look really nice. Thank you so much.
I have two general suggestions and a couple of nitpicks below ;)

Please move all predefined keys into one file. Their definitions aren't that large.

pairs is supposed to be sorted by key at encoding time. Maybe insert into the correct place in Set so it will be sorted at all times. If you do this you can use binary search in Load. Please also ensure that setting the same key multiple times will update it instead of including it twice.

[fjl]

UDP ports are uint16.

[nonsense]

Done.

[fjl]

You can remove rlp encoding methods from Discv5 because the codec 'sees through' named types and encodes them correctly.

[fjl]

Please rename DiscV5 -> DiscPort.

[nonsense]

Done.

[fjl]

Package rlp supports strings. Please make k a string to avoid conversions.

[nonsense]

Done.

[fjl]

You can remove rlp encoding methods from ID.

[nonsense]

Done.

[fjl]

Please use

type Secp256k1 ecdsa.PublicKey

You already need to parse the public key for signature verification, might as well do it at decoding time. The additional advantage will be that users can't get invalid curve points by accident.

[fjl]

You can be less verbose and write r.Set(ID(SECP256k1_KECCAK)).

[nonsense]

Done.

[fjl]

For your convenience, output of make run from p2p-drafts/testvec should be a lot more helpful now ;).

[fjl]

I'll re-review when the rest of the boxes are checked ;)

[nonsense]

@fjl great, will ping you when the TODO list is addressed.

[nonsense]

@fjl @zsfelfoldi I think I addressed all comments and TODOs. This should be ready for review now.

[RayPylant]

excellent

[RayPylant]

excellent

[zsfelfoldi]

Beautiful piece of code :) Left a few comments but nothing critical, LGTM.

[zsfelfoldi]

This name confused me at first, I thought it is just the key from the key/value pair. Not sure what I'd like to suggest instead though, I would probably call it Entry or Item but I am not good at naming stuff :)

[nonsense]

Entry is not a bad name for this interface. I can see how Key could confuse someone new to the package.

[nonsense]

I just pushed a commit that renames Key to Entry where I found it to actually mean Entry, rather than key from a key/value pair. I am not 100% convinced the new version is better, probably because I am already used to the Key interface, but I don't have strong opinion on the name.

@fjl @zsfelfoldi let me know what you think, I am fine with both names to be honest.

[zsfelfoldi]

Shouldn't this also invalidate r.raw? (probably you are only checking r.signature for nil but maybe it would be nicer to not leave an invalid r.raw there)

[nonsense]

I agree - we should invalidate it - someone could try to decode the record, after having set a key and changed its contents. Pushed a change.

[zsfelfoldi]

Why not just "port" or "udp"? Discovery v5-specific port will go away soon, the UDP port used by discovery will not depend on the version of the discovery protocol so we should not name this field after that.

[nonsense]

I honestly don't know enough about the roadmap to comment on that. Depending on how soon is soon, and when exactly ENR will be integrated with the rest of the codebase, we can decide whether to keep this DiscPort entry type, or remove it and just add Port.

I guess we can postpone this decision until Discovery v5-specific port actually goes away?

[nonsense]

@zsfelfoldi I think I addressed your comments. @fjl please review the Key rename to Entry and if you are not happy with it, feel free to just revert the commit. I think Entry reflects slightly better the semantics of the interface.

[karalabe]

Please bump the copyright headers to 2017 :)

[fjl]

The interface is called Key because it assigns the key to a key/value pair (that's what the method returns). I'll know if Entry is right after christmas. It doesn't really matter though.