Add options for record count and timeout for resolving DHT paths #4733

dirkmc · 2018-02-23T16:49:02Z

PR for issue #4723

EDIT: ~Kubuxu
Fixes #4723

License: MIT Signed-off-by: Dirk McCormick <[email protected]>

dirkmc · 2018-02-23T16:56:08Z

This PR adds two new parameters to ipfs name resolve and ipfs resolve:
ipfs name resolve <hash> --dht-record-count=5 --dht-timeout=20

Currently when resolving 16 IPNS records will be retrieved from the DHT (where the DHT k-value is 20). The dht-record-count parameter allows clients to retrieve less IPNS records from the DHT when resolving.

Currently the timeout for retrieving a public key for an IPNS record is one minute, and the timeout for retrieving the value itself is one minute. The public key and value are retrieved sequentially so it can take up to 2 minutes to time out. The dht-timeout parameter allows clients to set a timeout for retrieving both values (eg maximum of 30s to retrieve both).

License: MIT Signed-off-by: Dirk McCormick <[email protected]>

dirkmc · 2018-02-27T22:32:32Z

@vyzo @Stebalien @whyrusleeping could I ask you guys to take a look at this?
Thanks,
Dirk

dirkmc · 2018-02-27T22:33:24Z

core/commands/ipns.go

+			opts.Depth = 1
+		}
+		if rcok {
+			opts.DhtRecordCount = uint(rc)


Can get rid of this cast once ipfs/go-ipfs-cmdkit#14 is approved

vyzo

You might want to make a new method in namesys that takes the parameters, so that you don't have to rewrite all the references.

dirkmc · 2018-02-28T18:17:46Z

My concern was that we would then have three different methods for doing the same thing. Do you think that's not worth worrying about?

vyzo · 2018-02-28T20:31:03Z

My suggestion was more to add the new method that takes the options, and redirect the old method to the new one with default options. That way you wouldn't change the existing code at all.

dirkmc · 2018-02-28T20:34:24Z

I understand, my concern is about having three different methods in the API where we only really need one. Is that not worth worrying about?

vyzo · 2018-02-28T20:35:29Z

Hrm, dunno -- it's nice to have a simple method that takes default options.

whyrusleeping · 2018-02-28T20:37:20Z

When in doubt, variadic options can be a good interface

dirkmc · 2018-02-28T20:54:28Z

Yes you're (both) right, that is nicer, I'll use variadic options

License: MIT Signed-off-by: Dirk McCormick <[email protected]>

dirkmc · 2018-02-28T22:57:02Z

Ok changed to use variadic options, let me know if that makes sense

whyrusleeping · 2018-03-01T19:45:48Z

core/commands/ipns.go

 		}

 		if !strings.HasPrefix(name, "/ipns/") {
 			name = "/ipns/" + name
 		}

-		output, err := resolver.ResolveN(req.Context(), name, depth)


where did the depth arg go?

oh nvm, i found it

whyrusleeping · 2018-03-01T19:48:53Z

core/coreapi/name.go

 	if !strings.HasPrefix(name, "/ipns/") {
 		name = "/ipns/" + name
 	}

-	output, err := resolver.ResolveN(ctx, name, depth)
+	ropts := []nsopts.ResolveOpt{}


i prefer writing this as:
var ropts []nsopts.ResolveOpt

License: MIT Signed-off-by: Dirk McCormick <[email protected]>

whyrusleeping

I don't see anything wrong with this, pretty clean change overall.

I'd like to get one more 👍 from someone else before merging though, maybe @vyzo or @Kubuxu

vyzo

why is the options package a separate package?
It would be nicer from a usability perspective if it was inside namesys so that we wouldn't have to import it separately.

other than that LGTM.

dirkmc · 2018-03-02T15:46:18Z

I put it in a separate package as a way of namespacing, so that you call nsopts.Depth(1) instead of namespace.Depth(1), so that it's clear those methods are working with options. If you think that's not an issue I'm happy to put it all in the top level namesys package.

vyzo · 2018-03-02T16:32:30Z

Fair enough -- not really an issue :)

Kubuxu

LGTM, using Opts pattern fits here very well.

Stebalien · 2018-03-22T01:46:35Z

core/commands/ipns.go

@@ -57,9 +59,10 @@ Resolve the value of a dnslink:
 	Options: []cmdkit.Option{
 		cmdkit.BoolOption("recursive", "r", "Resolve until the result is not an IPNS name."),
 		cmdkit.BoolOption("nocache", "n", "Do not use cached entries."),
+		cmdkit.UintOption("dht-record-count", "dhtrc", "Number of records to request for DHT resolution."),
+		cmdkit.UintOption("dht-timeout", "dhtt", "Timeout in seconds for DHT resolution. Pass 0 for no timeout."),


So, I might be missing something but IPFS commands have a timeout flag (e.g., ipfs --timeout=30s). And we can always set a DHT resolution timeout by setting a timeout on the context. What's the usecase for the additional timeout?

The use case is that with --dht-timeout the DHT collects as many values as it can within the timeout, and then returns the best value it has found. The global IPFS command --timeout will simply kill the command:

> ./cmd/ipfs/ipfs name resolve --timeout=5s QmSgxPeqLrnM1ZB1pno3tYMQfqLB4cWXuAQNfSWsk77apK Error: Post http://127.0.0.1:5001/api/v0/name/resolve?arg=QmSgxPeqLrnM1ZB1pno3tYMQfqLB4cWXuAQNfSWsk77apK&encoding=json&stream-channels=true&timeout=5s: context deadline exceeded > ./cmd/ipfs/ipfs name resolve --dhtt=5 QmSgxPeqLrnM1ZB1pno3tYMQfqLB4cWXuAQNfSWsk77apK /ipfs/QmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn

Maybe it would be better for the --dht-timeout parameter to take a string though, eg --dht-timeout=5s instead of assuming seconds eg --dht-timeout=5

I see. It would be nice to explain that in the help but I can't think of a succinct way of putting it...

@dirkmc it will be better for it to parse the time.

I added a commit to parse the time, and changed the parameter description to
Max time to collect values during DHT resolution eg \"30s\". Pass 0 for no timeout
That's the most succinct way I can think of to say it

…olve License: MIT Signed-off-by: Dirk McCormick <[email protected]>

License: MIT Signed-off-by: Dirk McCormick <[email protected]>

whyrusleeping · 2018-03-23T06:15:00Z

namesys/ipns_validate_test.go

-	if err != ErrExpiredRecord {
-		t.Fatal("ValidateIpnsRecord should have returned ErrExpiredRecord")
+	_, err = resolver.resolveOnce(ctx, id.Pretty(), opts.DefaultResolveOpts())
+	if err == nil {


Why arent we checking for a specific error anymore?

The resolveOnce() method used to call routing.GetValue() but now we call routing.GetValues() directly.
The mock ValueStore implementation's GetValue() method would get a single record and call VerifyRecord() on it, which would return ErrExpiredRecord/ErrSignature etc. We now call the mock ValueStore implementation's GetValues() method, which gets multiple records, calls VerifyRecord() on each one individually, and adds the record to the list of valid records if there's no error. If there are no valid records it returns routing.ErrNotFound.

I looked into whether I can check for routing.ErrNotFound but the go-ipfs-routing mock returns datastore.ErrNotFound instead of routing.ErrNotFound when it doesn't find a key:
https://github.com/ipfs/go-ipfs-routing/blob/master/mock/centralized_client.go#L46

Alright, that seems fine. Thanks for the response!

whyrusleeping

LGTM, just one question on a test decision, then i'll merge.

Thanks @dirkmc :)

Add options for record count and timeout for resolving DHT paths This commit was moved from ipfs/kubo@41d82ee

Add options for record count and timeout for resolving DHT paths

1abf836

License: MIT Signed-off-by: Dirk McCormick <[email protected]>

dirkmc mentioned this pull request Feb 23, 2018

IPNS performance: name resolve record count parameter #4723

Closed

Fix gateway test

5096025

License: MIT Signed-off-by: Dirk McCormick <[email protected]>

dirkmc commented Feb 27, 2018

View reviewed changes

vyzo reviewed Feb 28, 2018

View reviewed changes

dirkmc added 3 commits February 28, 2018 16:57

Use variadic options

e8f79c8

License: MIT Signed-off-by: Dirk McCormick <[email protected]>

Document namesys options

d47ff79

License: MIT Signed-off-by: Dirk McCormick <[email protected]>

Fix namesys opts package name

350362c

License: MIT Signed-off-by: Dirk McCormick <[email protected]>

dirkmc mentioned this pull request Mar 1, 2018

IPNS republisher should not rely on DHT to store (possibly expired) records #4749

Closed

whyrusleeping reviewed Mar 1, 2018

View reviewed changes

Syntactic changes

5202f76

License: MIT Signed-off-by: Dirk McCormick <[email protected]>

whyrusleeping approved these changes Mar 1, 2018

View reviewed changes

dirkmc mentioned this pull request Mar 1, 2018

Move namesys republisher records into their own place in the datastore #4753

Closed

vyzo reviewed Mar 2, 2018

View reviewed changes

Kubuxu approved these changes Mar 3, 2018

View reviewed changes

Kubuxu added the RFM label Mar 8, 2018

Kubuxu added this to the go-ipfs 0.4.15 milestone Mar 8, 2018

Stebalien reviewed Mar 22, 2018

View reviewed changes

dirkmc added 2 commits March 21, 2018 22:58

Return ErrNotFound when zero values are returned from DHT namesys res…

32ef57c

…olve License: MIT Signed-off-by: Dirk McCormick <[email protected]>

Parse string time (eg 30s) instead of assuming seconds for —-dht-timeout

494f242

License: MIT Signed-off-by: Dirk McCormick <[email protected]>

dirkmc force-pushed the feat/namesys-value-count branch from b275d7b to 494f242 Compare March 22, 2018 16:52

dirkmc mentioned this pull request Mar 22, 2018

DHT Query Performance libp2p/go-libp2p-kad-dht#88

Closed

Kubuxu approved these changes Mar 23, 2018

View reviewed changes

whyrusleeping reviewed Mar 23, 2018

View reviewed changes

whyrusleeping approved these changes Mar 23, 2018

View reviewed changes

Kubuxu added need/author-input Needs input from the original author and removed RFM labels Mar 23, 2018

whyrusleeping merged commit 41d82ee into ipfs:master Mar 23, 2018

dirkmc deleted the feat/namesys-value-count branch March 23, 2018 17:45

Stebalien mentioned this pull request Apr 20, 2018

Tell DHT nodes if their IPNS records are out of date #4954

Closed

hacdias pushed a commit to ipfs/boxo that referenced this pull request Jan 27, 2023

Merge pull request ipfs/kubo#4733 from dirkmc/feat/namesys-value-count

5fe80b7

Add options for record count and timeout for resolving DHT paths This commit was moved from ipfs/kubo@41d82ee

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add options for record count and timeout for resolving DHT paths #4733

Add options for record count and timeout for resolving DHT paths #4733

dirkmc commented Feb 23, 2018 •

edited by Kubuxu

Loading

dirkmc commented Feb 23, 2018

dirkmc commented Feb 27, 2018

dirkmc Feb 27, 2018

vyzo left a comment

dirkmc commented Feb 28, 2018

vyzo commented Feb 28, 2018

dirkmc commented Feb 28, 2018

vyzo commented Feb 28, 2018

whyrusleeping commented Feb 28, 2018

dirkmc commented Feb 28, 2018

dirkmc commented Feb 28, 2018

whyrusleeping Mar 1, 2018

whyrusleeping Mar 1, 2018

whyrusleeping Mar 1, 2018

whyrusleeping left a comment

vyzo left a comment

dirkmc commented Mar 2, 2018

vyzo commented Mar 2, 2018

Kubuxu left a comment •

edited

Loading

Stebalien Mar 22, 2018

dirkmc Mar 22, 2018

dirkmc Mar 22, 2018

Stebalien Mar 22, 2018

Kubuxu Mar 22, 2018

dirkmc Mar 22, 2018

whyrusleeping Mar 23, 2018

dirkmc Mar 23, 2018

dirkmc Mar 23, 2018

whyrusleeping Mar 23, 2018

whyrusleeping left a comment

Add options for record count and timeout for resolving DHT paths #4733

Add options for record count and timeout for resolving DHT paths #4733

Conversation

dirkmc commented Feb 23, 2018 • edited by Kubuxu Loading

dirkmc commented Feb 23, 2018

dirkmc commented Feb 27, 2018

Choose a reason for hiding this comment

vyzo left a comment

Choose a reason for hiding this comment

dirkmc commented Feb 28, 2018

vyzo commented Feb 28, 2018

dirkmc commented Feb 28, 2018

vyzo commented Feb 28, 2018

whyrusleeping commented Feb 28, 2018

dirkmc commented Feb 28, 2018

dirkmc commented Feb 28, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

whyrusleeping left a comment

Choose a reason for hiding this comment

vyzo left a comment

Choose a reason for hiding this comment

dirkmc commented Mar 2, 2018

vyzo commented Mar 2, 2018

Kubuxu left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

whyrusleeping left a comment

Choose a reason for hiding this comment

dirkmc commented Feb 23, 2018 •

edited by Kubuxu

Loading

Kubuxu left a comment •

edited

Loading