Make Batch implement a NodeAdder interface #46

hsanjuan · 2018-10-22T14:31:35Z

See: * ipfs/go-unixfs#41 * ipfs/go-ipld-format#46 License: MIT Signed-off-by: Hector Sanjuan <[email protected]>

Stebalien

If we're implementing a DAGService, we need to behave like one. I can think of three ways to fix this:

Always flush before we perform a get/delete.
Remove queued blocks (complicated).
Split out a new NodeWriter interface.
Split out

Stebalien · 2018-10-23T19:06:34Z

batch.go

@@ -124,6 +126,39 @@ func (t *Batch) Add(nd Node) error {
 	return t.err
 }

+// Remove removes a node from the DAGService. This is a non-batching
+// operation.
+func (t *Batch) Remove(ctx context.Context, c cid.Cid) error {


This needs to remove the node from the queue (or flush the queue) and then remove the node.

Stebalien · 2018-10-23T19:06:56Z

batch.go

+
+// RemoveMany calls RemoveMany in the underlying DAGService.
+func (t *Batch) RemoveMany(ctx context.Context, nodes []cid.Cid) error {
+	return t.ds.RemoveMany(ctx, nodes)


Same, needs to remove from the queue.

Stebalien · 2018-10-23T19:07:04Z

batch.go

+
+// Get calls Get from the underlying DAGService.
+func (t *Batch) Get(ctx context.Context, c cid.Cid) (Node, error) {
+	return t.ds.Get(ctx, c)


Same. Needs to get from the queue.

Stebalien · 2018-10-23T19:07:10Z

batch.go

+
+// GetMany calls GetMany from the underlying DAGService.
+func (t *Batch) GetMany(ctx context.Context, nodes []cid.Cid) <-chan *NodeOption {
+	return t.ds.GetMany(ctx, nodes)


Stebalien · 2018-10-23T19:11:27Z

Personally, I think separating out a NodeWriter interface is probably the right way to go.

hsanjuan · 2018-10-23T23:39:02Z

@Stebalien you mean NodeWriter being an interface with a single Add method, and using NodeWriter directly in places which only Add() (importers) instead of a full featured DAGService (and Batch would implement it) ?

Stebalien · 2018-10-24T10:56:33Z

type NodeWriter interface {
    Add(context.Context, Node) error
    AddMany(context.Context, []Node) error
}

(we'd also need Batch to implement AddMany but that shouldn't be difficult).

We don't have to do it this way, I just don't want batch.Add(ctx, x); batch.Get(ctx, x.Cid()) to fail.

(as names, NodeWriter is even worse than NodeGetter...)

hsanjuan · 2018-10-24T11:34:11Z

So it's one more interface to the mix (NodeAdder ?) vs. having a suboptimal DAGService implementation for methods we don't use right now.

Before, doing Get on the DAGService while adding stuff with the batch-importers would cause the same problems (Get on a recently added node could fail it was still uncommitted).

I'm leaning towards just doing Commit before non-add operations and documenting that Batch does it this way and if someone ever have an interest in improving it can always be done, but I'm honestly not sure what's best (both work, both are easy to implement).

Stebalien · 2018-10-24T11:45:12Z

having a suboptimal DAGService implementation for methods we don't use right now.

No, having a broken DAGService implementation that sometimes works and sometimes doesn't. It doesn't matter if we're not using it, the issue is that it looks like it should work but won't.

Before, doing Get on the DAGService while adding stuff with the batch-importers would cause the same problems (Get on a recently added node could fail it was still uncommitted).

Yes, but that makes perfect sense. The "batch" is effectively a transaction. However, if we're going to claim that Batch implements DAGService, it needs to actually implement DAGService.

I'm leaning towards just doing Commit before non-add operations and documenting that Batch does it this way and if someone ever have an interest in improving it can always be done, but I'm honestly not sure what's best (both work, both are easy to implement).

We can, of course, do that. However, I really don't see the issue adding a new interface. It gives us a clean divide so applications can pick what they want to use:

NodeGetter - read-only
NodeAdder - append-only
DAGService - RW dag service

hsanjuan · 2018-10-24T12:31:17Z

No, having a broken DAGService...

I meant with the Commit before Get fix etc. It's suboptimal because it may make Gets really slow/defeat the purpose of batching.

However, I really don't see the issue adding a new interface

Ok i'll do that then

Stebalien · 2018-10-24T12:48:47Z

I meant with the Commit before Get fix etc. It's suboptimal because it may make Gets really slow/defeat the purpose of batching.

Ah, got it. Really, either way is fine with me as long as whatever we do is correct. However, this is really like the io.Reader/io.Writer split so I think it'll be fine.

hsanjuan · 2018-10-25T14:55:17Z

batch.go

+// commiting them as needed.
+func (t *Batch) AddMany(ctx context.Context, nodes []Node) error {
+	for _, n := range nodes {
+		err := t.Add(ctx, n)


Could potentially append all nodes directly but that might get us way over maxSize/maxNodes.

That shouldn't really be an issue. Going over max nodes/size should just trigger a flush (we don't want to buffer too much in memory).

Maybe we should just:

Make Add call AddMany`.

Move the main logic to AddMany (and process all nodes at once).

Stebalien

LGTM. We can mess with performance later. We should probably also be obeying the context but that'll be a bit tricky.

hsanjuan · 2018-10-25T17:27:29Z

Have a look to latest commit (cfe76e8) @Stebalien

I still need to update the gx tag before merging.

Stebalien · 2018-10-25T18:18:31Z

@hsanjuan LGTM.

This adds a NodeAdder interface (which partially implements a DAGService), and adjusts the Batch type to satisfy it.

hsanjuan self-assigned this Oct 22, 2018

hsanjuan requested review from Stebalien, magik6k and schomatis October 22, 2018 14:31

hsanjuan added a commit to ipfs/kubo that referenced this pull request Oct 22, 2018

Explicitally use Batch after removing it from importers

aa3f185

See: * ipfs/go-unixfs#41 * ipfs/go-ipld-format#46 License: MIT Signed-off-by: Hector Sanjuan <[email protected]>

hsanjuan mentioned this pull request Oct 22, 2018

Explicitally use BufferedDAG after removing Batch from importers ipfs/kubo#5626

Merged

6 tasks

ghost added the status/in-progress In progress label Oct 22, 2018

Stebalien requested changes Oct 23, 2018

View reviewed changes

hsanjuan changed the title ~~Make Batch implement a DAGService~~ Make Batch implement a NodeAdder interface Oct 25, 2018

hsanjuan commented Oct 25, 2018

View reviewed changes

Stebalien approved these changes Oct 25, 2018

View reviewed changes

hsanjuan force-pushed the fix/batchds branch from cfe76e8 to f48f39f Compare October 25, 2018 19:00

hsanjuan added 2 commits October 26, 2018 14:58

Add a NodeAdder interface and have Batch implement it

1b39e7b

This adds a NodeAdder interface (which partially implements a DAGService), and adjusts the Batch type to satisfy it.

gx publish 0.7.0

4f06ec0

hsanjuan force-pushed the fix/batchds branch from f48f39f to 4f06ec0 Compare October 26, 2018 12:58

hsanjuan merged commit c96181f into master Oct 26, 2018

ghost removed the status/in-progress In progress label Oct 26, 2018

hsanjuan deleted the fix/batchds branch October 26, 2018 13:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make Batch implement a NodeAdder interface #46

Make Batch implement a NodeAdder interface #46

hsanjuan commented Oct 22, 2018

Stebalien left a comment

Stebalien Oct 23, 2018

Stebalien Oct 23, 2018

Stebalien Oct 23, 2018

Stebalien Oct 23, 2018

Stebalien commented Oct 23, 2018

hsanjuan commented Oct 23, 2018

Stebalien commented Oct 24, 2018

hsanjuan commented Oct 24, 2018

Stebalien commented Oct 24, 2018

hsanjuan commented Oct 24, 2018

Stebalien commented Oct 24, 2018

hsanjuan Oct 25, 2018

Stebalien Oct 25, 2018

Stebalien left a comment

hsanjuan commented Oct 25, 2018

Stebalien commented Oct 25, 2018

Make Batch implement a NodeAdder interface #46

Make Batch implement a NodeAdder interface #46

Conversation

hsanjuan commented Oct 22, 2018

Stebalien left a comment

Choose a reason for hiding this comment

Stebalien Oct 23, 2018

Choose a reason for hiding this comment

Stebalien Oct 23, 2018

Choose a reason for hiding this comment

Stebalien Oct 23, 2018

Choose a reason for hiding this comment

Stebalien Oct 23, 2018

Choose a reason for hiding this comment

Stebalien commented Oct 23, 2018

hsanjuan commented Oct 23, 2018

Stebalien commented Oct 24, 2018

hsanjuan commented Oct 24, 2018

Stebalien commented Oct 24, 2018

hsanjuan commented Oct 24, 2018

Stebalien commented Oct 24, 2018

hsanjuan Oct 25, 2018

Choose a reason for hiding this comment

Stebalien Oct 25, 2018

Choose a reason for hiding this comment

Stebalien left a comment

Choose a reason for hiding this comment

hsanjuan commented Oct 25, 2018

Stebalien commented Oct 25, 2018