update for the go-ipfs-cmds refactor #5035

keks · 2018-05-24T11:33:00Z

This branch contains changes to go-ipfs so we can use it with this branch of go-ipfs-cmds: ipfs/go-ipfs-cmds#112

keks · 2018-08-22T10:55:13Z

@Stebalien sharness test t0272-urlstore.shfails and I'm not sure how to fix it.

The issue is that after adding a file through the urlstore, the subsequent get call will successfully fetch everything and then error with failed to fetch all nodes, so it seems like some if the DAG nodes are not properly added. However, the resulting file matches the file that was added (i.e. file3 matches file3.actual in the trash directory). I traced the error through the code and it is returned by res.Emit(reader), where reader is a *"unixfs/io".PBDagReader.

The furthest I have traced the error is that it is returned by a promise in the PBDagReader, and now I'm somewhat lost because that can only return either an error or the data, and we have all the data, so where does the error come from? I was hoping it was some race condition and the nodes are not written yet when the call to get comes in, but sleeping for a second before doing that doesn't help either.

I'm pretty puzzled and meandering through unfamiliar code, so I thought you might have an idea where that issue might come from.

Stebalien · 2018-08-22T23:12:48Z

I'm pretty puzzled and meandering through unfamiliar code, so I thought you might have an idea where that issue might come from.

The answer is: not your bug, your patch just makes us correctly report errors. I'll look into it.

keks · 2018-08-23T16:12:00Z

That did the trick, thanks! 🌈

keks · 2018-08-27T17:02:37Z

@Stebalien I removed the remaining debug output and rebased to master. This is now ready for review.

Stebalien · 2018-08-28T01:39:26Z

cmd/ipfs/daemon.go

 	// Inject metrics before we do anything
 	err := mprome.Inject()
 	if err != nil {
-		log.Errorf("Injecting prometheus handler for metrics failed with message: %s\n", err.Error())
+		return fmt.Errorf("Injecting prometheus handler for metrics failed with message %s", err.Error())


Don't want to return here. This can fail for various reasons.

Stebalien · 2018-08-28T01:45:41Z

commands/legacy/response.go

-	if ch, ok := out.(chan interface{}); ok {
-		out = (<-chan interface{})(ch)
+	// don't emit nil or Single{nil}
+	if out == nil || out == (cmds.Single{Value: nil}) {


Is not emitting nil correct?

The reason I did this is that I had some command sometimes emit a nil at the beginning. This led to the http response emitter call the preamble with nil, which led to wrong state in the http response. I don't see a reason to actually send nils, so I figured it's easiest to just ignore them. I don't remember which command that was, though.

Do you think this is wrong?

I'm worried we should be closing the output (if we're emitting a Single). However, this is probably fine (we'll close anyways).

Stebalien · 2018-08-28T01:53:25Z

core/commands/files.go


 		_, err := statGetFormatOptions(req)
 		if err != nil {
-			res.SetError(err, cmdkit.ErrClient)
+			// REVIEW NOTE: We didn't return here before, was that correct?
+			return cmdkit.Errorf(cmdkit.ErrClient, err.Error())


Looks like a bug.

Stebalien · 2018-08-28T01:56:28Z

core/commands/files.go

 		}

 		defer func() {
 			err := wfd.Close()
 			if err != nil {
-				re.SetError(err, cmdkit.ErrNormal)
+				re.CloseWithError(cmdkit.Errorf(cmdkit.ErrNormal, err.Error()))


Be careful not to swallow any errors that may have caused this condition. We should probably:

Use a named return value.

Store the close error in, e.g., err2.

Replace the returned error if it's nil.

Sound reasonable?

I think so. Is this what you had in mind?

Stebalien · 2018-08-28T01:59:37Z

core/commands/get.go

+			if !ok {
+				// TODO or just return the error here?
+				log.Error(e.New(e.TypeErr(outReader, v)))
+				return nil


Return the error (IMO).

Stebalien · 2018-08-28T02:02:41Z

core/commands/stat.go

 			}
 			select {
 			case <-time.After(interval):
 			case <-req.Context.Done():
-				return
+				break


return req.Context.Err()?

That would return an error and non-zero exit code to the user, even though everything was in order. Do you think this is a good idea?

Ah, I see. You're right.

Stebalien · 2018-08-28T02:09:49Z

core/commands/urlstore.go

 		}

-		cmds.EmitOnce(res, BlockStat{
+		err = cmds.EmitOnce(res, &BlockStat{


Can return directly.

Stebalien · 2018-08-28T02:10:14Z

test/sharness/lib/test-lib.sh

@@ -26,6 +26,7 @@ fi
 # it's too late to pass in --verbose, and --verbose is harder
 # to pass through in some cases.
 test "$TEST_VERBOSE" = 1 && verbose=t
+test "$TEST_IMMEDIATE" = 1 && immediate=t


Oh yeah, if you want we can remove this, but in that case I'd open a new PR for this.

When running single tests, I usually run them with ./txxxx-foo-bar.sh -v -i, which results in the test aborting immediately after an error. This makes finding the failing test in a large test file a lot easier. For some reason this behaviour could not be set using the env, meaning that it could not be used when invoked with make. This line allows enabling this mode using TEST_IMMEDIATE=1 make.

Got it, sounds useful, let's keep it (although we should probably document it in the sharness readme).

Stebalien · 2018-09-12T17:17:08Z

@keks: This has been unblocked. Once this is finished, we can merge ipfs/go-ipfs-cmds#112 and ship it.

keks · 2018-09-17T16:40:28Z

@Stebalien The sharness test t0252 consistently fails:

expecting success: 
  ipfs refs $ADIR_HASH &&
  ipfs repo gc &&
  ipfs refs $ADIR_HASH

Error: failed to fetch all nodes
not ok 6 - gc okay after adding incomplete node
#	
#	  ipfs refs $ADIR_HASH &&
#	  ipfs repo gc &&
#	  ipfs refs $ADIR_HASH

I can't tell whether this is because of something else of my changes. I don't really see a connection. Also, sometimes the robust repo gc (t0087) fails, but I got used to that one.

Stebalien · 2018-09-17T18:44:08Z

The test was broken. I've fixed it.

keks · 2018-09-18T13:04:57Z

core/commands/filestore.go

-					}
+				r, ok := v.(*filestore.ListRes)
+				if !ok {
+					// TODO or just return that error? why didn't we do that before?


@Stebalien What do you think should we do here?

Yes. I'm not sure why we didn't.

Stebalien

LGTM modulo the gc change.

Stebalien · 2018-09-18T18:16:01Z

core/commands/repo.go

@@ -99,7 +99,7 @@ order to reclaim hard disk space.
 					}
 				}
 				if errs {
-					res.SetError(fmt.Errorf("encountered errors during gc run"), cmdkit.ErrNormal)
+					outChan <- &GcResult{Error: "encountered errors during gc run"}


I think we actually want to return the error here. We spit out errors as we encounter them but this is supposed to cause the command to exit with an exit status of 1.

We should probably just switch this command away from the legacy commands system. Then we can drop the goroutine and just emit the values normally.

Yeah, not sure what I was thinking here...

Stebalien · 2018-09-18T18:24:33Z

Fixing the gc command should fix travis. I'm not sure why jenkins didn't fail...

keks · 2018-09-19T10:49:45Z

Fixing the gc command should fix travis. I'm not sure why jenkins didn't fail...

As far as I can see the issue is that it doesn't report all gc errors (some of the greps fail). That doesn't mean we shouldn't use SetError for the final error (we should), but I believe there is another error here, but I doubt it's in the commands library. I also had that test fail in other branches when reviewing PRs, so I'm confident it has nothing to do with this change set. I just ran the test with the race detector on and it didn't report anything 🤷‍♂️

keks · 2018-09-19T12:14:10Z

Looks like making repo gc use cmds 2 fixed the instability. @Stebalien I think I addressed all your concerns, want to merge this?

Stebalien · 2018-09-19T18:28:48Z

@keks LGTM but I'd like @magik6k to take a look.

@magik6k Any feedback and/or objections to these changes? I've done a pretty thorough code review (don't feel like you need to do a line-by-line audit) but I'd like to get a final sign-off from you before we actually merge this into go-ipfs.

Summary:

We now return errors instead of using SetError. This alone ends up removing a bunch of code.
We've removed support for multiple "error" responses. This was never really properly supported in, e.g., javascript, but go technically supported it.
We've made a bunch of low-level changes to reduce race conditions in the commands lib (esp around error handling).

excerpt of commit messages: - update postrun functions in core/commands - sharness: allow setting -i with TEST_IMMEDIATE=1 - cmds Run func returns error now - gx update cmdkit to 1.1.2 and cmds to 2.0.0-beta1 License: MIT Signed-off-by: keks <[email protected]>

License: MIT Signed-off-by: keks <[email protected]>

`ipfs refs` won't work because we don't have the referenced objects (ipfs refs fetches everything it prints). License: MIT Signed-off-by: Steven Allen <[email protected]>

License: MIT Signed-off-by: keks <[email protected]>

License: MIT Signed-off-by: Steven Allen <[email protected]>

Stebalien · 2018-09-19T22:33:06Z

🎉 🎉 🎉

keks · 2018-09-19T22:35:19Z

🎉 🎊

update for the go-ipfs-cmds refactor This commit was moved from ipfs/kubo@3538257

keks requested a review from Kubuxu as a code owner May 24, 2018 11:33

ghost assigned keks May 24, 2018

ghost added the status/in-progress In progress label May 24, 2018

keks force-pushed the feat/cmds2 branch 7 times, most recently from 8d7381a to 2daf766 Compare May 25, 2018 11:14

keks mentioned this pull request May 25, 2018

Bundle of changes ipfs/go-ipfs-cmds#112

Merged

keks force-pushed the feat/cmds2 branch from 91b1be3 to 4a3ebf4 Compare June 30, 2018 13:54

keks force-pushed the feat/cmds2 branch 2 times, most recently from 473884a to 9cd9bd4 Compare August 16, 2018 16:05

keks force-pushed the feat/cmds2 branch 4 times, most recently from d1e73de to 57e7b62 Compare August 27, 2018 16:53

Stebalien reviewed Aug 28, 2018

View reviewed changes

keks force-pushed the feat/cmds2 branch from 57e7b62 to 5bb074e Compare September 17, 2018 14:12

ghost assigned Stebalien Sep 17, 2018

keks commented Sep 18, 2018

View reviewed changes

Stebalien requested changes Sep 18, 2018

View reviewed changes

keks force-pushed the feat/cmds2 branch from 1eb7ddf to 95b5ec8 Compare September 19, 2018 10:56

Stebalien changed the title ~~[WIP] branch to track and test several changes to cmds library~~ branch to track and test several changes to cmds library Sep 19, 2018

Stebalien approved these changes Sep 19, 2018

View reviewed changes

Stebalien requested a review from magik6k September 19, 2018 18:24

Stebalien changed the title ~~branch to track and test several changes to cmds library~~ update for the go-ipfs-cmds refactor Sep 19, 2018

magik6k approved these changes Sep 19, 2018

View reviewed changes

keks and others added 10 commits September 19, 2018 14:35

big squash commit

908ffdd

excerpt of commit messages: - update postrun functions in core/commands - sharness: allow setting -i with TEST_IMMEDIATE=1 - cmds Run func returns error now - gx update cmdkit to 1.1.2 and cmds to 2.0.0-beta1 License: MIT Signed-off-by: keks <[email protected]>

update go-ipfs-cmds to 2.0.0-beta2

ef21c3b

License: MIT Signed-off-by: keks <[email protected]>

remove debug output

d14021d

License: MIT Signed-off-by: keks <[email protected]>

update cmds to 2.0.0-rc1

f4b32e3

License: MIT Signed-off-by: keks <[email protected]>

fix broken gc test

08cc855

`ipfs refs` won't work because we don't have the referenced objects (ipfs refs fetches everything it prints). License: MIT Signed-off-by: Steven Allen <[email protected]>

address stebalien's comments

74ec1a9

License: MIT Signed-off-by: keks <[email protected]>

fixes

caf855b

License: MIT Signed-off-by: keks <[email protected]>

repo gc: use cmds2

aa7494b

License: MIT Signed-off-by: keks <[email protected]>

document TEST_IMMEDIATE in test/sharness/README.md

480184f

License: MIT Signed-off-by: keks <[email protected]>

gx: update go-ipfs-cmds to the final release

b7484c1

License: MIT Signed-off-by: Steven Allen <[email protected]>

Stebalien force-pushed the feat/cmds2 branch from 0ae5413 to b7484c1 Compare September 19, 2018 21:36

fix resolve command for the new cmds refactor

d556ee2

License: MIT Signed-off-by: Steven Allen <[email protected]>

Stebalien merged commit 3538257 into master Sep 19, 2018

ghost removed the status/in-progress In progress label Sep 19, 2018

Stebalien deleted the feat/cmds2 branch September 19, 2018 22:30

kevina mentioned this pull request Sep 21, 2018

fix(pin): add select for channel handling #5480

Closed

hacdias pushed a commit to ipfs/boxo that referenced this pull request Jan 27, 2023

Merge pull request ipfs/kubo#5035 from ipfs/feat/cmds2

e9fb982

update for the go-ipfs-cmds refactor This commit was moved from ipfs/kubo@3538257

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

update for the go-ipfs-cmds refactor #5035

update for the go-ipfs-cmds refactor #5035

keks commented May 24, 2018

keks commented Aug 22, 2018

Stebalien commented Aug 22, 2018

keks commented Aug 23, 2018

keks commented Aug 27, 2018

Stebalien Aug 28, 2018

Stebalien Aug 28, 2018

keks Sep 18, 2018

Stebalien Sep 18, 2018

Stebalien Aug 28, 2018

Stebalien Aug 28, 2018

keks Sep 18, 2018

Stebalien Aug 28, 2018

Stebalien Aug 28, 2018

keks Sep 18, 2018

Stebalien Sep 18, 2018

Stebalien Aug 28, 2018

Stebalien Aug 28, 2018

keks Sep 18, 2018

Stebalien Sep 18, 2018

Stebalien commented Sep 12, 2018

keks commented Sep 17, 2018 •

edited

Loading

Stebalien commented Sep 17, 2018

keks Sep 18, 2018

Stebalien Sep 18, 2018

Stebalien left a comment

Stebalien Sep 18, 2018

keks Sep 19, 2018

Stebalien commented Sep 18, 2018

keks commented Sep 19, 2018

keks commented Sep 19, 2018

Stebalien commented Sep 19, 2018

Stebalien commented Sep 19, 2018

keks commented Sep 19, 2018

update for the go-ipfs-cmds refactor #5035

update for the go-ipfs-cmds refactor #5035

Conversation

keks commented May 24, 2018

keks commented Aug 22, 2018

Stebalien commented Aug 22, 2018

keks commented Aug 23, 2018

keks commented Aug 27, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Stebalien commented Sep 12, 2018

keks commented Sep 17, 2018 • edited Loading

Stebalien commented Sep 17, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Stebalien left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Stebalien commented Sep 18, 2018

keks commented Sep 19, 2018

keks commented Sep 19, 2018

Stebalien commented Sep 19, 2018

Stebalien commented Sep 19, 2018

keks commented Sep 19, 2018

keks commented Sep 17, 2018 •

edited

Loading