feat(actor): update go-hamt-ipld library to latest #3411

hannahhoward · 2019-09-11T22:46:12Z

Goals

Update Actor code to the latest go-hamt-ipld library, which supports fast CBOR decode/encode and also supports alternate bitwidths on HAMT.

Implementation

The main change from the previous version of go-ipld-hamt is the deferral of deserializing actual values stored in the HAMT. This means that when you lookup values, you need to pass a pointer to a value where raw CBOR is decoded into. Also, when iterating the values in a HAMT, it's neccesary to use a callback and a type so that deserialization happens properly.
go-hamt-ipld also now includes cbor-gen, which allows you to define fast-path deserialization/serialization for specific types. For the moment, it only applies to HAMT stuff but we could generate fast-path code for actor types as well.
This does not actually alter the bitwidth for the HAMT fanout -- that
will be a future PR.

For discussion

I've changed a couple methods that didn't have a lot of individual test coverage to begin with -- should that be added to get this merged?

See comment on lines below for the MakeGenisisFunc function -- I don't understand how this didn't fail the waiter_test in plumbing/msg before?

hannahhoward · 2019-09-12T21:29:04Z

consensus/genesis.go

@@ -169,6 +169,9 @@ func MakeGenesisFunc(opts ...GenOption) GenesisInitFunc {
 			if err = s.Commit(scid, a.Head); err != nil {
 				return nil, err
 			}
+			if err := st.SetActor(ctx, addr, a); err != nil {


Of particular note: I do not understand how this was working before -- the head gets committed, which updates the actor, but it isn't rewritten to the hamt -- it makes me think this was only working cause of in some memory caching? Not sure!

Presumably the problem is that the actor's Head is not getting updated in the state tree without this line. If this is the case, is line 160 still necessary? If so, we need a comment explaining why we're setting the actor twice.

anorth

Thanks. I don't like the deps on personal repos, but this didn't make the situation any worse, and we can address it later.

anorth · 2019-09-13T00:46:24Z

actor/builtin/paymentbroker/paymentbroker.go

@@ -177,7 +177,7 @@ func (pb *Actor) CreateChannel(vmctx exec.VMContext, target address.Address, eol

 	err := withPayerChannels(ctx, storage, payerAddress, func(byChannelID exec.Lookup) error {
 		// check to see if payment channel is duplicate
-		_, err := byChannelID.Find(ctx, channelID.KeyString())
+		err := byChannelID.Find(ctx, channelID.KeyString(), &PaymentChannel{})


In storagemarket.go you used nil for an unwanted out parameter. Do something consistent (nil seems fine if supported).

anorth · 2019-09-13T00:55:27Z

@hannahhoward your editor seems to be inserting unnecessary package aliases. I'd appreciate you figuring out how to avoid that.

acruikshank

This looks good. I have a couple question that may prompt minor modifications.

acruikshank · 2019-09-13T14:04:04Z

consensus/genesis.go

@@ -169,6 +169,9 @@ func MakeGenesisFunc(opts ...GenOption) GenesisInitFunc {
 			if err = s.Commit(scid, a.Head); err != nil {
 				return nil, err
 			}
+			if err := st.SetActor(ctx, addr, a); err != nil {


Presumably the problem is that the actor's Head is not getting updated in the state tree without this line. If this is the case, is line 160 still necessary? If so, we need a comment explaining why we're setting the actor twice.

acruikshank · 2019-09-13T14:04:43Z

state/tree.go

 	return &act, nil
 }

-func hackTransferObject(from, to interface{}) error {


acruikshank · 2019-09-13T14:11:25Z

vm/storage.go

-		nd, err = cbor.Decode(bytes, types.DefaultHashFunction, -1)
+	} else if raw, ok := v.([]byte); ok {
+		nd, err = cbor.Decode(raw, types.DefaultHashFunction, -1)
+	} else if cm, ok := v.(cbg.CBORMarshaler); ok {


Are we putting CBORMarshalers into storage? Is this new functionality? If so, and we aren't yet using it, we should wait to introduce this until it is exercised by code (and tests).

so a CBORMarshaler just means an object that conforms to that interface, meaning it has methods to fast path it's own serialization / deserialization -- this covers all hamt nodes, and in fact, they need to be fastpathed to encode properly. WrapObject won't work here because of the cbg.Deferred object. The actual solution is that go-ipld-cbor should check for fast path in its own calls to marshal/unmarshal, which there is an unmerged PR for: ipfs/go-ipld-cbor#64 . Absent that getting merged, this is neccesary for a couple direct calls to storage.Put

^^ @acruikshank

will add a comment to that effect.

frrist

@hannahhoward I will defer to whatever AC & AN suggest here. My feedback is primarily nits: combine if err :=, no unnecessary import renames, and a slight preference for select over else if in vm/storage.go.

Updates to the latest go-hamt-ipld library, which supports fast CBOR decode/encode and also supports alternate bitwidths on HAMT. This does not actually alter the bitwidth for the HAMT fanout -- that will be a future PR

hamt behavior was relying on actors being in memory

hannahhoward · 2019-09-16T21:12:22Z

@hannahhoward your editor seems to be inserting unnecessary package aliases. I'd appreciate you figuring out how to avoid that.

@anorth Hey so I finally tracked this down to not my editor but goimports, which does autoimporting, and specifically, this issue: golang/go#28428 & https://go-review.googlesource.com/c/tools/+/152000

They actually chose to add this feature and I honestly understand their reasoning -- the thought is even if the underlying package name is correct, if there's a mismatch with the filename there's no way to determine that it is without digging into the underlying source code of the file, so having the rename allows the imports of the file to be validated just by looking at the file itself.

Though also I am curious if I am the only one who uses goimports and if others have had the same issue.

Anyway, I only say this cause goimports is a pretty standard go tool, though I guess I am the only one who uses it on this project. But I'll turn it off and just do manual imports.

- remove unneccesary imports - delete set line - add comment

codecov-io · 2019-09-16T22:59:20Z

Codecov Report

Merging #3411 into master will decrease coverage by 22%.
The diff coverage is 15%.

@@           Coverage Diff            @@
##           master   #3411     +/-   ##
========================================
- Coverage      48%     26%    -23%     
========================================
  Files         230     230             
  Lines       14362   14296     -66     
========================================
- Hits         6959    3765   -3194     
- Misses       6375    9900   +3525     
+ Partials     1028     631    -397

hannahhoward force-pushed the feat/use-updated-ipld-hamt branch from 2854b12 to 31ca841 Compare September 12, 2019 21:25

hannahhoward commented Sep 12, 2019

View reviewed changes

hannahhoward requested review from acruikshank, anorth and frrist September 12, 2019 21:29

anorth approved these changes Sep 13, 2019

View reviewed changes

acruikshank approved these changes Sep 13, 2019

View reviewed changes

frrist reviewed Sep 13, 2019

View reviewed changes

hannahhoward added 2 commits September 16, 2019 12:02

feat(actor): update go-hamt-ipld library to latest

5323938

Updates to the latest go-hamt-ipld library, which supports fast CBOR decode/encode and also supports alternate bitwidths on HAMT. This does not actually alter the bitwidth for the HAMT fanout -- that will be a future PR

fix(consensus): rewrite hamt for actor state change

576762f

hamt behavior was relying on actors being in memory

refactor(deps): minor changes per PR comments

beb73c9

- remove unneccesary imports - delete set line - add comment

hannahhoward force-pushed the feat/use-updated-ipld-hamt branch from 31ca841 to beb73c9 Compare September 16, 2019 22:45

chore(deps): switched to tagged version

6f6b4a0

hannahhoward merged commit d756bd4 into master Sep 16, 2019

ZenGround0 deleted the feat/use-updated-ipld-hamt branch October 29, 2019 14:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(actor): update go-hamt-ipld library to latest #3411

feat(actor): update go-hamt-ipld library to latest #3411

hannahhoward commented Sep 11, 2019 •

edited

Loading

hannahhoward Sep 12, 2019

acruikshank Sep 13, 2019

anorth left a comment

anorth Sep 13, 2019

anorth commented Sep 13, 2019

acruikshank left a comment

acruikshank Sep 13, 2019

acruikshank Sep 13, 2019

acruikshank Sep 13, 2019

hannahhoward Sep 16, 2019

hannahhoward Sep 16, 2019

hannahhoward Sep 16, 2019

frrist left a comment

hannahhoward commented Sep 16, 2019 •

edited

Loading

codecov-io commented Sep 16, 2019

feat(actor): update go-hamt-ipld library to latest #3411

feat(actor): update go-hamt-ipld library to latest #3411

Conversation

hannahhoward commented Sep 11, 2019 • edited Loading

Goals

Implementation

For discussion

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anorth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anorth commented Sep 13, 2019

acruikshank left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

frrist left a comment

Choose a reason for hiding this comment

hannahhoward commented Sep 16, 2019 • edited Loading

codecov-io commented Sep 16, 2019

Codecov Report

hannahhoward commented Sep 11, 2019 •

edited

Loading

hannahhoward commented Sep 16, 2019 •

edited

Loading