Update gogo protobuf dependency

[whyrusleeping]

We havent updated our gogo-protobuf dep in a while. The primary reason for this is that its not as simple to do as a lot of our other deps. The repo itself is huge, and our current gogo-protobuf package is roughly 40MB. This make the gx install ux slow and really annoying.

We should look at repackaging a newer version with exactly what we need, or finding a better way to shrink this package to just what we need to make the process of keeping things updated easier.

cc @Kubuxu @parkan @vyzo

[vyzo]

This has escalated into a serious problem for us, because the vendored gogo-protobuf apparently generates code that can't marshall/unmarshall oneofs.

Is there any workaround with gx to make it not rewrite the gogo-protobuf dependency in our tree? We don't explicitly depend on the gx vendored package, but it gets rewritten because of the transitive dependency through libp2p.

[Kubuxu]

The huge part of gogo-protobuf are tests (there is tests that has 400k lines). We could add them to .gxingnore.

[Kubuxu]

Also it looks like we are using only /proto and /io.

[Kubuxu]

Removing /test form the gx export will leave us with about 1MiB code package.

[Kubuxu]

@vyzo you can try importing QmVsHZSaohNEZrzNuBPi9z63iiJxWc2rvN4PZKqJXQ9oU5 which is current master of gogo/protobuf but I have no idea how it will work (there might be backward incompatible changes).

[vyzo]

@Kubuxu thanks, the explicit import of this new version works with protos compiled with unvendored gogo-protobuf!

[vyzo]

And needless to say, it solves our marshaling woes :)

[Kubuxu]

The process became just a bit more complicated, as gogo protobuf fixed bug with order of values.

Version of protobuf used by us includes the bug, current protobuf 3 version is fixed.

Meaning that if we update the protobuf library hashes of new files will change.

[Kubuxu]

gogo/protobuf#42

[ribasushi]

Was browsing around and ran across this issue.

Since you are about start producing new hashes anyway ( due to the different order of protobuf "frames" ) may I suggest re-declaring repeated varint fields as packed? According to the spec packed fields are forward and backward compatible...

This will save 1~2 bytes per unixfs link (and likely for other higher protocols too)

[Kubuxu]

We won't be changing output hashes just to update protobuf Also as far as I know, we won't be touching protobuf-unixfs format. We plan deprecating it, and maybe creating ipld-unixfs in version 0.5 but it isn't still decided, AFAIK.

@whyrusleeping might know more about this.

[ribasushi]

@Kubuxu I meant the above suggestion given that hashes will change as per your own writeup here: #3214 (comment)

[Kubuxu]

As we don't want to change the hashes, we can't update for this new version of gogo-protobuf.

But we should update gogo-protobuf to a version with test vectors removed. It is 19MiB of 29MiB package, we probably could remove some other packages from it too.

@lgierth what is the status of the gx-worktree tool? Could it be used to painlessly create series of those PRs?

[Stebalien]

I'm currently working on this.

Plan:

• Publish a new version of gogo-prootbuf.
  • QmdxUuburamoF6zF9qjeQC4WYcWGbWuRmdLacMEsW8ioD8
• Update gogo for go-merkledag and go-unixfs, making sure we don't change the hashes. In go-merkeldag, this involves making sure we don't re-generate the marshaler code (and making sure we actually use it).
• File PRs against:
  • go-bitswap
    • update gogo protobuf go-bitswap#2
    • Merged
  • go-ipfs-routing
    • update protobuf go-ipfs-routing#8
    • Merged
  • go-ipns
    • update gogo protobuf go-ipns#16
    • Merged
  • go-libp2p-circuit
    • update gogo protobuf libp2p/go-libp2p-circuit#43
    • Merged
  • go-libp2p-crypto
    • update gogo protobuf libp2p/go-libp2p-crypto#37
    • Merged
  • go-libp2p-floodsub
    • update gogo protobuf libp2p/go-libp2p-pubsub#94
    • Merged
  • go-libp2p-kad-dht
    • upgrade protobuf and switch to bytes keys libp2p/go-libp2p-kad-dht#183
    • Merged
  • go-libp2p
    • update gogo protobuf libp2p/go-libp2p#390
    • Merged
  • go-libp2p-record
    • update gogo protobuf libp2p/go-libp2p-record#18
    • Merged
  • go-libp2p-secio
    • update gogo protobuf libp2p/go-libp2p-secio#29
    • Merged
  • go-log
    • update gogo protobuf go-log#39
    • Merged
  • go-multicodec
    • update gogo protobuf and regenerate multiformats/go-multicodec#32
    • Merged
• Merge PRs
• Gx-publish new versions
• Run a gx-workspace update: gx-workspace update start go-bitswap go-ipfs-routing go-ipns go-libp2p-circuit go-libp2p-crypto go-libp2p-floodsub go-libp2p-kad-dht go-libp2p go-libp2p-record go-libp2p-secio go-log go-merkledag go-multicodec go-unixfs.
• File a PR against go-ipfs.

[mib-kd743naq]

this involves making sure we don't re-generate the marshaler code

@Stebalien this seems like a suboptimal plan. What you want is to update gogo, apply the patch reverting the sorting change, and publish that in gx-go. Then you do regenerate the marshallers, taking advantage of improvements in the generated code. Perpahs even switching to the faster flavor ( no carriage of unrecognized field values )

The only reason I did not reopen/finish #5012 is because I ran out of time trying to figure out how gx-go works, and then never came back to it :(

[Stebalien]

Then you do regenerate the marshallers, taking advantage of improvements in the generated code. Perpahs even switching to the faster flavor ( no carriage of unrecognized field values )

We should do that as well however, we want all of this to work without gx. That's why I'm taking this approach (use upstream as much as possible).

[Stebalien]

The sharness tests pass. Running the interop tests now.

[Stebalien]

The interop tests pass (those that usually do, at least).