Add global option to specify the multibase encoding (server side)

[kevina]

Redo of #5464 but this time doing it (mostly) on the server side.

This may not be the most elegant implementation, but for now I think it is the most practical, espacally if we also want to support ?cid-base end points for API requests. We can clean this up later to make things more elegant.

It does all the work in the core/commands package and does not leak into other API's except for the ipfs filestore command where we need to pass an encoder function to FormatLong method.

See #5777 for an alternative client side implementation.

This adds support for --cid-base and --output-cidv1 option to the command line and HTTP API for the following commands:

ipfs add
ipfs files ls
ipfs files stat
ipfs ls
ipfs pin add
ipfs pin ls
ipfs pin rm
ipfs pin verify
ipfs refs
ipfs refs local
ipfs resolve
ipfs tar add
ipfs urlstore add
ipfs filestore dups
ipfs object get
ipfs object put
ipfs object links

Due to the fact that some commands return the CID itself and not a string, some command support the base conversion in text output only until we upgrade to something like refmt which will give us more control over the JSON output:

ipfs filestore ls
ipfs filestore verify
ipfs bitswap stat
ipfs bitswap wantlist
ipfs dag put
ipfs dag resolve

ipfs dag get currently does not support --cid-base due to the way it is implemented. This is considered a bug.

For some commands the original path string is preserved without any additional processing and is thus unaffected by --cid-base:

ipfs ls (only the original path displayed when multiple arguments are present, not the actual output) 
ipfs pin update

In addition all commands under bitswap, object, and dag do not, by default, auto-upgrade CIDv0 to CIDv1 as these commands are considered low-level and the conversion could cause problems (for example a dag will not survive a ipfs object get' / 'ipfs object put round trip. This can be changes by using --output-cidv1=true.

Todo:

• Remove cidenc.Interface for right now it doesn't serve a useful purpose and it may make more sense to define it to only have an Encode method (and no Recode method) so the existing cid.Encoder also implements the interface.
• Abandon the apicid concept all together as that will be a holdover from the client side implementation and move apicid.Cid to cidenc.Cid.
• Rebase to clean up commits and avoid depending on locally gx published versions of go-cidutil.
• Finalize --output-cidv1|--force-multibase-cids|--force-cid-base option name

Depends on ipfs/go-cidutil#10.

Closes #5233. Closes #5234. Closes #5349.

[magik6k]

I think there is a better way to integrate this using CoreAPI:

• Pick what to use for global options in CoreAPI - CoreApi.Offline #5654 (to be consistent)
• Define new CID interface:
  • Only for use in user-facing apis, internally we still want to use the string-backed version because performance
  • Instead of apicid.Hash we create implementation of this interface which keeps track of encoding
  • Calling .String on the encoding-tracking impl will get us the right base without having to remember to use cidenc everywhere
• We switch all coreapi methods to the new cid interface. This should be more contained as most of coreapi operates on path interface, which centralizes the change and makes it harder to miss places that care about encoding.
• Then we implement global coreapi option which allows user to specify encoding returned by default.
• If a coreapi method accepts path we use encoding from the given path instead of one defined by the global option
  • Or provide 2 global options, one allowing to override path encoding

@Stebalien @kevina Does that make sense?

[kevina]

@magik6k I think that perhaps that is something we can do later. As many of the command listed above have not been converted to the CoreAPI I don't see how it will help much for now. Also, I attempted to do something like you proposed (strong multibase encoding with CID) and received considerable push back from @Stebalien (see ipfs/go-cid#64 and the discussion in #5349). Although in all fairness I was proposing we use the interface type everywhere, if we limit the new CID type to the CoreAPI interface it may become more acceptable.

It sound to be like if we go with your proposal we will effectively be setting the base server side. Correct?

Above else, keeping this P.R. up to date has been a time sink. I would like to get something in as this is a component of the P0 base32 work and overdue by several months. My goal now is to simplify things as much as possible so this can get in quickly. We can work on a better interface later.

[eingenito]

@kevina what's the next steps for this PR? @magik6k do you have comments based on @kevina's response last comment? Do you think it's reasonable to merge based on the current implementation? Or do you want to kick it upstairs to @Stebalien to see what he thinks? If you guys want you might consider making time to chat amongst the three of you - Steven has a Calendly (which he's soon going to regret) available here: https://calendly.com/stebalien/30min/11-28-2018?back=1 so you can make time for a 15 minute sync conversation if that's useful.

[kevina]

@eingenito this p.r. is a change in direction from what me and @Stebalien agree on, the other p.r. #5777 is closer to the solution we discussed.

See the comments at the end of #5349 for more context.

I would like to get this p.r. pretty much as is and then if possible refine it later. I specially designed it to be an non-invasive as possible so that we will not be locked into a particular implementation.

We do need to agree on is the choice of flags --cid-base and --output-cidv1 and the fact that we want these command to effect the JSON output, however.

[magik6k]

Given the time constraints I think this is a decent solution.

We may also want:

• ipfs dag put - there is no way to specify encoding currently
• ipfs dag get - might be quite tricky, maybe as a separate PR

[Stebalien]

I'm actually quite impressed at how non-invasive you've been able to make this. This approach looks like it'll work quite well (baring unforeseen circumstances...).

[Stebalien]

@magik6k

dag get
dag get

See #5349 (comment) and #5349 (comment)

I think there is a better way to integrate this using CoreAPI:

I'd rather make the CoreAPI as clean as possible for libraries. That is, a go application using the CoreAPI as a library should also be able to use a local IPLD DAGService etc. without having to use multiple CID types.

---

@kevina is there any thing else that needs commenting on for me to unblock this?

[kevina]

@Stebalien see #5349 (comment).

[magik6k]

I'd rather make the CoreAPI as clean as possible for libraries. That is, a go application using the CoreAPI as a library should also be able to use a local IPLD DAGService etc. without having to use multiple CID types.

If we only apply this to exposed string hashes then yeah, this is fine. I don't think there is anything in CoreAPI that returns those, so 👍.

[kevina]

This is almost ready to go. I did some additional simplifications and updated the top-level description. Please reread it.

I will rebase to clean up my commits once near the end to let people who already started to review this a change to see what's new.

[kevina]

@Stebalien the test failures on Travis CI seam unrelated to my changes, and the shareness tests are passing on Cicle CI and Jenkins so we should be good to go test wise.

[kevina]

For the record, one of the reasons I objected to such a long name is in the command line help text, which is now (cut off at 80 cols):

USAGE
  ipfs - Global p2p merkle-dag filesystem.

SYNOPSIS
  ipfs [--config=<config> | -c] [--debug=<debug> | -D] [--help=<help>] [-h=<h>] 

OPTIONS

  -c,                      --config   string - Path to the configuration file to
  -D,                      --debug    bool   - Operate in debug mode.
  --help                              bool   - Show the full command help text.
  -h                                  bool   - Show a short version of the comma
  -L,                      --local    bool   - Run the command locally, instead 
  --offline                           bool   - Run the command offline.
  --api                               string - Use a specific API instance (defa
  --cid-base                          string - Multibase encoding used for versi
  --upgrade-cidv0-in-output           bool   - Upgrade version 0 to version 1 CI
  --enc,                   --encoding string - The encoding type the output shou
  --stream-channels                   bool   - Stream channel output.
  --timeout                           string - Set a global timeout on the comma

SUBCOMMANDS
...

Not a super big deal and something that we can perhaps fix later.

Also, for the record, my original objection to just --upgrade-cidv0 is that we might want to automatically upgrade CIDv0 pins to CIDv1, but now that I think of it, this is something that is best left to a config option as some sort of migration should be used to convert existing pins when this option is set.

But again, I don't think is is a big deal, especially since it won't be used much, and am more interested in getting this in, then anything else, so I am happy with --upgrade-cidv0-in-output.

[magik6k]

--upgrade-cidv0-in-output is a bit long, but since the alternatives weren't much shorter, so I'm ok with it.

Travis issue is #5845. Jenkins is another random fail. Also, needs a rebase.

[alanshaw]

I've not got strong feels on this one --upgrade-cidv0-in-output is long but descriptive and as previously observed is unlikely to be used often. 👍

[kevina]

@Stebalien let me know when you want to get this in and I will rebase just before.

[Stebalien]

@magik6k can you give this a ✔️?

Nvm. It probably doesn't need another review.

[Stebalien]

Mostly nits except the panic issue in CidEncoderFromPath. I'm find punting on the nits but they should be easy to fix.

I'll merge as soon as they're fixed and a rebase is done so rebase at will.

[Stebalien]

These functions should be named by what they do, not where they're used. GetLowLevelCidEncoder doesn't tell me how this is different from GetCidEncoder.

[kevina]

I hate naming, is it okay if we punt on this?

[kevina]

@Stebalien this should be good to go.

Not sure why t0080-repo.sh is failing on CircleCI, it works locally and with Jenkins.

[Stebalien]

(Everything passed, just not all at once. Jenkins is a not having a good day.)

[Stebalien]

🎆 🚀 🎆