refactor: require migration, remove path factory #1389

b5 · 2020-06-18T16:13:01Z

Put migration code to use, marking the migration as required before qri will start. This migration does a few things:

runs ipfs repo migration to support go-ipfs v0.5.0
converts ipfs configuration to use non-standard IPFS ports
moves the IPFS repo into the qri folder
adds a "Filesystems" configuration field

Qri now only honors QRI_PATH by default. Drop cmd.EnvPathFactory in favour of qri path functions & string values.

closes #1296, closes #1319, closes #1346

This commit puts migration code to use, marking the migration as required before qri will start. This migration does a few things: * runs ipfs repo migration to support go-ipfs v0.5.0 * converts ipfs configuration to use non-standard IPFS ports * moves the IPFS repo into the qri folder * adds a "Filesystems" configuration field Qri now only honors "QRI_PATH" by default. Drop cmd.EnvPathFactory in favour of qri path functions & string values.

feep · 2020-06-19T04:20:49Z

config/migrate/migrate.go

+			msg := `Your repo needs updating before qri can start. 
+Run migration now?`
+			if !confirm(streams.Out, streams.In, msg) {
+				return qerr.New(ErrNeedMigration, `your repo requires migration before it can run`)


Something clearer.

'Your repo needs to be migrated before any qri commands can be run.'

Or, from line 25:

'Your repo needs updating before qri can start.'

They should probably say the same thing.

Or maybe ErrNeedMigration should be more explainy since they answered ‘no’ to the simple explanation. Maybe also a linefeed before.

What it looks like currently:

❯ qri list Your repo needs updating before qri can start. Run migration now? [Y/n]: n your repo requires migration before it can run

feep · 2020-06-19T05:17:52Z

Notes here.

The error for network failure is a little ugly.

Probably not worth doing anything about it right now. @ me if you want me to file an issue.

Arqu · 2020-06-19T19:55:45Z

I've got this:

migrating configuration...done!
constructing "ipfs" filesystem: ipfs: need datastore migration

Also got:

$> qri list
getting store from filesystem: resolver of kind 'ipfs' does not exist on this filesystem

On a fresh pre-migration qri setup (ie just did qri setup and then updated)

b5 · 2020-06-22T15:53:55Z

here's what I suspect happened @Arqu:

my guess: you had a config file with a revision value of 2. The migration checked that file, assumed it was done, and moved on without actually doing anything. Mind setting config.yaml's revision to 1 & running any command like qri list?

Also got: getting store from filesystem: resolver of kind 'ipfs' does not exist on this filesystem
On a fresh pre-migration qri setup (ie just did qri setup and then updated)

This is a different (more concerning) issue, I'm going to try to reproduce now.

b5 · 2020-06-22T16:11:24Z

Ok, first effort to reproduce looks like it's working ok. Here's (roughly) what I ran:

$ export IPFS_PATH=/Users/b5/migration/test/ipfs
$ export QRI_PATH=/Users/b5/migration/test/qri
$ export PWD="$(pwd)"

# build from master@HEAD:
$ cd $QRI_REPO && git checkout 474151 && go install && cd $PWD

$ qri setup
choose a peername: [ruby_red_grand_basset_griffon_vendeen]:
set up qri repo at: /Users/b5/datasets/migration_test/qri

# build post-migration:
$ cd $QRI_REPO && git checkout 393473 && go install && cd $PWD

$ qri list
Your repo needs updating before qri can start.
Run migration now? [Y/n]:
migrating configuration...
  => Looking for suitable fs-repo-migrations binary.
  => None found, downloading.
  => Running: /var/folders/6p/sjn8bct92397mbgwmwwvcb4h0000gn/T/go-ipfs-migrate766176553/fs-repo-migrations -to 9 -y
Found fs-repo version 7 at /var/folders/6p/sjn8bct92397mbgwmwwvcb4h0000gn/T/ipfs_temp036298354
===> Running migration 7 to 8...
applying 7-to-8 repo migration
locking repo at "/var/folders/6p/sjn8bct92397mbgwmwwvcb4h0000gn/T/ipfs_temp036298354"
  - verifying version is '7'
> Upgrading config to new format
updated version file
===> Migration 7 to 8 succeeded!
===> Running migration 8 to 9...
applying 8-to-9 repo migration
updated version file
===> Migration 8 to 9 succeeded!
  => Success: fs-repo has been migrated to version 9.
done!
you have no datasets

Arqu · 2020-06-22T16:24:17Z

Yup, changing the config revision to 1 and re-doing the migration did the trick.

dustmop

There's a big move away from the term "repoPath" here. Is that motivated by deemphasizing the repo package? Because I think referring to the place that stores block data, logbook, refs, config, etc as the "repository" is still useful as a piece of terminology. In fact, it's even more useful once "ipfs/" lives inside "qri/", since then there's no confusion over whether "repo" refers to "qri/" or "ipfs/" or the parent of one or both of these.

While QRI_PATH as an environment variable makes sense - it matches of a standard convention of specifying configuration and data - out of that context, seeing "qriPath" as a method name is clear obvious. It may not be too confusing within this isolated change, but for example, in Desktop it could mean "path to the qri binary", which would become tricky when working on the "qri cli" and "Desktop" at the same time.

dustmop · 2020-06-22T18:25:17Z

cmd/qri.go

@@ -36,8 +34,7 @@ https://github.com/qri-io/qri/issues`,
 	cmd.SetUsageTemplate(rootUsageTemplate)
 	cmd.PersistentFlags().BoolVarP(&opt.NoPrompt, "no-prompt", "", false, "disable all interactive prompts")
 	cmd.PersistentFlags().BoolVarP(&opt.NoColor, "no-color", "", false, "disable colorized output")
-	cmd.PersistentFlags().StringVar(&opt.RepoPath, "repo", qriPath, "provide a path to load qri from")
-	cmd.PersistentFlags().StringVar(&opt.IpfsPath, "ipfs-path", ipfsPath, "override IPFS path location")
+	cmd.PersistentFlags().StringVar(&opt.qriPath, "path", qriPath, "provide a path to load qri from")


This description and flag name seems too generic and/or confusing. This sounds like it could be used to load an executable or framework or something. And path doesn't narrow down what is happening. What if we use qri-path and describe it as "path where qri data and config lives"

dustmop · 2020-06-22T18:29:33Z

cmd/qri.go

@@ -162,6 +155,11 @@ func (o *QriOptions) Instance() *lib.Instance {
 	return o.inst
 }

+// QriPath returns the path to the qri directory


Would prefer some more specificity. This could mean the "qri directory" where the binary is installed, or where Desktop exists, or where I store my datasets. How about "QriPath returns the path where qri data and configuration exist"

dustmop · 2020-06-22T18:41:36Z

lib/lib.go

@@ -291,7 +287,7 @@ func NewInstance(ctx context.Context, repoPath string, opts ...Option) (qri *Ins
 		opts = []Option{
 			OptStdIOStreams(),
 			OptSetIPFSPath(""),
-			OptCheckConfigMigrations(),
+			OptCheckConfigMigrations(false),


What are your thoughts on changing the migration UI such that after migrations run successfully the qri binary exits with a message like "migration successful, please run command again". As it is now, if I run qri list then I get the migration prompt, confirm, see some messages about progress, then soon after the pager pops up and obscures what happened. I think it would be fine to bail out inside and ask the user to start over, it's less cognitive load than having to remember what was going on before the migration started.

"migration successful, please run command again"

fully agree. Ok if I do this as a follow-up PR?

nm, managed to fit this in

Yes totally

dustmop · 2020-06-22T18:43:06Z

repo/test/temp_repo.go

@@ -51,8 +51,14 @@ func newTempRepo(peername, prefix string, g gen.CryptoGenerator) (r TempRepo, er
 		return r, err
 	}

+	// Create directory for new Qri repo.
+	QriPath := filepath.Join(RootPath, "qri")


lower-case this var

Hmmm, why is RootPath upper-case?

uhhh... dunno. will fix

dustmop · 2020-06-22T19:01:25Z

cmd/factory.go

-		return filepath.Join(dir, "qri"), filepath.Join(dir, "ipfs")
-	}
+// RootedQriPath gives a "/qri" directory from a given root path
+func RootedQriPath(root string) string {


This is only used in one place, might as well remove it and inline at the call site.

Arqu

Besides the existing comments mostly LGTM. I just have a few more to make sure we don't slip.

Arqu · 2020-06-22T18:57:14Z

config/config.go

+		{
+			Type: "ipfs",
+			Config: map[string]interface{}{
+				"path": "./ipfs",


Is this path "correct"? In the sense that I'm not sure what it's relative to and also not sure whether this works well on windows.

really good point about windows. At a minimum we should audit to make sure this value is always generated with file path.Join(cfg.Path(), "ipfs").

Regarding "correct", this is what we said in RFC0029:

Relative Paths in configuration files are relative to the file
If a configuration value specifies a filepath, and the given path is relative, that path should be relative to the location of the config file. We should add a note of this to $ qri config --help.

Arqu · 2020-06-22T18:59:01Z

config/migrate/migrate.go

@@ -87,7 +108,6 @@ func OneToTwo(cfg *config.Config) error {
 	}

 	// TODO(ramfox): remove original ipfs repo after all migrations were successful


Is this handled now? If yes we should remove the comment, if not, think this definitely needs to get in.
Also thinking about it now, what happens if users use ipfs for other stuff too? Should we or should we not remove it?

It's not yet, and I agree we need to figure out an answer here. whatever we come up with, I'd like to handle it in a separate PR

Works for me, it's safe for the end user even in this state, just that we waste some disk space currently.

Arqu · 2020-06-22T19:02:08Z

lib/lib.go

@@ -291,7 +287,7 @@ func NewInstance(ctx context.Context, repoPath string, opts ...Option) (qri *Ins
 		opts = []Option{
 			OptStdIOStreams(),
 			OptSetIPFSPath(""),
-			OptCheckConfigMigrations(),
+			OptCheckConfigMigrations(false),


b5 · 2020-06-22T19:38:37Z

There's a big move away from the term "repoPath" here. Is that motivated by deemphasizing the repo package? Because I think referring to the place that stores block data, logbook, refs, config, etc as the "repository" is still useful as a piece of terminology. In fact, it's even more useful once "ipfs/" lives inside "qri/", since then there's no confusion over whether "repo" refers to "qri/" or "ipfs/" or the parent of one or both of these.

Definitely motivated by de-emphasizing the term "repo". I was hoping to make the change in terminology incrementally. After reading your note I agree this PR doesn't even address that change in the notes. which isn't good. If we're going to drop the "repo" term, we should do it explicitly. I'll revert changes away from the repoPath term

dustmop

LGTM, one comment about the "return successfully after migration".

dustmop · 2020-06-23T01:15:12Z

cmd/cmd.go

@@ -60,6 +61,9 @@ func Execute() {
 // ErrExit writes an error to the given io.Writer & exits
 func ErrExit(w io.Writer, err error) {
 	log.Debug(err.Error())
+	if errors.Is(err, migrate.ErrMigrationSucceeded) {
+		printSuccess(w, "migration succeeded, re-run your command to continue")


Add os.Exit(0) here? Otherwise looks like the err gets printed again. Also, return code should be 0 on success. Though I'm not sure if "the original command didn't run" should count as success?

damn nice catch. I vote we exit with status 0 (success), a thing did succeed, and only because the user chose Yes. I've intentionally carved around a non interactive version of this so we can write qri connect --migrate which will auto-migrate and continue

this way users don't get super confused. restore "RepoPath" to describe qri data directory

b5 added the refactor A code change that neither fixes a bug nor adds a feature label Jun 18, 2020

b5 self-assigned this Jun 18, 2020

b5 force-pushed the feat-enable-migration branch 2 times, most recently from b471d00 to 647d019 Compare June 19, 2020 01:15

b5 force-pushed the feat-enable-migration branch from 647d019 to 7863cd6 Compare June 19, 2020 01:28

b5 changed the title ~~WIP: enable migration~~ refactor: require migration, remove path factory Jun 19, 2020

b5 requested a review from ramfox June 19, 2020 01:29

b5 marked this pull request as ready for review June 19, 2020 01:29

b5 requested review from Arqu, chriswhong and feep June 19, 2020 01:30

feep reviewed Jun 19, 2020

View reviewed changes

feep approved these changes Jun 19, 2020

View reviewed changes

b5 requested review from feep and dustmop June 19, 2020 14:48

fix(migrate): don't copy entire home directory if IPFS_PATH isn't set

393473b

b5 mentioned this pull request Jun 22, 2020

refactor(store): drop Store from config, muxfs owns cafs.Filestore #1393

Merged

b5 added this to the v0.9.9 milestone Jun 22, 2020

b5 mentioned this pull request Jun 22, 2020

feat(config): set up config migration w/ OneToTwo func #1332

Closed

dustmop reviewed Jun 22, 2020

View reviewed changes

Arqu mentioned this pull request Jun 22, 2020

Migrating IPFS and config #1376

Closed

9 tasks

dustmop reviewed Jun 22, 2020

View reviewed changes

Arqu reviewed Jun 22, 2020

View reviewed changes

b5 force-pushed the feat-enable-migration branch from 63dcf97 to a4b8868 Compare June 23, 2020 00:52

dustmop approved these changes Jun 23, 2020

View reviewed changes

refactor(cmd): exit after successful migration without running command

4aafc9c

this way users don't get super confused. restore "RepoPath" to describe qri data directory

b5 force-pushed the feat-enable-migration branch from a4b8868 to 4aafc9c Compare June 23, 2020 01:23

b5 merged commit 5430405 into migration-rev2 Jun 23, 2020

b5 deleted the feat-enable-migration branch June 23, 2020 01:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor: require migration, remove path factory #1389

refactor: require migration, remove path factory #1389

b5 commented Jun 18, 2020 •

edited

Loading

feep Jun 19, 2020 •

edited

Loading

feep commented Jun 19, 2020

Arqu commented Jun 19, 2020 •

edited

Loading

b5 commented Jun 22, 2020

b5 commented Jun 22, 2020

Arqu commented Jun 22, 2020

dustmop left a comment

dustmop Jun 22, 2020

dustmop Jun 22, 2020

dustmop Jun 22, 2020

Arqu Jun 22, 2020

b5 Jun 22, 2020 •

edited

Loading

b5 Jun 23, 2020

dustmop Jun 23, 2020

dustmop Jun 22, 2020

b5 Jun 22, 2020

dustmop Jun 22, 2020

Arqu left a comment

Arqu Jun 22, 2020

b5 Jun 22, 2020

Arqu Jun 22, 2020

b5 Jun 22, 2020

Arqu Jun 22, 2020

Arqu Jun 22, 2020

b5 commented Jun 22, 2020

dustmop left a comment

dustmop Jun 23, 2020

b5 Jun 23, 2020

		@@ -87,7 +108,6 @@ func OneToTwo(cfg *config.Config) error {
		}

		// TODO(ramfox): remove original ipfs repo after all migrations were successful

refactor: require migration, remove path factory #1389

refactor: require migration, remove path factory #1389

Conversation

b5 commented Jun 18, 2020 • edited Loading

feep Jun 19, 2020 • edited Loading

Choose a reason for hiding this comment

feep commented Jun 19, 2020

Arqu commented Jun 19, 2020 • edited Loading

b5 commented Jun 22, 2020

b5 commented Jun 22, 2020

Arqu commented Jun 22, 2020

dustmop left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

b5 Jun 22, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Arqu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

b5 commented Jun 22, 2020

dustmop left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

b5 commented Jun 18, 2020 •

edited

Loading

feep Jun 19, 2020 •

edited

Loading

Arqu commented Jun 19, 2020 •

edited

Loading

b5 Jun 22, 2020 •

edited

Loading