Use --format=json in sql:sync

[greg-1-anderson]

Use format json instead of format string when fetching database credentials in sql:sync

[weitzman]

LGTM as long as its been tested against Drush 9.5 and Drush 8.

[larowlan]

I tried to test this, remote drush is 8 (global) but we also have a project-relative drush 9.
It dies, hard - but no harder than it currently does - I think we're going to have to remove the global drush (which would be my preference anyway).

[greg-1-anderson]

It should be safe to go from Drush 9 (local) -> global Drush 8 (remote) -> site local Drush 9.

Removing the global Drush 8 is an option, but the inconvenience here is that you need to set %drush-script in all of your aliases.

[greg-1-anderson]

@larowlan What is the output of the sql:sync with --debug?

[larowlan]

  The command "ssh -o PasswordAuthentication=no {redacted}@{redacted} 'drush core
  -status db-name --format=json --root=/var/www/{redacted}/current/app'" failed.

  Exit Code: 127(Command not found)

  Working directory: /data

Note I have - drush-script: /var/www/{redacted}/current/vendor/drush/drush/drush in my alias but is looks its calling with just 'drush' which doesn't resolve on the other host

[greg-1-anderson]

@larowlan Try:

live:
  paths:
    drush-script: /path/to/drupal/vendor/bin/drush

It's also okay to use /vendor/drush/drush/drush, but I'd say vendor/bin is preferable, as it does not assume the location of the executable in the sources.

[larowlan]

spoke too soon
it got further - here's the output

[notice] Starting to dump database on source. [20.08 sec, 10.35 MB]
 [info] Executing: ssh -o PasswordAuthentication=no redacted@redacted'/var/www/redacted/current/vendor/drush/drush/drush sql-dump --strict=0 --gzip --result-file=auto --backend=json --root=/var/www/redacted/current/app' [20.09 sec, 10.35 MB]
Enter passphrase for key '/root/.ssh/sf_id':
 [info] Undefined index: log backend.inc:286 [162.44 sec, 10.54 MB]
 [info] Undefined index: output backend.inc:306 [162.44 sec, 10.54 MB]
 [notice] Starting to discover temporary files directory on target. [162.44 sec, 10.51 MB]
 [info] Executing: /data/vendor/drush/drush/drush core-status drush-temp --format=string --uri=default --root=/data/app [162.45 sec, 10.51 MB]
 [notice] Copying dump file from source to target. [163.58 sec, 10.51 MB]
 [info] Executing: /data/vendor/drush/drush/drush core-rsync @self.dev:/home/deployer/drush-backups/redacted/20190227020908/redacted.sql.gz @self:/tmp/redacted.sql.gz --uri=default --root=/data/app -- --remove-source-files [163.58 sec, 10.52 MB]
>
> In HostPath.php line 88:
>
>   Site alias @self.dev not found.
>
>
> core:rsync [--exclude-paths EXCLUDE-PATHS] [--include-paths INCLUDE-PATHS] [--mode [MODE]] [-h|--help] [-q|--quiet] [-v|vv|vvv|--verbose] [-V|--version] [--ansi] [--no-ansi] [-n|--no-interaction] [-d|--debug] [-y|--yes] [--no] [--remote-host REMOTE-HOST] [--remote-user REMOTE-USER] [-r|--root ROOT] [-l|--uri URI] [--simulate] [--pipe] [-D|--define DEFINE] [--] <command> <source> <target> [<extra>]...
>

In Process.php line 239:

  [Symfony\Component\Process\Exception\ProcessFailedException]
  The command "/data/vendor/drush/drush/drush core-rsync @self.dev:/home/deployer/drush-backups
  /redacted/20190227020908/redacted.sql.gz @self:/tmp/redacted.sql.gz
   --uri=default --root=/data/app -- --remove-source-files" failed.

  Exit Code: 1(General error)

  Working directory:

  Output:
  ================


  Error Output:
  ================

  In HostPath.php line 88:

    Site alias @self.dev not found.


  core:rsync [--exclude-paths EXCLUDE-PATHS] [--include-paths INCLUDE-PATHS] [--mode [MODE]] [-
  h|--help] [-q|--quiet] [-v|vv|vvv|--verbose] [-V|--version] [--ansi] [--no-ansi] [-n|--no-int
  eraction] [-d|--debug] [-y|--yes] [--no] [--remote-host REMOTE-HOST] [--remote-user REMOTE-US
  ER] [-r|--root ROOT] [-l|--uri URI] [--simulate] [--pipe] [-D|--define DEFINE] [--] <command>
   <source> <target> [<extra>]...

[greg-1-anderson]

--backend=json?! What version of Drush are you running locally again? That looks more like Drush 8 than Drush 9, but I don't think either version of Drush should say --backend=json. Drush 8 should say --backend=2 and Drush 9 should say --format=json. Something seems off here.

[weitzman]

Thats actually how Drush9 does it. See

drush/src/Commands/sql/SqlSyncCommands.php

Lines 140 to 144 in 9dacc08

	$this->logger()->notice(dt('Starting to dump database on source.'));
	// Set --backend=json. Drush 9.6+ changes that to --format=json. See \Drush\Preflight\PreflightArgs::setBackend.
	// Drush 9.5- handles this as --backend.
	$process = $this->processManager()->drush($sourceRecord, 'sql-dump', [], $dump_options + ['backend' => 'json']);
	$process->mustRun();

[larowlan]

the local is this branch, remote is now 9.5.2

[greg-1-anderson]

OK I forgot all about --backend=json. I'm going to have to dig into this a little deeper when I have a chance.

[larowlan]

No hurry @greg-1-anderson i've got a workaround in our Makefile that uses ssh/rsync etc

Happy to keep lending a hand testing

[greg-1-anderson]

@larowlan : Looked at this a bit more. I was able to successfully run an sql-sync from a Drupal site with a site-local Drush 9.5 to a Drupal site with a site-local Drush 9.6 when both sites were local. It appears that the bug you are seeing is limited to remote-source sites with Drush 9.5 (or earlier).

Your error says "Site alias @self.dev not found." What is your sql-sync commandline? Does it include @self.dev? It looks like @self.dev is a remote site, and @self of course is the local site. Is that right?

[larowlan]

I was running drush sql:sync @dev @self which somehow ended up as @self.dev in the later commands

[greg-1-anderson]

@larowlan Do you have your @dev alias defined in self.sites.yml?

[larowlan]

Yes

…

On Wed, Mar 6, 2019, 2:44 PM Greg Anderson ***@***.***> wrote: @larowlan <https://github.com/larowlan> Do you have your @dev alias defined in self.sites.yml? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3936 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAh49s7VTaID39jHzSmKXyuqES3nYxWnks5vT0eogaJpZM4a_ufe> .

[greg-1-anderson]

@larowlan I think I was able to reproduce and fix the problem preventing @self.dev from being found. Could you try the latest commit and see if it works for you? Please be sure to run composer install after updating the branch to ensure you get the site-alias dependency.

[weitzman]

clever ... change this before merge?

[greg-1-anderson]

Yes, I want feedback on this PR before I merge the site-alias PR, but I will merge and make a beta2 release before I merge here.

[weitzman]

still tests PRs before they merge?

[greg-1-anderson]

Yes. Without the branch limitation, pull requests are tested twice.

[greg-1-anderson]

Tested this and it seems to be working fine. I will merge this after I make the stable releases of site-alias 3.0.0 and site-process 2.0.0, probably tomorrow morning.

[greg-1-anderson]

Released another beta of site-alias