create a database copying script

[Pomax]

No related issue: this adds a new script to our repo to make database copying easier, as well as two new invoke commands, inv copy-stage-db and inv copy-prod-db, with associated docs updates.

This does require you have the Heroku CLI utility installed, so that we don't need to start stage/prod credentials anywhere, and can just pull them from heroku automatically, provided you have access to the staging and prod apps.

(If you need to install the heroku CLI first, remember to run heroku login before running this script, which will kick off a browser based login process)

[fessehaye]

Got the following error

$ inv copy-prod-db
←[1;37mcd C:\Users\Simon\Documents\GitHub\foundation.mozilla.org && node copy-db.js --prod←[0m
 »   Warning: heroku update available from 7.53.0 to 7.59.0.
heroku: Press any key to open up the browser to login or q to exit: C:\Users\Simon\Documents\GitHub\foundation.mozilla.org\copy-db.js:17
const HEROKU_TEXT = HEROKU_OUTPUT.toString().replaceAll(`\n`, ` `);
                                             ^

TypeError: HEROKU_OUTPUT.toString(...).replaceAll is not a function
    at Object.<anonymous> (C:\Users\Simon\Documents\GitHub\foundation.mozilla.org\copy-db.js:17:46)
    at Module._compile (internal/modules/cjs/loader.js:1138:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1158:10)
    at Module.load (internal/modules/cjs/loader.js:986:32)
    at Function.Module._load (internal/modules/cjs/loader.js:879:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
    at internal/main/run_main_module.js:17:47
    ```

[TheoChevalier]

Nice :) It worked here, just had a first failure, probably because docker was still running something. I made sure to shut everything down in Docker, then tried again and it worked 👍

Tested with:
Heroku 7.59
Node 16.10.0
Docker 4.0.1
macOS 11.6

Sharing the initial error just as an FYI:

theochevalier@MBP-2018:foundation.mozilla.org (sync-inline-fields=)$ inv copy-stage-db
cd /Users/theochevalier/Documents/GitHub/foundation.mozilla.org && node copy-db.js
Running db copy for staging, run with --prod to run for production
Starting postgres docker image...
Starting backend docker image...
Downloading foundation-mofostaging-net database (this may take a while)...
Resetting db...
dropdb: error: database removal failed: ERROR:  database "wagtail" is being accessed by other users
DETAIL:  There is 1 other session using the database.
/Users/theochevalier/Documents/GitHub/foundation.mozilla.org/copy-db.js:32
    throw e;
    ^

Error: Command failed: docker exec foundation_postgres_1 dropdb -hpostgres -Ufoundation --if-exists wagtail
dropdb: error: database removal failed: ERROR:  database "wagtail" is being accessed by other users
DETAIL:  There is 1 other session using the database.

    at checkExecSyncError (node:child_process:826:11)
    at execSync (node:child_process:900:15)
    at run (/Users/theochevalier/Documents/GitHub/foundation.mozilla.org/copy-db.js:29:12)
    at docker (/Users/theochevalier/Documents/GitHub/foundation.mozilla.org/copy-db.js:41:10)
    at Object.<anonymous> (/Users/theochevalier/Documents/GitHub/foundation.mozilla.org/copy-db.js:56:1)
    at Module._compile (node:internal/modules/cjs/loader:1101:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:79:12) {
  status: 1,
  signal: null,
  output: [
    null,
    Buffer(0) [Uint8Array] [],
    Buffer(154) [Uint8Array] [
      100, 114, 111, 112, 100,  98,  58,  32, 101, 114, 114, 111,
      114,  58,  32, 100,  97, 116,  97,  98,  97, 115, 101,  32,
      114, 101, 109, 111, 118,  97, 108,  32, 102,  97, 105, 108,
      101, 100,  58,  32,  69,  82,  82,  79,  82,  58,  32,  32,
      100,  97, 116,  97,  98,  97, 115, 101,  32,  34, 119,  97,
      103, 116,  97, 105, 108,  34,  32, 105, 115,  32,  98, 101,
      105, 110, 103,  32,  97,  99,  99, 101, 115, 115, 101, 100,
       32,  98, 121,  32, 111, 116, 104, 101, 114,  32, 117, 115,
      101, 114, 115,  10,
      ... 54 more items
    ]
  ],
  pid: 80654,
  stdout: Buffer(0) [Uint8Array] [],
  stderr: Buffer(154) [Uint8Array] [
    100, 114, 111, 112, 100,  98,  58,  32, 101, 114, 114, 111,
    114,  58,  32, 100,  97, 116,  97,  98,  97, 115, 101,  32,
    114, 101, 109, 111, 118,  97, 108,  32, 102,  97, 105, 108,
    101, 100,  58,  32,  69,  82,  82,  79,  82,  58,  32,  32,
    100,  97, 116,  97,  98,  97, 115, 101,  32,  34, 119,  97,
    103, 116,  97, 105, 108,  34,  32, 105, 115,  32,  98, 101,
    105, 110, 103,  32,  97,  99,  99, 101, 115, 115, 101, 100,
     32,  98, 121,  32, 111, 116, 104, 101, 114,  32, 117, 115,
    101, 114, 115,  10,
    ... 54 more items
  ]
}

[Pomax]

@fessehaye you need to be logged into the heroku toolchain, so remember to run heroku login (you should only need to do this once after installing the CLI utility) before you run the database copy commands.

[Pomax]

@TheoChevalier good point, let me update the docs for now to remind folks to make sure nothing dockerish is running before issueing this command, because it does need exclusive db access since it needs to literally destroy the existing db as part of the copy.

[TheoChevalier]

Looks good, thanks for creating that!

noted

[danielfmiranda]

Hi @Pomax!

I am getting an error as well:

Error: No such container: foundation_postgres_1
/Users/daniel/Development/foundation.mozilla.org/copy-db.js:32
    throw e;
    ^

Error: Command failed: docker exec foundation_postgres_1 dropdb -hpostgres -Ufoundation --if-exists wagtail
Error: No such container: foundation_postgres_1

    at checkExecSyncError (node:child_process:690:11)
    at execSync (node:child_process:727:15)
    at run (/Users/daniel/Development/foundation.mozilla.org/copy-db.js:29:12)
    at docker (/Users/daniel/Development/foundation.mozilla.org/copy-db.js:41:10)
    at Object.<anonymous> (/Users/daniel/Development/foundation.mozilla.org/copy-db.js:56:1)
    at Module._compile (node:internal/modules/cjs/loader:1092:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1121:10)
    at Module.load (node:internal/modules/cjs/loader:972:32)
    at Function.Module._load (node:internal/modules/cjs/loader:813:14)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:76:12) {
  status: 1,
  signal: null,
  output: [
    null,
    Buffer(0) [Uint8Array] [],
    Buffer(48) [Uint8Array] [
       69, 114, 114, 111, 114,  58,  32,  78, 111,
       32, 115, 117,  99, 104,  32,  99, 111, 110,
      116,  97, 105, 110, 101, 114,  58,  32, 102,
      111, 117, 110, 100,  97, 116, 105, 111, 110,
       95, 112, 111, 115, 116, 103, 114, 101, 115,
       95,  49,  10
    ]
  ],
  pid: 13758,
  stdout: Buffer(0) [Uint8Array] [],
  stderr: Buffer(48) [Uint8Array] [
     69, 114, 114, 111, 114,  58,  32,  78, 111,
     32, 115, 117,  99, 104,  32,  99, 111, 110,
    116,  97, 105, 110, 101, 114,  58,  32, 102,
    111, 117, 110, 100,  97, 116, 105, 111, 110,
     95, 112, 111, 115, 116, 103, 114, 101, 115,
     95,  49,  10
  ]

I have tried to run inv new-env to see if it will rename my docker container to "foundation_postgres_1" but it appears that it continues to name the docker container to "foundationmozillaorg_postgres_1".

I also tried to see if there was an error on my part by checking the docker files and stuff but cannot seem to find the root of the issue

[Pomax]

I've added an explicit docker-compose down at the start of the script to force the issue, can you check if that solves things @danielfmiranda?

Also remember to use the "request changes" review function, so that the PR state gets updated.

[danielfmiranda]

Hi @Pomax apologies for the delayed response!

I pulled the latest rendition of this branch and gave it another go. However it appears that the initial error regarding the name of the container still persists:

Running db copy for staging, run with --prod to run for production
Making sure no docker containers are running...
Starting postgres docker image...
Starting backend docker image...
Downloading foundation-mofostaging-net database (this may take a while)...
Resetting db...

Error: No such container: foundation_postgres_1
/Users/daniel/Development/foundation.mozilla.org/copy-db.js:32
    throw e;
    ^

Error: Command failed: docker exec foundation_postgres_1 dropdb -hpostgres -Ufoundation --if-exists wagtail
Error: No such container: foundation_postgres_1

    at checkExecSyncError (node:child_process:690:11)
    at execSync (node:child_process:727:15)
    at run (/Users/daniel/Development/foundation.mozilla.org/copy-db.js:29:12)
    at docker (/Users/daniel/Development/foundation.mozilla.org/copy-db.js:41:10)
    at Object.<anonymous> (/Users/daniel/Development/foundation.mozilla.org/copy-db.js:60:1)
    at Module._compile (node:internal/modules/cjs/loader:1092:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1121:10)
    at Module.load (node:internal/modules/cjs/loader:972:32)
    at Function.Module._load (node:internal/modules/cjs/loader:813:14)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:76:12) {
  status: 1,
  signal: null,
  output: [
    null,
    Buffer(0) [Uint8Array] [],
    Buffer(48) [Uint8Array] [
       69, 114, 114, 111, 114,  58,  32,  78, 111,
       32, 115, 117,  99, 104,  32,  99, 111, 110,
      116,  97, 105, 110, 101, 114,  58,  32, 102,
      111, 117, 110, 100,  97, 116, 105, 111, 110,
       95, 112, 111, 115, 116, 103, 114, 101, 115,
       95,  49,  10
    ]
  ],
  pid: 42597,
  stdout: Buffer(0) [Uint8Array] [],
  stderr: Buffer(48) [Uint8Array] [
     69, 114, 114, 111, 114,  58,  32,  78, 111,
     32, 115, 117,  99, 104,  32,  99, 111, 110,
    116,  97, 105, 110, 101, 114,  58,  32, 102,
    111, 117, 110, 100,  97, 116, 105, 111, 110,
     95, 112, 111, 115, 116, 103, 114, 101, 115,
     95,  49,  10
  ]
}

I then ran docker ps -a to look at all the containers and found that my containers were being named foundationmozillaorg_postgres.

So I changed the name in the script files to see if that would help but then got this other error in return:

    throw e;
    ^

Error: Command failed: docker exec foundationmozillaorg_postgres_1 pg_restore -hpostgres -Ufoundation -dwagtail vsccktmymxzgoc.db.archive
pg_restore: error: input file is too short (read 0, expected 5)

    at checkExecSyncError (node:child_process:690:11)
    at execSync (node:child_process:727:15)
    at run (/Users/daniel/Development/foundation.mozilla.org/copy-db.js:29:12)
    at docker (/Users/daniel/Development/foundation.mozilla.org/copy-db.js:41:10)
    at Object.<anonymous> (/Users/daniel/Development/foundation.mozilla.org/copy-db.js:70:1)
    at Module._compile (node:internal/modules/cjs/loader:1092:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1121:10)
    at Module.load (node:internal/modules/cjs/loader:972:32)
    at Function.Module._load (node:internal/modules/cjs/loader:813:14)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:76:12) {
  status: 1,
  signal: null,
  output: [
    null,
    Buffer(0) [Uint8Array] [],
    Buffer(64) [Uint8Array] [
      112, 103,  95, 114, 101, 115, 116, 111, 114, 101,  58,
       32, 101, 114, 114, 111, 114,  58,  32, 105, 110, 112,
      117, 116,  32, 102, 105, 108, 101,  32, 105, 115,  32,
      116, 111, 111,  32, 115, 104, 111, 114, 116,  32,  40,
      114, 101,  97, 100,  32,  48,  44,  32, 101, 120, 112,
      101,  99, 116, 101, 100,  32,  53,  41,  10
    ]
  ],
  pid: 42708,
  stdout: Buffer(0) [Uint8Array] [],
  stderr: Buffer(64) [Uint8Array] [
    112, 103,  95, 114, 101, 115, 116, 111, 114, 101,  58,
     32, 101, 114, 114, 111, 114,  58,  32, 105, 110, 112,
    117, 116,  32, 102, 105, 108, 101,  32, 105, 115,  32,
    116, 111, 111,  32, 115, 104, 111, 114, 116,  32,  40,
    114, 101,  97, 100,  32,  48,  44,  32, 101, 120, 112,
    101,  99, 116, 101, 100,  32,  53,  41,  10
  ]
}

Please let me know if you need any more info, thanks!

[danielfmiranda]

Please see comments thread for error details

[danielfmiranda]

Hi Pomax!

With the new updates, the process got further than before and seemed to have downloaded the staging DB. However, a little further into the process I have recieved this error message. I also tried a second time by running "inv new-env" and then "inv copy-stage-db" and recieved the same error:

Downloading foundation-mofostaging-net database (this may take a while)...
Resetting db...
Building user roles...
Importing snapshot...
no such directory

/Users/daniel/Development/foundation.mozilla.org/copy-db.js:32
    throw e;
    ^

Error: Command failed: docker cp vsccktmymxzgoc.db.archive foundation_postgres_1:/
no such directory

    at checkExecSyncError (node:child_process:690:11)
    at execSync (node:child_process:727:15)
    at run (/Users/daniel/Development/foundation.mozilla.org/copy-db.js:29:12)
    at Object.<anonymous> (/Users/daniel/Development/foundation.mozilla.org/copy-db.js:69:1)
    at Module._compile (node:internal/modules/cjs/loader:1092:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1121:10)
    at Module.load (node:internal/modules/cjs/loader:972:32)
    at Function.Module._load (node:internal/modules/cjs/loader:813:14)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:76:12)
    at node:internal/main/run_main_module:17:47 {
  status: 1,
  signal: null,
  output: [
    null,
    Buffer(0) [Uint8Array] [],
    Buffer(18) [Uint8Array] [
      110, 111,  32, 115, 117,  99,
      104,  32, 100, 105, 114, 101,
       99, 116, 111, 114, 121,  10
    ]
  ],
  pid: 44587,
  stdout: Buffer(0) [Uint8Array] [],
  stderr: Buffer(18) [Uint8Array] [
    110, 111,  32, 115, 117,  99,
    104,  32, 100, 105, 114, 101,
     99, 116, 111, 114, 121,  10
  ]
}

[danielfmiranda]

Hi Pomax! Thanks for taking the time to resolve the errors I was having.

I have tested the command now and can confirm it works! Approving :D

[github-actions]

This PR introduces visual differences. Click here to inspect the diffs.