diff --git a/documentation/automated_recovery.md b/documentation/automated_recovery.md deleted file mode 100644 index 08afe0fb..00000000 --- a/documentation/automated_recovery.md +++ /dev/null @@ -1,73 +0,0 @@ -# Recovery procedures - -These instructions provide automated procedures for recovering from select failures of PE components which are managed by PEADM. - -Manual procedures are documented in [recovery.md](recovery.md) - -## Recover from failed primary Puppet server - -1. Promote the replica ([official docs](https://puppet.com/docs/pe/2019.8/dr_configure.html#dr-promote-replica)) -2. [Replace missing or failed replica Puppet server](#replace-missing-or-failed-replica-puppet-server) - -## Replace missing or failed replica Puppet server - -This procedure uses the following placeholder references. - -* _\_ - The FQDN and certname of the primary Puppet server -* _\_ - The FQDN and certname of the PE-PostgreSQL server which resides in the same availability group as the replacement replica Puppet server -* _\_ - The FQDN and certname of the replacement replica Puppet server - -1. Run `peadm::add_replica` plan to deploy replacement replica Puppet server - 1. For Standard and Large deployments - - bolt plan run peadm::add_replica primary_host= replica_host= - - 2. For Extra Large deployments - - bolt plan run peadm::add_replica primary_host= replica_host= replica_postgresql_host= - -## Replace failed PE-PostgreSQL server (A or B side) - -The procedure for replacing a failed PE-PostgreSQL server is the same regardless of which PE-PostgreSQL server is missing or if the name of the PE-PostgrSQL server is the same or different. This procedure uses the following placeholder references. - -* _\_ - The FQDN and certname of the new server being brought in to replace the failed PE-PostgreSQL server -* _\_ - The FQDN and certname of the still-working PE-PostgreSQL server -* _\_ - The FQDN and certname of the failed PE-PostgreSQL server -* _\_ - The FQDN and certname of the primary Puppet server -* _\_ - The FQDN and certname of the replica Puppet server - -Procedure: - -1. Stop `puppet.service` on Puppet server primary and replica - - bolt task run service name=puppet.service action=stop --targets , - -2. Temporarily set both primary and replica server nodes so that they use the remaining healthy PE-PostgreSQL server - - bolt plan run peadm::util::update_db_setting --target , postgresql_host= override=true - -3. Restart `pe-puppetdb.service` on Puppet server primary and replica - - bolt task run service name=pe-puppetdb.service action=restart --targets , - -4. Purge failed PE-PostgreSQL node from PuppetDB - - bolt command run "/opt/puppetlabs/bin/puppet node purge " --targets - -5. Run `peadm::add_database` plan to deploy replacement PE-PostgreSQL server - - bolt plan run peadm::add_database -t primary_host= - -## Replace failed replica puppet server AND failed replica pe-postgresql server - -This procedure uses the following placeholder references. - -* _\_ - The FQDN and certname of the primary Puppet server -* _\_ - The FQDN and certname of the failed replica Puppet server - -1. Ensure the old replica server is forgotten. - - bolt command run "/opt/puppetlabs/bin/puppet infrastructure forget " --targets - -2. [Replace failed PE-PostgreSQL server (A or B side)](#replace-failed-pe-postgresql-server-a-or-b-side) -3. [Replace missing or failed replica Puppet server](#replace-missing-or-failed-replica-puppet-server) diff --git a/documentation/recovery/automated_recovery.md b/documentation/recovery/automated_recovery.md new file mode 100644 index 00000000..01ebec55 --- /dev/null +++ b/documentation/recovery/automated_recovery.md @@ -0,0 +1,95 @@ +# Recovery procedures + +These instructions provide automated procedures for recovering from select failures of PE components which are managed by PEADM. + +Manual procedures are documented in [recovery.md](recovery.md) + +## Recover from failed Puppet primary server + +1. Promote the replica ([official docs](https://puppet.com/docs/pe/2019.8/dr_configure.html#dr-promote-replica)) +2. [Replace missing or failed replica Puppet primary server](#replace-missing-or-failed-replica-puppet-server) + +## Replace missing or failed replica Puppet primary server + +This procedure uses the following placeholder references. + +* _\_ - The FQDN and certname of the Puppet primary server +* _\_ - The FQDN and certname of the PE-PostgreSQL server which resides in the same availability group as the replacement replica Puppet primary server +* _\_ - The FQDN and certname of the replacement replica Puppet primary server + +1. Run `peadm::add_replica` plan to deploy replacement replica Puppet primary server + 1. For Standard and Large deployments: + + bolt plan run peadm::add_replica primary_host= replica_host= + + 2. For Extra Large deployments: + + bolt plan run peadm::add_replica primary_host= replica_host= replica_postgresql_host= + +## Replace failed PE-PostgreSQL server (A or B side) + +The procedure for replacing a failed PE-PostgreSQL server is the same regardless of which PE-PostgreSQL server is missing or whether the name of the PE-PostgreSQL server is the same or different. This procedure uses the following placeholder references. + +* _\_ - The FQDN and certname of the new server being brought in to replace the failed PE-PostgreSQL server +* _\_ - The FQDN and certname of the still-working PE-PostgreSQL server +* _\_ - The FQDN and certname of the failed PE-PostgreSQL server +* _\_ - The FQDN and certname of the Puppet primary server +* _\_ - The FQDN and certname of the replica Puppet primary server + +Procedure: + +1. Run the `peadm::replace_failed_postgresql` plan to replace the failed PE-PostgreSQL server: + + bolt plan run peadm::replace_failed_postgresql \ + primary_host= \ + replica_host= \ + working_postgresql_host= \ + failed_postgresql_host= \ + replacement_postgresql_host= + +## Replace failed replica Puppet primary server AND failed replica PE-PostgreSQL server + +This procedure uses the following placeholder references. + +* _\_ - The FQDN and certname of the Puppet primary server +* _\_ - The FQDN and certname of the failed replica Puppet primary server + +1. Ensure the old replica server is forgotten. + + bolt command run "/opt/puppetlabs/bin/puppet infrastructure forget " --targets + +2. [Replace failed PE-PostgreSQL server (A or B side)](#replace-failed-pe-postgresql-server-a-or-b-side) +3. [Replace missing or failed replica Puppet primary server](#replace-missing-or-failed-replica-puppet-server) + +## Add or replace compilers + +This procedure uses the following placeholder references. + +* _\_ - Either A or B; whichever of the two letter designations the compiler is being assigned to +* _\_ - A comma-separated list of FQDN and certname of the new compiler(s) +* _\_ - A comma-separated list of DNS alt names for the compiler +* _\_ - The FQDN and certname of the Puppet primary server +* _\_ - The FQDN and certname of the PE-PostgreSQL server with availability group _\_ + +Procedure: + +1. Run the `peadm::add_compilers` plan to add the compilers: + + bolt plan run peadm::add_compilers \ + primary_host= \ + compiler_hosts= \ + avail_group_letter= \ + dns_alt_names= \ + primary_postgresql_host= + +Please note, the optional parameters and values of the plan are as follows: + + + +| Parameter | Default value | Description | +| ------------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------ | +| `avail_group_letter` | `A` | By default, each compiler will be added to the primary group A. | +| `dns_alt_names` | `undef` | | +| `primary_postgresql_host` | `undef` | By default, this will pre-populate to the required value depending on whether your architecture contains HA and or external databases. | + +For more information around adding compilers to your infrastructure [Expanding Your Deployment](expanding.md#adding-compilers-with-peadmadd_compiler) \ No newline at end of file diff --git a/documentation/recovery.md b/documentation/recovery/recovery.md similarity index 96% rename from documentation/recovery.md rename to documentation/recovery/recovery.md index 282a09c1..f735af10 100644 --- a/documentation/recovery.md +++ b/documentation/recovery/recovery.md @@ -4,7 +4,9 @@ These instructions all assume that the failed server is destroyed, and being rep The new system needs to be provisioned with the same certificate name as the system it is replacing. -## Recover from failed primary Puppet server +Automated procedures are documented in [automated_recovery.md](automated_recovery.md) + +## Recover from failed Puppet primary server 1. Promote the replica ([official docs](https://puppet.com/docs/pe/2019.8/dr_configure.html#dr-promote-replica)) 2. Purge the failed primary server @@ -18,7 +20,7 @@ The new system needs to be provisioned with the same certificate name as the sys This procedure uses the following placeholder references. -* _\_ - The FQDN and certname of the primary Puppet server +* _\_ - The FQDN and certname of the Puppet primary server * _\_ - The FQDN and certname of the old replica Puppet server that has failed or is missing * _\_ - The FQDN and certname of the new replica Puppet server * _\_ - Either A or B; whichever of the two letter designations is appropriate for the replacement server. It will be the opposite of the primary server. @@ -81,7 +83,7 @@ The procedure for replacing a failed PE-PostgreSQL server is the same regardless * _\_ - The FQDN and certname of the new server being brought in to replace the failed PE-PostgreSQL server * _\_ - The FQDN and certname of the still-working PE-PostgreSQL server * _\_ - Either A or B; whichever of the two letter designations is appropriate for the server being replaced. It will be the opposite of the still-working PE-PostgreSQL server -* _\_ - The FQDN and certname of the primary Puppet server +* _\_ - The FQDN and certname of the Puppet primary server Procedure: @@ -200,7 +202,7 @@ This procedure uses the following placeholder references. * _\_ - Either A or B; whichever of the two letter designations the compiler is being assigned to * _\_ - The FQDN and certname of the new compiler * _\_ - A comma-separated list of DNS alt names for the compiler -* _\_ - The FQDN and certname of the primary Puppet server +* _\_ - The FQDN and certname of the Puppet primary server * _\_ - The FQDN and certname of the PE-PostgreSQL server with availability group _\_ 1. On _\_: diff --git a/manifests/setup/legacy_compiler_group.pp b/manifests/setup/legacy_compiler_group.pp index e3601200..0fb161e4 100644 --- a/manifests/setup/legacy_compiler_group.pp +++ b/manifests/setup/legacy_compiler_group.pp @@ -67,6 +67,6 @@ } node_group { 'PE Compiler': - rule => ['and', ['=', ['trusted', 'extensions', peadm::oid('peadm_legacy_compiler')], 'false']], + rule => ['and', ['=', ['trusted', 'extensions', peadm::oid('peadm_legacy_compiler')], 'false']], } }