update_pgcluster.yml: Update pgBackRest package on the backup server

[vitabaks]

Related issue: #644

This task updates the pgBackRest package on the backup server (Dedicated Repository Host) before updating the Postgres cluster hosts.
It runs only if the 'pgbackrest' group is defined in the inventory and the update target is set to 'system'.

[vitabaks]

FYI @chuegel

[chuegel]

FYI @chuegel

awesome, thanks!

[vitabaks]

Test

PLAY [update_pgcluster.yml | Update PostgreSQL HA Cluster (based on "Patroni")] ***

TASK [Gathering Facts] *********************************************************
ok: [10.172.0.21]
ok: [10.172.0.20]
ok: [10.172.0.22]

TASK [Include main variables] **************************************************
ok: [10.172.0.20]
ok: [10.172.0.21]
ok: [10.172.0.22]

TASK [[Prepare] Get Patroni Cluster Leader Node] *******************************
ok: [10.172.0.21]
ok: [10.172.0.22]
ok: [10.172.0.20]

TASK [[Prepare] Add host to group "primary" (in-memory inventory)] *************
ok: [10.172.0.20] => (item=10.172.0.20)

TASK [[Prepare] Add hosts to group "secondary" (in-memory inventory)] **********
ok: [10.172.0.20] => (item=10.172.0.21)
ok: [10.172.0.20] => (item=10.172.0.22)

TASK [Print Patroni Cluster info] **********************************************
ok: [10.172.0.20] => {
    "msg": [
        "Cluster Name: postgres-cluster",
        "Cluster Leader: pgnode01"
    ]
}

PLAY [(1/4) PRE-UPDATE: Perform pre-update tasks] ******************************

TASK [Gathering Facts] *********************************************************
ok: [10.172.0.20]
ok: [10.172.0.21]
ok: [10.172.0.22]

TASK [Include role variables] **************************************************
ok: [10.172.0.20]
ok: [10.172.0.21]
ok: [10.172.0.22]

TASK [Include main variables] **************************************************
ok: [10.172.0.20]
ok: [10.172.0.21]
ok: [10.172.0.22]

TASK [Include OS-specific variables] *******************************************
ok: [10.172.0.20]
ok: [10.172.0.21]
ok: [10.172.0.22]

TASK [Running Pre-Checks] ******************************************************

TASK [update : [Pre-Check] (ALL) Test PostgreSQL DB Access] ********************
ok: [10.172.0.20]
ok: [10.172.0.21]
ok: [10.172.0.22]

TASK [update : [Pre-Check] Make sure that physical replication is active] ******
ok: [10.172.0.20]

TASK [update : [Pre-Check] Make sure there is no high replication lag (more than 10.00 MB)] ***
ok: [10.172.0.20]

TASK [update : [Pre-Check] Make sure there are no long-running transactions (more than 15 seconds)] ***
ok: [10.172.0.22]
ok: [10.172.0.21]
ok: [10.172.0.20]

TASK [Update pgBackRest package (Dedicated Repository Host)] *******************

TASK [update : Gather facts from pgbackrest server] ****************************
ok: [10.172.0.20 -> 10.172.0.200]

TASK [update : Update apt cache] ***********************************************
ok: [10.172.0.20 -> 10.172.0.200]

TASK [update : Install the latest version of pgbackrest package] ***************
ok: [10.172.0.20 -> 10.172.0.200]

PLAY [(2/4) UPDATE: Secondary] *************************************************

TASK [Gathering Facts] *********************************************************
ok: [10.172.0.21]

TASK [Include role variables] **************************************************
ok: [10.172.0.21]

TASK [Include main variables] **************************************************
ok: [10.172.0.21]

TASK [Include OS-specific variables] *******************************************
ok: [10.172.0.21]

TASK [Stop read-only traffic] **************************************************

TASK [update : Edit patroni.yml | enable noloadbalance, nosync, nofailover] ****
changed: [10.172.0.21] => (item=noloadbalance: true)
changed: [10.172.0.21] => (item=nosync: true)
changed: [10.172.0.21] => (item=nofailover: true)

TASK [update : Reload patroni service] *****************************************
changed: [10.172.0.21]
FAILED - RETRYING: [10.172.0.21]: Make sure replica endpoint is unavailable (30 retries left).
FAILED - RETRYING: [10.172.0.21]: Make sure replica endpoint is unavailable (29 retries left).

TASK [update : Make sure replica endpoint is unavailable] **********************
ok: [10.172.0.21]

TASK [update : Wait for active transactions to complete] ***********************
ok: [10.172.0.21]

TASK [Stop Services] ***********************************************************

TASK [update : Check PostgreSQL is started and accepting connections] **********
ok: [10.172.0.21]

TASK [update : Execute CHECKPOINT before stopping PostgreSQL] ******************
changed: [10.172.0.21]

TASK [update : Stop Patroni service on the Cluster Replica (pgnode02)] *********
changed: [10.172.0.21]

TASK [Update PostgreSQL] *******************************************************

TASK [update : Update apt cache] ***********************************************
ok: [10.172.0.21]

TASK [update : Install the latest version of PostgreSQL packages] **************
ok: [10.172.0.21] => (item=postgresql-16)
ok: [10.172.0.21] => (item=postgresql-client-16)
ok: [10.172.0.21] => (item=postgresql-contrib-16)
ok: [10.172.0.21] => (item=postgresql-server-dev-16)

TASK [Update Patroni] **********************************************************

TASK [update : Install the latest version of Patroni] **************************
ok: [10.172.0.21]

TASK [Update all system packages] **********************************************

TASK [update : Update apt cache] ***********************************************
ok: [10.172.0.21]

TASK [update : Update all system packages] *************************************
ok: [10.172.0.21]

TASK [Start Services] **********************************************************

TASK [update : Start Patroni service] ******************************************
changed: [10.172.0.21]

TASK [update : Wait for port 8008 to become open on the host] ******************
ok: [10.172.0.21]

TASK [update : Check that the Patroni is healthy] ******************************
ok: [10.172.0.21]

TASK [update : Check PostgreSQL is started and accepting connections] **********
ok: [10.172.0.21]

TASK [Start read-only traffic] *************************************************

TASK [update : Edit patroni.yml | disable noloadbalance, nosync, nofailover] ***
changed: [10.172.0.21] => (item=noloadbalance: false)
changed: [10.172.0.21] => (item=nosync: false)
changed: [10.172.0.21] => (item=nofailover: false)

TASK [update : Reload patroni service] *****************************************
changed: [10.172.0.21]
FAILED - RETRYING: [10.172.0.21]: Make sure replica endpoint is available (30 retries left).
FAILED - RETRYING: [10.172.0.21]: Make sure replica endpoint is available (29 retries left).
FAILED - RETRYING: [10.172.0.21]: Make sure replica endpoint is available (28 retries left).
FAILED - RETRYING: [10.172.0.21]: Make sure replica endpoint is available (27 retries left).

TASK [update : Make sure replica endpoint is available] ************************
ok: [10.172.0.21]

PLAY [(2/4) UPDATE: Secondary] *************************************************

TASK [Gathering Facts] *********************************************************
ok: [10.172.0.22]

TASK [Include role variables] **************************************************
ok: [10.172.0.22]

TASK [Include main variables] **************************************************
ok: [10.172.0.22]

TASK [Include OS-specific variables] *******************************************
ok: [10.172.0.22]

TASK [Stop read-only traffic] **************************************************

TASK [update : Edit patroni.yml | enable noloadbalance, nosync, nofailover] ****
changed: [10.172.0.22] => (item=noloadbalance: true)
changed: [10.172.0.22] => (item=nosync: true)
changed: [10.172.0.22] => (item=nofailover: true)

TASK [update : Reload patroni service] *****************************************
changed: [10.172.0.22]
FAILED - RETRYING: [10.172.0.22]: Make sure replica endpoint is unavailable (30 retries left).

TASK [update : Make sure replica endpoint is unavailable] **********************
ok: [10.172.0.22]

TASK [update : Wait for active transactions to complete] ***********************
ok: [10.172.0.22]

TASK [Stop Services] ***********************************************************

TASK [update : Check PostgreSQL is started and accepting connections] **********
ok: [10.172.0.22]

TASK [update : Execute CHECKPOINT before stopping PostgreSQL] ******************
changed: [10.172.0.22]

TASK [update : Stop Patroni service on the Cluster Replica (pgnode03)] *********
changed: [10.172.0.22]

TASK [Update PostgreSQL] *******************************************************

TASK [update : Update apt cache] ***********************************************
ok: [10.172.0.22]

TASK [update : Install the latest version of PostgreSQL packages] **************
ok: [10.172.0.22] => (item=postgresql-16)
ok: [10.172.0.22] => (item=postgresql-client-16)
ok: [10.172.0.22] => (item=postgresql-contrib-16)
ok: [10.172.0.22] => (item=postgresql-server-dev-16)

TASK [Update Patroni] **********************************************************

TASK [update : Install the latest version of Patroni] **************************
ok: [10.172.0.22]

TASK [Update all system packages] **********************************************

TASK [update : Update apt cache] ***********************************************
ok: [10.172.0.22]

TASK [update : Update all system packages] *************************************
ok: [10.172.0.22]

TASK [Start Services] **********************************************************

TASK [update : Start Patroni service] ******************************************
changed: [10.172.0.22]

TASK [update : Wait for port 8008 to become open on the host] ******************
ok: [10.172.0.22]

TASK [update : Check that the Patroni is healthy] ******************************
ok: [10.172.0.22]

TASK [update : Check PostgreSQL is started and accepting connections] **********
ok: [10.172.0.22]

TASK [Start read-only traffic] *************************************************

TASK [update : Edit patroni.yml | disable noloadbalance, nosync, nofailover] ***
changed: [10.172.0.22] => (item=noloadbalance: false)
changed: [10.172.0.22] => (item=nosync: false)
changed: [10.172.0.22] => (item=nofailover: false)

TASK [update : Reload patroni service] *****************************************
changed: [10.172.0.22]
FAILED - RETRYING: [10.172.0.22]: Make sure replica endpoint is available (30 retries left).
FAILED - RETRYING: [10.172.0.22]: Make sure replica endpoint is available (29 retries left).
FAILED - RETRYING: [10.172.0.22]: Make sure replica endpoint is available (28 retries left).
FAILED - RETRYING: [10.172.0.22]: Make sure replica endpoint is available (27 retries left).

TASK [update : Make sure replica endpoint is available] ************************
ok: [10.172.0.22]

PLAY [(3/4) UPDATE: Primary] ***************************************************

TASK [Gathering Facts] *********************************************************
ok: [10.172.0.20]

TASK [Include role variables] **************************************************
ok: [10.172.0.20]

TASK [Include main variables] **************************************************
ok: [10.172.0.20]

TASK [Include OS-specific variables] *******************************************
ok: [10.172.0.20]

TASK [Switchover Patroni leader role] ******************************************

TASK [update : Perform switchover of the leader for the Patroni cluster "postgres-cluster"] ***
ok: [10.172.0.20]
FAILED - RETRYING: [10.172.0.20]: Make sure that the Patroni is healthy and is a replica (300 retries left).

TASK [update : Make sure that the Patroni is healthy and is a replica] *********
ok: [10.172.0.20]

TASK [Stop read-only traffic] **************************************************

TASK [update : Edit patroni.yml | enable noloadbalance, nosync, nofailover] ****
changed: [10.172.0.20] => (item=noloadbalance: true)
changed: [10.172.0.20] => (item=nosync: true)
changed: [10.172.0.20] => (item=nofailover: true)

TASK [update : Reload patroni service] *****************************************
changed: [10.172.0.20]
FAILED - RETRYING: [10.172.0.20]: Make sure replica endpoint is unavailable (30 retries left).
FAILED - RETRYING: [10.172.0.20]: Make sure replica endpoint is unavailable (29 retries left).
FAILED - RETRYING: [10.172.0.20]: Make sure replica endpoint is unavailable (28 retries left).

TASK [update : Make sure replica endpoint is unavailable] **********************
ok: [10.172.0.20]

TASK [update : Wait for active transactions to complete] ***********************
ok: [10.172.0.20]

TASK [Stop Services] ***********************************************************

TASK [update : Check PostgreSQL is started and accepting connections] **********
ok: [10.172.0.20]

TASK [update : Execute CHECKPOINT before stopping PostgreSQL] ******************
changed: [10.172.0.20]

TASK [update : Stop Patroni service on the old Cluster Leader (pgnode01)] ******
changed: [10.172.0.20]

TASK [Update PostgreSQL] *******************************************************

TASK [update : Update apt cache] ***********************************************
ok: [10.172.0.20]

TASK [update : Install the latest version of PostgreSQL packages] **************
ok: [10.172.0.20] => (item=postgresql-16)
ok: [10.172.0.20] => (item=postgresql-client-16)
ok: [10.172.0.20] => (item=postgresql-contrib-16)
ok: [10.172.0.20] => (item=postgresql-server-dev-16)

TASK [Update Patroni] **********************************************************

TASK [update : Install the latest version of Patroni] **************************
ok: [10.172.0.20]

TASK [Update all system packages] **********************************************

TASK [update : Update apt cache] ***********************************************
ok: [10.172.0.20]

TASK [update : Update all system packages] *************************************
ok: [10.172.0.20]

TASK [Start Services] **********************************************************

TASK [update : Start Patroni service] ******************************************
changed: [10.172.0.20]

TASK [update : Wait for port 8008 to become open on the host] ******************
ok: [10.172.0.20]

TASK [update : Check that the Patroni is healthy] ******************************
ok: [10.172.0.20]

TASK [update : Check PostgreSQL is started and accepting connections] **********
ok: [10.172.0.20]

TASK [Start read-only traffic] *************************************************

TASK [update : Edit patroni.yml | disable noloadbalance, nosync, nofailover] ***
changed: [10.172.0.20] => (item=noloadbalance: false)
changed: [10.172.0.20] => (item=nosync: false)
changed: [10.172.0.20] => (item=nofailover: false)

TASK [update : Reload patroni service] *****************************************
changed: [10.172.0.20]
FAILED - RETRYING: [10.172.0.20]: Make sure replica endpoint is available (30 retries left).
FAILED - RETRYING: [10.172.0.20]: Make sure replica endpoint is available (29 retries left).
FAILED - RETRYING: [10.172.0.20]: Make sure replica endpoint is available (28 retries left).
FAILED - RETRYING: [10.172.0.20]: Make sure replica endpoint is available (27 retries left).

TASK [update : Make sure replica endpoint is available] ************************
ok: [10.172.0.20]

PLAY [(4/4) POST-UPDATE: Update extensions] ************************************

TASK [Gathering Facts] *********************************************************
ok: [10.172.0.21]
ok: [10.172.0.20]
ok: [10.172.0.22]

TASK [Include role variables] **************************************************
ok: [10.172.0.20]
ok: [10.172.0.21]
ok: [10.172.0.22]

TASK [Include main variables] **************************************************
ok: [10.172.0.20]
ok: [10.172.0.21]
ok: [10.172.0.22]

TASK [Include OS-specific variables] *******************************************
ok: [10.172.0.20]
ok: [10.172.0.21]
ok: [10.172.0.22]

TASK [Update extensions] *******************************************************
[WARNING]: Module remote_tmp /var/lib/postgresql/.ansible/tmp did not exist and
was created with a mode of 0700, this may cause issues when running as another
user. To avoid this, create the remote_tmp dir with the correct permissions
manually

TASK [update : Get the current Patroni Cluster Leader Node] ********************
ok: [10.172.0.22]
ok: [10.172.0.21]
ok: [10.172.0.20]

TASK [update : Get a list of databases] ****************************************
ok: [10.172.0.22]

TASK [update : Update extensions in each database] *****************************
included: /workspace/postgresql_cluster/roles/update/tasks/update_extensions.yml for 10.172.0.22 => (item=postgres)
included: /workspace/postgresql_cluster/roles/update/tasks/update_extensions.yml for 10.172.0.22 => (item=template1)

TASK [update : Get a list of old PostgreSQL extensions] ************************
ok: [10.172.0.22]

TASK [update : The extensions are up-to-date] **********************************
ok: [10.172.0.22] => {
    "msg": [
        "The extension versions are up-to-date for the database postgres",
        "No update is required."
    ]
}

TASK [update : Get a list of old PostgreSQL extensions] ************************
ok: [10.172.0.22]

TASK [update : The extensions are up-to-date] **********************************
ok: [10.172.0.22] => {
    "msg": [
        "The extension versions are up-to-date for the database template1",
        "No update is required."
    ]
}

TASK [Check for any update failure] ********************************************
ok: [10.172.0.20] => (item=10.172.0.20)
ok: [10.172.0.20] => (item=10.172.0.21)
ok: [10.172.0.20] => (item=10.172.0.22)

TASK [Check the Patroni cluster state] *****************************************
ok: [10.172.0.20]

TASK [Check the current PostgreSQL version] ************************************
ok: [10.172.0.20]

TASK [List the Patroni cluster members] ****************************************
ok: [10.172.0.20] => {
    "msg": [
        "+ Cluster: postgres-cluster (7364751812621002672) --+-----------+-----------------+",
        "| Member   | Host        | Role    | State     | TL | Lag in MB | Tags            |",
        "+----------+-------------+---------+-----------+----+-----------+-----------------+",
        "| pgnode01 | 10.172.0.20 | Replica | streaming |  3 |         0 | datacenter: dc1 |",
        "|          |             |         |           |    |           | key1: value1    |",
        "+----------+-------------+---------+-----------+----+-----------+-----------------+",
        "| pgnode02 | 10.172.0.21 | Replica | streaming |  3 |         0 | datacenter: dc1 |",
        "|          |             |         |           |    |           | key1: value1    |",
        "+----------+-------------+---------+-----------+----+-----------+-----------------+",
        "| pgnode03 | 10.172.0.22 | Leader  | running   |  3 |           | datacenter: dc1 |",
        "|          |             |         |           |    |           | key1: value1    |",
        "+----------+-------------+---------+-----------+----+-----------+-----------------+"
    ]
}

TASK [Update completed] ********************************************************
ok: [10.172.0.20] => {
    "msg": [
        "PostgreSQL HA cluster update completed.",
        "Current version: 16.2 (Ubuntu 16.2-1.pgdg22.04+1)"
    ]
}

PLAY RECAP *********************************************************************
10.172.0.20                : ok=319  changed=102  unreachable=0    failed=0    skipped=700  rescued=0    ignored=0
10.172.0.200               : ok=60   changed=17   unreachable=0    failed=0    skipped=57   rescued=0    ignored=0
10.172.0.21                : ok=260  changed=96   unreachable=0    failed=0    skipped=657  rescued=0    ignored=0
10.172.0.22                : ok=267  changed=96   unreachable=0    failed=0    skipped=663  rescued=0    ignored=0

The following tasks demonstrate the fact that the pgbackrest package has been updated on the backup server (10.172.0.200)

TASK [Update pgBackRest package (Dedicated Repository Host)] *******************

TASK [update : Gather facts from pgbackrest server] ****************************
ok: [10.172.0.20 -> 10.172.0.200]

TASK [update : Update apt cache] ***********************************************
ok: [10.172.0.20 -> 10.172.0.200]

TASK [update : Install the latest version of pgbackrest package] ***************
ok: [10.172.0.20 -> 10.172.0.200]

passed