#1882 - Recovery and backup startegy patroni #2057

guru-aot · 2023-06-28T16:00:08Z

Updated the base image of database backup container to support till Postgres 14 - https://github.com/BCDevOps/backup-container/blob/master/docker/Dockerfile
The base build template was changed from sample build given in https://github.com/BCDevOps/backup-container/blob/master/openshift/templates/backup/backup-build.yaml
Removed the mongo db backup from makefile and removed in all the namespaces
The backup.conf should be created for databases seperately, so renamed it for simsdb and refactored to reflect the changes in the config names, pvc's and containers created
The deploy.yaml is updated with the latest from bcgov recommendations https://github.com/BCDevOps/backup-container/blob/master/openshift/templates/backup/backup-deploy.yaml
Backup stratergy is followed by rolling backup with default bcgov recommendations on backup

andrewsignori-aot · 2023-06-28T19:22:32Z

devops/Makefile

-export BACKUP_MONGODB_APP_NAME := $(or $(BACKUP_MONGODB_APP_NAME), mongodb-backup)
-export BACKUP_CONFIGMAP_NAME := $(or $(BACKUP_CONFIGMAP_NAME), backup-conf)
+export BACKUP_POSTGRESQL_APP_NAME := $(or $(BACKUP_POSTGRESQL_APP_NAME), patroni-simsdb-backup)
+export PATRONI_SIMSDB_BACKUP_CONFIGMAP_NAME := $(or $(PATRONI_SIMSDB_BACKUP_CONFIGMAP_NAME), patroni-simsdb-backup-conf)


When this was mentioned "The backup.conf should be created for databases separately" does it means that they need to be created per database or per database type? For instance, Mongo/Postgres? In case we have a second DB on Postgres, do we need to create a second backup structure?
IMO, the answer to this question can help determine the variable's names.

Yeah for seperate database not the database type, because individual databases can have different schedule of backups

So, "The backup.conf should be created for databases separately" actually means that the backups can be configured for multiple databases but we are actually structuring it in a way that it will be per database assuming that a new database will have a different backup schedule, right?

Does it mean that in the future, if we have more than one DB, instead of a new line in a config file we will create a separate structure to handle the new backup?

yes you got it right, that was the plan, as individual backup stratergy for different databases.

I went through the conversation @guru-aot @andrewsignori-aot .

@guru-aot what could be the benefit of having individual backup strategy per db?

andrewsignori-aot · 2023-06-28T19:40:14Z

devops/openshift/database-backup/backup-build.yaml

@@ -73,4 +71,4 @@ parameters:
    displayName: FROM Image Tag
    description: Base image to build from.  Docker creds or Artificatory setup may be needed to alleviate docker rate-limiting
    required: true
-    value: artifacts.developer.gov.bc.ca/docker-remote/centos/postgresql-12-centos7:20200917-804ef01
+    value: docker.io/centos/postgresql-12-centos7:20200917-804ef01


artifacts.developer.gov.bc.ca/docker-remote is not the same as docker.io?
Are we not supposed to be using the artifacts URL as much as possible (if not for everything)?

Yes you are right, but in this case, the build.yaml has been copied from the https://github.com/BCDevOps/backup-container/blob/master/openshift/templates/backup/backup-build.yaml. And the base image is actually not uses this too instead it uses quay.io/fedora/postgresql-14:14, as per the suggestion from the bcgov developer who did the changes to the dockerfile, which supports postgres 14.

@guru-aot does artifacts.developer.gov.bc.ca/docker-remote/centos/postgresql-12-centos7:20200917-804ef01 work? as artifacts.developer.gov.bc.ca/docker-remote is just a proxy.
Correct me if wrong.

IMO only if it does not work we should start thinking about docker.io IMO.

andrewsignori-aot · 2023-06-28T19:48:54Z

devops/openshift/database-backup/backup-deploy.yaml

@@ -170,16 +169,16 @@ objects:
                  value: ${MONGODB_AUTHENTICATION_DATABASE}
                - name: TABLE_SCHEMA
                  value: ${TABLE_SCHEMA}
-                - name: ${DATABASE_SERVER_NAME}_USER
+                - name: DATABASE_USER


Are we removing the "_USER" because we will no longer support the backup for multiple DBs?
"_USER" reference: https://github.com/BCDevOps/backup-container/tree/master#backupconf

Should we decide to move away from back-up of multiple db in same backup container?

If there is a requirement in future to have a another db, or we if decide to back up redis(by any chance) current way will be easily adaptable right?

andrewsignori-aot · 2023-06-28T19:50:10Z

devops/Makefile

 # Create the POSTGRES structure
-db-backup-build-postgresql:
+db-backup-build-patroni:


Should this also include SIMS to be aligned with the other renamed variables?

i specifically did not add SIMS, becuase build is going to be same for all, but while deploying we can have multiple databases with different names.

andrewsignori-aot

Thanks for the research on this. Few comments to get moe clarity 😉

ann-aot · 2023-06-28T20:10:56Z

devops/Makefile

@@ -143,16 +143,15 @@ oc-build-forms: | print-status build-forms
 oc-deploy-ha-mongo: | print-status deploy-ha-mongo
 oc-deploy-forms: | print-status deploy-forms

+# Build backup structure


minor - preriod

ann-aot

Whenever available, I would like to have a walk-through for my information

dheepak-aot · 2023-06-28T22:54:15Z

devops/openshift/database-backup/backup-build.yaml

@@ -73,4 +71,4 @@ parameters:
    displayName: FROM Image Tag
    description: Base image to build from.  Docker creds or Artificatory setup may be needed to alleviate docker rate-limiting
    required: true
-    value: artifacts.developer.gov.bc.ca/docker-remote/centos/postgresql-12-centos7:20200917-804ef01
+    value: docker.io/centos/postgresql-12-centos7:20200917-804ef01


Can we claim that current backup supports postgres 14 with this image?

Yes it supports Postgres 14, as per the fedora postgres 14 base image

dheepak-aot · 2023-06-28T22:56:40Z

devops/openshift/database-backup/patroni-simsdb-backup.conf

@@ -1,7 +1,4 @@
 postgres=patroni-master:5432/SIMSDB
-postgres=patroni-master:5432/FFA_API_DB
-postgres=patroni-master:5432/FFA_BPM_DB


dheepak-aot · 2023-06-28T23:03:40Z

Great work. Added some comments for my understanding and clarity.

Also, as we are in production curious to know about enabling FTP. I know it is outside the PR context but when I looked at the readme of back up container it rang a bell to ask about it.

devops/Makefile

dheepak-aot

Thanks for the walk through 👍

ann-aot

Thanks for the walk through @guru-aot 👍

andrepestana-aot

Thanks for the explanations!

andrewsignori-aot

Thanks for making this happen. Looks good 👍

sonarqubecloud · 2023-06-30T17:30:16Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

github-actions · 2023-06-30T17:31:06Z

Backend Unit Tests Coverage Report

Totals
Statements:	17.91% ( 2127 / 11875 )
Methods:	8.33% ( 126 / 1513 )
Lines:	20.71% ( 1863 / 8997 )
Branches:	10.11% ( 138 / 1365 )

github-actions · 2023-06-30T17:32:50Z

E2E Queue Consumers Coverage Report

Totals
Statements:	71% ( 399 / 562 )
Methods:	61.97% ( 44 / 71 )
Lines:	72.97% ( 351 / 481 )
Branches:	40% ( 4 / 10 )

github-actions · 2023-06-30T17:32:59Z

E2E Workflow Workers Coverage Report

Totals
Statements:	49.81% ( 267 / 536 )
Methods:	41.56% ( 32 / 77 )
Lines:	55.33% ( 218 / 394 )
Branches:	26.15% ( 17 / 65 )

github-actions · 2023-06-30T17:39:21Z

E2E SIMS API Coverage Report

Totals
Statements:	50.18% ( 3543 / 7061 )
Methods:	45.63% ( 418 / 916 )
Lines:	55.38% ( 2915 / 5264 )
Branches:	23.84% ( 210 / 881 )

guru-aot added 10 commits June 27, 2023 14:06

updated

7a82c9e

updated

633ab60

updated

cf507e8

updated

776fbd6

updated

ef1537b

updated

9d8e91d

updated

1849c1e

updated

b854b56

updated

653f2ec

updated

3c180d7

guru-aot requested review from andrewsignori-aot and dheepak-aot June 28, 2023 18:02

guru-aot self-assigned this Jun 28, 2023

guru-aot added the Devops Devops label Jun 28, 2023

guru-aot requested review from sh16011993, ann-aot, andrepestana-aot and JasonCTang June 28, 2023 18:03

andrewsignori-aot reviewed Jun 28, 2023

View reviewed changes

updated

79b9f71

andrewsignori-aot reviewed Jun 28, 2023

View reviewed changes

ann-aot reviewed Jun 28, 2023

View reviewed changes

updated

93fdd3c

ann-aot reviewed Jun 28, 2023

View reviewed changes

dheepak-aot reviewed Jun 28, 2023

View reviewed changes

updated

a1644e8

andrepestana-aot reviewed Jun 29, 2023

View reviewed changes

devops/Makefile Show resolved Hide resolved

dheepak-aot approved these changes Jun 29, 2023

View reviewed changes

ann-aot approved these changes Jun 29, 2023

View reviewed changes

andrepestana-aot approved these changes Jun 29, 2023

View reviewed changes

Updated

be48c0c

andrewsignori-aot approved these changes Jun 30, 2023

View reviewed changes

andrewsignori-aot changed the title ~~#1882-Recovery and backup startegy patroni~~ #1882 - Recovery and backup startegy patroni Jun 30, 2023

guru-aot merged commit 0f24257 into main Jun 30, 2023

guru-aot deleted the feature/#1882-Recovery-and-backup-strategy-of-Patroni branch June 30, 2023 18:12

guru-aot temporarily deployed to DEV June 30, 2023 18:45 — with GitHub Actions Inactive

guru-aot temporarily deployed to DEV June 30, 2023 18:55 — with GitHub Actions Inactive

guru-aot temporarily deployed to DEV June 30, 2023 19:04 — with GitHub Actions Inactive

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

#1882 - Recovery and backup startegy patroni #2057

#1882 - Recovery and backup startegy patroni #2057

guru-aot commented Jun 28, 2023 •

edited

Loading

andrewsignori-aot Jun 28, 2023 •

edited

Loading

guru-aot Jun 28, 2023

andrewsignori-aot Jun 28, 2023

guru-aot Jun 28, 2023

dheepak-aot Jun 28, 2023

andrewsignori-aot Jun 28, 2023

guru-aot Jun 28, 2023

dheepak-aot Jun 28, 2023

andrewsignori-aot Jun 28, 2023

dheepak-aot Jun 28, 2023

andrewsignori-aot Jun 28, 2023

guru-aot Jun 28, 2023

andrewsignori-aot left a comment

ann-aot Jun 28, 2023

ann-aot left a comment •

edited

Loading

dheepak-aot Jun 28, 2023

guru-aot Jun 28, 2023

dheepak-aot Jun 28, 2023

dheepak-aot commented Jun 28, 2023 •

edited

Loading

dheepak-aot left a comment

ann-aot left a comment

andrepestana-aot left a comment

andrewsignori-aot left a comment

sonarqubecloud bot commented Jun 30, 2023

github-actions bot commented Jun 30, 2023

github-actions bot commented Jun 30, 2023

github-actions bot commented Jun 30, 2023

github-actions bot commented Jun 30, 2023

#1882 - Recovery and backup startegy patroni #2057

#1882 - Recovery and backup startegy patroni #2057

Conversation

guru-aot commented Jun 28, 2023 • edited Loading

andrewsignori-aot Jun 28, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewsignori-aot left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ann-aot left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dheepak-aot commented Jun 28, 2023 • edited Loading

dheepak-aot left a comment

Choose a reason for hiding this comment

ann-aot left a comment

Choose a reason for hiding this comment

andrepestana-aot left a comment

Choose a reason for hiding this comment

andrewsignori-aot left a comment

Choose a reason for hiding this comment

sonarqubecloud bot commented Jun 30, 2023

github-actions bot commented Jun 30, 2023

Backend Unit Tests Coverage Report

github-actions bot commented Jun 30, 2023

E2E Queue Consumers Coverage Report

github-actions bot commented Jun 30, 2023

E2E Workflow Workers Coverage Report

github-actions bot commented Jun 30, 2023

E2E SIMS API Coverage Report

guru-aot commented Jun 28, 2023 •

edited

Loading

andrewsignori-aot Jun 28, 2023 •

edited

Loading

ann-aot left a comment •

edited

Loading

dheepak-aot commented Jun 28, 2023 •

edited

Loading