#1963 - Database Name prefixed #2152

guru-aot · 2023-08-01T17:07:42Z

Updated database name as a parameter for deployments, the sims_db_name is connected from every environment github secrets.
The existing environment github secret database_name_key will be removed, as it is not used.
The sims_db_name, is a new environment github secret, which needs to be defined in every environment, which is connected to sims-creds secret in openshift. In-turn used in the deployments in github actions as an application secret.

Note: Connecting the existing patroni locally and creating a database in the name '{sims_db_name}' is required as a prerequisite, so that the github actions can find the new database when run.

dheepak-aot · 2023-08-01T20:50:52Z

.github/workflows/release-deploy-all.yml

@@ -88,7 +88,7 @@ env:
  QUEUE_PREFIX: ${{ secrets.QUEUE_PREFIX }}
  API_PORT: ${{ secrets.API_PORT }}
  WEB_PORT: ${{ secrets.WEB_PORT }}
-  DATABASE_NAME_KEY: ${{ secrets.DATABASE_NAME_KEY }}
+  DATABASE_NAME_PREFIX: ${{ secrets.DATABASE_NAME_PREFIX }}


Do we really need the prefix? DATABASE_NAME could be something which is mapped from github env to deployment directly right?

like this DATABASE_NAME: ${{ vars.DATABASE_NAME }}

Also should database name be in secrets in github? we can use vars

yes we can have and we already discussed this i guess. The thing is even without the database_name or the database_name_prefix in the github secrets, the database name should be taken as SIMSDB and all the actions should work. This change is for an exception scenario where we have to create a new set of db for new environment. So introduced the prefix, rather going with the database_name and repeating with the same value everywhere.

This inturn does not need the change in the prod environment deployment, if the secret database_name_key is removed from the Prod github secrets

We can talk as a team and settle this. Maybe I am missing something but I was under the assumption that the DB name would go under the sims-env-creds maybe, but the current approach would also be acceptable.

dheepak-aot · 2023-08-01T21:02:08Z

.github/workflows/release-deploy-all.yml

@@ -103,6 +103,7 @@ jobs:
          echo Git Red: ${{ inputs.gitRef }}
          echo BUILD NAMESPACE: $BUILD_NAMESPACE
          echo HOST_PREFIX: $HOST_PREFIX
+          echo DATABASE_NAME_PREFIX: $DATABASE_NAME_PREFIX


Just an echo but we need the same for deploy-sims-api, workers and queue-consumers right?

yes, but i guess, we can remove all the echos in the deploy-sims-api, workers and queue-consumers too, as they depends on run-db-migrations. its okay to have at one place :). I will add it in other places too

Since the DATABASE_NAME_PREFIX is in the secrets GitHub will not print its value at all, the same for $HOST_PREFIX.

okay then we can remove them in all the entires.

dheepak-aot · 2023-08-01T21:13:17Z

Great job on having the staging environment ready. Added 2 comments out of which only first one is something I am confused about.

andrewsignori-aot

Nice work, let's try to settle the comments quickly 😉
I do not see a major concern between the options.

andrewsignori-aot · 2023-08-01T22:30:22Z

.github/workflows/release-deploy-all.yml

@@ -103,6 +103,7 @@ jobs:
          echo Git Red: ${{ inputs.gitRef }}
          echo BUILD NAMESPACE: $BUILD_NAMESPACE
          echo HOST_PREFIX: $HOST_PREFIX
+          echo DATABASE_NAME_PREFIX: $DATABASE_NAME_PREFIX


Since the DATABASE_NAME_PREFIX is in the secrets GitHub will not print its value at all, the same for $HOST_PREFIX.

andrewsignori-aot · 2023-08-01T22:33:59Z

.github/workflows/release-deploy-all.yml

@@ -88,7 +88,7 @@ env:
  QUEUE_PREFIX: ${{ secrets.QUEUE_PREFIX }}
  API_PORT: ${{ secrets.API_PORT }}
  WEB_PORT: ${{ secrets.WEB_PORT }}
-  DATABASE_NAME_KEY: ${{ secrets.DATABASE_NAME_KEY }}
+  DATABASE_NAME_PREFIX: ${{ secrets.DATABASE_NAME_PREFIX }}


We can talk as a team and settle this. Maybe I am missing something but I was under the assumption that the DB name would go under the sims-env-creds maybe, but the current approach would also be acceptable.

dheepak-aot · 2023-08-02T21:42:16Z

.github/workflows/release-deploy-all.yml

@@ -101,8 +100,6 @@ jobs:
        run: |
          echo Git Environment: ${{ inputs.environment }}
          echo Git Red: ${{ inputs.gitRef }}
-          echo BUILD NAMESPACE: $BUILD_NAMESPACE


Why did we remove the echo of the build_namespace ?

its also not printing so removed.

Correct me if wrong. I see the echo happening just under that.

My bad missed it in the printing below, reverted the change back :)

dheepak-aot · 2023-08-02T21:52:56Z

devops/openshift/queue-consumers-deploy.yml

@@ -276,8 +276,10 @@ parameters:
    value: superuser-username
  - name: DB_PASSWORD_KEY
    value: superuser-password
-  - name: DATABASE_NAME_KEY
-    required: true
+  - name: API_SECRET_NAME


API_SECRET_NAME is not required to be added here. QUEUE_CONSUMERS_SECRET_NAME can be used.

dheepak-aot · 2023-08-02T21:53:11Z

devops/openshift/queue-consumers-deploy.yml

-                      key: ${DATABASE_NAME_KEY}
-                      name: ${DB_SECRET_NAME}
+                      key: ${SIMS_DB_NAME}
+                      name: ${API_SECRET_NAME}


QUEUE_CONSUMERS_SECRET_NAME can be used.

dheepak-aot · 2023-08-02T21:53:56Z

devops/openshift/workers-deploy.yml

@@ -124,8 +124,10 @@ parameters:
    value: superuser-username
  - name: DB_PASSWORD_KEY
    value: superuser-password
-  - name: DATABASE_NAME_KEY
-    required: true
+  - name: API_SECRET_NAME


API_SECRET_NAME is not required to be added here. WORKERS_SECRET_NAME can be re-used.

dheepak-aot · 2023-08-02T21:54:13Z

devops/openshift/workers-deploy.yml

-                      key: ${DATABASE_NAME_KEY}
-                      name: ${DB_SECRET_NAME}
+                      key: ${SIMS_DB_NAME}
+                      name: ${API_SECRET_NAME}


WORKERS_SECRET_NAME can be used.

dheepak-aot · 2023-08-02T22:04:24Z

.github/workflows/release-deploy-all.yml

@@ -88,7 +88,6 @@ env:
  QUEUE_PREFIX: ${{ secrets.QUEUE_PREFIX }}
  API_PORT: ${{ secrets.API_PORT }}
  WEB_PORT: ${{ secrets.WEB_PORT }}
-  DATABASE_NAME_KEY: ${{ secrets.DATABASE_NAME_KEY }}


Not part of the PR. I see that the build namespace and keycloak realm are hard coded here. I believe we will be changing it to read from github secrets in future.

We have the BUILD_NAMESPACE already available and used, for instance, in .github\workflows\env-setup-build-forms-server.yml
We can do the same for the KEYCLOAK_REALM.
By the way, @dheepak-aot for these ones that were hard-code, we used the vars.

I have added a repository variable and changed it.

dheepak-aot

Thanks for doing the additional refactors to replace the tools namespace and keycloak realm. 👍

andrewsignori-aot · 2023-08-02T22:56:48Z

devops/openshift/workers-deploy.yml

@@ -137,6 +137,7 @@ parameters:
  - name: BUILD_TAG
    value: "0"
  - name: WORKERS_SECRET_NAME
+    value: sims-api-creds


I am not following this default value. Do we need it?

Just following the steps follwed in api-deploy.yml, also the name was not self understanding what it is, the value just gives an idea what this secret name is.

andrewsignori-aot · 2023-08-02T22:57:02Z

devops/openshift/queue-consumers-deploy.yml

@@ -289,6 +289,7 @@ parameters:
  - name: BUILD_TAG
    value: "0"
  - name: QUEUE_CONSUMERS_SECRET_NAME
+    value: sims-api-creds


Same from https://github.com/bcgov/SIMS/pull/2152/files#r1282487715

Just following the steps follwed in api-deploy.yml, also the name was not self understanding what it is, the value just gives an idea what this secret name is.

IMO, the standard is old and does not represent well the change that we did when we introduced the sims-ENV-creds.

andrewsignori-aot

Thanks for doing the changes, only one last comment to be checked.

sonarqubecloud · 2023-08-02T23:08:51Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

andrewsignori-aot

Thanks for doing the changes, looks good 👍
Please ensure that we have the instructions updated in the next release instructions
https://app.zenhub.com/workspaces/student-information-management-system-5fce9df5aa1b45000e937014/reports/release?release=Z2lkOi8vcmFwdG9yL1JlbGVhc2UvOTQxODU

github-actions · 2023-08-02T23:13:18Z

Backend Unit Tests Coverage Report

Totals
Statements:	17.87% ( 2137 / 11957 )
Methods:	8.26% ( 126 / 1525 )
Lines:	20.68% ( 1873 / 9059 )
Branches:	10.05% ( 138 / 1373 )

github-actions · 2023-08-02T23:14:48Z

E2E Workflow Workers Coverage Report

Totals
Statements:	49.81% ( 267 / 536 )
Methods:	41.56% ( 32 / 77 )
Lines:	55.33% ( 218 / 394 )
Branches:	26.15% ( 17 / 65 )

github-actions · 2023-08-02T23:15:32Z

E2E Queue Consumers Coverage Report

Totals
Statements:	72.5% ( 406 / 560 )
Methods:	63.38% ( 45 / 71 )
Lines:	74.53% ( 357 / 479 )
Branches:	40% ( 4 / 10 )

github-actions · 2023-08-02T23:21:20Z

E2E SIMS API Coverage Report

Totals
Statements:	52.65% ( 3757 / 7136 )
Methods:	49.3% ( 457 / 927 )
Lines:	57.73% ( 3074 / 5325 )
Branches:	25.57% ( 226 / 884 )

ann-aot

👍 good work @guru-aot

guru-aot added 3 commits July 28, 2023 15:31

updated

dd5fcb8

updated

12cac5c

updated

675d322

guru-aot requested review from dheepak-aot, andrewsignori-aot, ann-aot and sh16011993 August 1, 2023 17:07

guru-aot marked this pull request as ready for review August 1, 2023 17:16

dheepak-aot reviewed Aug 1, 2023

View reviewed changes

updated

5d5bffb

guru-aot self-assigned this Aug 1, 2023

guru-aot added the Devops Devops label Aug 1, 2023

andrewsignori-aot reviewed Aug 1, 2023

View reviewed changes

andrewsignori-aot changed the title ~~Database Name prefixed~~ #1963 - Database Name prefixed Aug 1, 2023

guru-aot added 5 commits August 1, 2023 16:55

updated

6e44444

updated

2697355

updated

1bc3a5e

updated

4e3c7b2

updated

1c144c9

dheepak-aot reviewed Aug 2, 2023

View reviewed changes

guru-aot added 2 commits August 2, 2023 15:58

updated

bb64a7e

updated

40b55a4

dheepak-aot reviewed Aug 2, 2023

View reviewed changes

updated

c73c401

updated

23a3583

dheepak-aot approved these changes Aug 2, 2023

View reviewed changes

andrewsignori-aot reviewed Aug 2, 2023

View reviewed changes

updated

ace31a1

andrewsignori-aot approved these changes Aug 2, 2023

View reviewed changes

ann-aot approved these changes Aug 2, 2023

View reviewed changes

guru-aot merged commit 9f31199 into main Aug 3, 2023

guru-aot deleted the #1963-Create_Staging_Environment branch August 3, 2023 18:28

guru-aot temporarily deployed to DEV August 3, 2023 18:44 — with GitHub Actions Inactive

guru-aot temporarily deployed to DEV August 3, 2023 18:45 — with GitHub Actions Inactive

guru-aot temporarily deployed to DEV August 3, 2023 18:47 — with GitHub Actions Inactive

#1963 - Database Name prefixed #2152

#1963 - Database Name prefixed #2152

Conversation

guru-aot commented Aug 1, 2023 • edited Loading

dheepak-aot Aug 1, 2023 • edited Loading

Choose a reason for hiding this comment

guru-aot Aug 1, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dheepak-aot Aug 1, 2023 • edited Loading

Choose a reason for hiding this comment

guru-aot Aug 1, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

guru-aot Aug 1, 2023 • edited Loading

Choose a reason for hiding this comment

dheepak-aot commented Aug 1, 2023

andrewsignori-aot left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dheepak-aot left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewsignori-aot left a comment

Choose a reason for hiding this comment

sonarqubecloud bot commented Aug 2, 2023

andrewsignori-aot left a comment

Choose a reason for hiding this comment

github-actions bot commented Aug 2, 2023

Backend Unit Tests Coverage Report

github-actions bot commented Aug 2, 2023

E2E Workflow Workers Coverage Report

github-actions bot commented Aug 2, 2023

E2E Queue Consumers Coverage Report

github-actions bot commented Aug 2, 2023

E2E SIMS API Coverage Report

ann-aot left a comment

Choose a reason for hiding this comment

guru-aot commented Aug 1, 2023 •

edited

Loading

dheepak-aot Aug 1, 2023 •

edited

Loading

guru-aot Aug 1, 2023 •

edited

Loading

dheepak-aot Aug 1, 2023 •

edited

Loading

guru-aot Aug 1, 2023 •

edited

Loading

guru-aot Aug 1, 2023 •

edited

Loading