#2180 - Queued Assessments - Assessment Workflow Enqueuer Scheduler - Start Assessment (Part 1) #2285

andrewsignori-aot · 2023-09-08T22:21:11Z

Created the new queue configuration for the scheduler, executed every 30 seconds, and cleaned up every 24 hours.
Removed from the code the points where the queue to start the workflow was invoked. From now on only the scheduler will invoke the workflow start queue.
- Method startAssessment removed since it is no longer needed.
- Method hasIncompleteAssessment and assertAllAssessmentsCompleted removed since there is no purpose anymore.
- Multiple workflows can now be saved and the scheduler will coordinate their execution.
StartApplicationAssessment queue can now be started without providing the workflowName.
Created a migration to update the assessment status based on assessment_workflow_id, and assessment_data. The rollback is less meaningful and it was created to try to bring the column back to its original state where the column was set with its default value Submitted.

TODO:

E2E tests
Logs
- Move ProcessSummaryResult from integrations to lib to allow its use outside integrations.

sources/packages/backend/apps/db-migrations/src/main.ts

sources/packages/backend/apps/queue-consumers/src/services/application/application.service.ts

andrepestana-aot

Good job. Left some questions.

.../apps/db-migrations/src/sql/StudentAssessments/Rollback-update-student-assessment-status.sql

...ces/packages/backend/apps/queue-consumers/src/services/workflow/workflow-enqueuer.service.ts

sources/packages/backend/apps/api/src/services/student-appeal/student-appeal.service.ts

...d/apps/api/src/services/student-scholastic-standings/student-scholastic-standings.service.ts

...packages/backend/apps/db-migrations/src/sql/Queue/Add-assessment-workflow-enqueuer-queue.sql

sources/packages/backend/apps/queue-consumers/src/services/application/application.service.ts

ann-aot

Good work @andrewsignori-aot 👍 Thanks for the explanation.

dheepak-aot · 2023-09-12T18:40:05Z

...s/backend/apps/db-migrations/src/sql/StudentAssessments/Update-student-assessment-status.sql

.../apps/db-migrations/src/sql/StudentAssessments/Rollback-update-student-assessment-status.sql

dheepak-aot · 2023-09-12T18:56:49Z

...end/apps/queue-consumers/src/processors/assessment/start-application-assessment.processor.ts


  @Process()
  async startAssessment(job: Job<StartAssessmentQueueInDTO>) {
+    let workflowName = job.data.workflowName;


dheepak-aot · 2023-09-12T19:11:42Z

sources/packages/backend/apps/queue-consumers/src/services/application/application.service.ts

+      )
+      .getQuery();
+    return (
+      this.applicationRepo


Just wondering, this query could have come from student assessments instead of applications right? so that it will just return the assesments(assessment id) which needs to be processed.

True. It was originally designed to allow the later update of the application + assessment together but it did not work as expected and I ended up doing the individual updates in a transaction.
Right now the only benefit is that it makes it pretty easy to have the assessments grouped to allow the processing and also pick the next assessment to be processed if multiple exists under the same application.

dheepak-aot · 2023-09-12T19:20:44Z

...ces/packages/backend/apps/queue-consumers/src/services/workflow/workflow-enqueuer.service.ts

+            currentProcessingAssessment: { id: nextAssessment.id },
+          });
+        if (!applicationUpdateResult.affected) {
+          throw new Error("Application update did not affected any records.");


The error message could be more specific e.g. currentProcessingAssessment id was already set or incorrect application id. something like that.

That log itself may not provide the entire picture, but please check below the logs around it. As shown below there is some additional info that would make it easier to have it traced.

About the suggestions:

"currentProcessingAssessment id was already set" would not be accurate.

"incorrect application id" can be checked using the additional logs also.

Queueing next pending assessment for application id 123.
Found 2 pending assessment(s). Queueing assessment 999.
Associating application currentProcessingAssessment as assessment id 999.
(...)
Application update did not affected any records.

By the way, planning to do some logging improvement in the next PR.

dheepak-aot · 2023-09-12T19:21:02Z

...ces/packages/backend/apps/queue-consumers/src/services/workflow/workflow-enqueuer.service.ts

+          });
+        if (!assessmentUpdateResults.affected) {
+          throw new Error(
+            "Student assessment update did not affected any records.",


https://github.com/bcgov/SIMS/pull/2285/files#r1323461343

#2285 (comment)

dheepak-aot · 2023-09-12T19:25:20Z

Great job with creating the queue to process assessments. Great to see the depth of analysis and considerations involved in the effort. Thanks for explaining the issue with migrations. Please have a look at the comments.

sonarqubecloud · 2023-09-12T20:01:15Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
1 Code Smell

No Coverage information
0.0% Duplication

github-actions · 2023-09-12T20:02:11Z

Backend Unit Tests Coverage Report

Totals
Statements:	17.54% ( 2181 / 12437 )
Methods:	8.02% ( 126 / 1572 )
Lines:	20.32% ( 1917 / 9432 )
Branches:	9.63% ( 138 / 1433 )

github-actions · 2023-09-12T20:03:26Z

E2E Workflow Workers Coverage Report

Totals
Statements:	46.73% ( 300 / 642 )
Methods:	40% ( 32 / 80 )
Lines:	51.02% ( 251 / 492 )
Branches:	24.29% ( 17 / 70 )

github-actions · 2023-09-12T20:03:41Z

E2E Queue Consumers Coverage Report

Totals
Statements:	67.54% ( 437 / 647 )
Methods:	58.02% ( 47 / 81 )
Lines:	69.93% ( 386 / 552 )
Branches:	28.57% ( 4 / 14 )

andrepestana-aot

Looks good! Thanks for the explanation.

github-actions · 2023-09-12T20:12:54Z

E2E SIMS API Coverage Report

Totals
Statements:	54.48% ( 3995 / 7333 )
Methods:	51.31% ( 488 / 951 )
Lines:	59.3% ( 3245 / 5472 )
Branches:	28.79% ( 262 / 910 )

dheepak-aot

Thanks for making the changes 👍

guru-aot

LGTM, nice work @andrewsignori-aot

andrewsignori-aot added 5 commits September 7, 2023 17:26

Starting the workflow on application submission.

224d811

Stop calling the queue for features

d348b75

DB migration to update student assessment

9d9497e

Temporary logs added.

a698573

Code cleanup

c6b3aed

andrewsignori-aot added Queue Consumers DB DB migration involved Camunda Worflow Involves camunda workflow changes labels Sep 8, 2023

andrewsignori-aot self-assigned this Sep 8, 2023

andrewsignori-aot changed the title ~~#2180 - Queued Assessments - Assessment Workflow Enqueuer Scheduler - Start Assessment (Part 1)~~ #2180 - Queued Assessments - Assessment Workflow Enqueuer Scheduler - Start Assessment (Part 1) Sep 8, 2023

andrewsignori-aot added 7 commits September 8, 2023 16:09

Typeorm DB migration transaction mode changed

8842e00

E2E test fix

33a9ea9

E2E test fix

cda732c

Pre-review adjustments

751b9a5

Update application/assessment changes.

0165fed

Error improvement

df7e3e7

DB migration improvements

51248f6

andrewsignori-aot requested review from andrepestana-aot, dheepak-aot, ann-aot, sh16011993, JasonCTang and guru-aot September 11, 2023 20:31

andrewsignori-aot marked this pull request as ready for review September 11, 2023 20:32

andrepestana-aot reviewed Sep 11, 2023

View reviewed changes

sources/packages/backend/apps/db-migrations/src/main.ts Outdated Show resolved Hide resolved

andrepestana-aot reviewed Sep 11, 2023

View reviewed changes

sources/packages/backend/apps/queue-consumers/src/services/application/application.service.ts Show resolved Hide resolved

andrepestana-aot reviewed Sep 11, 2023

View reviewed changes

ann-aot reviewed Sep 11, 2023

View reviewed changes

.../apps/db-migrations/src/sql/StudentAssessments/Rollback-update-student-assessment-status.sql Show resolved Hide resolved

ann-aot reviewed Sep 11, 2023

View reviewed changes

...ces/packages/backend/apps/queue-consumers/src/services/workflow/workflow-enqueuer.service.ts Outdated Show resolved Hide resolved

ann-aot reviewed Sep 11, 2023

View reviewed changes

...ces/packages/backend/apps/queue-consumers/src/services/workflow/workflow-enqueuer.service.ts Outdated Show resolved Hide resolved

guru-aot reviewed Sep 12, 2023

View reviewed changes

sources/packages/backend/apps/api/src/services/student-appeal/student-appeal.service.ts Show resolved Hide resolved

guru-aot reviewed Sep 12, 2023

View reviewed changes

...d/apps/api/src/services/student-scholastic-standings/student-scholastic-standings.service.ts Show resolved Hide resolved

guru-aot reviewed Sep 12, 2023

View reviewed changes

...packages/backend/apps/db-migrations/src/sql/Queue/Add-assessment-workflow-enqueuer-queue.sql Show resolved Hide resolved

PR comments

345050f

guru-aot reviewed Sep 12, 2023

View reviewed changes

sources/packages/backend/apps/queue-consumers/src/services/application/application.service.ts Show resolved Hide resolved

ann-aot approved these changes Sep 12, 2023

View reviewed changes

Reverted DB migration transaction mode.

a053969

dheepak-aot reviewed Sep 12, 2023

View reviewed changes

...s/backend/apps/db-migrations/src/sql/StudentAssessments/Update-student-assessment-status.sql Outdated Show resolved Hide resolved

dheepak-aot reviewed Sep 12, 2023

View reviewed changes

.../apps/db-migrations/src/sql/StudentAssessments/Rollback-update-student-assessment-status.sql Show resolved Hide resolved

dheepak-aot reviewed Sep 12, 2023

View reviewed changes

Removed submitted from DB migration

f5b5579

dheepak-aot reviewed Sep 12, 2023

View reviewed changes

Log improvement

c8578d6

andrepestana-aot approved these changes Sep 12, 2023

View reviewed changes

dheepak-aot approved these changes Sep 12, 2023

View reviewed changes

guru-aot approved these changes Sep 12, 2023

View reviewed changes

andrewsignori-aot merged commit 672f0a4 into main Sep 12, 2023

andrewsignori-aot deleted the feature/#2180-enqueuer-start-assessment branch September 12, 2023 20:32

andrewsignori-aot had a problem deploying to DEV September 12, 2023 20:44 — with GitHub Actions Failure

andrewsignori-aot had a problem deploying to DEV September 12, 2023 22:12 — with GitHub Actions Failure

andrewsignori-aot had a problem deploying to DEV September 12, 2023 22:50 — with GitHub Actions Failure

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

#2180 - Queued Assessments - Assessment Workflow Enqueuer Scheduler - Start Assessment (Part 1) #2285

#2180 - Queued Assessments - Assessment Workflow Enqueuer Scheduler - Start Assessment (Part 1) #2285

andrewsignori-aot commented Sep 8, 2023 •

edited

Loading

andrepestana-aot left a comment

ann-aot left a comment

dheepak-aot Sep 12, 2023

dheepak-aot Sep 12, 2023

dheepak-aot Sep 12, 2023

andrewsignori-aot Sep 12, 2023

dheepak-aot Sep 12, 2023

andrewsignori-aot Sep 12, 2023

andrewsignori-aot Sep 12, 2023

dheepak-aot Sep 12, 2023

andrewsignori-aot Sep 12, 2023

dheepak-aot commented Sep 12, 2023

sonarqubecloud bot commented Sep 12, 2023

github-actions bot commented Sep 12, 2023

github-actions bot commented Sep 12, 2023

github-actions bot commented Sep 12, 2023

andrepestana-aot left a comment

github-actions bot commented Sep 12, 2023

dheepak-aot left a comment

guru-aot left a comment

#2180 - Queued Assessments - Assessment Workflow Enqueuer Scheduler - Start Assessment (Part 1) #2285

#2180 - Queued Assessments - Assessment Workflow Enqueuer Scheduler - Start Assessment (Part 1) #2285

Conversation

andrewsignori-aot commented Sep 8, 2023 • edited Loading

andrepestana-aot left a comment

Choose a reason for hiding this comment

ann-aot left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dheepak-aot commented Sep 12, 2023

sonarqubecloud bot commented Sep 12, 2023

github-actions bot commented Sep 12, 2023

Backend Unit Tests Coverage Report

github-actions bot commented Sep 12, 2023

E2E Workflow Workers Coverage Report

github-actions bot commented Sep 12, 2023

E2E Queue Consumers Coverage Report

andrepestana-aot left a comment

Choose a reason for hiding this comment

github-actions bot commented Sep 12, 2023

E2E SIMS API Coverage Report

dheepak-aot left a comment

Choose a reason for hiding this comment

guru-aot left a comment

Choose a reason for hiding this comment

andrewsignori-aot commented Sep 8, 2023 •

edited

Loading