Process assessments in the queue

[andrewsignori-aot]

• Camunda workflows will no longer be invoked when new student_assessment records are created. Triggering a workflow will be a responsibility of a new scheduler.
• All the student_assessment records will be associated with the application's current assessment alongside its creation, which means that the current assessment will always point to the most recent assessment record.
• Multiple student_assessment records can be created and all will be executed in the creation order.
• The student_assessment records for the same application should be executed in order, but different applications for the same students can be executed in parallel.
• Two new schedulers will be created.
  • Assessment workflow execution (every few seconds): used to start and cancel workflows.
  • Assessment workflow execution retry (every few hours): used to retry start and cancelation operations that are taking too long.
• Two existing queues will be still used.
  • start-application-assessment
  • cancel-application-assessment
• New status will be created for the student_assessment table to allow its control by the schedulers.
  • Add also an extra column, similar to sims.applications.application_status_updated_on to track the date that the status was updated (e.g. student_assessment_status_updated_on).
• Add a new column current_processing_assessment_id to the sims.applications to be updated by the scheduler. Every time that a record is queued for execution this new column will also be updated.
• Student Application cancellation will happen only before the application is in "Completed" status. Right now there is no scenario where a completed application or an application with reassessment records would be canceled.

New status for the assessment

• Created: when the student_assessment is created.
  • Set by the code creating the record. Can be the default value for the new column.
• Queued: when the student_assessment is queued to be executed on Camunda. Change the status to Queued and call the queue after, in this way, if the Camunda call fails it will retry later because the status is already set.
  • Set by the new scheduler: Assessment workflow execution.
• In progress: when the workflow id is associated with the student_assessment.
  • Se from a Camunda worker (e.g. associate-workflow-instance-task).
• Completed: at the end by a dedicated worker.
  • Set by a new Camunda worker to be created.
• Cancellation requested: when a workflow needs to be canceled, for instance, when the student cancels the application or the "Bulk offering update" happens and the application is in progress. The student_assessment can have the status updated to Cancellation requested to be selected by the specific queue and then have its status finally updated to Cancelled.
  • Set by the application code requesting the cancellation.
• Cancellation queued: create a new item in the queue cancel-application-assessment. This queue should also be changed to update the student_assessment to Cancelled.
  • Set by the new scheduler Assessment workflow execution.
• Cancelled: when the workflow is canceled or a call to Camunda to cancel the workflow returns a "Not found".
  • Set by the existing queue cancel-application-assessment.

Regular Assessment Lifecycle

For a single workflow execution, the assessments will progress as shown below.

Stage 1

The student_assessment record is saved to the database with the default Created status.

Stage 2

The record with Created status:

1. is selected by the assessment-workflow-execution-queue scheduler.
2. its status is updated to Queued.
3. Its id is also set in the application's current_processing_assessment_id column.
4. it is added to the start-application-assessment queue.

Stage 3

1. The workflow has its execution started by Camunda.
2. The associate-workflow-instance worker, besides setting the workflow_id in DB (already happening), also sets the workflow status to In progress.

During this stage, alongside the associate-workflow-instance execution, if another workflow was already set to process the student_assessment, use the job.cancelWorkflow(); to terminate its execution.

Stage 4

A new workflow worker is created at the end of the workflow to have the student_assessment record changed to completed.

Queued Assessment Lifecycle

On the occasion when more than one student_assessment record is created, a scheduler will coordinate their ordered execution.
At any given time, assessments for one particular application can have 0 to 1 "Queued" records and 1 to n created records waiting to be processed.
Every time that a student_assessment record is queued its id is also set in the application's current_processing_assessment_id column.

Stage 1

Private Zenhub Image

Stage 2

Stage 3

Assessment Cancellation Lifecycle

The cancellation of a workflow is currently executed by the queue cancel-application-assessment. The Offering bulk change, for instance, executes also the bulk cancellation of any In progress application.
To allow the cancellation to also be handled by the schedulers, we can use the assessment status Cancellation requested to update any student-assessment to be canceled. The scheduler can then look for these records and try to execute the process and set the student-assessment as canceled.
The workflow cancellation process happens immediately and right now it is processed by cancel-application-assessment.processor.ts.
Student Application cancellation will happen only before the application is in "Completed" status. Right now there is no scenario where a completed application or an application with reassessment records would be canceled.

Schedulers

Assessment workflow execution (assessment-workflow-execution-queue)

Coordinate the execution of all the student_assessments to ensure its execution. Allow multiple student_assessments to be created and ensures their ordered execution and its association with the application's current assessment.

Execution interval

Executed every minute or some short interval to be discussed.

Target records

• Any application where the current assessment is in Created status.
• Any application where the current assessment is in Completed status and there is 1 to n Created assessment (waiting to be executed). Only the oldest assessment record in Created status will be selected for each application.

Process description

• If the current assessment is in Created status.
  1 - Change the status to Queued.
  2 - Queue the assessment for execution (already existing start-application-assessment).
• If the current assessment is NOT in Created status.
  1 - Change the status to Queued.
  2 - Update the application's current assessment to point to this record.
  3 - Queue the assessment for execution (already existing start-application-assessment).

Assessment Workflow execution retry (assessment-workflow-execution-queue-retry)

Checks for any stalled Queued student_assessments that did not start after a long period.

Execution interval

Executed every few hours or some long interval to be discussed.

Target records

Any application where the current assessment is in Queued status for some long period of time (e.g. 6 hours).

Process description

Queue the assessment for execution (already existing start-application-assessment).

Refactors

Schedulers should trigger workflows

Stop triggering the start-application-assessment and the cancel-application-assessment from the applications. Only the new schedulers would be responsible for triggering the Camunda workflows and canceling them. We can consider maybe letting the "Original Assessment" trigger the assessment for a better user experience once the application is submitted.

Current points triggering the assessment workflow:

• Application submission application.students.controller.ts
• Student Appeal Request student-appeal.aest.controller.ts
• Offering bulk change education-program-offering.service.ts
• Scholastic standing student-scholastic-standings.institutions.controller.ts

Current points for canceling the assessment workflow:

• Offering bulk change education-program-offering.service.ts

Notifications

Some assessment creations, like Scholastic Standing, also create notifications for the students. We should consider triggering these notifications when the specific assessment becomes the application's current assessment.

Other refactors

• Check if the code current inside method startAssessment can be moved to the start-application-assessment queue.
• Remove the no longer needed code assertAllAssessmentsCompleted.

[andrewsignori-aot]

Friendly business description

Summary of the technical work done so far for this epic (#1896, #2179, #2183, #2180).

• Workflows are no longer initiated as soon as they are created. For instance, submitting a Student application can take up to 30s to start the process.
• Applications (even from the same student) can have assessments processed at the same time (in parallel). A single application will have only one assessment being processed at a single time, which means, if two assessments are pending for the same application, the oldest will be processed and, once this is done, the next one will start.
• Any batch operation that can potentially create multiple assessments is now supported, like the below ones.
  • Business validations for Bulk Withdrawal Upload (Business validations for Bulk Withdrawal Upload #2112)
  • Institution Edits Offering (Institution Edits Offering #1281)
• Assessments can now be approved while the same application has already been with one assessment in progress. Before if the Ministry tried to approve one assessment while the other was in progress it would generate an issue.
• Better error handling when the same assessment is started twice. Before an incident was generated in Camunda. Now the system can smart detect it and cancel the second incorrect instance.
• Logs improvements.