Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

e-Cert Implementation for Individual Student Processing #2510

Closed
14 tasks done
andrewsignori-aot opened this issue Nov 15, 2023 · 0 comments
Closed
14 tasks done

e-Cert Implementation for Individual Student Processing #2510

andrewsignori-aot opened this issue Nov 15, 2023 · 0 comments
Assignees
Labels

Comments

@andrewsignori-aot
Copy link
Collaborator

andrewsignori-aot commented Nov 15, 2023

Allow the processing of the student disbursements, individually processing its calculations and committing them to the database, defining them as "Ready to send" (new status to be added), which means, should be included in the next e-Cert generated. The news status would allow the e-Cert calculations and the file generation to happen at different times.

Benefits

  • Allow each disbursement to be individually processed and committed.
  • Allow code simplifications that no longer need to deal with bulk processing.
  • Allow the processing of multiple disbursements for the same students where account data as overawards deductions and maximums, must be updated (address #1761 - Handling full time BCSL lifetime and federal maximums #1886 (comment)).
  • In case one disbursement result has an issue it would not affect the other student disbursements.

Acceptance Criteria

  • Introduce a new Ready to send status for the DisbursementScheduleStatus. This status will be set when the calculations are done and the record should be part of the next e-Cert.
  • Add a new date column to capture the date when the Ready to send status was set. The suggested name ready_to_send_date in sims.disbursement_schedules.
  • Update the below reference codes to have the Ready to Send with the same effect as Sent, which means that SIMS will consider the money as sent either way (money was paid).
  • Adjust the current schedulers for full-time-e-cert-integration and part-time-e-cert-integration to execute the process with the proposed separation between calculations and the e-Cert generation.
    • Create a process to execute the individual calculations
      • Select all disbursements not sent yet and not calculated, which means in 'Pending' status.
      • Ensure that only non-archived applications will be selected. Once the applications are archived the system should stop checking them, as per confirmed with @HRAGANBC.
      • Disbursement date in the past or in the near future.
      • Application status must be 'Completed'.
    • For every select record, execute the calculation in parallel but ensure that records for the same student are processed sequentially by disbursement date ascendant.
      1. Create a DB transaction.
      2. Apply overawards deductions.
      3. Calculate effective value.
      4. Calculate the tuition remittance effective amount.
      5. Create BC Total grants award.
      6. Set the status to Ready to Send
      7. Commit the transaction.
    • Create smaller services to handle the individual e-Cert steps. There is already a separation into methods and a further separation into services would make it easier to apply unit tests (source code reference).
    • Allow selecting records to be processed with fewer conditions to generate a better individual log and easily capture the reason why a disbursement is not part of the e-Cert. For instance, allow students with canceled MSFAA or invalid SIN to be selected and execute an in-memory check to validate MSFAA and SIN, producing a log if necessary.
    • Create scoped logs for every single student calculation processed and ensure that the output is sent to the POD log and job log.
    • Create scoped logs for every single e-cert record generated and ensure that the output is sent to the POD log and job log.
@andrewsignori-aot andrewsignori-aot self-assigned this Nov 16, 2023
@andrewsignori-aot andrewsignori-aot changed the title E-cert Implementation for Individual Student Processing e-Cert Implementation for Individual Student Processing Nov 17, 2023
andrewsignori-aot added a commit that referenced this issue Nov 17, 2023
…rations and Entity Models Added (#2513)

Added the DB migrations needed for the e-Cert process changes.
- New `Ready to send` status type for the enum
`DisbursementScheduleStatus`.
- New column `ready_to_send_date` in table
`sims.disbursement_schedules`.
andrewsignori-aot added a commit that referenced this issue Nov 28, 2023

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
…essing-part-2
andrewsignori-aot added a commit that referenced this issue Dec 1, 2023
…2531)

### Overview
- Created the first E2E test for part-time e-Cert generation with basic
validations **and not validating** the file generated since it would
require better control of the saved data.
- Updated the E2E test for full-time using the most recently introduced
factories **and not validating** the file generated since it would
require better control of the saved data.
- Full-time/part-time schedulers were updated to use the most recent
logging process.
- e-Cert is now segregated in the calculation process and e-Cert file
generation process.

### New e-Cert calculations process based on steps
- The calculations are now segregated into isolated steps that can be
shared or custom for full-time/part-time.
- Validations were removed from the SQL query to in-memory to allow
better logging.
- The SQL query used by `prepareDisbursementsForECertGeneration` is now
segregated into two different queries for calculations and e-Cert
generation.
- The new queries are now using Typeorm object query syntax.
- The logs were extended to allow capturing information at every step of
the calculation process. **The logs for the e-Cert generation were not
updated.**
- The calculation steps related files in this PR have references to the
previous codes where they belonged before. The code was moved and
simplified.
- All students with eligible disbursements are processed in parallel and
disbursements for the same student are executed sequentially.
- Every student disbursement is individually part of a single DB
transaction that will be also individually committed.
andrewsignori-aot added a commit that referenced this issue Dec 8, 2023
… Tests (#2566)

### Local Load Test
- Executed tests with 15,000 eligible disbursements and executed
refactors to improve performance and fix issues.
- Some `save` operations were converted to `update`, for performance
improvement.
- During e-Cert generation, the records were updated to `Sent` at one
single operation which for a large amount of records was leading to an
idle transaction error because the SQL command was taking too long to be
executed.

### New E2Es added for part-time/full-time.

#### Full-time

_Should disburse BC funding for a close-to-maximum disbursement, reduce
BC funding when passing the maximum, and withhold BC Funding when a
restriction was applied due to the maximum configured value for the year
being reached._

This E2E was useful to validate a restriction being added to a
disbursement and affecting the upcoming disbursement for the same
student.

#### Part-time
_Should create an e-Cert with three disbursements for two different
students with two disbursements each where three records are eligible._

#### New E2E helpers

- Created a full-time/part-time parser to alow verification of
individual fields in an e-Cert.
- Created a helper to mock the Bull Job and also allow access to check
the generated logs.

### Minor refactor
as pointed out by @guru-aot, there was a type on
`DisbursementScheduleStatus.ReadyToSend` enum. The migration file names
were not updated.
andrewsignori-aot added a commit that referenced this issue Dec 12, 2023
…2574)

- Created the new based class `ECertProcessIntegrationBaseScheduler` to
be shared by the full-time and part-time e-Cert schedulers.
- Created also the base class `ECertFileHandler` to support the
above-mentioned change.
- Changed the e-Cert file generation to also use the `ProcessSummary`
logs.

Please note that refactors for the e-Cert feedback are out of the scope
of this effort.
andrewsignori-aot added a commit that referenced this issue Dec 14, 2023
… and Investigation (#2592)

- Fixed an issue for the BC `assert-life-time-maximum-full-time-step`
that would fail during the log due to a null exception while accessing
`bcLoan.valueCode`.
- Added `cause` to the errors generated during e-Cert record generation
for further investigation. The `cause` is present in `es2022` but only
`es2022.error` was updated right now because extra changes are needed to
completely update the backend to use `es2022`.
andrewsignori-aot added a commit that referenced this issue Dec 14, 2023
…ed application number to logs (#2597)

Adding the application number to e-Cert logs to make it easier to find
application-related issues.
The log entry would be changing from:
Processing disbursement id 56209 scheduled for 2023-12-17.
To
Processing application number 1077887860, disbursement id 56209 scheduled for 2023-12-17.
andrewsignori-aot added a commit that referenced this issue Dec 15, 2023
…ed COE Declined Constraint (#2599)

Added COE declined constraint to e-Cert eligibility criteria to ensure
that disbursements with declined COEs will not be considered for e-Cert
generations.
@michesmith michesmith added this to the 2.0 Part-Time Students MVP milestone May 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants