-
Notifications
You must be signed in to change notification settings - Fork 14
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CAS Integration 3A - Create new Supplier and Site #3260
Comments
@CarlyCotton I would like to request the change of the below AC, from
to
The point is that "dashes" can also be present on names and it is not an ASCII character hence it would make sense to have it added in the current ASCII conversion table that we use for the SFTP. Please let me know if there is any concern. |
@andrewsignori-aot Updated the AC. Happy to keep things consistent across SIMS when possible rather than have one-offs. |
…ier (#3784) - Centralized the method to convert "special" characters into ASCII. - Moved the `FILE_DEFAULT_ENCODING` const to the `utilities` lib to be consumed by the now shared `translateToASCII` method (around 9 files impacted because of this change). It was previously defined at the shared services level that has a dependency on the utility library. - Created unit tests for the `convertToASCII` and for the method `getSupplierInfoFromCAS` that is mocked in E2E tests. - Added the unit test mock helper [@suites/unit](https://github.com/suites-dev/suites), following the instructions from [here](https://github.com/suites-dev/suites). Please see the [unit test implemented](https://github.com/bcgov/SIMS/blob/2c52e6efb32c49b3be2af53eb5e2ebac6a8fb832/sources/packages/backend/libs/integrations/src/cas/_tests_/unit/cas.service.getSupplierInfoFromCAS.spec.ts) and/or the Nestjs doc to see the benefits. This was the one recommended in the [Nestjs page](https://docs.nestjs.com/recipes/automock) (please keep in mind that automock was renamed to suites). - Create the DB migration for the new type `Manual intervention` to be used in the upcoming PR. ### Migration revert ![image](https://github.com/user-attachments/assets/269fc593-2b8b-4707-8441-9e69d60af797)
…all (#3804) ## Main Changes - The current process is now segregated into different classes (called processors) that are responsible for taking care of the below scenarios. - `CASActiveSupplierNotFoundProcessor`: 🆕 - Supplier does not exist on CAS; - CAS API should be called to create supplier and site; - Student CAS supplier should be updated. - `CASPreValidationsProcessor`: 🆕 - Given name or not a Canadian address. - Update the student supplier for manual intervention. - CASActiveSupplierFoundProcessor: (🆕 refactored) - Supplier found on CAS - Student CAS supplier should be updated. ### Address match - Included the address match logic in the evaluation processes following the logic already decided in #3729. If an active address with a valid match is found it will be persisted and the CAS supplier will be considered valid. If the address does not exist or does not match, the supplier will be saved but the CAS record will be considered invalid. ### Error handling API call properly reports errors using HTTP Status 400 and adding meaningful error messages. Please see below samples. These error messages were used to define the max length validation for the submitted data. - [0069] Length of field SupplierName exceeded. Maximum length is 80. - [0034] SIN is already in use. - [0069] Length of field AddressLine1 exceeded. Maximum length is 35. ### GetSupplier 404 The `GetSupplier` method from CAS API does not return a 404 error when no supplier is found as stated in the documentation. The method seems to be implemented as a search and it will just return an empty array if no items were found. ```json { "items": [], "hasMore": false, "limit": 0, "offset": 0, "count": 0, } ``` ### Secondary PR changes - Created a small helper to identify if the student's address is from Canada. The idea is to avoid confusion about which property to use and how the value is set on DB. This is the reason the method receives the `AddressInfo` itself which contains `country` and `selectedCountry` properties. - New CAS mocks added to allow the CAS responses mocks. ### New E2E Tests - Should update CAS supplier table to manual intervention when student does not have a first name. - Should update CAS supplier table to manual intervention when student address is not from Canada. - Should create a new supplier and site on CAS and update CAS suppliers table when the student was not found on CAS and the request to create the supplier and site was successful. ### Log samples ```console [2024-10-17 22:53:31]--INFO: Log details [2024-10-17 22:53:31]----INFO: Log details [2024-10-17 22:53:31]----INFO: Logging on CAS. [2024-10-17 22:53:31]----INFO: Logon successful. [2024-10-17 22:53:31]------INFO: Log details [2024-10-17 22:53:31]------INFO: Processing student CAS supplier ID: 69. [2024-10-17 22:53:31]------INFO: CAS evaluation result status: NotFound. [2024-10-17 22:53:31]------INFO: No active CAS supplier found. Reason: Supplier not found. [2024-10-17 22:53:31]------INFO: Created supplier and site on CAS. [2024-10-17 22:53:31]------INFO: Updated CAS supplier and site for the student. [2024-10-17 22:53:31]INFO: CAS supplier integration executed. ``` ```console [2024-10-17 23:04:38]INFO: Log details [2024-10-17 23:04:38]INFO: Executing CAS supplier integration... [2024-10-17 23:04:38]INFO: Found 1 records to be updated. [2024-10-17 23:04:38]--INFO: Log details [2024-10-17 23:04:38]----INFO: Log details [2024-10-17 23:04:38]----INFO: Logging on CAS. [2024-10-17 23:04:38]----INFO: Logon successful. [2024-10-17 23:04:38]------INFO: Log details [2024-10-17 23:04:38]------INFO: Processing student CAS supplier ID: 86. [2024-10-17 23:04:38]------INFO: CAS evaluation result status: PreValidationsFailed. [2024-10-17 23:04:38]------WARN: Not possible to retrieve CAS supplier information because some pre-validations were not fulfilled. Reason(s): Non-Canadian address. [2024-10-17 23:04:38]------INFO: Updated CAS supplier for manual intervention. [2024-10-17 23:04:38]INFO: CAS supplier integration executed. ``` ```console [2024-10-17 22:55:57]INFO: Log details [2024-10-17 22:55:57]INFO: Executing CAS supplier integration... [2024-10-17 22:55:57]INFO: Found 1 records to be updated. [2024-10-17 22:55:57]--INFO: Log details [2024-10-17 22:55:57]----INFO: Log details [2024-10-17 22:55:57]----INFO: Logging on CAS. [2024-10-17 22:55:57]----INFO: Logon successful. [2024-10-17 22:55:57]------INFO: Log details [2024-10-17 22:55:57]------INFO: Processing student CAS supplier ID: 74. [2024-10-17 22:55:57]------INFO: CAS evaluation result status: ActiveSupplierFound. [2024-10-17 22:55:57]------INFO: Active CAS supplier found. [2024-10-17 22:55:57]------INFO: Updated CAS supplier for the student. [2024-10-17 22:55:57]INFO: CAS supplier integration executed. ``` ### TODO - Adjust the scheduler cron expression. - Add more E2E tests. - Run in parallel. - If some error is present throw in the scheduler to allow it to be retried.
…CAS authentication (#3822) As agreed in the previous PR, the CAS authentication is now encapsulated inside the CAS API service and the auth token no longer needs to be passed to every method. The current CAS token expiration is set to 1 hour and the token will be considered "about to expire" one minute before it.
…Error and Scheduler Time (#3830) - Changed the to allow throwing an error when some error is present in the process summary error attending the below AC. ``` If some error is present throw in the scheduler to allow it to be retried. ``` - Added the below E2E to test the above change. - Should throw an error for the first student and process the second one when the CAS API call failed for the first student but worked for the second one. - Adjusted the scheduler to be executed during weekdays only attending the below AC (`0 20 * * 1-5`). ``` Schedule these to go once a day at noon during weekdays for now. ``` ### Rollback ![image](https://github.com/user-attachments/assets/59a242ed-72ec-465b-a52f-8848144568f0) ### Fix - `COUNTRY_CANADA` was defined as "canada" when it is defined as "Canada" is all form.io definitions leading to an incorrect logic. - SFTP issue while providing the file content as a string instead of a Buffer as identified by @dheepak-aot.
…n Parallel (#3837) - Adjusted supplier's processes to happen in parallel. - Reduced the amount of allowed parallel processes to 2 to 4. Previously it was using the "os cores" as the default value which could potentially be up to 32 while running on Openshift.
DemoNew student created (New Supplier and Site Created)Basic BCeID used: andrew_signori_cas_1 New Supplier and Site CreationCompleted on Ministry PortalBull Logs{
"jobData": {},
"returnValue": [
"Process finalized with success.",
"Pending suppliers to update found: 1.",
"Records updated: 1."
]
} Log details
Dabase Persisted FormatsNew student created (No first name and no Canadian Address)Basic BCeID used: andrew_signori_cas_2 Completed on Ministry PortalBull Logs{
"jobData": {},
"returnValue": [
"Process finalized with success.",
"Pending suppliers to update found: 1.",
"Records updated: 1.",
"Attention, process finalized with success but some errors and/or warnings messages may require some attention.",
"Error(s): 0, Warning(s): 1, Info: 10"
]
} Log details
New student created (active student present on CAS, and active matching address)Basic BCeID used: andrew_signori_cas_3 Completed on Ministry PortalBull Logs{
"jobData": {},
"returnValue": [
"Process finalized with success.",
"Pending suppliers to update found: 1.",
"Records updated: 1."
]
} Log details
@CarlyCotton from the above scenarios, which ones would you like to see on a demo? Some, all? |
…And Error Log Improvement (#3843) - Fixed the order that first and last names were provided to the formatter. - Added the HTTP bad request logs for better troubleshooting while the final error handling ticket is pending. ### Sample logged error (Invalid SIN). ![image](https://github.com/user-attachments/assets/04626581-6a3b-4a9e-a312-2bb194e670e7)
…e Sites / No matching address) (#3884) - If a Get Supplier Response is Successful and returns Active Supplier and NO active Sites, **Create New Site** - If a Get Supplier Response is Successful and returns Active Supplier and ONE OR MANY active Site, Compare the Response AddressLine1 and Postal Code to SIMS address - The address match logic was already added during the implementation of [#3260](#3260). - Created the new API `createExistingSupplierSite` call on CAS service. - Fields to include: - SupplierNumber, - AddressLine1: Use first addressline from SIMS. Truncate if ours is longer than supported. Convert or omit special characters (like is done with other integrations). - City: Use as-is, - Province: Use abbreviated versions AB,BC,MB,NB,NL,NS,NT,NU,ON,PE,QC,YT, - Country, - PostalCode: ALL CAPS, no space, - EmailAddress: Use as-is - Created Unit test - "Should invoke CAS API createSiteForExistingSupplier with formatted payload when all data was provided as expected." - Created E2E test - "Should create a new site and update the student CAS supplier when an active CAS supplier exists with no match addresses." - "Should create a new site and update the student CAS supplier when an inactive CAS supplier exists with matching addresses."
User Story:
As an SABC Finance team member (Ministry user), I need all students in SIMS to have a supplier and site number attached. If we do not receive a match from the initial CAS supplier search after SIN validation, we need to create a new supplier and site for the student.
Acceptance Criteria:
Get Supplier (Last Name and SIN)
Get Supplier Response "None found/0 Count":
Schedule these to go once a day at noon during weekdays for now.
Call Create New Supplier and Site (command 1)
Fields to include (See Field Requirements File below): SIN, AddressLine1, City, Province, Country, PostalCode, EmailAddress (Optional/Conditional fields can be left out or left null)
If successful response to Create New Site & Supplier, then update SIMS CAS table with the returned
supplierNumber
,SiteCode
and any other required info and setisValid
=trueIf error response, retry for now. Manual intervention may be required and will be dealt with on ticket CAS Integration 4A - Supplier Maintenance Errors #3258
Business Context
Supplier Web Service RESTful Client Onboarding v02_01.pdf
Individual Supplier Web Service Field Requirements (All Requests) JAN2023.xlsx
Technical Context
Sample CURL to create
Response:
{"SUPPLIER_NUMBER":"2007638","SUPPLIER_SITE_CODE":"[001] ","CAS-Returned-Messages":"SUCCEEDED"}
Business Context
Command 1 - Create New Supplier and Site Call
SupplierName:
Successful Response
Technical
hyphen-minus
is the ASCII character 45 while the "dash" is a Unicode character, hence it would make sense to add it to the conversation shared with the SFTP."CAS-Returned-Messages":"SUCCEEDED"
.SUCCEEDED
log the error message.Manual intervention required
.Manual Intervention Required
, set theis_valid
column tofalse
.sims.cas_suppliers
table when the student was not found on CAS and the request to create the supplier was successful.The text was updated successfully, but these errors were encountered: