Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CAS Integration 3A - Create new Supplier and Site #3260

Closed
34 of 36 tasks
HRAGANBC opened this issue May 9, 2024 · 3 comments
Closed
34 of 36 tasks

CAS Integration 3A - Create new Supplier and Site #3260

HRAGANBC opened this issue May 9, 2024 · 3 comments

Comments

@HRAGANBC
Copy link
Collaborator

HRAGANBC commented May 9, 2024

User Story:

As an SABC Finance team member (Ministry user), I need all students in SIMS to have a supplier and site number attached. If we do not receive a match from the initial CAS supplier search after SIN validation, we need to create a new supplier and site for the student.

Acceptance Criteria:

  • Create new Supplier Status: "Manual Intervention" Set the flag isValid=FALSE

Get Supplier (Last Name and SIN)

  • Ensure initial Get Supplier call uses the follow name format to search:
    • ALL upper case letters
    • Hyphens accepted, not dashes
    • Special characters: Please use our current translation strategy to convert special characters where possible, or omit them.
    • If the student does NOT have a Given name, do NOT attempt the Get Supplier Call. Status: Manual Intervention Required
    • If the student does NOT have a Canada address, do NOT attempt the Get Supplier Call. Status: Manual Intervention Required

Get Supplier Response "None found/0 Count":

  • Schedule these to go once a day at noon during weekdays for now.

  • Call Create New Supplier and Site (command 1)

  • Fields to include (See Field Requirements File below): SIN, AddressLine1, City, Province, Country, PostalCode, EmailAddress (Optional/Conditional fields can be left out or left null)

    • Name Field: Use ALL CAPS. LASTNAME, GIVENNAMES (must have a space after the comma). Hyphens accepted, not dashes (CAS only translation).
    • AddressLine1: Use first addressline from SIMS. Truncate if ours is longer than supported. Convert or omit special characters (like is done with other integrations).
    • City: Use as-is
    • Province: Use abbreviated versions AB,BC,MB,NB,NL,NS,NT,NU,ON,PE,QC,YT
    • Country: For Canada: CA
    • Postal Code: ALL CAPS, no space.
    • Email Address: Use as-is
  • If successful response to Create New Site & Supplier, then update SIMS CAS table with the returned supplierNumber, SiteCode and any other required info and set isValid=true

  • If error response, retry for now. Manual intervention may be required and will be dealt with on ticket CAS Integration 4A - Supplier Maintenance Errors #3258

image.png

Business Context
Supplier Web Service RESTful Client Onboarding v02_01.pdf

Individual Supplier Web Service Field Requirements (All Requests) JAN2023.xlsx

Technical Context

Sample CURL to create

curl --location '${BASEURL}/cfs/supplier/' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer ${TOKEN}' \
--data-raw '{
    SupplierName: '\''Skywalker, Leia'\'',
    SubCategory: '\''Individual'\'',
    Sin: '\''422233197'\'',
    SupplierAddress: [
    {
        AddressLine1: '\''1012 Douglas St'\'',
        City: '\''Victoria'\'',
        Province: '\''BC'\'',
        Postalcode: '\''V8W 2C3'\'',
        Country: '\''Canada'\'',
        EmailAddress: '\''[email protected]'\'',
    }]
}'

Response:

{"SUPPLIER_NUMBER":"2007638","SUPPLIER_SITE_CODE":"[001] ","CAS-Returned-Messages":"SUCCEEDED"}

Business Context

Command 1 - Create New Supplier and Site Call

SupplierName:

{
    "SupplierName":"SMITH, SUSAN",
    "SubCategory":"Individual",
    "Sin":"",
    "BusinessNumber":"",
    "SupplierAddress":[
       {
           "AddressLine1":"5522 Oyama Lake Rd",
           "AddressLine2":"",
           "AddressLine3":"",
           "City":"Lake Country",
           "Province":"BC",
           "Country":"CA",
           "PostalCode":"V4V2C9",
           "EmailAddress":[email protected],
           "EftAdvicePref": 
       }
    ]
}

Successful Response

{
    "Supplier_Number":"123456789",
    "Supplier_Site_Code":"001",
    "CAS-Returned-Messages":"SUCCESS"
}
**Error Response**
{
    "Supplier_Number":"123456789",
    "Supplier_Site_Code":"001",
    "CAS-Returned-Messages":"Error Message"
}

Technical

  • While creating the new supplier on CAS ensure the name also has the special characters removed.
  • To handle the " Special characters", extract the logic from method convertRawContent to the string-utils.ts.
    • The method should receive a string and return a string.
    • Add the hyphen/dash conversion. hyphen-minus is the ASCII character 45 while the "dash" is a Unicode character, hence it would make sense to add it to the conversation shared with the SFTP.
    • Check if the request was successful by inspecting the response property "CAS-Returned-Messages":"SUCCEEDED".
      • If the value is different than SUCCEEDED log the error message.
  • Create the new status Manual intervention required.
    • For the business AC that leads to Manual Intervention Required, set the is_valid column to false.
  • Log to the ProcessSummary the individual errors for the students.
    • Ensure that all the students will have at least one attempt, meaning do not stop processing at the first student error.
    • If some error is present throw in the scheduler to allow it to be retried.
  • Expand existing E2E tests. Please add at least the below positive scenario.
    • Should create a new supplier on CAS and update sims.cas_suppliers table when the student was not found on CAS and the request to create the supplier was successful.
    • Should set the case supplier record status as 'Manual Intervention Required' when the student does not have the first name.
  • Breaking the PR.
    • Adjust the "Get Supplier", including the characters translations.
    • Create the CAS call.
@HRAGANBC HRAGANBC changed the title Copy of CAS Integration - Add CAS Supplier Info to Student Profile (SFAS Import & Get Supplier Number) CAS Integration - Add CAS Supplier Info to Student Profile (Create new Supplier/Site & Update Existing Site) May 9, 2024
@HRAGANBC HRAGANBC changed the title CAS Integration - Add CAS Supplier Info to Student Profile (Create new Supplier/Site & Update Existing Site) CAS Integration 2 - Add CAS Supplier Info to Student Profile (Create new Supplier/Site & Update Existing Site) May 9, 2024
@HRAGANBC HRAGANBC changed the title CAS Integration 2 - Add CAS Supplier Info to Student Profile (Create new Supplier/Site & Update Existing Site) CAS Integration 3 - Add CAS Supplier Info to Student Profile (Create new Supplier/Site & Update Existing Site) May 13, 2024
@sslaws sslaws added the Business Items under Business Consideration label May 17, 2024
@HRAGANBC HRAGANBC changed the title CAS Integration 3 - Add CAS Supplier Info to Student Profile (Create new Supplier/Site & Update Existing Site) CAS Integration 3 - Create new Supplier, Create new Site, & Update Existing Site Jul 9, 2024
@HRAGANBC HRAGANBC assigned CarlyCotton and unassigned HRAGANBC Jul 24, 2024
@CarlyCotton CarlyCotton changed the title CAS Integration 3 - Create new Supplier, Create new Site, & Update Existing Site CAS Integration 3A - Create new Supplier and Site Aug 1, 2024
@CarlyCotton CarlyCotton added Dev & Architecture Development and Architecture and removed Business Items under Business Consideration labels Aug 20, 2024
@CarlyCotton CarlyCotton added Business Items under Business Consideration and removed Dev & Architecture Development and Architecture labels Sep 10, 2024
@CarlyCotton CarlyCotton added Dev & Architecture Development and Architecture and removed Business Items under Business Consideration labels Sep 26, 2024
@andrewsignori-aot andrewsignori-aot removed the Dev & Architecture Development and Architecture label Oct 3, 2024
@andrewsignori-aot andrewsignori-aot self-assigned this Oct 7, 2024
@andrewsignori-aot
Copy link
Collaborator

@CarlyCotton I would like to request the change of the below AC, from

Hyphens accepted, not dashes (CAS only translation)

to

Hyphens accepted, not dashes.

The point is that "dashes" can also be present on names and it is not an ASCII character hence it would make sense to have it added in the current ASCII conversion table that we use for the SFTP. Please let me know if there is any concern.

@CarlyCotton
Copy link
Collaborator

@andrewsignori-aot Updated the AC. Happy to keep things consistent across SIMS when possible rather than have one-offs.

github-merge-queue bot pushed a commit that referenced this issue Oct 11, 2024
…ier (#3784)

- Centralized the method to convert "special" characters into ASCII.
- Moved the `FILE_DEFAULT_ENCODING` const to the `utilities` lib to be
consumed by the now shared `translateToASCII` method (around 9 files
impacted because of this change). It was previously defined at the
shared services level that has a dependency on the utility library.
- Created unit tests for the `convertToASCII` and for the method
`getSupplierInfoFromCAS` that is mocked in E2E tests.
- Added the unit test mock helper
[@suites/unit](https://github.com/suites-dev/suites), following the
instructions from [here](https://github.com/suites-dev/suites). Please
see the [unit test
implemented](https://github.com/bcgov/SIMS/blob/2c52e6efb32c49b3be2af53eb5e2ebac6a8fb832/sources/packages/backend/libs/integrations/src/cas/_tests_/unit/cas.service.getSupplierInfoFromCAS.spec.ts)
and/or the Nestjs doc to see the benefits. This was the one recommended
in the [Nestjs page](https://docs.nestjs.com/recipes/automock) (please
keep in mind that automock was renamed to suites).
- Create the DB migration for the new type `Manual intervention` to be
used in the upcoming PR.

### Migration revert


![image](https://github.com/user-attachments/assets/269fc593-2b8b-4707-8441-9e69d60af797)
github-merge-queue bot pushed a commit that referenced this issue Oct 21, 2024
…all (#3804)

## Main Changes

- The current process is now segregated into different classes (called
processors) that are responsible for taking care of the below scenarios.
  - `CASActiveSupplierNotFoundProcessor`: 🆕
    - Supplier does not exist on CAS;
    - CAS API should be called to create supplier and site;
    - Student CAS supplier should be updated.
  - `CASPreValidationsProcessor`: 🆕
    - Given name or not a Canadian address.
    - Update the student supplier for manual intervention.
  - CASActiveSupplierFoundProcessor: (🆕 refactored)
    - Supplier found on CAS
    - Student CAS supplier should be updated. 

### Address match
- Included the address match logic in the evaluation processes following
the logic already decided in #3729. If an active address with a valid
match is found it will be persisted and the CAS supplier will be
considered valid. If the address does not exist or does not match, the
supplier will be saved but the CAS record will be considered invalid.

### Error handling
API call properly reports errors using HTTP Status 400 and adding
meaningful error messages. Please see below samples.
These error messages were used to define the max length validation for
the submitted data.
- [0069] Length of field SupplierName exceeded. Maximum length is 80.
- [0034] SIN is already in use.
- [0069] Length of field AddressLine1 exceeded. Maximum length is 35.

### GetSupplier 404
The `GetSupplier` method from CAS API does not return a 404 error when
no supplier is found as stated in the documentation.
The method seems to be implemented as a search and it will just return
an empty array if no items were found.
```json
{
    "items": [],
    "hasMore": false,
    "limit": 0,
    "offset": 0,
    "count": 0,
}
```

### Secondary PR changes
- Created a small helper to identify if the student's address is from
Canada. The idea is to avoid confusion about which property to use and
how the value is set on DB. This is the reason the method receives the
`AddressInfo` itself which contains `country` and `selectedCountry`
properties.
- New CAS mocks added to allow the CAS responses mocks.

### New E2E Tests

- Should update CAS supplier table to manual intervention when student
does not have a first name.
- Should update CAS supplier table to manual intervention when student
address is not from Canada.
- Should create a new supplier and site on CAS and update CAS suppliers
table when the student was not found on CAS and the request to create
the supplier and site was successful.

### Log samples

```console
[2024-10-17 22:53:31]--INFO: Log details
[2024-10-17 22:53:31]----INFO: Log details
[2024-10-17 22:53:31]----INFO: Logging on CAS.
[2024-10-17 22:53:31]----INFO: Logon successful.
[2024-10-17 22:53:31]------INFO: Log details
[2024-10-17 22:53:31]------INFO: Processing student CAS supplier ID: 69.
[2024-10-17 22:53:31]------INFO: CAS evaluation result status: NotFound.
[2024-10-17 22:53:31]------INFO: No active CAS supplier found. Reason: Supplier not found.
[2024-10-17 22:53:31]------INFO: Created supplier and site on CAS.
[2024-10-17 22:53:31]------INFO: Updated CAS supplier and site for the student.
[2024-10-17 22:53:31]INFO: CAS supplier integration executed.
```
```console
[2024-10-17 23:04:38]INFO: Log details
[2024-10-17 23:04:38]INFO: Executing CAS supplier integration...
[2024-10-17 23:04:38]INFO: Found 1 records to be updated.
[2024-10-17 23:04:38]--INFO: Log details
[2024-10-17 23:04:38]----INFO: Log details
[2024-10-17 23:04:38]----INFO: Logging on CAS.
[2024-10-17 23:04:38]----INFO: Logon successful.
[2024-10-17 23:04:38]------INFO: Log details
[2024-10-17 23:04:38]------INFO: Processing student CAS supplier ID: 86.
[2024-10-17 23:04:38]------INFO: CAS evaluation result status: PreValidationsFailed.
[2024-10-17 23:04:38]------WARN: Not possible to retrieve CAS supplier information because some pre-validations were not fulfilled. Reason(s): Non-Canadian address.
[2024-10-17 23:04:38]------INFO: Updated CAS supplier for manual intervention.
[2024-10-17 23:04:38]INFO: CAS supplier integration executed.
```
```console
[2024-10-17 22:55:57]INFO: Log details
[2024-10-17 22:55:57]INFO: Executing CAS supplier integration...
[2024-10-17 22:55:57]INFO: Found 1 records to be updated.
[2024-10-17 22:55:57]--INFO: Log details
[2024-10-17 22:55:57]----INFO: Log details
[2024-10-17 22:55:57]----INFO: Logging on CAS.
[2024-10-17 22:55:57]----INFO: Logon successful.
[2024-10-17 22:55:57]------INFO: Log details
[2024-10-17 22:55:57]------INFO: Processing student CAS supplier ID: 74.
[2024-10-17 22:55:57]------INFO: CAS evaluation result status: ActiveSupplierFound.
[2024-10-17 22:55:57]------INFO: Active CAS supplier found.
[2024-10-17 22:55:57]------INFO: Updated CAS supplier for the student.
[2024-10-17 22:55:57]INFO: CAS supplier integration executed.
```

### TODO
- Adjust the scheduler cron expression.
- Add more E2E tests.
- Run in parallel.
- If some error is present throw in the scheduler to allow it to be
retried.
github-merge-queue bot pushed a commit that referenced this issue Oct 23, 2024
…CAS authentication (#3822)

As agreed in the previous PR, the CAS authentication is now encapsulated
inside the CAS API service and the auth token no longer needs to be
passed to every method.
The current CAS token expiration is set to 1 hour and the token will be
considered "about to expire" one minute before it.
github-merge-queue bot pushed a commit that referenced this issue Oct 24, 2024
…Error and Scheduler Time (#3830)

- Changed the to allow throwing an error when some error is present in
the process summary error attending the below AC.
```
If some error is present throw in the scheduler to allow it to be retried.
```
- Added the below E2E to test the above change.
- Should throw an error for the first student and process the second one
when the CAS API call failed for the first student but worked for the
second one.
- Adjusted the scheduler to be executed during weekdays only attending
the below AC (`0 20 * * 1-5`).
```
Schedule these to go once a day at noon during weekdays for now.
```

### Rollback

![image](https://github.com/user-attachments/assets/59a242ed-72ec-465b-a52f-8848144568f0)

### Fix
- `COUNTRY_CANADA` was defined as "canada" when it is defined as
"Canada" is all form.io definitions leading to an incorrect logic.
- SFTP issue while providing the file content as a string instead of a
Buffer as identified by @dheepak-aot.
github-merge-queue bot pushed a commit that referenced this issue Oct 25, 2024
…n Parallel (#3837)

- Adjusted supplier's processes to happen in parallel.
- Reduced the amount of allowed parallel processes to 2 to 4. Previously
it was using the "os cores" as the default value which could potentially
be up to 32 while running on Openshift.
@andrewsignori-aot
Copy link
Collaborator

andrewsignori-aot commented Oct 25, 2024

Demo

New student created (New Supplier and Site Created)

Basic BCeID used: andrew_signori_cas_1

image.png

New Supplier and Site Creation

Completed on Ministry Portal

image.png

Bull Logs

{
  "jobData": {},
  "returnValue": [
    "Process finalized with success.",
    "Pending suppliers to update found: 1.",
    "Records updated: 1."
  ]
}
Log details
[2024-10-28 18:59:53]INFO: Log details
[2024-10-28 18:59:45]INFO: Executing CAS supplier integration...
[2024-10-28 18:59:45]INFO: Found 1 records to be updated.
[2024-10-28 18:59:53]--INFO: Log details
[2024-10-28 18:59:53]----INFO: Log details
[2024-10-28 18:59:53]------INFO: Log details
[2024-10-28 18:59:45]------INFO: Processing student CAS supplier ID: 119.
[2024-10-28 18:59:47]------INFO: CAS evaluation result status: NotFound.
[2024-10-28 18:59:47]------INFO: No active CAS supplier found. Reason: Supplier not found.
[2024-10-28 18:59:53]------INFO: Created supplier and site on CAS.
[2024-10-28 18:59:53]------INFO: Updated CAS supplier and site for the student.
[2024-10-28 18:59:53]INFO: CAS supplier integration executed.

Dabase Persisted Formats

image.png

New student created (No first name and no Canadian Address)

Basic BCeID used: andrew_signori_cas_2

image.png

Completed on Ministry Portal

image.png

Bull Logs

{
  "jobData": {},
  "returnValue": [
    "Process finalized with success.",
    "Pending suppliers to update found: 1.",
    "Records updated: 1.",
    "Attention, process finalized with success but some errors and/or warnings messages may require some attention.",
    "Error(s): 0, Warning(s): 1, Info: 10"
  ]
}
Log details
[2024-10-28 19:49:56]INFO: Log details
[2024-10-28 19:49:55]INFO: Executing CAS supplier integration...
[2024-10-28 19:49:56]INFO: Found 1 records to be updated.
[2024-10-28 19:49:56]--INFO: Log details
[2024-10-28 19:49:56]----INFO: Log details
[2024-10-28 19:49:56]------INFO: Log details
[2024-10-28 19:49:56]------INFO: Processing student CAS supplier ID: 120.
[2024-10-28 19:49:56]------INFO: CAS evaluation result status: PreValidationsFailed.
[2024-10-28 19:49:56]------WARN: Not possible to retrieve CAS supplier information because some pre-validations were not fulfilled. Reason(s): Given names not present, Non-Canadian address.
[2024-10-28 19:49:56]------INFO: Updated CAS supplier for manual intervention.
[2024-10-28 19:49:56]INFO: CAS supplier integration executed.

New student created (active student present on CAS, and active matching address)

Basic BCeID used: andrew_signori_cas_3

image.png

Completed on Ministry Portal

image.png

Bull Logs

{
  "jobData": {},
  "returnValue": [
    "Process finalized with success.",
    "Pending suppliers to update found: 1.",
    "Records updated: 1."
  ]
}
Log details
[2024-10-28 20:10:25]INFO: Log details
[2024-10-28 20:10:20]INFO: Executing CAS supplier integration...
[2024-10-28 20:10:20]INFO: Found 1 records to be updated.
[2024-10-28 20:10:25]--INFO: Log details
[2024-10-28 20:10:25]----INFO: Log details
[2024-10-28 20:10:25]------INFO: Log details
[2024-10-28 20:10:20]------INFO: Processing student CAS supplier ID: 121.
[2024-10-28 20:10:25]------INFO: CAS evaluation result status: ActiveSupplierAndSiteFound.
[2024-10-28 20:10:25]------INFO: Active CAS supplier and site found.
[2024-10-28 20:10:25]------INFO: Updated CAS supplier for the student.
[2024-10-28 20:10:25]INFO: CAS supplier integration executed.

@CarlyCotton from the above scenarios, which ones would you like to see on a demo? Some, all?
If all I can reuse the users and present the UI results and logs.
If there is a missing scenario please let me know that I can prepare for tomorrow.

github-merge-queue bot pushed a commit that referenced this issue Oct 28, 2024
…And Error Log Improvement (#3843)

- Fixed the order that first and last names were provided to the
formatter.
- Added the HTTP bad request logs for better troubleshooting while the
final error handling ticket is pending.

### Sample logged error (Invalid SIN).

![image](https://github.com/user-attachments/assets/04626581-6a3b-4a9e-a312-2bb194e670e7)
github-merge-queue bot pushed a commit that referenced this issue Nov 12, 2024
…e Sites / No matching address) (#3884)

- If a Get Supplier Response is Successful and returns Active Supplier
and NO active Sites, **Create New Site**
- If a Get Supplier Response is Successful and returns Active Supplier
and ONE OR MANY active Site, Compare the Response AddressLine1 and
Postal Code to SIMS address
- The address match logic was already added during the implementation of
[#3260](#3260).
- Created the new API `createExistingSupplierSite` call on CAS service.
   - Fields to include: 
     - SupplierNumber, 
- AddressLine1: Use first addressline from SIMS. Truncate if ours is
longer than supported. Convert or omit special characters (like is done
with other integrations).
     - City: Use as-is, 
- Province: Use abbreviated versions
AB,BC,MB,NB,NL,NS,NT,NU,ON,PE,QC,YT,
     - Country, 
     - PostalCode: ALL CAPS, no space, 
     - EmailAddress: Use as-is
- Created Unit test
- "Should invoke CAS API createSiteForExistingSupplier with formatted
payload when all data was provided as expected."
- Created  E2E test
- "Should create a new site and update the student CAS supplier when an
active CAS supplier exists with no match addresses."
- "Should create a new site and update the student CAS supplier when an
inactive CAS supplier exists with matching addresses."
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants