Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

#3745 - Modify process that reads SIN & CRA verification response files #3835

Merged
merged 25 commits into from
Oct 30, 2024

Conversation

bidyashish
Copy link
Collaborator

@bidyashish bidyashish commented Oct 24, 2024

Acceptance Criteria

  • Jobs must continue processing remaining records in a file after it fails to process a record.
  • SIN and CRA response files will need to be archived after processing even where there are failed records. (IMB staff will need to monitor the jobs to confirm if errors occur processing SIMS response files and address them accordingly)
  • Disable error reporting when SIMS Reference_IDX fails to match.
  • If records are identified as SIMS records (based on the project area information being present), errors will be generated as usual.
    • If records are identified as SFAS records (based on the project area information NOT being present), records should be skipped.
  • For the SIN validation (source reference), if the referenceIndex is not found on SIMS, assume the record is from SFAS and do not log as an error.
  • For income verifications, the records should already be skipped. Execute a manual test to ensure that a file with records that do not contain the VERIFICATION_ID in the free project area is skipped. Consider creating an E2E to execute the test.
  • Demo for both cases with a sample file.

@bidyashish bidyashish self-assigned this Oct 24, 2024
@bidyashish bidyashish marked this pull request as draft October 24, 2024 21:24
@bidyashish bidyashish changed the title add test sample files #3745 - Modify process that reads SIN & CRA verification response files Oct 25, 2024
@bidyashish bidyashish marked this pull request as ready for review October 28, 2024 16:03
@guru-aot guru-aot self-requested a review October 28, 2024 20:58
@@ -47,7 +47,7 @@ export class SINValidation extends RecordDataModel {
type: "timestamptz",
nullable: true,
})
dateReceived?: Date;
dateReceived?: Date | null;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The dateReceived is already marked as nullable and declared as "dateReceived?". Please revert this change.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reverted

Copy link
Collaborator

@andrewsignori-aot andrewsignori-aot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for making the changes, looks good 👍

Copy link

Copy link

Backend Unit Tests Coverage Report

Totals Coverage
Statements: 22.35% ( 3692 / 16516 )
Methods: 10.33% ( 212 / 2053 )
Lines: 25.67% ( 3202 / 12475 )
Branches: 13.98% ( 278 / 1988 )

Copy link

E2E Workflow Workers Coverage Report

Totals Coverage
Statements: 65.43% ( 583 / 891 )
Methods: 59.26% ( 64 / 108 )
Lines: 68.54% ( 464 / 677 )
Branches: 51.89% ( 55 / 106 )

Copy link

E2E Queue Consumers Coverage Report

Totals Coverage
Statements: 85.88% ( 1216 / 1416 )
Methods: 87.86% ( 123 / 140 )
Lines: 86.8% ( 1032 / 1189 )
Branches: 70.11% ( 61 / 87 )

Copy link

E2E SIMS API Coverage Report

Totals Coverage
Statements: 66.01% ( 5571 / 8439 )
Methods: 63.3% ( 683 / 1079 )
Lines: 70.12% ( 4401 / 6276 )
Branches: 44.93% ( 487 / 1084 )

Copy link
Collaborator

@andrepestana-aot andrepestana-aot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Thanks for doing the changes!

@bidyashish bidyashish added this pull request to the merge queue Oct 30, 2024
Merged via the queue into main with commit 20fcda4 Oct 30, 2024
20 checks passed
@bidyashish bidyashish deleted the feature/#3745-modify-CRA-SIN-response branch October 30, 2024 19:00
dheepak-aot pushed a commit that referenced this pull request Oct 30, 2024
…es (#3835)

***Acceptance Criteria***
- [x] Jobs must continue processing remaining records in a file after it
fails to process a record.
- [x] SIN and CRA response files will need to be archived after
processing even where there are failed records. (IMB staff will need to
monitor the jobs to confirm if errors occur processing SIMS response
files and address them accordingly)
- [x] Disable error reporting when SIMS Reference_IDX fails to match.
- [x] If records are identified as SIMS records (based on the project
area information being present), errors will be generated as usual.
- [x] If records are identified as SFAS records (based on the project
area information NOT being present), records should be skipped.
- [x] For the SIN validation ([source
reference](https://github.com/bcgov/SIMS/blob/8864eaa8a5f62360f5e3c91c6de06f319e008250/sources/packages/backend/libs/integrations/src/services/sin-validation/sin-validation.service.ts#L119)),
if the `referenceIndex` is not found on SIMS, assume the record is from
SFAS and do not log as an error.
- [x] For income verifications, the records should already be skipped.
Execute a manual test to ensure that a file with records that do not
contain the `VERIFICATION_ID` in the free project area is skipped.
Consider creating an E2E to execute the test.
- [x] Demo for both cases with a sample file.
sh16011993 added a commit that referenced this pull request Nov 5, 2024
### The following are achieved in this hotfix:

- SIN Validation - Gender Field Limit
(#3839)
- SINF restriction bridge mapping and rename to SINR
(#3846)
- Virus Scanning - Not functioning PROD
(#3854)
- Modify process that reads SIN & CRA verification response files
(#3835)
- Hotfix: Bulk uploads: Unexpected error while uploading the file
displays when "Validate" or "Create now" are selected for valid bulk
upload
- Virus Scanning - Empty File Check
- Hotfix: Change file naming convention of the SIN validation request
file
- Hotfix: Fix for withdraw upload
- Hotfix: Name change for CRA Response file
- Parse the last 2 characters in NEB as decimal places
- Update PTMSFAA File Name

---------

Co-authored-by: Ashish <[email protected]>
Co-authored-by: Andre Pestana <[email protected]>
Co-authored-by: Lewis Chen <[email protected]>
Co-authored-by: Andrew Boni Signori <[email protected]>
Co-authored-by: Dheepak Ramanathan <[email protected]>
github-merge-queue bot pushed a commit that referenced this pull request Nov 5, 2024
### The following are the achieved in this hotfix:

- SIN Validation - Gender Field Limit
(#3839)
- SINF restriction bridge mapping and rename to SINR
(#3846)
- Virus Scanning - Not functioning PROD
(#3854)
- Modify process that reads SIN & CRA verification response files
(#3835)
- Hotfix: Bulk uploads: Unexpected error while uploading the file
displays when "Validate" or "Create now" are selected for valid bulk
upload
- Virus Scanning - Empty File Check
- Hotfix: Change file naming convention of the SIN validation request
file
- Hotfix: Fix for withdraw upload
- Hotfix: Name change for CRA Response file
- Parse the last 2 characters in NEB as decimal places
- Update PTMSFAA File Name

---------

Co-authored-by: Ashish <[email protected]>
Co-authored-by: Andre Pestana <[email protected]>
Co-authored-by: Lewis Chen <[email protected]>
Co-authored-by: Andrew Boni Signori <[email protected]>
Co-authored-by: Dheepak Ramanathan <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants