Hotfix: Modify process that reads SIN & CRA verification response files if required

[JasonCTang]

The transmission SIN verification & CRA income verification request files to their respective federal agencies will be handled by SFAS as there is a file sequencing number that must align between our two systems. The data itself, will not be merged and our file will be sent as is. However, we will be receiving individual response files for both SIMS and SFAS and there will be no way to distinguish between the two.

We need to ensure the SIMS jobs that read the response files can handle the SFAS response files gracefully. In other words, make sure when the SIMS job reads the SFAS response file and fails to find a corresponding record, the job will report the error but continue processing the remaining records in the file until the end and then move the file to the archive folder as if it were completed successfully.

Also, with the SIN file specifically, we attempt to match the SIMS Reference_IDX which will not be present in the SFAS file. We want to disable to error reporting of this particular error.

Acceptance Criteria

• Jobs must continue processing remaining records in a file after it fails to process a record.
• SIN and CRA response files will need to be archived after processing even where there are failed records. (IMB staff will need to monitor the jobs to confirm if errors occur processing SIMS response files and address them accordingly)
• Disable error reporting when SIMS Reference_IDX fails to match.
• If records are identified as SIMS records (based on the project area information being present), errors will be generated as usual.
  • If records are identified as SFAS records (based on the project area information NOT being present), records should be skipped.
• For the SIN validation (source reference), if the referenceIndex is not found on SIMS, assume the record is from SFAS and do not log as an error.
• For income verifications, the records should already be skipped. Execute a manual test to ensure that a file with records that do not contain the VERIFICATION_ID in the free project area is skipped. Consider creating an E2E to execute the test.
• Demo for both cases with a sample file.

[sslaws]

@JasonCTang to complete ticket with ACs and then this will go to Decomp and merge #3746 into this ticket as well.

[CarlyCotton]

CRA response file with non-SIM records was read with no errors.
SIN response file with non-SIM records was read in and the correct SIMS records verified. (And not moving to Archive)
HOWEVER, the mixed file is failing the hash check.

[CarlyCotton]

Alll good to go. Processed SIN file with mixed records after I fixed my incoming test file to the right spec.