LG-7306 Redacted logging of ThreatMetrix response

[stevegsa]

Why: To prevent PII from appearing in log files. Fields like first_name appear when their retention is not set to zero. It will be set that way in prod but we want to be doubly sure PII-ish fields will never appear in log files.
How: Create a redacter service that finds fields that should be filtered and puts the response 'redacted' in them. Note: only first level fields will be processed

Example ThreatMetrix response logging event:

{"name":"IdV: doc auth optional verify_wait submitted","properties":{"event_properties":{"success":true,"errors":{},"address_edited":false,"proofing_results":{"messages":[],"exception":null,"transaction_id":"resolution-mock-transaction-id-123","reference":"aaa-bbb-ccc","timed_out":false,"context":{"should_proof_state_id":true,"stages":{"resolution":{"client":"ResolutionMock","errors":{},"exception":null,"success":true,"timed_out":false,"transaction_id":"resolution-mock-transaction-id-123","reference":"aaa-bbb-ccc"},"state_id":{"client":"StateIdMock","errors":{},"success":true,"timed_out":false,"exception":null,"transaction_id":"state-id-mock-transaction-id-456","state":"MT","state_id_jurisdiction":"ND"},
  "threatmetrix":{"client":"lexisnexis:ddp","errors":{},"exception":null,"success":true,"timed_out":false,
    "transaction_id":"7e76104d-448a-4546-851c-199cbe2ab7b1","response_body":{
      "account_address_city": [redacted],
      "account_address_state": "fl",
      "account_address_street1": [redacted],
      "account_address_zip": [redacted],
      "request_duration":"1445",
      "request_id":"7e76104d-448a-4546-851c-199cbe2ab7b1",
      "request_result":"success",
      "review_status":"pass",
      "risk_rating":"low"

[zachmargolis]

was this not caught as part of earlier refactor? can we add a regression spec that would have caught this?

[stevegsa]

yeah this was a bug. it was adding costing for tmx at the resolution step and checking for proofingcomponent in the spec but was using the transaction_id from resolution not tmx. i'll update the spec

[zachmargolis]

Any update on this? haven't seen specs for it yet

[zachmargolis]

where do we pass back the alerts for AAMVA, InstantVerify? Is it in this structure?

[stevegsa]

yes. callback_log_data.result[:context][:stages][:resolution and :state_id...

[jskinne3]

That's a nice way to test!

[stevegsa]

yes. if you have suggestions on how to make that message more clear they are welcome. essentially if we see that in our log files we know there is a bug and we'll simply assume that successful responses always contain PII fields

[zachmargolis]

@stevegsa before I approve, can you address this comment: #6857 (comment)

[zachmargolis]

LGTM, thanks for fixing all this!

[stevegsa]

LGTM, thanks for fixing all this!

omg thanks for staying on that one. found some more costing bugs!!! that should have been a separate story. i tried to slip it in...

[stevegsa]

Costing fix will be put in a separate pr...