LG-14253 Track time to complete proofing workflow#11179
Merged
Conversation
We want to start tracking how long it takes people to complete identity proofing. Part of that is understanding how long it takes people to go from the start of the process to the enter password step where their profile is created. At this point we want to know: 1. How long did it take the person to complete the workflow 2. What steps remain before person finishes proofing (i.e. verify-by-mail, in-person-proofing, or fraud review) The enter password submit events already logs enough information to determine the second. This commit starts tracking the duration of the proofing session by adding a timestamp at the welcome step and logging the elapsed seconds since on the enter password events. [skip changelog]
matthinz
approved these changes
Aug 30, 2024
n1zyy
approved these changes
Aug 30, 2024
jmhooper
added a commit
that referenced
this pull request
Sep 3, 2024
We are working to better measure the time it takes users to complete identity proofing. In #11179 we began logging how long it takes people to complete the identity proofing workflow. This proofing workflow time only tells a part of the story. Many users have to complete out-of-band events to finish identity proofing. We need to know how long these workflows take to complete to fully understand how long identity proofing takes. When a user finishes the proofing workflow we create a `Profile` record. This profile represents the proofing attempt and is in a “pending” state until the user completes those events. By logging the age of the pending profile we can determine how long it has been since the user finished the proofing workflow and thus how long they have been in a pending state. This commit opts to log the age of the profile in seconds. This will simplify the creation of the Cloudwatch dashboard to represent this information since CloudWatch does not have good tooling for dealing with timestamps. There are 3 out-of-band processes that can be required to complete proofing: 1. Verify-by-mail code entry 2. In-person proofing results 3. Fraud review pass These are represented by the following events: 1. `IdV: enter verify by mail code submitted` 2. `GetUspsProofingResultsJob: Enrollment status updated` 3. `Fraud: Profile review passed` This commit adds a `profile_age_in_seconds` property to all of these events. To measure this a `profile_age_in_seconds` method was also added to the `Profile` model. The `profile_age_in_seconds` property was also added to the `Fraud: Profile review rejected` events to maintain existing symmetry between arguments there. [skip changelog]
jmhooper
added a commit
that referenced
this pull request
Sep 3, 2024
We are working to better measure the time it takes users to complete identity proofing. In #11179 we began logging how long it takes people to complete the identity proofing workflow. This proofing workflow time only tells a part of the story. Many users have to complete out-of-band events to finish identity proofing. We need to know how long these workflows take to complete to fully understand how long identity proofing takes. When a user finishes the proofing workflow we create a `Profile` record. This profile represents the proofing attempt and is in a “pending” state until the user completes those events. By logging the age of the pending profile we can determine how long it has been since the user finished the proofing workflow and thus how long they have been in a pending state. This commit opts to log the age of the profile in seconds. This will simplify the creation of the Cloudwatch dashboard to represent this information since CloudWatch does not have good tooling for dealing with timestamps. There are 3 out-of-band processes that can be required to complete proofing: 1. Verify-by-mail code entry 2. In-person proofing results 3. Fraud review pass These are represented by the following events: 1. `IdV: enter verify by mail code submitted` 2. `GetUspsProofingResultsJob: Enrollment status updated` 3. `Fraud: Profile review passed` This commit adds a `profile_age_in_seconds` property to all of these events. To measure this a `profile_age_in_seconds` method was also added to the `Profile` model. The `profile_age_in_seconds` property was also added to the `Fraud: Profile review rejected` events to maintain existing symmetry between arguments there. [skip changelog]
jmhooper
added a commit
that referenced
this pull request
Sep 4, 2024
…nts (#11186) We are working to better measure the time it takes users to complete identity proofing. In #11179 we began logging how long it takes people to complete the identity proofing workflow. This proofing workflow time only tells a part of the story. Many users have to complete out-of-band events to finish identity proofing. We need to know how long these workflows take to complete to fully understand how long identity proofing takes. When a user finishes the proofing workflow we create a `Profile` record. This profile represents the proofing attempt and is in a “pending” state until the user completes those events. By logging the age of the pending profile we can determine how long it has been since the user finished the proofing workflow and thus how long they have been in a pending state. This commit opts to log the age of the profile in seconds. This will simplify the creation of the Cloudwatch dashboard to represent this information since CloudWatch does not have good tooling for dealing with timestamps. There are 3 out-of-band processes that can be required to complete proofing: 1. Verify-by-mail code entry 2. In-person proofing results 3. Fraud review pass These are represented by the following events: 1. `IdV: enter verify by mail code submitted` 2. `GetUspsProofingResultsJob: Enrollment status updated` 3. `Fraud: Profile review passed` This commit adds a `profile_age_in_seconds` property to all of these events. To measure this a `profile_age_in_seconds` method was also added to the `Profile` model. The `profile_age_in_seconds` property was also added to the `Fraud: Profile review rejected` events to maintain existing symmetry between arguments there.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We want to start tracking how long it takes people to complete identity proofing. Part of that is understanding how long it takes people to go from the start of the process to the enter password step where their profile is created. At this point we want to know:
The enter password submit events already logs enough information to determine the second. This commit starts tracking the duration of the proofing session by adding a timestamp at the welcome step and logging the elapsed seconds since on the enter password events.