Allow confirming email in a different browser

[monfresh]

Why: Some people might visit the site via a Service Provider (SP)
in one browser, but open the email confirmation link in a different
browser. This means that anything that was previously stored in the
session in the first browser won't be available in the second browser.
The consequence is that the user won't be redirected back to the SP
after they finish creating their account.

How:

• Add a new ServiceProviderRequest DB table

When a request is made from an SP, whether it's via SAML or OpenID
Connect, we create a new ServiceProviderRequest with the following
attributes: the full request URL, a random unique UUID (because it's
not guaranteed that separate requests will have a unique identifier
attached to them), the issuer, and LOA. We also store these same
attributes in the session to support the sign in scenario (as opposed
to account creation).

Then, we check to see if the user is fully authenticated, and if not,
we redirect them to the /sign_up/start page, and include the
request_id in the params, as well as in the links on that page that
point to the account creation and sign in pages. Note that I moved the
logic that determines whether or not the landing page should appear,
from the sessions controller to the SAML and OpenID Connect controllers,
since they are the ones that know where the user should go. This adheres
to the Tell, Don't Ask principle, and allows us to remove the
show_start_page key from the session.

When the user chooses to create an account, the form contains the
request_id in a hidden field so it can pass it on to the registrations
controller. To include the request_id in the confirmation email, I had
to create a new method based on the one Devise uses, but with the
ability to pass in the request_id. Since Devise automatically sends the
confirmation email when the user is created (via a callback), I had to
override the original method to do nothing.

By having the request_id in the email, that means that if the user
visits the confirmation URL in a different browser, we can go back to
storing the SP info in the session by looking up the
ServiceProviderRequest whose uuid matches the _request_id parameter
from the URL in the email. By going back to using the session from then
on, it frees us from having to keep passing in the request_id in URLs
and in forms. This is done in
EmailConfirmationsController#process_valid_confirmation_token.

Note that the request_id is prefixed with an underscore in the email to
make sure that the URL ends with the confirmation token. By default,
Rails sorts params in link_to helpers alphabetically, but we don't
want request_id to come last because if the user chooses to copy the
link instead of clicking it, and if they don't copy it correctly, the
app won't find the ServiceProviderRequest, whereas an invalid
confirmation token will display an error to the user.

Once the user finishes creating their account, when they click the
button to continue to the SP, the app makes the original SP request
again, calling the before_actions in the SAML and OpenId Connect
controllers a second time. However, we don't want to create a new
ServiceProviderRequest because we want to be able to delete the
original one after the user goes back to the SP. We don't want the
ServiceProviderRequests table to grow indefinitely. The only way
to know for sure whether or not a ServiceProviderRequest that matches
the request URL already exists is to query on its url field. The
problem is, given that we want to index that field since it will be
queried on every SP request, the url field value is too big for the
Postgres btree index. There are solutions to this problem (that I
haven't implemented before), but I didn't feel it was worth the
trouble when we can rely on the presence of sp_session[:request_id]
to determine whether or not a new ServiceProviderRequest should be
created or not. We also use the sp_session[:request_id] to determine
which ServiceProviderRequest to delete by find a matching uuid. Then,
we can safely delete the :sp Hash from the session.

By keeping track of the SP via the request_id, we can also remove the
session timeout code and flash message that dealt with the scenario
where a user makes a request from an SP, but then remains on the
sign in page for longer than 8 minutes. Note that all this does is
preserve the branded experience on the sign in page. Allowing the
SP info to be restored after sign in will be implemented in a
separate PR.

Note that this doesn't include the same fix for the scenario where a
user opens the password reset link in a different browser. That will be
in a separate PR.

[zachmargolis]

is this method extraction to reduce the ABC size of the #create method?

[monfresh]

Correct.

[zachmargolis]

this will yield twice when true, is that intentional?

[monfresh]

That was not intentional, but it doesn't seem to cause a problem. However, I'll fix it so that it reads yield needs_idv. That's what I get for coding late at night.

[zachmargolis]

Where does @ raw_confirmation_token get set? Where does generate_confirmation_token! come from? Devise?

[monfresh]

Yes, I copied Devise's send_confirmation_instructions with the only modification being the ability to pass in the request_id to the mailer. I'll add a comment.

[zachmargolis]

I think overriding find_by can lead to confusing behavior especially since will never return nil.

What about making a new method?

def self.find_by_or_null(*args)
  find_by(*args) || NullServiceProviderRequest.new
end

[monfresh]

Sure.

[zachmargolis]

update the comment?

it 'redirects to login with the request_id' or something?

[monfresh]

Yes, will do.

[zachmargolis]

is this commented out on purpose?

[monfresh]

Yes. I noticed that we currently do not preserve the branded experience on the recovery code page. I opened an issue asking if that was intentional or not but haven't heard back yet. I would assume that we would show the branded nav bar throughout the whole creation process.

[jessieay]

huh! create a ticket?

[monfresh]

Yep. In my previous comment I said that I opened an issue already. https://github.com/18F/identity-private/issues/1692

[zachmargolis]

this line is red from codeclimate, do we not have test coverage?

[monfresh]

Probably not for this scenario, which is where a user makes a request from an SP while they are already signed in with email and password only and haven't entered their OTP yet. I'll add a test.

[jessieay]

link to commit with spec?

[monfresh]

6e1ff63

[jessieay]

darn looks like that was squashed?

[monfresh]

Here you go: d1ba3b9
Scroll down to loa1_sso_spec

[monfresh]

Thanks for the feedback. I addressed it in second commit. PTAL. I'll also fix conflicts with master and push again.

[zachmargolis]

LGTM!

[jessieay]

Left a round of feedback. Could probably comb through more but figured I'd submit my comments so you have time to respond :)

[jessieay]

should an href be a _path?

[monfresh]

I believe it needs to be _url to include params. Plus, I think we're moving to using _url exclusively. I would assume this would fail if it had to be _path.

[jessieay]

We are moving to _url for redirects since that is in line with the HTTP spec but we are not moving to _url for everything. There are some benefits to _path :)

AFAIK, you don't need _url to include params/

[pkarman]

my memory is that _url came up only because the HTTP spec requires it for Redirect responses. That's all. I agree that _path is fine for href values etc, since those are all client-side anyway.

[monfresh]

I just tried changing to _path and it failed. _url is required if you want to check for params as well.

[jessieay]

huh! create a ticket?

[jessieay]

I wouldn't necessarily expect to see expectations in a setup step like this.

[jessieay]

this method is AWESOME and LONG - any way to break it up into smaller named chunks for easier reading?

[monfresh]

Yep, I knew you would bring this up, but thought I'd get this out for review and then clean it up.

[jessieay]

you know me soooo well 🥇

[jessieay]

did we change this behavior intentionally? If so, seems potentially unrelated to moving SP info from the session into the DB

[monfresh]

It is very related, and it was intentional. As mentioned in my commit message, Devise automatically calls send_confirmation_instructions when the user is created (via a model callback). Unfortunately, Devise does not make it possible to send it additional parameters that can be included in the email. Therefore, I had to create a new send_custom_confirmation_instructions method that accepts an argument, and at the same time override Devise's send_confirmation_instructions to do nothing. Otherwise, the user would receive 2 emails.

[jessieay]

I wouldn't expect to see DB queries in a feature spec. My rule of thumb is that the expectations should be looking for things that the end user can see or experience, like page content or a redirect.

Any way to test this in a controller or other type of spec instead?

[monfresh]

Not that I can think of. I'm open to ideas if you have them. I want to be able to isolate the creation of the ServiceProviderRequest and then make sure it gets deleted. In a controller spec, the entire action will run in one step. AFAIK, I can't use a controller spec to verify that a particular ServiceProviderRequest was created since it will get deleted after calling the action. This controller is special in that it gets called twice, but we only want some of what it does to be called once. I think this is best tested with a feature spec.

[jessieay]

maybe a request spec? I usually think of feature specs as "end user" focused, so no database-related expectations. Request specs are more backend-y so perhaps this would work there?

[monfresh]

It might, but I'm not sure it's worth writing a whole new spec that in essence duplicates what the feature spec does, just to test a single expectation.

[pkarman]

I would expect to see a feature test using perform_in_browser to show the full end-to-end multiple browser interplay, from the user's perspective. Am I missing seeing that?

[monfresh]

Ah, yes, thanks for reminding me. I meant to add a step that deletes the session. Doing that now.

[monfresh]

Fixed in c630146
PTAL

[jessieay]

unique: true ?

[monfresh]

Good call

[jessieay]

👍

[jessieay]

nice! glad we are removing this (assuming it's unused). Thoughts about removing in a separate PR for clarity?

[monfresh]

This is not removing the A/B testing entirely. It is removing a duplicate div that was causing the LOA3 spec to fail because it found 2 elements matching the "Get started" button. I suppose I could edit the spec to use the first match and then remove this in a separate PR. What do you think?

[jessieay]

separate might be nice since this PR is quite large as-is, but up to you!

[jessieay]

now that we have the sp data in the database, why do we need to store anything other than the UUI Din the session?

[monfresh]

Good question. For several reasons.

• We need to store the request URL to make sure that existing users who sign in are redirected back to the SP. It's easier and faster to query the session rather than the DB, and easier than making sure the request_id is available on all forms and URLs throughout the auth process.
• The branded nav bar depends on the presence of either the issuer in the session or the request_id in the params. In the scenario where a user is signing in, we don't include the request_id params throughout the sign in process because that's only necessary for scenarios where users are switching browsers. Therefore, we need the issuer in the session.
• We have several places that check for sp_session[:loa3] in order to do stuff. In order to replace that, we'd need to pass in the request_id on every form and URL, and it would also mean a lot more DB lookups.

[pkarman]

I think @monfresh has taken the right approach. Populating the session once from the db is cleaner, since there is already a single link between the session and the browser (the session cookie).

[jessieay]

Hey @monfresh I just pulled this down locally to test.

CHROME:

• visit http://localhost:3000/test/saml
• entered my email

Firefox:

• confirmed email
• set password
• confirmed phone
• confirmed recovery code

after this flow, I was redirected to my profile page rather than the SP, which makes me think that the SP-related info was not being saved properly in Firefox. Am I doing something wrong?

[monfresh]

@jessieay You need to come from the sp-rails app. The test/saml endpoint doesn't call SamlIdpController#auth, which is where all the magic happens.

[monfresh]

Here are the scenarios I could think of and wrote tests for. Please see if I missed any.

• User comes from SP, clicks "Get started", submits form with invalid email => page displays inline error as before, and URL still contains request_id param, and the form still contains the correct request_id in a hidden field.

• User comes from SP, clicks "Get started", submits form with valid email => user is redirected to /sign_up/verify_email and receives an email with a link containing the correct request id in a param named _request_id (note the underscore prefix) and the order of the params should be _request_id, then confirmation_token.

  • User clicks "Resend" on the /sign_up/verify_email page. => User receives another confirmation email with the same correct params in the link
  • User clicks "try a different email" on the /sign_up/verify_email page, then enters a different email => User receives an email with the correct link
  • User clicks the link in the email => User is confirmed and is redirected to /sign_up/enter_password with the request_id in the params
  • User sits idle until session times out => SP branding should remain and continuing the account creation should redirect back to SP (I just thought of this new scenario and the second part is failing)

• User comes from SP, clicks "Get started", submits form with valid email, opens email link in a different browser, disables JS in the browser, then enters an invalid password => page displays inline error as before, and URL still contains request_id param, and the form still contains the correct request_id in a hidden field.

  • Enter correct password and complete the account creation => you should be redirected to SP

• User comes from SP, clicks "Get started", then stays idle until session times out => branded nav should remain visible, and completing the account creation should redirect back to SP

[monfresh]

@jessieay I believe I addressed your feedback. We need to get this in to the RC so it can be deployed and tested. If you find anything else, let me know and I'll address it in a separate PR. Thanks!

[pkarman]

why not actually switch browsers? perform_in_browser lets you do that.

[monfresh]

Used Perform in browser here: 4e61357

[jessieay]

why are we calling yield here instead of just returning needs_idv? It looks like that is a boolean value?

[monfresh]

Because this method is meant to be called with a block so that the auth method can return if the user needs_idv.

[jessieay]

why are we naming this param _request_id instead of request_id?

[monfresh]

Please read the initial commit message 😄

[jessieay]

to recap for others:

Note that the request_id is prefixed with an underscore in the email to
make sure that the URL ends with the confirmation token. By default,
Rails sorts params in link_to helpers alphabetically, but we don't
want request_id to come last because if the user chooses to copy the
link instead of clicking it, and if they don't copy it correctly, the
app won't find the ServiceProviderRequest, whereas an invalid
confirmation token will display an error to the user.

I find this logic sensible, although I am wondering how we can communicate the reasoning for this underscore to future people / our future selves, since it does feel a bit mysterious when browsing the code.

[monfresh]

Good point. I typically use git log -S or I look up the file in question in GitHub and look at the history to find out which commit introduced something and then I look up the PR, assuming the commit message explains the change since we've been pretty good about doing that.

[jessieay]

Yeah I think that is ideal, but that cannot always be depended on. I will noodle on ways we can make this more obvious via a future PR, perhaps through some specs

[jessieay]

wondering if sp_request_id would be a clearer param name?

[monfresh]

Yes, current_sp_request. Good catch.

[monfresh]

Fixed here: cdf1da9
PTAL

[jessieay]

I know that nil.to_s is empty string, but perhaps it would be clearer to just use '' here?

[monfresh]

Where does nil.to_s come from? I want to pass in a nil value to the form. When the form renders, the request_id hidden input field won't have a value at all.

[jessieay]

anything in the view is going to inherently have to_s called on it. So this will render in the hidden input field as '', I believe

[monfresh]

That doesn't seem to be the case for nil:

<input class="block col-12 field hidden" type="hidden" name="user[request_id]" id="user_request_id">

It works either way, but I think nil conveys the intent better, which is that by default, we don't want any value at all, not even an empty string, because it can imply presence. For example, if we had a conditional in the view that asked if request_id, then it would pass with an empty string, but not if it was nil.

[jessieay]

I always thought Nonexistent was two words - TIL!

[jessieay]

This comment is very clear / accurate...what worries me about this comment is that what if we change the method and don't update the comment? I might just leave the comment as "Overrride send_confirmation_instructions from Devise" or no comment at all ;)

[jessieay]

thoughts on my comment about this comment, @monfresh ?

[monfresh]

Yes, just remembered: 03a4683

[monfresh]

Thoughts?

[jessieay]

looks good!

[monfresh]

Thanks for the thorough review, @jessieay! Is this good to go now?

[jessieay]

so is the plan to break this up in a future PR? I commented about doing that before and I remember that you replied but the thread is gone now...

[jessieay]

Oh I see you did break it up...WOW still so long.

[jessieay]

💯 this is awesme - so glad we are going to fix this before release number 1!