OAuth2 token handling

[IntranetFactory]

I'm currently researching an alternative for PostMan. Your extension is very impressive. We use PostMan mostly with endpoints using OAuth2 code flow to provide tokens. So I currently try to understand if there is already any way to request arbitrary OAuth2 tokens with httpyac.

I just discovered openIdVariableReplacer.ts and I'm wondering how to use it? Are you planning to add some kind of OAuth2 token handling and/or are you open to PR in that area?

[AnWeber]

The support for OpenID is more or less the reason why I don't use vscode-restclient. And I don't really like Postman v8 either. OpenId is supported out of the box. You can view the requests in the output channel of the extension.
If changes are needed, pull requests are welcome. A description of the reason for the change or further documentation would be helpful.

All flows are tested with a keycloak server and openid configuration.

client_credentials

.env in workspace root

local_tokenEndpoint=http://localhost:8080/auth/realms/myRealm/protocol/openid-connect/token
local_clientId=...
local_clientSecret=...

service.http

GET http://localhost:8080/service/api/v1/foo
Authorization: openid client_credentials local

Output

POST http://localhost:8080/auth/realms/myRealm/protocol/openid-connect/token

authorization: Basic ...
content-type: application/x-www-form-urlencoded

grant_type=client_credentials
-----
GET http://localhost:8080/service/api/v1/foo
Authorization: Bearer ...

password

.env in workspace root

local_tokenEndpoint=http://localhost:8080/auth/realms/myRealm/protocol/openid-connect/token
local_clientId=...
local_clientSecret=...
local_username=...
local_password=...

service.http

GET /service/api/v1/foo
Authorization: openid password local

Authorization Code

.env in workspace root

local_authorizationEndpoint=http://localhost:8080/auth/realms/myRealm/protocol/openid-connect/auth
local_tokenEndpoint=http://localhost:8080/auth/realms/myRealm/protocol/openid-connect/token
local_clientId=...
local_clientSecret=...

service.http

GET /service/api/v1/foo
Authorization: openid authorization_code local

on this flow a local http server is opened receiving redirect from keycloak on port 3000

[IntranetFactory]

Great starting point. I think it should be fairly easy to extend that to not only handle open id but also generic OAuth2. The main differences I would currently expect is that a) it should be possible to define which scopes should be used b) the returned token might not be a JWT.

I'm wondering about the purpose of the local_* name prefix? Aren't these endpoints usually provided by a remote service?

Are the returned access and refresh tokens persisted or just kept in memory?

[AnWeber]

The prefix more or less came about because I needed too many variables for the flows. And I also have the use case with the token exchange where I need a second OpenId endpoint. It is possible to pass it also by object (realm.tokenEndpoint).
The returned access and refresh tokens are only kept in memory. I have considered using keytar, but have currently discarded this.
I have seen the use case with the scopes, but I did not have a meaningful test for it and therefore ignored it currently. We can extend the current implementation if you can provide the test or a test scenario.

[IntranetFactory]

I'm going to setup a client id in MS AAD for both OpenId sign in and to access the MS Graph API. Their Graph API uses dozens of scopes to control access to all kind of graph endpoints.

[IntranetFactory]

I'm sorry - I'm a little lost. I registered a new client in AAD, I used http://localhost:3000/callback as the Redirect Uri. I created prod.env and selected the file:

I created a .http file to call the MS Graph /me endpoint (https://docs.microsoft.com/en-us/graph/api/user-get?view=graph-rest-1.0&tabs=http)

I didn't figure out how to now a) to start the Open Id sign in flow and then b) to use the returned access token for the bearer authorization.

[IntranetFactory]

I made some progress. I missed Authorization: openid authorization_code local - my assumption was something like {{access_token_local}} local is the prefix for the variables in the .env file?

[AnWeber]

Authorization: openid authorization_code local

After send OpenIdVariableReplacer matches auth header by regex and opens a browser with authorizationEndpoint

[IntranetFactory]

That gives me now the url https://login.microsoftonline.com/common/oauth2/v2.0/authorize?client_id=0c1d9732-466e-458f-85d7-260e448831a8&response_type=code&state=L1PaqDwpNg0fKHilfrEJceFaYS8PF0&redirect_uri=http%3A%2F%2Flocalhost%3A3000%2Fcallback

A working url we generate for our site is https://login.microsoftonline.com/common/oauth2/v2.0/authorize?response_type=code&access_type=offline&client_id=798429c7-66a7-462b-9bc7-88e124e680b0&state=b2ZmaWNlLTM2NXwvfG9pZGNzaWduaW58MjAyMS0wMi0yMVQyMDo0NTo1Nlp8ZDYwMWE3MDYwMGFhZmFjNjdlZGFlMjg1MjhjMGI4Mjc2Zjc3ZjRmZQ%3d%3d&redirect_uri=https%3a%2f%2fapp.adenin.com%2foauth2connector%2freturnUrl&scope=openid+profile+email

So it seems that access_type and scope parameters are added.

[IntranetFactory]

Currently I assume that the next steps to generalize bearer authorization could be:

1. add option parameters local_access_type and local_scope When present &access_type and/or &scope should be added to authorize url
2. support also Authorization: oauth2 authorization_code local I think the only difference between oauth2 doesn't expect the id_token in the response and that the login failure ("Anmeldung gescheitert") should probably be "Login failed" for openid, and "Authorization failed" for oauth2,

[AnWeber]

1. on authorization_cod]e(https://openid.net/specs/openid-connect-core-1_0.html#AuthRequest) flow scope is required. I added param `local_scope'. If it is empty value openid is used.
2. I did not add parameter access_type because it is not mentioned in OpenID Flow. It is only relevant for Google Identify Service, (stackoverflow). Please just add the parameter to the authorizationEndpoint http://localhost/auth?access_type=offline
3. I added support if access_token is not a valid jwt token. In this case timeskew (estimated time diff between local and server) is set to 0.
4. login failure message is changed to Authorization failed. This message gives the best indication of problems with the authorization.
5. Since I don't currently validate the id token, oauth2 and openid don't really differ and I'll spare myself this distinction.

[IntranetFactory]

Thank you for your fast enhancement. I've installed 1.14.0 (2021-02-22) and add added the line local_scope=openid profile email to my prod.env file.

It generates now the url https://login.microsoftonline.com/common/oauth2/v2.0/authorize?client_id=0c1d9732-466e-458f-85d7-260e448831a8&response_type=code&state=34jcLubOQLW9vsnxA9RPREY8Js2AqD&redirect_uri=http%3A%2F%2Flocalhost%3A3000%2Fcallback which neither contains my scope nor the new default openid. Did I miss anything?

[IntranetFactory]

5. Since I don't currently validate the id token, oauth2 and openid don't really differ

From my understanding oidc is an specific, limited use case of oauth2. As soon as scope works I think you'll also support oauth2. For our use cases oidc is not relevant. I had been searching for a tool supporting oauth2 and nearly skipped your outstanding solution as I didn't see oauth2. So using openid as the only keyword is just fine, but I think that using the oauth2 as the main keyword might attract more users.

[AnWeber]

Did you reload vscode after install? On my computer it works.

Thanks for the recommendation with OAuth2. I will add it to the README.md.

[IntranetFactory]

code showed a "restart" button - which I think I had clicked. I now restarted it manually, but I get now the following error when clicking on send

TypeError: Cannot read property 'indexOf' of undefined at c:\Users\ma\.vscode\extensions\anweber.vscode-httpyac-1.14.0\dist\extension.js:2:433316 at 
Generator.next (<anonymous>) at c:\Users\ma\.vscode\extensions\anweber.vscode-httpyac-1.14.0\dist\extension.js:2:430055 at new Promise (<anonymous>) at 
n (c:\Users\ma\.vscode\extensions\anweber.vscode-httpyac-1.14.0\dist\extension.js:2:429800) at 
c:\Users\ma\.vscode\extensions\anweber.vscode-httpyac-1.14.0\dist\extension.js:2:432899 at 
new Promise (<anonymous>) at w (c:\Users\ma\.vscode\extensions\anweber.vscode-httpyac-1.14.0\dist\extension.js:2:432879) 
at authorization_code (c:\Users\ma\.vscode\extensions\anweber.vscode-httpyac-1.14.0\dist\extension.js:2:434934) 
at c:\Users\ma\.vscode\extensions\anweber.vscode-httpyac-1.14.0\dist\extension.js:2:432103 at Generator.next (<anonymous>) 
at c:\Users\ma\.vscode\extensions\anweber.vscode-httpyac-1.14.0\dist\extension.js:2:43005...

[AnWeber]

I experienced this error myself. Did you select your environment? Parameter authorizationEndpoint is not found. It will get fixed in next release Martin Amm <[email protected]> schrieb am Mo., 22. Feb. 2021, 17:52:

…

code showed a "restart" button - which I think I had clicked. I now restarted it manually, but I get now the following error when clicking on send TypeError: Cannot read property 'indexOf' of undefined at c:\Users\ma\.vscode\extensions\anweber.vscode-httpyac-1.14.0\dist\extension.js:2:433316 at Generator.next (<anonymous>) at c:\Users\ma\.vscode\extensions\anweber.vscode-httpyac-1.14.0\dist\extension.js:2:430055 at new Promise (<anonymous>) at n (c:\Users\ma\.vscode\extensions\anweber.vscode-httpyac-1.14.0\dist\extension.js:2:429800) at c:\Users\ma\.vscode\extensions\anweber.vscode-httpyac-1.14.0\dist\extension.js:2:432899 at new Promise (<anonymous>) at w (c:\Users\ma\.vscode\extensions\anweber.vscode-httpyac-1.14.0\dist\extension.js:2:432879) at authorization_code (c:\Users\ma\.vscode\extensions\anweber.vscode-httpyac-1.14.0\dist\extension.js:2:434934) at c:\Users\ma\.vscode\extensions\anweber.vscode-httpyac-1.14.0\dist\extension.js:2:432103 at Generator.next (<anonymous>) at c:\Users\ma\.vscode\extensions\anweber.vscode-httpyac-1.14.0\dist\extension.js:2:43005... — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADGP643GXBJ6HRVEZ7QK62DTAKDUPANCNFSM4X7IBUUQ> .

[IntranetFactory]

You are right, I didn't select an environment. The error is gone now.

[IntranetFactory]

I'm now login into O365, but then I'm prompted, and after clicking Open I get the error below. Is there any url on http://localhost:3000 I can invoke to see it the server responds?

[AnWeber]

Oh man, that hurts. i am an enemy of usability. This is the good case. It worked and there is a redirect to vscode//{filename}}. The behavior occurs because you were already logged in. I change it to a boring html page:-)

After that, the http server will be terminated immediately, so you will not be able to call it again. This behavior is intentional, as I still find opening an http server very unexpected, so I wanted to minimize its use (=> error page).

The deadlock (browser does not open) you observed is probably due to the fact that for each request I start the server independently. I guess I will have to improve the management of the server. For me it was enough, but if the behavior is a black box, it should become more comfortable.

[IntranetFactory]

I didn't check what the VS Code WebViews do exactly - but maybe they can be used to launch the auth flow inline instead of launching an external web browser? PostMan also has a simple web browser included, and they seem to just detect the final redirect. So when the server responds with 302 to redirect_uri they just detect that, but do not redirect to the page at all. Which is quite neat as you can just use any existing redirect_uri and you don't need to register an additional client id for the http://localhost:3000/callback.

[AnWeber]

No, Webview cannot fulfill this functionality. That was my first idea. Best support I can get from vscode is Authentication Provider but this is VSCode Insiders.
But I also want to support cli with this project, so I don't want to use VS code functionality at this central point. I just need to make the http server more chatty, then it should work.
After the redirect the code gets exchanged for access_token. How can Postman support this without its own ClientId. Do you have a source for this behavior?

[IntranetFactory]

In Postman I can use any existing client id, secret & redirect_uri. It seems that e.g. https://stackoverflow.com/questions/58156957/how-postman-complete-the-oauth-2-0-flow-without-actually-redirect-to-the-redirec explains how they do it. From my understanding their "trick" is that they have an internal web browser, which can detect the 302 response from the server.

[AnWeber]

In this case you really trust Postman with a lot (ClientSecret, ClientId, Username and Password). I also considered using a headless chrome like in Browser Preview. But it was too much effort for me and with the current implementation I can also use the password management of the browser.

[IntranetFactory]

Headless would work, but it's a pain to maintain and you would need to know the users password. We already use Puppeteer in visual regression tests. But you need a script for each partner, and every time the page where their authorize the access changes, the script needs to be changed.

VS Code uses a relay/proxy server https://github.com/microsoft/vscode/blob/a699ffaee62010c4634d301da2bbdb7646b8d1da/extensions/github-authentication/src/githubServer.ts#L17 to handle GitHub authorization, which stores the received token in keytar. For using a similar concept, setting up a proxy server like Grant might be an option, but would be another dependency to setup and maintain.

I think using short-lived mini web server to receive the token is a brilliant idea and I also like the idea to not persist the received token at all.

[AnWeber]

new version with fix is released. may you please test. thanks:-)

[IntranetFactory]

Thank you. I now always got the tokens and accessing the MS Graph endpoints works as well.

In my tests I needed to use different scopes. But after changing the .env no new token was requested. The only option to request a new token I found was restarting VS code.

So it would be great if a changes of the scope value since the token was requested would be detected, or having an "request token" code lens would probably be even better.

[AnWeber]

Presumably the scope is deifned as an inline variable. This was not considered in the cache. I have now changed the CacheKey to the actual values, so it should work after the next release.
Unfortunately I didn't understand your idea with the Request Token CodeLens. My approach is to remember the token after use and if there is a refresh token, use it to keep the login alive.

[IntranetFactory]

Using the scope values as part of the cache key will solve the problem as well. I was wondering if it would make sense to be able to explicitly request a new token even if a token is already cached. Currently VS code needs to be restarted if I want to get a new token. Having a code lens would allow to request a new one without having to restart.

[AnWeber]

OAuthTokenStore is cleared on reset environment. Use command "refresh/ reset environment" Martin Amm <[email protected]> schrieb am Sa., 27. Feb. 2021, 15:53:

…

Using the scope values as part of the cache key will solve the problem as well. I was wondering if it would make sense to be able to explicitly request a new token even if a token is already cached. Currently VS code needs to be restarted if I want to get a new token. Having a code lens would allow to request a new one without having to restart. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADGP645OI5ZW3P5AMV66TWDTBEBPXANCNFSM4X7IBUUQ> .

[IntranetFactory]

I think using the scope as part of the cache key will not always work. Let's have say

1. I have a client id "A" and scope "read"
2. I authorize access and get a token for client id "A" -> "token1"
3. I change scope to "read write" and authorize access; get a new token -> "token2"
4. I change scope back to "read"

In a quick test it seems that current result is that "token1" is used again. But a new token should be requested, most OAuth implementations I'm aware of manage one token per user and client id. So as soon as "token2" was issued, "token1" became invalid and cannot be used again.

[AnWeber]

One token per user, client and token endpoint should work. I switch between different oauth server with same user and clientId and don't want to login everytime. Do you agree?

[IntranetFactory]

I now repeated my test:

1. I started with local_scope: openid profile email User.Read in my .env
2. requested GET https://graph.microsoft.com/v1.0/me/people which failed as expected
3. I added People.Read to scope
4. I requested GET https://graph.microsoft.com/v1.0/me/people again
5. a new token is requested for the changed scope, and access to the endpoint works
6. I changed scope in .env back to local_scope: openid profile email User.Read and I requested GET https://graph.microsoft.com/v1.0/me/people again

My assumption was that access would fail as the current scopes do not allow access to the endpoint. But access was successful. I understand why it worked. I don't know how to get rid of the previously cached token, without restarting.

I'm not sure if the current behavior is a bug or a feature.

[AnWeber]

I have now refactored the management of OAuth tokens. These can now be viewed and also deleted using the command httpyac.logout. For example for user session detection the ClientId, TokenEndpoint and OAuth Flow type is used. This should remove the previous token when the token is changed. I have also prepared the integration of the Implicit Flow. (5bb91c9)

[IntranetFactory]

Works great. It would be nice if the logout command could also be started from code lens.

[AnWeber]

released new version with code lens support

[IntranetFactory]

The code lens is very helpful.

An idea for a minor improvement would be to show the environment name e.g. currently I don't see that the active token is based on "local2":

[AnWeber]

Great idea. I did not like the display of the tokenEndpoint, only the display of the environment was not easily possible. The prefix is a good compromise.