documentation on how to use upvote locally as library with service account creds

[thehesiod]

I've been trying to create a helper script by using upvote locally as an API, and was able to get to retrieving Blockables: https://gist.github.com/thehesiod/0ffde6e56ea56aede919fe6068a73f89
after fixing this issue: googleapis/google-cloud-datastore#211

The way I have it is by adding google-cloud-sdk/platform/google_appengine and the root upvote folder to the pythonpath, as well as creating a bunch of __init__.py files in the upvote source folders.

The issue I ran into is the following: AttributeError: 'BeginTransactionRequest' object has no attribute 'transaction_options'

I believe because by installing the googledatastore module, there are now two sets of datastore protos: site-packages/google/cloud/datastore_v1/proto/datastore_pb2.py and site-packages/google/cloud/proto/datastore/v1/datastore_pb2.py

The datastore_v1 one declares the transaction_options field descriptor, while the v1 does not. It's kinda odd that both are v1 and yet different.

I have the following google modules installed with the latest cloud sdk:

google-api-core                 1.3.0    
google-api-python-client        1.5.5    
google-auth                     1.5.0    
google-auth-httplib2            0.0.3    
google-cloud                    0.34.0   
google-cloud-bigquery           1.5.0    
google-cloud-core               0.28.1   
google-cloud-datastore          1.7.0    
google-resumable-media          0.3.1    
googleapis-common-protos        1.5.3    
googledatastore                 7.0.1    
proto-google-cloud-datastore-v1 0.90.4   
proto-google-datastore-v1       1.4.0

I tried substituting one for the other but then got errors about mismatched string types. Ideas?

[msuozzo]

Couple things:

• It should be possible to hit the voting endpoint (e.g. /api/web/votes/cast/<sha>) instead of directly interfacing with datastore. That will both simplify the code you'll need to write and still automate the whitelisting process.
• I haven't looked at googledatastore before but I'd doubt that it would play nicely with google-cloud-datastore. Have you tried using the latter instead? The latter uses the newer, non-deprecated google-auth library so authenticating with a service account should be much more straightforward.

[thehesiod]

ya I was thinking about that. Another was the remote api I just discovered, I'm about to try that, thoughts? I'm guessing the preferred is the /votes/ endpoint. It would be nice if all the APIs were documented.

[thehesiod]

ugh, even with remote api (which took awhile to get to work) has the same issue :( Guess I'm left with the Vote HTTPS API. I'll have to figure out how to get auth working correctly. If/when I get it to work I'll post an update

[thehesiod]

btw, googledatastore is a hard requirement in /google-cloud-sdk/platform/google_appengine/google/appengine/ext/ndb/tasklets.py:_make_cloud_datastore_context

[thehesiod]

@msuozzo ok, so I tried calling the /api/web/votes/query API via the browser and it works, however then tried calling it via a service account: https://gist.github.com/thehesiod/c51e3e39850763d4883772ac4b8435ca and it just redirects me to the oauth login page. How would I go about calling this API from a service account cred? Thanks!

[msuozzo]

So I looked into this for a bit and have come away with the impression that service accounts aren't able to properly authenticate to App Engine's default auth API (especially considering it was developed long before Cloud IAM). AFAICT, default auth requires the authenticating user have a full Google Account which service accounts do not.

The best I can think to do with the current setup is to extract the session token and use that as a parameterized value in your script. Hacky? Yes. Functional? Also yes.

Looking to an actual solution, there are a number of other authentication options available on App Engine (see this page). We've looked into migrating to Firebase Auth in the past to enable arbitrary email-based auth but it doesn't look like we'll have the staffing/support to do the migration in the near future. When I last looked, there were really only questions of getting it to play nice with Angular 1.X so I'd imagine it would be simple to get a quick-and-dirty prototype up. (Essentially just following this tutorial.)

Let me know what you decide! (and if you manage to get anything cool built 😄)

[thehesiod]

extract the session auth token from a regular user's login? we'd like to avoid that since we want all our users to have 2 factor enabled. I found this: https://cloud.google.com/endpoints/docs/openapi/service-account-authentication#using_jwt_signed_by_service_account does this sound like it will work?
There's also this: https://developers.google.com/identity/protocols/OAuth2ServiceAccount

[msuozzo]

Would 2FA really affect your ability to use the raw credential? Wouldn't it just affect the process by which you obtain said credential?

Yeah so unfortunately the first link is only related to Cloud Endpoints, an API definition product/system that Upvote does not use. It could but the trickiest part would be changing the auth system to accommodate Endpoints and at that point, you might as well isolate your effort to changing the existing auth system used in the current API instead of porting it to another structure.

The second link appears to be related to the authentication process that you'd need to employ once you ported the auth to accept OAuth (as described in this section of the docs).

[thehesiod]

The issue is credentials expire (theoretically at any time), so we couldn't have a manual process to re-obtain them. Thanks for all the help btw! Hmm will have too do more research with what you presented when I get back to the States thanks!

[msuozzo]

The issue is credentials expire (theoretically at any time), so we couldn't have a manual process to re-obtain them.

Yeah. I was thinking of it mainly in the context of "I want to be able to use a CLI in one-off bulk whitelisting tasks and I'm willing to go through a bit of pain to do that." The strategy doesn't really scale beyond that...

Thanks for all the help btw!

Absolutely! Good Luck!

[thehesiod]

ok, going to close this because I think we're going to be able to do everything we need directly in upvote....also I have a new question which I'll log shortly which I hope I can get some help on :)