Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement new client credentials for Google Geo API's #2058

Closed
mkoo opened this issue Apr 27, 2019 · 30 comments
Closed

Implement new client credentials for Google Geo API's #2058

mkoo opened this issue Apr 27, 2019 · 30 comments
Assignees
@mkoo
Labels
Function-Locality/Event/Georeferencing Priority-Critical (Arctos is broken) Critical because it is breaking functionality.

Comments

@mkoo
Copy link
Member

mkoo commented Apr 27, 2019

To ensure that we have all our background service calls up and running (and unlimited), I need to institute new client credentials for Arctos. Which means understanding the new content terms requirements then implementing.

This impacts:

  • maps for Media
  • elevation, distance-from, locality, etc suggestions; basically the internal Edit Locality stuff in the Google Box on the righthand side of user screen.

Many of these have no direct Leaflet equivalent (please correct me if I'm wrong!)

On my to-do list....
@mkoo mkoo self-assigned this Apr 27, 2019
@dustymc
Copy link
Contributor

dustymc commented Apr 29, 2019

There's still some relevant stuff in #2011, most critical being the decision to stay with Google or find another solution. That might be a good AWG topic.

The "background" calls keep finding ways to break other things. Currently it looks like any Agent with an Address can't be altered; we geocode them so we can do more with shipments, and the geocoder isn't happy. If I'm going to get access to a functional client_id/private_key I don't really want to dig out a bunch of complex code. If not, or not soon, I think I'm going to have to prioritize some way of fixing or disabling the background stuff I thought I could ignore.

Help?

@ccicero
Copy link

ccicero commented May 2, 2019

What are we trying to do with shipments that involves geocoding?

@dustymc
Copy link
Contributor

dustymc commented May 2, 2019

Not much - there's a map shipment button from loan search results.

Origins are #1325

The big question is if I can get the same type of credentials we've had from Google, in which case fixing this is about 5 seconds (literally - copy/paste/save/done!) or if I need to find another way, which (if it's even possible) would likely involve rebuilding a bunch of cryptography stuff that Google uses as a "signature."

@ccicero
Copy link

ccicero commented May 2, 2019

OK. I can see how that will be useful. Never noticed that!

Let's discuss at this morning's issue meeting.

@campmlc
Copy link

campmlc commented May 2, 2019

David Phau Manager of Developer Relations for Google Earth Engines and Google Earth Outreach
was at iDigBio at Berkeley [email protected]
May be a good contact point at google for this issue and for future Arctos / Google interactions

@campmlc
Copy link

campmlc commented May 2, 2019

This issue is related to a switchover of the Arctos account from Google Business to Enterprise to something else - Berkeley Natural History Museum Project, we get in free under their quota
paid for on Michelle's card
Or
We go to something more open source: Leaf(let)?
Will do similar things but not support the services we use; not large datasets
Should we start a new project with Google that is specific to Arctos
We had a google enterprise account that was a grant from google that gave us business access at no cost
If we do this for Arctos, then we'd need to submit a new proposal, have a contact there we can work with.
Does TACC have a relationship with Google? or serious GIS in Oracle?

@campmlc
Copy link

campmlc commented May 2, 2019

Michelle says she will create a client ID and implement Friday and will make recommendations to the AWG for next week.

@mkoo
Copy link
Member Author

mkoo commented May 2, 2019 via email

@campmlc
Copy link

campmlc commented May 2, 2019

Would he know someone we could contact for future reference?

@atrox10
Copy link

atrox10 commented May 2, 2019 •
edited
Loading

For now, can we just disable the google agent shipping address geocoding? I can't send loans if I can't add a shipping address to an agent, which usually is a very easy task.

@atrox10
Copy link

atrox10 commented May 7, 2019

URGENT!! CAn someone fix this or just disable this function!! I can't add addresses to Agents for loans and we are at a LOAN impasse. This is the last week with some of my students doing loans, so I need to be able to add addresses to Agents!!

@dustymc
Copy link
Contributor

dustymc commented May 7, 2019

I'm not sure what to do here. The address integration is causing a lot of problems.

@mkoo can I get a client_id??

Should I "temporarily" remove the geocoding??

Should I permanently remove it and the tools that use it?? (I'd rather not.)

Should I look for another solution??

HELP!!

@ccicero
Copy link

ccicero commented May 7, 2019

What's involved in temporarily removing the geocoding? I will talk with @mkoo this afternoon about the client ID, and see what the timeframe might be for doing that.

We also can't edit an existing agent address (which I was trying to do so I could use the same address, but for a different student); existing address has an "Attn:" to a person which should really be in the outside contact information. So I also can't move forward on this loan.

@dustymc
Copy link
Contributor

dustymc commented May 7, 2019

I think the biggest problem with anything "temporary" would be catching back up - I think I'd have to re-geocode everything to make sure everything is current. I severely dislike the idea of 'map some arbitrary things that may have more to do with service credentials than reality' so I think we'd need to disable the shipment mapping until everything can be put back together and refreshed.

If we want to go there, I think I can disable that part of the code in an hour or so.

You can't ever alter used addresses - that would corrupt the history of shipments, the logic to prevent that is built into the DB.

I can SQL in a new address, but I think that takes us back to having inconsistent data. If it's just one I guess I could manually geocode it first....

Can we wait until you can talk to Michell before doing anything drastic?

@ccicero
Copy link

ccicero commented May 7, 2019

Yes, let's wait to talk with Michelle.

@mkoo
Copy link
Member Author

mkoo commented May 8, 2019 via email

@mkoo
Copy link
Member Author

mkoo commented May 8, 2019 via email

@dustymc
Copy link
Contributor

dustymc commented May 8, 2019

Still getting https://developers.google.com/maps/documentation/javascript/error-messages#invalid-client-id-map-error

It should be gme-{stuff}

@mkoo
Copy link
Member Author

mkoo commented May 8, 2019 via email

@dustymc
Copy link
Contributor

dustymc commented May 8, 2019

Looks like around 10K/m.

Screen Shot 2019-05-08 at 9 48 24 AM

but who knows how that translates to maps - I'd assume most of them see a map or two, some of them see LOTS of maps, etc.

I think firefox is recently blocking analytics by default too, just to mess with you...

@mkoo
Copy link
Member Author

mkoo commented May 8, 2019 via email

@dustymc
Copy link
Contributor

dustymc commented May 8, 2019

THis is probably better - it's from https://google.secure.force.com/apex/MapsReport which I think you have access to.

Screen Shot 2019-05-08 at 9 57 37 AM

@dustymc dustymc mentioned this issue May 9, 2019
Closed
@campmlc
Copy link

campmlc commented May 9, 2019 •
edited
Loading

from Michelle at AWG meeting 5-9-19
Arctos uses google maps webservices, embedded thumbnaiis but also locations, addresses, elevation lookup apis; each one hits google services and gets authenticated in different ways; google maps was easy, simple authentication key; but on the backend google has gone to pay as you go to generate new authenticate code; there is no specific quote; affects Berkeley Mapper, Symbiota; Michelle just put in her credit card, ended up being charged $8; no costs incurred over $200 because of Michelle's personal .edu account at Berkeley; could be moved under Berkeley umbrella
Other option is to have client ID:
The only way to get a client ID of the proper type is to go through an application process for an educational grant; or use Berkeley grant; need to know how many hits per second and overall daily use, and Arctos is way low.
We can authenticate with an api key for features, elevation
Can we do this for addresses?
Or wait to have client ID?
Michelle will answer by this afternoon

@dustymc
Copy link
Contributor

dustymc commented May 13, 2019

I am trying to figure out how to use the API key instead of client ID, and I'm getting

API keys with referer restrictions cannot be used with this API.

Google says

https://developers.google.com/maps/faq#browser-keys-blocked-error

Arctos IP is

129.114.60.95

but for some reason nslookup thinks it's

129.114.52.171

so I'm not sure which of those Google would see.

@mkoo is this something that can be changed?

@mkoo
Copy link
Member Author

mkoo commented May 13, 2019 via email

@mkoo
Copy link
Member Author

mkoo commented May 13, 2019 via email

@mkoo
Copy link
Member Author

mkoo commented May 14, 2019 via email

@dustymc
Copy link
Contributor

dustymc commented May 15, 2019

I think our mapping capabilities are now fully restored. Please reopen if you find something I've missed.

@dustymc dustymc closed this as completed May 15, 2019
@mkoo
Copy link
Member Author

mkoo commented May 15, 2019

Quick summary for future reference:
After spending quality time with two google support staff and having Google accept our request for upgraded quota, we learned some new stuff: no longer are client id/ secret handshakes needed or will work for the API's we are using (this was the opposite of what is in the Documentation BTW). We can and should use authentication api keys that we generate. BUT also NOT in the docs is that these should be unrestricted keys (previous one was restricted for calls from our domain). Not sure I understand why but it works! Requests for any more capacity (upgrade to quota) can be made by filling out the application again and requesting more (demonstrated by our usage-- see the console).

@dustymc
Copy link
Contributor

dustymc commented May 16, 2019

Also for posterity: I think I'm now piping all map/api calls through googleSignURL(), which has been modified to just assemble the URL and append the APIKey. That should make maintenance much easier, lightens the code a bit, makes it possible to temporarily disable the cache, etc.

Note that there is no encryption in this approach, and the API key apparently has to be unrestricted to authorize some calls - anyone could map something in Arctos, grab the key from their network logs, and use it. We should be monitoring usage, and Google recommends periodically pulling a new API key (which can be entered in Arctos' Global Settings).

@tucotuco tucotuco mentioned this issue Dec 15, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mkoo mkoo

Labels
Function-Locality/Event/Georeferencing Priority-Critical (Arctos is broken) Critical because it is breaking functionality.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@ccicero @mkoo @atrox10 @dustymc @campmlc