UTM and data entry

[dustymc]

Arctos currently does not have the capability to convert UTM to coordinates.

Locality requires all-or-none of (coordinates, datum, georeference protocol, etc.).

Therefore it's not currently possible to enter UTM data with specimen records. Possibilities:

1. figure out how to convert UTM to coordinates
2. store UTM+metadata in collecting_event.verbatim_coordinates (this happens now, but it's not very visible) and push NULL coordinates+metadata to Locality (this is the part that does not currently happen)
3. ?????

Unknown: what is the relationship between UTM data and datum?

@ccicero

[ccicero]

I'm not sure what it means to push NULL coordinates to Locality. You would have the UTM metadata but with no decimal degree coords?

We need to have it more visible in data entry that you are entering verbatim coords, and you should be able to enter UTMs as the verbatim values.

How to convert to decimal degrees is another question, as is the relationship between UTM and datum - @mkoo ?

[dustymc]

I'm not sure what it means to push NULL coordinates to Locality. You would have the UTM metadata but with no decimal degree coords?

Table LOCALITY would have NO spatial data. It fits in COLLECTING_EVENT just fine, but that generally isn't displayed (and can't be displayed graphically - it's string data, not spatial).

We need to have it more visible in data entry that you are entering verbatim coords, and you should be able to enter UTMs as the verbatim values.

That's almost certainly correct, but I don't know how to do it without eg, making you enter everything in two places (and that would conflict with the generated coordinates). See also above - who/where/etc. should see verbatim data post-entry? I am VERY open to ideas!

[ccicero]

Let's discuss at next AWG meeting.

[dustymc]

AWG

Added to project

[mkoo]

Two separate issues seem to be here: 1) storing verbatim UTMs -- Arctos needs to be required as any transformation will lose precision (just the nature of the math) so the question is where. Since we store UTM+metadata in collecting_event.verbatim_coordinates, can this just not be visible if it exists? Maybe add a tooltip on data entry that UTMs' are stored as text and are not mappable until converted to DD? 2) converting UTM+metadata to DD -- there is more than one CRS called UTMs so magic one-stop button to transform would probably not be trivial for Dusty and Arctos (we have data everywhere in the world). I personally have had to convert up to 5 different UTMs systems in MVZ's data and the collectors were no help (they got coordinates from others and had no idea what the relevant metadata were). So lots of trial/ error and sleuthing. BTW UTM and Datum-- take my georeferencing workshop! UTM is the coordinate reference system; datum is the mathematical model of the planet for conversion purposes, so an important part of the metadata. Often you can safely assume WGS84 but not if coordinates were plucked off a topoquad since there maybe more than one datum's grid being used (eg. USGS usually have both NAD27 and NAD83). Also regional high precision mapping may be using another datum that is better for local use. I think we need documentation on conversion sources and how-to's to accomplish in a GIS (best for batch converting especially non-US localities)-- on my to-do but Geo group can discuss more too.

…

On Wed, Mar 21, 2018 at 12:00 PM, dustymc ***@***.***> wrote: AWG Added to project — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1486 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACZ_0f9uLTcdACopr4EhKA9xrl2g7lf8ks5tgqNhgaJpZM4S07Be> .

[dustymc]

storing verbatim

This is not an issue - verbatim everything ends up in collecting_event.verbatim_coordinates. How those data may be accessed seems to be part of this. (It may be an immediate issue which revolves around the incorrect assumption that I can pull DD.d coordinate from input, but that's easy enough to fix.)

converting..not be trivial

That is my understanding as well. I think that leaves about two possibilities:

1. don't accept UTM in data entry
2. add a bunch of complexity (and perhaps some clever way of ignoring it most of the time???) to the data entry form so we can be explicit about what's verbatim vs. what's mapping coordinates.

(2) implies that we'd allow the entry of both, which is likely to result in...

1. verbatim
2. verbatim converted to DD.d coordinates
3. user-supplied DD.d coordinates which don't match the auto-converted

In this case I have no idea what we'd do.

Keep doing what we do except with UTM, and throw up a "this can't be converted" warning when a user selects UTM???

datum

I wasn't talking about conversion, but UTM data itself (eg, what you might read off a GPS). 64.148, -148.52 is ambiguous - it's datum-dependent. Are you saying that 26052.4E 14389.2N Z6 is also ambiguous/that datum changes it? I somehow got the idea that datum was built into UTM - that UTM + datum is nonsense.

[jtgiermakowski]

I have a whole set of specimens which have the coordinates indicated in UTM and NAD83 (yes, UTM needs datum for precision). It would be nice to accept UTM and force the user to convert those to Lat/Long in WGS84. Best scenario would be to have Georeferencing method added as "Converted from supplied verbatim UTM", thus no different than assigning coordinates from a verbatim locality. Burden is on user to convert properly, including datum transformations.

[Jegelewicz]

A UTM to coordinate tool: https://awsm-tools.com/geo/utm-to-geographic

Georeferencing method added as "Converted from supplied verbatim UTM"

Do you mean this should be an option for GEOREFERENCE PROTOCOL? I do think that the protocol options provided are limited, but this might already be part of the MANIS guidelines, if so then it is covered. If not, do we need a new protocol.

[dustymc]

per AWG:

• add explicit verbatim coordinate to data entry
  -- and TRS
• expose verbatim_coordinates

[campmlc]

From AWG meeting Sept 13, 2018

1. Interface issues: need to add UTM coordinates to specimen record display as verbatim coordinates and search results option
2. Dusty can do automated georeferencing of these - make this clear in georeference protocol
   How many UTM entries are there without coordinates? @dustymc
3. Big issue - what does this do to the data entry interface? At load, Arctos can't convert UTMs, so load fails; only pathway would be to add a separate pathway to data entry to add both verbatim and derived data;
4. Public Lands Survey System; same issue -we can't handle; so process as the same? make sure these data can be loaded as verbatim; TRS data
5. create as formated text fields?
6. All this must be searchable
7. consider international UTM issues at some point if the text fields are ever to be used for automated conversion
   Figure this out in geography committee

[dustymc]

Interface issues: need to add UTM coordinates to specimen record display as verbatim coordinates and search results option

http://arctos.database.museum/guid/MSB:Mamm:124903

Verbatim Coordinates: 4266462N 298460E ZONE 13

Dusty can do automated georeferencing of these - make this clear in georeference protocol

We already do that. You can pull it into the locality with a click. #1109. Should we actively push it (eg, to specimens with no coordiante-bearing event)? I have no idea how current sources deal with UTM (perhaps not at all??) - the automation uses all available information.

How many UTM entries are there without coordinates?

Zero - anything that's made it that far with (structured) UTM data was also georeferenced. The question here is if we want to change that, and if so how.

Big issue - what does this do to the data entry interface? At load, Arctos can't convert UTMs, so load fails; only pathway would be to add a separate pathway to data entry to add both verbatim and derived data;

Correct. Currently you provide verbatim data and at load it's converted to D.dd format and pushed to locality (along with being retained in collecting_event.verbatim_coordinates). I think you're asking to bypass that push - to allow ONLY verbatim. How should that work?

• we could just insert NULL into locality coordinates if they can't be converted, but that'll probably lead to silent failure for other conversions.
• we could add explicit "map coordinates" but you'd have to fill them out (in the bulkloader as well)
• ?????????

Public Lands Survey System

http://handbook.arctosdb.org/documentation/locality.html#trs

One option: treat UTM the same way, change nothing in the interfaces.

create as formated text fields?

That's always been in the model. It's squished together in one text field, but I'm the only one who's ever written to it so I can un-squish it.

searchable

This is a bit like searching a remarks field - you can string-match some stuff, but you're never going to know if you're actually found all of what you want or not.

consider international UTM issues at some point if the text fields are ever to be used for automated conversion

There's no standardization in the system; fully automating this stuff just doesn't seem likely. (And it's not just international - there are at least two "official" UTM-like things that get called UTM used in the US.)

[campmlc]

Question regarding this statement: Currently you provide verbatim data and at load it's converted to D.dd format and pushed to locality (along with being retained in collecting_event.verbatim_coordinates). I think you're asking to bypass that push - to allow ONLY verbatim. How should that work? I don't think we want to change anything for any of the other georeferencing data types, only UTM. Take this out of the dropdown, treat it separately as formatted text fields. That way we can continue using the autoconvert push for other data types. Am I understanding correctly?

…

On Thu, Sep 13, 2018 at 1:31 PM, dustymc ***@***.***> wrote: Interface issues: need to add UTM coordinates to specimen record display as verbatim coordinates and search results option http://arctos.database.museum/guid/MSB:Mamm:124903 Verbatim Coordinates: 4266462N 298460E ZONE 13 [image: screen shot 2018-09-13 at 12 14 54 pm] <https://user-images.githubusercontent.com/5720791/45510295-a813c480-b74e-11e8-9e23-c1e19409967d.png> Dusty can do automated georeferencing of these - make this clear in georeference protocol We already do that. You can pull it into the locality with a click. #1109 <#1109>. Should we actively push it (eg, to specimens with no coordiante-bearing event)? I have no idea how current sources deal with UTM (perhaps not at all??) - the automation uses all available information. How many UTM entries are there without coordinates? Zero - anything that's made it that far with (structured) UTM data was also georeferenced. The question here is if we want to change that, and if so how. Big issue - what does this do to the data entry interface? At load, Arctos can't convert UTMs, so load fails; only pathway would be to add a separate pathway to data entry to add both verbatim and derived data; Correct. Currently you provide verbatim data and at load it's converted to D.dd format and pushed to locality (along with being retained in collecting_event.verbatim_coordinates). I think you're asking to bypass that push - to allow ONLY verbatim. How should that work? - we could just insert NULL into locality coordinates if they can't be converted, but that'll probably lead to silent failure for other conversions. - we could add explicit "map coordinates" but you'd have to fill them out (in the bulkloader as well) - ????????? Public Lands Survey System http://handbook.arctosdb.org/documentation/locality.html#trs One option: treat UTM the same way, change nothing in the interfaces. create as formated text fields? That's always been in the model. It's squished together in one text field, but I'm the only one who's ever written to it so I can un-squish it. searchable This is a bit like searching a remarks field - you can string-match some stuff, but you're never going to know if you're actually found all of what you want or not. consider international UTM issues at some point if the text fields are ever to be used for automated conversion There's no standardization in the system; fully automating this stuff just doesn't seem likely. (And it's not just international - there are at least two "official" UTM-like things that get called UTM used in the US.) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1486 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AOH0hPHJX_SINyVHUlWWzAllnyjBVxFcks5uarJ1gaJpZM4S07Be> .

[Jegelewicz]

I don't think we want to change anything for any of the other georeferencing data types, only UTM. Take this out of the dropdown, treat it separately as formatted text fields. That way we can continue using the autoconvert push for other data types. Am I understanding correctly?

Agree. I get the complexity of making any conversions using this information. We just want it recorded in a way that allows it to be found without searching multiple free-text fields. I'd like to be able to find ALL of the UTEP:Inv records that include a UTM, because if I had that, Artie Metcalf's maps and field notes, and an on-line conversion tool, I could have someone do a Georeference that could be checked against whatever GeoLocate or Google comes up with from the text locality description. I would think I'd get a better outcome from the two than one or the other.

Bottom line - if you have a discreet data item, doesn't it always make more sense to place it in a discreet field? It may seem worthless now, but some day it might be just what someone is looking for....

[campmlc]

I think this goes back to Michelle's post from March 22 on this thread. What is being request now is ust some place we can enter these data and have them appear on the record and be searchable, e.g. from Michelle's post: 1) storing verbatim UTMs -- Arctos needs to be required as any transformation will lose precision (just the nature of the math) so the question is where. Since we store UTM+metadata in collecting_event.verbatim_coordinates, can this just not be visible if it exists? Maybe add a tooltip on data entry that UTMs' are stored as text and are not mappable until converted to DD? #2 is obviously the ultimate goal, but can we get something functional for #1 in the short term?

…

On Thu, Sep 13, 2018 at 5:22 PM, Teresa Mayfield ***@***.***> wrote: I don't think we want to change anything for any of the other georeferencing data types, only UTM. Take this out of the dropdown, treat it separately as formatted text fields. That way we can continue using the autoconvert push for other data types. Am I understanding correctly? Agree. I get the complexity of making any conversions using this information. We just want it recorded in a way that allows it to be found without searching multiple free-text fields. I'd like to be able to find ALL of the UTEP:Inv records that include a UTM, because if I had that, Artie Metcalf's maps and field notes, and an on-line conversion tool, I could have someone do a Georeference that could be checked against whatever GeoLocate or Google comes up with from the text locality description. I would think I'd get a better outcome from the two than one or the other. Bottom line - if you have a discreet data item, doesn't it always make more sense to place it in a discreet field? It may seem worthless now, but some day it might be just what someone is looking for.... — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1486 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AOH0hEzBAfWzUTloMExTTUteichHcdnpks5uaui3gaJpZM4S07Be> .

[dustymc]

The bulkloader will now deal with "coordinates" that cannot be converted to locality coordinates.

[Jegelewicz]

So for stuff I already have entered like this, where do I put the UTM?

[Jegelewicz]

OH - collecting event?

[dustymc]

Yup, collecting event is where most verbatim place-time-stuff lives.

[dustymc]

Reopening.

UTM data do not produce coordinates, coordinate metadata cannot exist without coordinates, so we have no place for max_error_distance and max_error_units.

I think those data make the most sense in locality remarks, and propose just throwing an error when the data are in UTM and something for which we don't have a verbatim_... field is included. I'm up for better ideas.

Help!

[Jegelewicz]

Can you show us what happens? I'm not certain I understand the problem or the solution....

[dustymc]

makes

while

makes

[Jegelewicz]

Just so I'm sure I get it.

Someone enters UTM coordinates with a max error value. Because we can't convert UTM to decimal degrees, the error busts the import. The solution is to put the error in the locality remarks.

Correct?

If so, I think that's OK, but not an obvious solution when one gets the error on bulkload. I don't know how many UTM coordinates come with an error anyway. The ones at UTEP usually don't, but that might not be true for other collections. The only other suggestion would be to have additional fields that go along with UTM Zone, UTM E/W, and UTM N/S - "UTM max error" and "UTM max error units" that would stick with the verbatim UTM stuff rather than be part of the "coordinates" fields - if that makes any sense at all.

[dustymc]

Close enough - "because we don't store verbatim_error" is more precise, but same effect.

I don't like the idea of the format of verbatim coordinates being context-dependent.

IF throwing an error is acceptable, it would be a less-vague error (eg, "you can't provide both UTM and error"). I could hide fields or something on data entry, but that won't help with bulkloading - I need a solution that works on the back end, not (just) the front.

I'm not sure how it's possible to end up with UTM and a defensible error, that's just what was entered....

[Jegelewicz]

I'm not sure how it's possible to end up with UTM and a defensible error, that's just what was entered....

I'm sure someone has a method somewhere - or a decision was made that null error means throw in 1000m.

"because we don't store verbatim_error"

Maybe we should? I'd have to mull that one over. However, a less-vague error would probably be OK. Can we also suggest that any error on UTM should be placed in locality remarks?

"you can't provide both UTM and error, please move error data to locality remarks"

[campmlc]

If we can add instructions with the error message, e.g. "you can't provide both UTM and error in bulkload format. Please move UTM error information to locality remarks and reload" then I think throwing an error should be fine. Errors that lead to acceptable solutions are workable. Alternately, can we automate this? If an error is entered with UTM data, automatically add to locality remarks column for reload? Error message: "you can't provide both UTM and error in bulkload format. UTM error information has been moved to locality remarks. If acceptable, mark to load."

…

On Wed, Oct 10, 2018 at 12:35 PM dustymc ***@***.***> wrote: Close enough - "because we don't store verbatim_error" is more precise, but same effect. I don't like the idea of the format of verbatim coordinates being context-dependent. *IF* throwing an error is acceptable, it would be a less-vague error (eg, "you can't provide both UTM and error"). I could hide fields or something on data entry, but that won't help with bulkloading - I need a solution that works on the back end, not (just) the front. I'm not sure how it's possible to end up with UTM and a defensible error, that's just what was entered.... — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1486 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AOH0hDymRe4UleUmQA8km4C3a1AVxlQAks5ujj2_gaJpZM4S07Be> .

[dustymc]

Maybe we should?

For this, I don't see the point. It is a denormalizer though - "1000 m" and "1 km" and "0.621371 miles" are all the same THING, but require 3 localities. Not worth thinking about as long as things like #1705 keep popping up.

automatically

I can, but I think it would confuse people - I think I'd rather be very plain in what the implications of UTM data are.

I added an error handler for now.

UTM may not be accompanied by MAX_ERROR_DISTANCE or MAX_ERROR_UNITS