Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Load spatial data #750

Merged
merged 8 commits into from
Feb 7, 2024
Merged

Load spatial data #750

merged 8 commits into from
Feb 7, 2024

Conversation

greg-does-weather
Copy link
Collaborator

@greg-does-weather greg-does-weather commented Feb 6, 2024

What does this PR do? 🛠️

  • Adds scripts and documentation for loading geospatial data locally and in cloud environments.
  • Adds our current geospatial data sources as git LFS files

closes #723

@loganmcdonald-noaa
Copy link
Member

Which environments have we done this for already? Just greg? Is it ok if I try to use the script to proceed in the other environments?

@greg-does-weather
Copy link
Collaborator Author

I loaded greg and staging. It shouldn't hurt anything to load the data into other environments. They won't use it until the code changes go in.

@greg-does-weather
Copy link
Collaborator Author

After merging, local developers will need to install and enable git LFS in order to get the spatial data. I'll drop a corresponding note in Slack when the time comes.

geospatial queries. In order for the site to work correctly locally, the data
needs to be loaded first. The spatial data is stored in our repo using
[git LFS](https://www.atlassian.com/git/tutorials/git-lfs).

Copy link
Member

@loganmcdonald-noaa loganmcdonald-noaa Feb 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might go without saying but - make sure the docker container is running and healthy?


3. Load the spatial data into your local database by running
```sh
node load-shapefiles.js
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have to also have the dependencies from this script locally since this is running locally and not in the container? i.e. mariadb, shapefiles, etc.

Copy link
Collaborator

@eric-gade eric-gade left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍 and it's working locally, as well as in my sandbox

Code Review Checklist

Partial checklist since this is an infra change

The Basics

  • Checks are passing
  • I read the code
  • I ran the code
  • (if applicable) Post deploy steps are run
  • (if applicable) I validated the change on the deployed version in

Documentation

  • changes to “how we do things” are documented in READMEs
  • all new functions and methods are commented using plain language
  • any new modules added documented in modules.md

Security

  • security false positives are documented
  • data from external sources is cleaned and clearly marked

Reliability

  • error handling exists for unusual or missing values
  • interactions with external systems are wrapped in try/except
  • functionality is tested with unit or integration tests
  • dependency updates in composer.json also got changed in composer-lock.json

Infrastructure

  • all changes are auditable and documented via a script
  • it is clear who can and should run the script
  • (if applicable) diagrams have been updated or added in PlantUML

Accessibility

  • New pages have been added to cypress-axe file so that they will be tested with our automated accessibility testing
  • Meets WCAG 2.0 AA or 2.1 AA for Section 508 compliance
    • Site is keyboard accessible. All interactions can be accessed with a keyboard
    • Site is free of keyboard traps. The keyboard focus is never trapped in a loop
    • All form inputs have explicit labels
    • Form instructions are associated with inputs
    • All relevant images use an img tag
    • All images have appropriate alt attributes
    • Multimedia is tagged. All multimedia has appropriate captioning and audio description
    • Text has sufficient color contrast. All text has a contrast ratio of 4.5:1 with the background
    • Site never loses focus. Focus is always visible when moving through the page with the keyboard
    • Tab order is logical
    • Tables are coded properly. Tables have proper headers and column attributes
    • Headings are nested properly. Heading elements are nested in a logical way
    • Language is set. The language for the page is set
    • CSS is not required to use the page. The page makes sense with or without CSS
    • Links are unique and contextual. All links can be understood taken alone, e.g., ‘Read more - about 508’
    • Page titles are descriptive

Device Matrix

  • firefox/gecko (renders correctly and user interactions work)
  • chrome/chromium/edge (renders correctly and user interactions work)
  • safari/webkit (renders correctly and user interactions work)
  • web page is readable and usable
    • at 480px (mobile)
    • at 640px (tablet)
    • at 1024px (desktop)

Comment on lines 22 to 24
// MariaDB supports IF EXISTS with indices but MySQL does not, so use this more
// convoluted way to drop indices if they already exists. (If you try to drop an
// index that doesn't exist, that's an error.)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, well that's annoying

trap "kill 0" EXIT

# Get database service connection details from cf
cf curl "/v2/apps/$(cf app --guid $TARGET)/env" | jq -r '.system_env_json.VCAP_SERVICES["aws-rds"][0].credentials' | jq -r '[.host, .port, .db_name, .username, .password]|@tsv' |
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Masterful.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I stole this from one of Logan's scripts and just tweaked it a little. It's very nice!

git lfs install
```

3. Load the spatial data into your local database by running
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe for the future not-usses (or us, when we've long forgotten), we can reiterate that you need to zap the application locally here and also git-lfs pull if needed. Or at least just to say, make sure the containers are running locally before doing this step.

Copy link
Collaborator Author

@greg-does-weather greg-does-weather Feb 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Went ahead and updated the doc, but also added spatial loading into the make zap target.

Copy link
Member

@loganmcdonald-noaa loganmcdonald-noaa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have also read and tested this locally with the container and it works great! Thank you!

@greg-does-weather greg-does-weather added this pull request to the merge queue Feb 7, 2024
Merged via the queue into main with commit 93d3737 Feb 7, 2024
9 of 10 checks passed
@greg-does-weather greg-does-weather deleted the mgwalker/load-gis-data branch February 7, 2024 21:04
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[GIS] Load geospatial data into cloud environments
3 participants