Make data sources readable and incorporate sha256 hash checks #48
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
weiji14
force-pushed
the
data_list
branch
2 times, most recently
from
95bd90b to
4632010
Compare
Switch to a format that is both human and machine readable - YAML. See http://yaml.org/spec/1.2/spec.html. Datasets are grouped into blocks, with each block having its own metadata (e.g. dataset and literature DOI links). Aiming to automatically generate README.md files from this YAML file into the highres/lowres/misc folders.
Standardize the README.md files across the highres/lowres/misc folders. Has columns Filename, Location, Resolution, Literature Citation and Data Citation. Wrote some Python pandas code to parse the data_list.yml. Note that high resolution datasets have 'nan' as their resolution as they are point datasets (i.e. this was placed in too hard basket).
weiji14
changed the title
weiji14
added a commit
that referenced
this pull request
Nov 15, 2018
Did not realize that my local copy of bed_WGS84_grid.txt was corrupted, so changing the hash. Also remove extra whitespace in the data_list.yml file. Patches #48.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The work in this Pull Request ensures that our data source list should be both human readable and machine readable. People wishing to examine the original data sources should be able to do so with a visible document (README.md). Scripts should be able to easily parse the url links and sha256 hashes tied to the data via something like a Comma Separated Value file (data_list.csv) or YAML file (data_list.yml).
This is also where we get serious about data integrity. Explicitly listing all the data files with their SHA256 checksums (required) and their download URL (if available). Note only the Pine Island Glacier datasets are missing a public URL.
Links upstream to #7, #8, #20, #21.
TODO:
Create data_list.csv listing all the current data files(b55a7c7)