Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add dataset mapping and reference it in s3_to_json_s3 script #27

Merged
merged 2 commits into from
Dec 7, 2021
Merged

Add dataset mapping and reference it in s3_to_json_s3 script #27

merged 2 commits into from
Dec 7, 2021

Conversation

philerooski
Copy link
Contributor

@philerooski philerooski commented Dec 6, 2021

This addresses https://sagebionetworks.jira.com/browse/ETL-74

  • Added a dataset mapping file. Right now the only app version present in the file is the most recent test build, version 1.0.2, build 57
  • Added a function to get the dataset mapping file from S3 and reference it when bucketing the JSON file. The location of the dataset mapping file is assumed to be in the same bucket as the s3_to_json_s3 job script (as deployed by sceptre) and the key is hardcoded to its deployment location. I updated the deployment configuration so that both glue jobs and their resources (just dataset_mapping file, so far...) are included in the sceptre deployment.

@philerooski philerooski requested a review from tthyer December 6, 2021 22:46
Copy link
Contributor

@tthyer tthyer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@philerooski philerooski merged commit 9df1b14 into Sage-Bionetworks:main Dec 7, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants