[ETL-245] Add script for resolving ETL-245 #90
Conversation
| except FileNotFoundError: | ||
| study_counts[study]["parquet"].append((dataset, 0)) | ||
| study_counts[study]["json"] = sorted(study_counts[study]["json"]) | ||
| study_counts[study]["parquet"] = sorted(study_counts[study]["parquet"]) |
There was a problem hiding this comment.
What is the purpose of the comparison? Is this just to make sure all the file counts match the unique record counts?
There was a problem hiding this comment.
Each record (zip file) from Bridge contain a number of JSON. The only required JSON in each record is metadata.json (which conforms to the ArchiveMetadata schema) so comparing file counts of the ArchiveMetadata JSON dataset to the record counts in the ArchiveMetadata Parquet dataset tells us if any records failed to process during the export to Parquet. I think every record from MTB thus far includes a taskData.json (sharedSchema) as well, so we can look at these counts and compare them to the ArchiveMetadata counts. The other datasets are less useful as QA unless you count the strict subset of records which contain AudioLevelRecord, MotionRecord, or WeatherResult data, but still interesting to count.
No description provided.