NLS scraper has inaccurate number of records calculation #259
Labels
bug
Something isn't working
data engineering
Things related to data: scraping, cleaning, labelling, transformation
Describe the bug
The NLS scraper appears to incorrectly calculate the number of records field on opendata.scot based on the file contents description which just gives you the number of files in a zip. It also judges this based on the first file upload on a page and doesn't take into account multiple file uploads.
To Reproduce
See
fetch_num_recs()
method innls_scraper.py
Expected behavior
Number of records should reflect the number of records in a dataset (e.g. number of rows in a CSV)
Screenshots


Hardware and software used
N/A
Additional context
This functionality was patched out in 59dca44 but the function still remains
The text was updated successfully, but these errors were encountered: