Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace .csv in web-scraper scripts #225

Open
4 tasks
KarenJewell opened this issue Feb 18, 2023 · 0 comments
Open
4 tasks

Replace .csv in web-scraper scripts #225

KarenJewell opened this issue Feb 18, 2023 · 0 comments

Comments

@KarenJewell
Copy link
Member

Is your feature request related to a problem? Please describe.
Currently, the webscraper scripts (/web-scrapers) output to .csvs which is then ingested into merge_data.py. This should be converted to .json output to keep consistent with the rest of the project.

Describe the solution you'd like

  • Change the output of /web-scrapers to be .json files
  • Update merge_data.py to ingest the new .json files
  • format code with Black before PR
  • update docs (architecture.md and datapipeline.png)

Describe alternatives you've considered
None

Additional context
Original ticket triggering this change is #163

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Development

No branches or pull requests

1 participant