Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Move overrides from scrapers/package.py into a separate file, scrapers/overrides.json #76

Open
wants to merge 14 commits into
base: main
Choose a base branch
from
Open

Move overrides from scrapers/package.py into a separate file, scrapers/overrides.json #76

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
bc633a9
add overrides.json to scrapers folder
duck-master Nov 4, 2024
464639b
use overrides.json within package.py
duck-master Nov 4, 2024
d395ee9
refactor package.py to only use overrides.json
duck-master Nov 4, 2024
585e58f
refactor package.py to move overrides.json logic into get_overrides()
duck-master Nov 4, 2024
5c8674b
add mention of scrapers/overrides.json to scrapers/README.md
duck-master Nov 4, 2024
e472ae4
get_overrides() -> get_json_data(jsonfile) in scrapers/package.py
duck-master Nov 4, 2024
ab4b74c
refactor scrapers/package.py to use get_json_data
duck-master Nov 11, 2024
cd44271
set current working directory in scrapers/package.py
duck-master Nov 11, 2024
3b7ff1d
reorder imports in scrapers/package.py; replace dict() with {}
duck-master Nov 11, 2024
abad611
specify encoding of public/latest.json in scrapers/package.py
duck-master Nov 11, 2024
f47e40b
update scrapers/package.py docstring
duck-master Nov 11, 2024
4b7600c
add merge_data function in scrapers/package.py
duck-master Nov 13, 2024
5ac9ed3
refactor scrapers/package.py to use merge_data
duck-master Nov 13, 2024
69c3a2c
decapitalize poverrides in scrapers/package.py
duck-master Nov 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion scrapers/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ This folder contains several files. The files tracked by git are:
* utils.py
* README.md - this very file!
* .pylintrc
* overrides.json - some manual override data for cases where web scraping fails

The files intentionally left out of git are:

Expand All @@ -36,7 +37,7 @@ It is mainly intended to serve as a data source for the frontend, which is the r

* `fireroad.py` creates `fireroad.json`
* `catalog.py` creates `catalog.json`
* `package.py` combines these to create `../public/latest.json`. (This is the final product that our frontend ingests.)
* `package.py` combines these with `overrides.json` to create `../public/latest.json`. (This is the final product that our frontend ingests.)

`math_dept.py` is an irregularly run file that helps create override data for courses in the MIT math department (since those are formatted slightly differently). `utils.py` contains a few utility functions and variables, which in turn are used by `fireroad.py` and `package.py`. The file `__init__.py` is empty but we include it anyways for completeness.

Expand Down
Loading