Webscrapers and dashboard for viewing energy/power related data from the ISO/RTOs (Independent/Regional System/Transmission Operators).
Currently only MISO data is avilable. Realtime data for load, generation, and LMP are shown in the dashboard, but historical data is retrievable using the "MISOClient". Examples of data retrieval are in "examples/miso_example.py".
- Make sure docker + docker-compose are installed
- Add a password to the ".env_example" file and rename the file to ".env"
- Run "sudo docker compose up -d"
- Open a browser and navigate to "http://0.0.0.0:8050"
Dashboard is made using Dash (Flask + Plotly). Data is stored using a PostgreSQL database. The dockerized Dash app, includes a cron job that runs a python script which retrieves and inserts data into the database every minute. The docker PostgreSQL database is separate from the docker Dash app.
The webscrapers utilize Aiohttp for asynchronous web requests.
demonstration.mp4
Historical data retrieval for MISO realtime LMP is quite slow (~10s per file on my machine). The archived/zip MISO market report files are large and parsing xlsx files can be slow (calamine engine helps -- requires pandas >= 2.2).