Change data source to apidatos.ree.es

.pre-commit-config.yaml

@@ -1,14 +1,14 @@
 minimum_pre_commit_version: "2.10.0"
 repos:
   - repo: https://github.com/pycqa/isort
-    rev: 5.8.0
+    rev: 5.10.1
     hooks:
       - id: isort
         name: isort (python)
         args:
           - --dont-order-by-type
   - repo: https://github.com/psf/black
-    rev: "21.6b0"
+    rev: "21.11b1"
     hooks:
       - id: black
         name: Format code (black)
@@ -21,7 +21,7 @@ repos:
       - id: check-toml
       - id: check-yaml
   - repo: https://github.com/pycqa/flake8
-    rev: "3.9.2"
+    rev: "4.0.1"
     hooks:
       - id: flake8
         name: Lint code (flake8)
@@ -39,7 +39,7 @@ repos:
           - flake8-variables-names>=0.0.3
           - pep8-naming>=0.11.1
   - repo: "https://github.com/pre-commit/mirrors-mypy"
-    rev: "v0.902"
+    rev: "v0.910-1"
     hooks:
       - id: "mypy"
         name: "Check type hints (mypy)"

CHANGELOG.md

@@ -1,5 +1,38 @@
 # Changelog
 
+## [v3.0.0](https://github.com/azogue/aiopvpc/tree/v3.0.0) - Change Data Source to apidatos.ree.es (2021-12-05)
+
+[Full Changelog](https://github.com/azogue/aiopvpc/compare/v3.0.0...v2.3.0)
+
+🔥 **BREAKING-CHANGE**: this release **removes support for the old PVPC tariffs**
+(prices < 2021-06-01), and the extra methods to use this library as a _dataloader_
+(`.download_prices_for_range(...)`), leaving only the **code to support the HA Core integration**.
+
+Motivated by recent successful attempts to kick us out from `api.esios.ree.es`,
+we are changing the data source to another REE public server, at `apidatos.ree.es`,
+with the same information than the current one, available without authentication 👌
+
+**This release implements the new data-source**, but also maintains the _legacy_ one.
+
+* Initial configuration is set with a new `data_source` parameter, **with the new source as default**.
+* If a 403 status-code is received, the **data source is switched** (new to legacy / legacy to new), no retry is done,
+  and the User-Agent loop trick is only used for the legacy data-source.
+
+**Changes:**
+
+* :fire: Remove support for old PVPC tariffs and range download methods,
+and make `tariff` and `websession` required arguments
+
+* :sparkles: Add alternative data-source from 'apidatos.ree.es'
+  * Implement data parsing from `apidatos.ree.es`, using endpoint at `/es/datos/mercados/precios-mercados-tiempo-real`
+  * Add `data_source` parameter with valid keys 'apidatos' and 'esios_public', setting the new one as default ;-)
+  * Remove retry call if 403 status is received, but maintain the User-Agent loop, and also toggle the data-source for the next call
+  * Move old ATTRIBUTION to `.attribution` property, as function of the data-source
+
+* :truck: Change test patterns to new tariffs by substituting old examples in DST days from 2019 to equivalent days since 2021-06, using the new tariff keys
+
+* :truck: Add test patterns from new data-source, and adjust tests
+
 ## [v2.3.0](https://github.com/azogue/aiopvpc/tree/v2.3.0) - Decrease API refresh rate and try to avoid banning (2021-12-01)
 
 [Full Changelog](https://github.com/azogue/aiopvpc/compare/v2.3.0...v2.2.4)

README.md

@@ -28,17 +28,12 @@ Install with `pip install aiopvpc` or clone it to run tests or anything else.
 ## Usage
 
 ```python
+import aiohttp
 from datetime import datetime
 from aiopvpc import PVPCData
 
-pvpc_handler = PVPCData(tariff="discrimination", zone_ceuta_melilla=False)
-
-start = datetime(2021, 5, 20, 22)
-end = datetime(2021, 6, 7, 16)
-prices_range: dict = await pvpc_handler.async_download_prices_for_range(start, end)
+async with aiohttp.ClientSession() as session:
+    pvpc_handler = PVPCData(session=session, tariff="2.0TD")
+    prices: dict = await pvpc_handler.async_update_prices(datetime.utcnow())
+print(prices)
 ```
-
-Check [this example on a jupyter notebook](https://github.com/azogue/aiopvpc/blob/master/Notebooks/Download%20PVPC%20prices.ipynb), where the downloader is combined with pandas and matplotlib to plot the electricity prices.
-To play with it, clone the repo and install the project with `poetry install -E jupyter`, and then `poetry run jupyter notebook`.
-
-![sample_pvpc_plot.png](https://github.com/azogue/aiopvpc/blob/master/Notebooks/sample_pvpc_plot.png)

aiopvpc/__init__.py

@@ -1,4 +1,5 @@
 """Simple aio library to download Spanish electricity hourly prices."""
-from .pvpc_data import DEFAULT_POWER_KW, PVPCData, TARIFFS
+from .const import DEFAULT_POWER_KW, TARIFFS
+from .pvpc_data import PVPCData
 
 __all__ = ("DEFAULT_POWER_KW", "PVPCData", "TARIFFS")

aiopvpc/const.py

@@ -3,6 +3,7 @@
 """
 import sys
 from datetime import date
+from typing import Dict, Literal
 
 if sys.version_info[:2] >= (3, 9):  # pragma: no cover
     import zoneinfo  # pylint: disable=import-error
@@ -13,14 +14,11 @@
 
 # Tariffs as internal keys in esios API data
 TARIFF_20TD_IDS = ["PCB", "CYM"]
-OLD_TARIFS_IDS = ["GEN", "NOC", "VHC"]
 
 # Tariff names used in HomeAssistant integration
 TARIFFS = ["2.0TD", "2.0TD (Ceuta/Melilla)"]
-OLD_TARIFFS = ["normal", "discrimination", "electric_car"]
 
 TARIFF2ID = dict(zip(TARIFFS, TARIFF_20TD_IDS))
-OLD_TARIFF2ID = dict(zip(OLD_TARIFFS, OLD_TARIFS_IDS))
 
 # Contracted power
 DEFAULT_POWER_KW = 3.3
@@ -31,8 +29,19 @@
 
 DEFAULT_TIMEOUT = 5
 PRICE_PRECISION = 5
+
+DataSource = Literal["esios_public", "apidatos"]  # , "esios"
 URL_PVPC_RESOURCE = (
     "https://api.esios.ree.es/archives/70/download_json"
     "?locale=es&date={day:%Y-%m-%d}"
 )
-ATTRIBUTION = "Data retrieved from api.esios.ree.es by REE"
+URL_APIDATOS_PRICES_RESOURCE = (
+    "https://apidatos.ree.es/es/datos/mercados/precios-mercados-tiempo-real"
+    "?time_trunc=hour"
+    "&geo_ids={geo_id}"
+    "&start_date={start:%Y-%m-%dT%H:%M}&end_date={end:%Y-%m-%dT%H:%M}"
+)
+ATTRIBUTIONS: Dict[DataSource, str] = {
+    "esios_public": "Data retrieved from api.esios.ree.es by REE",
+    "apidatos": "Data retrieved from apidatos.ree.es by REE",
+}

aiopvpc/parser.py

@@ -5,16 +5,59 @@
 * Parser for the contents of the JSON files
 """
 from datetime import datetime, timedelta
-from typing import Any, Dict, Optional, Union
+from typing import Any, Dict, TypedDict
 
-from aiopvpc.const import PRICE_PRECISION, REFERENCE_TZ, UTC_TZ, zoneinfo
+from aiopvpc.const import (
+    DataSource,
+    PRICE_PRECISION,
+    REFERENCE_TZ,
+    URL_APIDATOS_PRICES_RESOURCE,
+    URL_PVPC_RESOURCE,
+    UTC_TZ,
+    zoneinfo,
+)
 
 
-def extract_pvpc_data(
-    data: Dict[str, Any],
-    key: Optional[str] = None,
-    tz: zoneinfo.ZoneInfo = REFERENCE_TZ,
-) -> Union[Dict[datetime, float], Dict[datetime, Dict[str, float]]]:
+class PricesResponse(TypedDict):
+    """Data schema for parsed prices coming from `apidatos.ree.es`."""
+
+    name: str
+    data_id: str
+    last_update: datetime
+    unit: str
+    series: Dict[str, Dict[datetime, float]]
+
+
+def extract_prices_from_apidatos_ree(
+    data: Dict[str, Any], tz: zoneinfo.ZoneInfo = REFERENCE_TZ
+) -> PricesResponse:
+    """Parse the contents of a query to 'precios-mercados-tiempo-real'."""
+    ref_ts = datetime(2021, 1, 1, tzinfo=REFERENCE_TZ).astimezone(UTC_TZ)
+    loc_ts = datetime(2021, 1, 1, tzinfo=tz).astimezone(UTC_TZ)
+    loc_ts - ref_ts.astimezone(UTC_TZ)
+    offset_timezone = loc_ts - ref_ts
+
+    def _parse_dt(ts: str) -> datetime:
+        return datetime.fromisoformat(ts).astimezone(UTC_TZ) + offset_timezone
+
+    return PricesResponse(
+        name=data["data"]["type"],
+        data_id=data["data"]["id"],
+        last_update=_parse_dt(data["data"]["attributes"]["last-update"]),
+        unit="€/kWh",
+        series={
+            data_series["type"].replace(" (€/MWh)", ""): {
+                _parse_dt(price["datetime"]): round(price["value"] / 1000.0, 5)
+                for price in data_series["attributes"]["values"]
+            }
+            for data_series in data["included"]
+        },
+    )
+
+
+def extract_prices_from_esios_public(
+    data: Dict[str, Any], key: str, tz: zoneinfo.ZoneInfo = REFERENCE_TZ
+) -> PricesResponse:
     """Parse the contents of a daily PVPC json file."""
     ts_init = datetime(
         *datetime.strptime(data["PVPC"][0]["Dia"], "%d/%m/%Y").timetuple()[:3],
@@ -24,19 +67,42 @@ def extract_pvpc_data(
     def _parse_tariff_val(value, prec=PRICE_PRECISION) -> float:
         return round(float(value.replace(",", ".")) / 1000.0, prec)
 
-    def _parse_val(value) -> float:
-        return float(value.replace(",", "."))
-
-    if key is not None:
-        return {
-            ts_init + timedelta(hours=i): _parse_tariff_val(values_hour[key])
-            for i, values_hour in enumerate(data["PVPC"])
-        }
-
-    return {
-        ts_init
-        + timedelta(hours=i): {
-            k: _parse_val(v) for k, v in values_hour.items() if k not in ("Dia", "Hora")
-        }
+    pvpc_prices = {
+        ts_init + timedelta(hours=i): _parse_tariff_val(values_hour[key])
         for i, values_hour in enumerate(data["PVPC"])
     }
+
+    return PricesResponse(
+        name="PVPC ESIOS",
+        data_id="legacy",
+        last_update=datetime.utcnow().replace(microsecond=0, tzinfo=UTC_TZ),
+        unit="€/kWh",
+        series={"PVPC": pvpc_prices},
+    )
+
+
+def extract_pvpc_data(
+    data: Dict[str, Any], url: str, key: str, tz: zoneinfo.ZoneInfo = REFERENCE_TZ
+) -> Dict[datetime, float]:
+    """Parse the contents of a daily PVPC json file."""
+    if url.startswith("https://api.esios.ree.es/archives"):
+        prices_data = extract_prices_from_esios_public(data, key, tz)
+    elif url.startswith("https://apidatos.ree.es"):
+        prices_data = extract_prices_from_apidatos_ree(data, tz)
+    else:
+        raise NotImplementedError(f"Data source not known: {url} >{data}")
+    return prices_data["series"]["PVPC"]
+
+
+def get_url_prices(
+    source: DataSource, zone_ceuta_melilla: bool, now_local_ref: datetime
+) -> str:
+    """Make URL for PVPC prices."""
+    if source == "esios_public":
+        return URL_PVPC_RESOURCE.format(day=now_local_ref.date())
+
+    start = now_local_ref.replace(hour=0, minute=0)
+    end = now_local_ref.replace(hour=23, minute=59)
+    return URL_APIDATOS_PRICES_RESOURCE.format(
+        start=start, end=end, geo_id=8744 if zone_ceuta_melilla else 8741
+    )