error sensor bad data

[Lordfas]

The problem

Logger: homeassistant.components.pvpc_hourly_pricing.sensor
Source: components/pvpc_hourly_pricing/sensor.py:136
Integration: Spain electricity hourly pricing (PVPC) (documentation, issues)
First occurred: 21:22:10 (2 occurrences)
Last logged: 21:26:06

sensor.precio_luz: repeated bad data update, mark component as unavailable source

What version of Home Assistant Core has the issue?

core-2021.11.5

What was the last working version of Home Assistant Core?

core-2021.11.5

What type of installation are you running?

Home Assistant Core

Integration causing the issue

esios

Link to integration documentation on our website

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

[ad-ha]

I am finding the same issues. The PVPC has become unavailable.

My log shows the following:

Logger: aiopvpc.pvpc_data
Source: /usr/local/lib/python3.9/site-packages/aiopvpc/pvpc_data.py:231
First occurred: 7:09:26 PM (12 occurrences)
Last logged: 11:39:00 PM

Forbidden error with 'https://api.esios.ree.es/archives/70/download_json?locale=es&date=2021-11-29' -> Headers: <CIMultiDictProxy('Content-Type': 'text/html', 'Cache-Control': 'no-cache, no-store', 'Connection': 'close', 'Content-Length': '860', 'X-Iinfo': '6-18602997-0 0CNN RT(1638218339535 9) q(0 -1 -1 1) r(0 -1) B15(11,1720275,0)', 'Strict-Transport-Security': 'max-age=31536000', 'Set-Cookie': 'incap_ses_1396_1885724=o8iGaACy+k2RLpQe3pZfE2M6pWEAAAAAWlqKbJ+gbdejF3g5h4njzw==; path=/; Domain=.esios.ree.es; Secure; SameSite=None')>
Forbidden error with 'https://api.esios.ree.es/archives/70/download_json?locale=es&date=2021-11-29' -> Headers: <CIMultiDictProxy('Content-Type': 'text/html', 'Cache-Control': 'no-cache, no-store', 'Connection': 'close', 'Content-Length': '862', 'X-Iinfo': '13-87936528-0 0CNN RT(1638220139538 10) q(0 -1 -1 0) r(0 -1) B15(11,1720275,0)', 'Strict-Transport-Security': 'max-age=31536000', 'Set-Cookie': 'incap_ses_1396_1885724=Bu1qb1HNzFwyhaYe3pZfE2tBpWEAAAAAxWu7KSQ4MF8ZsdIxasTD7A==; path=/; Domain=.esios.ree.es; Secure; SameSite=None')>
Forbidden error with 'https://api.esios.ree.es/archives/70/download_json?locale=es&date=2021-11-29' -> Headers: <CIMultiDictProxy('Content-Type': 'text/html', 'Cache-Control': 'no-cache, no-store', 'Connection': 'close', 'Content-Length': '862', 'X-Iinfo': '12-77704484-0 0CNN RT(1638221939533 12) q(0 -1 -1 2) r(0 -1) B15(11,1720275,0)', 'Strict-Transport-Security': 'max-age=31536000', 'Set-Cookie': 'incap_ses_1396_1885724=/GNLbgJv0j12obge3pZfE3NIpWEAAAAAVNiv2o243jqlz6Ggp8VmCg==; path=/; Domain=.esios.ree.es; Secure; SameSite=None')>
Forbidden error with 'https://api.esios.ree.es/archives/70/download_json?locale=es&date=2021-11-29' -> Headers: <CIMultiDictProxy('Content-Type': 'text/html', 'Cache-Control': 'no-cache, no-store', 'Connection': 'close', 'Content-Length': '862', 'X-Iinfo': '12-78019039-0 0CNN RT(1638223739534 13) q(0 -1 -1 0) r(0 -1) B15(11,1720275,0)', 'Strict-Transport-Security': 'max-age=31536000', 'Set-Cookie': 'incap_ses_1396_1885724=0+FbXmDkPwp+qske3pZfE3tPpWEAAAAAMnnAkf0TAyzMjFBDA4bTCw==; path=/; Domain=.esios.ree.es; Secure; SameSite=None')>
Forbidden error with 'https://api.esios.ree.es/archives/70/download_json?locale=es&date=2021-11-29' -> Headers: <CIMultiDictProxy('Content-Type': 'text/html', 'Cache-Control': 'no-cache, no-store', 'Connection': 'close', 'Content-Length': '861', 'X-Iinfo': '14-98378988-0 0CNN RT(1638225539532 9) q(0 -1 -1 1) r(0 -1) B15(11,1720275,0)', 'Strict-Transport-Security': 'max-age=31536000', 'Set-Cookie': 'incap_ses_1396_1885724=QRqTU0BhkH6NVdce3pZfE4NWpWEAAAAA4CL3P2stUmXc+XTbwqhBFA==; path=/; Domain=.esios.ree.es; Secure; SameSite=None')>

and

Logger: homeassistant.components.pvpc_hourly_pricing.sensor
Source: components/pvpc_hourly_pricing/sensor.py:136
Integration: Spain electricity hourly pricing (PVPC) (documentation, issues)
First occurred: 7:10:27 PM (1 occurrences)
Last logged: 7:10:27 PM

sensor.pvpc: repeated bad data update, mark component as unavailable source

[probot-home-assistant]

pvpc_hourly_pricing documentation
pvpc_hourly_pricing source
_{^{(message by IssueLinks)}}

[probot-home-assistant]

Hey there @azogue, mind taking a look at this issue as it has been labeled with an integration (pvpc_hourly_pricing) you are listed as a code owner for? Thanks!
_{^{(message by CodeOwnersMention)}}

[lazynoda]

Same problem here. I've noticed that accessing directly to https://api.esios.ree.es/archives/70/download_json?locale=es&date=2021-11-30 via curl works.

System info

version	core-2021.11.5
installation_type	Home Assistant Container
dev	false
hassio	false
docker	true
user	root
virtualenv	false
python_version	3.9.7
os_name	Linux
os_version	5.10.63-v7l+
arch	armv7l
timezone	Europe/Madrid

[azogue]

It's also happening for me (and probably for all)

Looks like the ESIOS server is banning us again 😭

It returns a 403 status error for a request at a PUBLIC url 😤, and it's just banning us based on the content of the User-Agent header 😢, that's why the URL works with curl, or with a browser

I'll try to make a quickfix to try to avoid the banning, again 😭, but it looks like the webadmins at ESIOS are kind of angry at us 🙈

[vz-john]

Can you add an option to add an API Token from REE?
https://www.esios.ree.es/es/pagina/api

[jjprz]

Can you add an option to add an API Token from REE? https://www.esios.ree.es/es/pagina/api

It could be interesting

[martinezpenya]

Same problem here!

[davefx]

Alternately, maybe you could add an option to set the used User Agent, so they cannot easily identify us!

[fernandodpr]

I've just received my personal token for the REE API. They also sent the following message asking for a responsible use of the resource.

Este token se emite para uso personal del solicitante.

· Si va a dar acceso público a los proyectos desarrollados por Vd. (aplicaciones, páginas web, scripts…) que utilizan datos recuperados de la API de e·sios, debe llevarse la información que devuelve la API a un servidor de su propiedad, de tal forma que esas aplicaciones, páginas o scripts accedan a los datos almacenados en su servidor y no a los sistemas de REE.

· Uso responsable de la API: no realice peticiones masivas, redundantes o innecesarias, tal como información ya descargada con anterioridad o información que se sabe que no se ha modificado, así como peticiones a indicadores inexistentes

Would be great to have the possibility in the integration to insert the API token and send all petitions authenticated.

[lazynoda]

If "esios" API is banning us, we could also use REData API: https://www.ree.es/es/apidatos. Example for today: https://apidatos.ree.es/en/datos/mercados/precios-mercados-tiempo-real?start_date=2021-12-01T00:00&end_date=2021-12-01T23:59&time_trunc=hour

From JSON response, take one of the objects inside "include" (maybe filtering by id == "1001"?) and that data is in €/MWh. I don't see any rate limit restriction or anything else.

The REData API provides a simple REST service to allow third parties to access the backend data used in the REData application. By using this API, you will be able to retrieve data from the REData widgets and use them for your own purposes.

[lazynoda]

But totally agree on using "esios" personal API keys so we don't run on the same problem again. How much time it took @fernandodpr for them to provide a key? I guess we need to write it in HA documentation.

[MrMarble]

But totally agree on using "esios" personal API keys so we don't run on the same problem again. How much time it took @fernandodpr for them to provide a key? I guess we need to write it in HA documentation.

I asked for a token last night, and they replied today, exactly 14h apart

[fernandodpr]

But totally agree on using "esios" personal API keys so we don't run on the same problem again. How much time it took @fernandodpr for them to provide a key? I guess we need to write it in HA documentation.

Mostly the same as @MrMarble, I asked for it yesterday at 17h and received an email today at 11 h. The method it's a little rudimentary, but it seems to work.
I don't have a lot of spare time to test the new API, but as soon as I can I'll try It.

[martinezpenya]

I've requested it today, once I receive where can I test it? I'm new in this but I want to help, if I can.
I've been looking around the code and haven't seen where is the url and de token that the component is using now.

[azogue]

Quickfix for those in a hurry: azogue/aiopvpc#35 (comment)