Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 140: ordinal not in range(128) #80

Open
lakovacs78 opened this issue May 11, 2021 · 11 comments

Comments

@lakovacs78
Copy link

lakovacs78 commented May 11, 2021

Hi @ndokter and @lowdef

I'm from Hungary and my power meter: Sanxing SX631 (S34U18)

I'm trying to use DSMR slimme meter integration in HA but I received an error message:

Traceback (most recent call last):
File "/usr/local/lib/python3.8/asyncio/selector_events.py", line 862, in _read_ready__data_received
self._protocol.data_received(data)
File "/usr/local/lib/python3.8/site-packages/dsmr_parser/clients/protocol.py", line 91, in data_received
data = data.decode('ascii')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 140: ordinal not in range(128)

Here is an example from my telegram data:

/AUX59902730044

0-0:1.0.0(210511102330S)
0-0:42.0.0(AUX1030302730044)
0-0:96.1.0(9902730044)
0-0:96.14.0(0001)
0-0:96.50.68(ON)
0-0:17.0.0(90.000kW)
1-0:1.8.0(000101.821
kWh)
1-0:1.8.1(000040.496kWh)
1-0:1.8.2(000061.325
kWh)
1-0:1.8.3(000000.000kWh)
1-0:1.8.4(000000.000
kWh)
1-0:2.8.0(000653.426kWh)
1-0:2.8.1(000398.181
kWh)
1-0:2.8.2(000255.245kWh)
1-0:2.8.3(000000.000
kWh)
1-0:2.8.4(000000.000kWh)
1-0:3.8.0(000002.626
kvarh)
1-0:4.8.0(000143.011kvarh)
1-0:5.8.0(000002.247
kvarh)
1-0:6.8.0(000000.379kvarh)
1-0:7.8.0(000075.324
kvarh)
1-0:8.8.0(000067.687kvarh)
1-0:15.8.0(000755.250
kWh)
1-0:32.7.0(237.2V)
1-0:52.7.0(235.5
V)
1-0:72.7.0(237.1V)
1-0:31.7.0(008
A)
1-0:51.7.0(008A)
1-0:71.7.0(008
A)
1-0:13.7.0(0.996)
1-0:33.7.0(0.996)
1-0:53.7.0(0.995)
1-0:73.7.0(0.997)
1-0:14.7.0(50.00Hz)
1-0:1.7.0(00.000
kW)
1-0:2.7.0(05.868kW)
1-0:5.7.0(00.000
kvar)
1-0:6.7.0(00.000kvar)
1-0:7.7.0(00.473
kvar)
1-0:8.7.0(00.000kvar)
0-0:98.1.0(210501000000S)(000041.191
kWh)(000015.425kWh)(000025.766kWh)(000274.705kWh)(000189.336kWh)(000085.369kWh)(000001.034kvarh)(000059.658kvarh)(000000.866kvarh)(000000.168kvarh)(000031.678kvarh)(000027.980kvarh)(000315.899kWh)(03.644kW)(03.188kW)(03.644kW)(06.872kW)(06.872kW)(06.544kW)
0-0:96.13.0(????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????)
!6D64

Could you help me pls?

Thank you in advance!

@ndokter
Copy link
Owner

ndokter commented May 13, 2021

The field with the question marks seems to be called 'Text message max 1024 characters.'. The parser currently assumes all byte data can be parsed as ASCII but it's not clear to me currently why that would fail

@lowdef
Copy link
Contributor

lowdef commented May 13, 2021

hi @lakovacs78,

What makes you think this should be parsable with DSMR parser?
Nearly none of the lines parses with any of the DSMR Specification variants.
What specification is your meter claiming to follow?

In principle I think the telegrams could be made to parse, but that requires adding a new specification type and maybe some new value types.

With following code you can investigate it yourself:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
issue78.py
@author: lowdef
"""

from dsmr_parser import telegram_specifications
from dsmr_parser.objects import Telegram
from dsmr_parser.parsers import TelegramParser
from dsmr_parser.parsers import MBusParser
from dsmr_parser.parsers import ValueParser
from dsmr_parser import obis_references
from dsmr_parser.value_types import timestamp
from decimal import Decimal


telegram_issue  = r"""/AUX59902730044

0-0:1.0.0(210511102330S)
0-0:42.0.0(AUX1030302730044)
0-0:96.1.0(9902730044)
0-0:96.14.0(0001)
0-0:96.50.68(ON)
0-0:17.0.0(90.000kW)
1-0:1.8.0(000101.821kWh)
1-0:1.8.1(000040.496kWh)
1-0:1.8.2(000061.325kWh)
1-0:1.8.3(000000.000kWh)
1-0:1.8.4(000000.000kWh)
1-0:2.8.0(000653.426kWh)
1-0:2.8.1(000398.181kWh)
1-0:2.8.2(000255.245kWh)
1-0:2.8.3(000000.000kWh)
1-0:2.8.4(000000.000kWh)
1-0:3.8.0(000002.626kvarh)
1-0:4.8.0(000143.011kvarh)
1-0:5.8.0(000002.247kvarh)
1-0:6.8.0(000000.379kvarh)
1-0:7.8.0(000075.324kvarh)
1-0:8.8.0(000067.687kvarh)
1-0:15.8.0(000755.250kWh)
1-0:32.7.0(237.2V)
1-0:52.7.0(235.5V)
1-0:72.7.0(237.1V)
1-0:31.7.0(008A)
1-0:51.7.0(008A)
1-0:71.7.0(008A)
1-0:13.7.0(0.996)
1-0:33.7.0(0.996)
1-0:53.7.0(0.995)
1-0:73.7.0(0.997)
1-0:14.7.0(50.00Hz)
1-0:1.7.0(00.000kW)
1-0:2.7.0(05.868kW)
1-0:5.7.0(00.000kvar)
1-0:6.7.0(00.000kvar)
1-0:7.7.0(00.473kvar)
1-0:8.7.0(00.000kvar)
0-0:98.1.0(210501000000S)(000041.191kWh)(000015.425kWh)(000025.766kWh)(000274.705kWh)(000189.336kWh)(000085.369kWh)(000001.034kvarh)(000059.658kvarh)(000000.866kvarh)(000000.168kvarh)(000031.678kvarh)(000027.980kvarh)(000315.899kWh)(03.644kW)(03.188kW)(03.644kW)(06.872kW)(06.872kW)(06.544kW)
0-0:96.13.0(????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????)
!6D64
"""


sample = telegram_issue.replace('\n', '\r\n')

telegram_specification = telegram_specifications.V5
telegram_specification['checksum_support']=False

parser = TelegramParser(telegram_specification)
telegram = Telegram(sample, parser, telegram_specification)

print(telegram)

@lakovacs78
Copy link
Author

lakovacs78 commented May 17, 2021

hi @lakovacs78,

What makes you think this should be parsable with DSMR parser?

Nearly none of the lines parses with any of the DSMR Specification variants.

What specification is your meter claiming to follow?

In principle I think the telegrams could be made to parse, but that requires adding a new specification type and maybe some new value types.

With following code you can investigate it yourself:

#!/usr/bin/env python3

# -*- coding: utf-8 -*-

"""

issue78.py

@author: lowdef

"""



from dsmr_parser import telegram_specifications

from dsmr_parser.objects import Telegram

from dsmr_parser.parsers import TelegramParser

from dsmr_parser.parsers import MBusParser

from dsmr_parser.parsers import ValueParser

from dsmr_parser import obis_references

from dsmr_parser.value_types import timestamp

from decimal import Decimal





telegram_issue  = r"""/AUX59902730044



0-0:1.0.0(210511102330S)

0-0:42.0.0(AUX1030302730044)

0-0:96.1.0(9902730044)

0-0:96.14.0(0001)

0-0:96.50.68(ON)

0-0:17.0.0(90.000kW)

1-0:1.8.0(000101.821kWh)

1-0:1.8.1(000040.496kWh)

1-0:1.8.2(000061.325kWh)

1-0:1.8.3(000000.000kWh)

1-0:1.8.4(000000.000kWh)

1-0:2.8.0(000653.426kWh)

1-0:2.8.1(000398.181kWh)

1-0:2.8.2(000255.245kWh)

1-0:2.8.3(000000.000kWh)

1-0:2.8.4(000000.000kWh)

1-0:3.8.0(000002.626kvarh)

1-0:4.8.0(000143.011kvarh)

1-0:5.8.0(000002.247kvarh)

1-0:6.8.0(000000.379kvarh)

1-0:7.8.0(000075.324kvarh)

1-0:8.8.0(000067.687kvarh)

1-0:15.8.0(000755.250kWh)

1-0:32.7.0(237.2V)

1-0:52.7.0(235.5V)

1-0:72.7.0(237.1V)

1-0:31.7.0(008A)

1-0:51.7.0(008A)

1-0:71.7.0(008A)

1-0:13.7.0(0.996)

1-0:33.7.0(0.996)

1-0:53.7.0(0.995)

1-0:73.7.0(0.997)

1-0:14.7.0(50.00Hz)

1-0:1.7.0(00.000kW)

1-0:2.7.0(05.868kW)

1-0:5.7.0(00.000kvar)

1-0:6.7.0(00.000kvar)

1-0:7.7.0(00.473kvar)

1-0:8.7.0(00.000kvar)

0-0:98.1.0(210501000000S)(000041.191kWh)(000015.425kWh)(000025.766kWh)(000274.705kWh)(000189.336kWh)(000085.369kWh)(000001.034kvarh)(000059.658kvarh)(000000.866kvarh)(000000.168kvarh)(000031.678kvarh)(000027.980kvarh)(000315.899kWh)(03.644kW)(03.188kW)(03.644kW)(06.872kW)(06.872kW)(06.544kW)

0-0:96.13.0(????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????)

!6D64

"""





sample = telegram_issue.replace('\n', '\r\n')



telegram_specification = telegram_specifications.V5

telegram_specification['checksum_support']=False



parser = TelegramParser(telegram_specification)

telegram = Telegram(sample, parser, telegram_specification)



print(telegram)

@lowdef Thank you for your help!

I tried to parse my "const" telegram data based on your example script (sorry I'm not a Python expert). Now I have a theory what is wrong in my telegram but I need to connect somehow to my DSMR reader in a Python script and then parse my data with dsmr parser. Could you help me in this script (how I should set the serial connection) pls?

@lowdef
Copy link
Contributor

lowdef commented May 17, 2021 via email

@Zolli
Copy link

Zolli commented Sep 18, 2021

Hey, @lowdef i encountered the same problem using this specific meter, also in Hungary, if i'm able to help somehow please let me know, to receive the data i use a SlimmeLezer falsed with esp-link so i able to acces the data using telnet

@Bubi504
Copy link

Bubi504 commented Nov 4, 2021

@Zolli jutottál ezzel valamire ?

@Zolli
Copy link

Zolli commented Nov 4, 2021

@Bubi504 Sure, but not with esphome sadly. I ended up with the DSMRReader project. This project discards unicode characters, but obviously makes the CRC check fail, but fortunately you can disable CRC checks. I configured it as a DSMR5 compatible meter. It works like a charm

@Bubi504
Copy link

Bubi504 commented Nov 4, 2021

@Zolli Now it's working without DSMRReader, the solution is here : esphome/issues#2393 (comment)

@Aeroid
Copy link
Contributor

Aeroid commented Nov 12, 2021

I had the same issue with my tcp-bridge. Adding errors="ignore" to decode("ascii") helped
see #92

@p0macs
Copy link

p0macs commented Feb 3, 2024

Hi!
This is an old issue, but I have a similar error now, using homeassistant (Core: 2024.1.6) and trying to add the DSMR Slimme meter integration.
After filling in the configuration (using network with port 23) the integration cannot be added.
Looking into the logs, I can see error messages like this:
File "/usr/local/lib/python3.11/site-packages/dsmr_parser/clients/protocol.py", line 132, in data_received telegram = telegram.encode("latin1").decode("ascii") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 1371: ordinal not in range(128)
I have a hungarian EON meter too. I can parse the telegram using DSMR Reader without issues.
I guess the problem is very similar to the original issue, at the last telegram message:
0-0:96.13.0(ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ)
This version of homeassistant should use the 1.3.1 version of dsmr-parser.
Do you have an idea how to add this integration?

@p0macs
Copy link

p0macs commented Feb 4, 2024

The problem with the above telegram message is, that the 0xFF characters are not a trailing character in the message, but they are part of the last 0-0:96.13.0 OBIS ID code. This is some kind of placeholder for a missing text message.
One problem is, that the telegram.encode("latin1").decode("ascii") will not accept the 0xFF codes.
The other problem is, that parser regexp will accept only letters and number and some special characters like .*().
One possible workaround could be to simply remove this placeholder from the telegram message - after the checksum calculation, but before parsing the lines - using something like replace(chr(0xff),'')

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants