Attempt to set variable to body of email fails with core dump

[scstraus]

Hi, when I try to set one of the variable attributes to the value of the body of an email from IMAP email sensor, I get the following (traceback at the bottom)

2019-05-14 22:56:07 DEBUG (MainThread) [homeassistant.core] Bus:Handling <Event call_service[L]: domain=variable, service=set_variable, service_data=variable=dpd_variable, value_template=delivery, attributes_template={
  "from": "REDACTED",
  "date": "Tue, 14 May 2019 21:42:10 +0200",
  "subject": "Fwd: DPD Kurýr: Dnes Vám doručíme balíček",
  "body": "---------- Forwarded message ---------
From: <[email protected]>
Date: Wed, Apr 17, 2019 at 8:55 AM
Subject: DPD Kur=C3=BDr: Dnes V=C3=A1m doru=C4=8D=C3=ADme bal=C3=AD=C4=8Dek
To: <REDACTED>


[image: Header image] <https://dpdkuryr.cz/r?Parcel=3D15505908155927>
P=C4=9Bkn=C3=BD den,
V=C3=A1=C5=A1 bal=C3=AD=C4=8Dek =C4=8D. *15505908155927* od SHIPITO C/O MAP=
CARGO

doru=C4=8D=C3=ADme dnes mezi 11:04-12:04.
*Doru=C4=8Dovac=C3=AD adresa: * REDACTED
V aplikaci DPD Kur=C3=BDr si st=C3=A1le m=C5=AF=C5=BEete zm=C4=9Bnit term=
=C3=ADn i m=C3=ADsto
Va=C5=A1e p=C5=99ihla=C5=A1ovac=C3=AD =C3=BAdaje: www.dpdkuryr.cz
<https://dpdkuryr.cz/r?Parcel=3D15505908155927>, osobn=C3=AD k=C3=B3d 8326
Nebo si m=C5=AF=C5=BEete bal=C3=AD=C4=8Dek vyzvednout na n=C4=9Bkter=C3=A9m=
 z na=C5=A1ich v=C3=BDdejn=C3=ADch m=C3=ADst
Pickup!
Tady na V=C3=A1s po=C4=8Dk=C3=A1 7 dn=C3=AD.
P=C5=99ihlaste se do na=C5=A1=C3=AD
aplikace DPD Kur=C3=BDr
<https://dpdkuryr.cz/Home/Parcel?utm_source=3Dnotifikace&utm_medium=3De-mai=
l&utm_content=3Dofd&utm_campaign=3DDPD_Kuryr&Parcel=3D15505908155927>


Vyberte si z t=C3=A9m=C4=9B=C5=99 1 000
v=C3=BDdejn=C3=ADch m=C3=ADst Pickup <https://pickup.dpd.cz>


[image: Chatbot image]
<https://bot.dpd.cz/?utm_source=3Dnotifikace&utm_medium=3Demail>
*Jak jste byli spokojeni s doru=C4=8Den=C3=ADm?*
Budeme r=C3=A1di za Va=C5=A1i zp=C4=9Btnou vazbu. Sta=C4=8D=C3=AD, kdy=C5=
=BE tady
<http://public.dpd.cz/customers/feedback/whitelistpage.aspx?email=3DZ0SYAKO=
GAASFW01GXZbzV09S>
odsouhlas=C3=ADte zasl=C3=A1n=C3=AD kr=C3=A1tk=C3=A9ho dotazn=C3=ADku. D=C4=
=9Bkujeme.

S radost=C3=AD,
Va=C5=A1e DPD

dpd.cz <http://www.dpd.cz>  |  Sledujte n=C3=A1=C5=A1 Facebook
<https://www.facebook.com/dpd.cz>  a  Twitter
<https://twitter.com/dpd_cz?lang=3Dcs>  |  Nej=C4=8Dast=C4=9Bj=C5=A1=C3=AD =
odpov=C4=9Bdi
<http://dorucujemeradost.dpd.cz/kontaktujte-nas?utm_source=3DNotifikace&utm=
_medium=3DLinkZapati&utm_campaign=3DFAQ>
 d=C5=99=C3=ADve,
ne=C5=BE se stihnete zeptat
Toto je automaticky generovan=C3=BD e-mail, proto na n=C4=9Bj pros=C3=ADm n=
eodpov=C3=ADdejte.

[image: Zname zvyky a chovani online nakupujicich]
<https://eshopperbarometer.dpd.com/>


Tento e-mail je urcen pouze pro jeho adresata/adresaty a muze obsahovat
duverne informace, jejichz ochrana muze byt vyzadovana pravnimi predpisy.
Jestlize jste zpravu obdrzel(a) omylem, neprodlene informujte jejiho
odesilatele a tuto zpravu, jeji prilohy a pripadne kopie ihned vymazte.
Jakakoli forma uziti, zverejneni, reprodukce, kopirovani, distribuce a
sireni teto zpravy je v takovem pripade zakazana. DPD CZ s.r.o., neodpovida
za mozne skody zpusobene neuplnym prenosem, moznou modifikaci ci zpozdenim
teto zpravy behem prenosu od odesilatele k adresatovi.
---
This e-mail transmission is intended solely for the ordinary user of the
e-mail address to which it was addressed. It may contain legally privileged
and/or confidential information. If you have received this e-mail in error
or are not an intended recipient please inform the sender with-out delay
and delete this e-mail, attachments and possible copies immediately. The
unauthorised use, disclosure, distribution and/or copying of this e-mail or
any information it contains is prohibited. DPD CZ s.r.o., does not accept
liability for any damage caused by incomplete transmission, possible
modification or delay of this e-mail during the transmission from the
sender to the recipient.
"
}>
2019-05-14 22:56:07 ERROR (MainThread) [homeassistant.core] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/config/custom_components/variable/__init__.py", line 179, in async_set_variable
    attributes = json.loads(attributes_template.async_render({ 'variable': current_state }))
  File "/usr/local/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.7/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Invalid control character at: line 5 column 50 (char 186)
2019-05-14 22:56:07 DEBUG (MainThread) [homeassistant.core] Bus:Handling <Event state_changed[L]: entity_id=automation.dpd_state_change, old_state=<state automation.dpd_state_change=on; last_triggered=None, id=dpd_state_change, friendly_name=DPD state change, hidden=True @ 2019-05-14T22:55:42.905821+02:00>, new_state=<state automation.dpd_state_change=on; last_triggered=2019-05-14T22:56:07.720688+02:00, id=dpd_state_change, friendly_name=DPD state change, hidden=True @ 2019-05-14T22:55:42.905821+02:00>>

[scstraus]

Here's another one, I tried to take out the body but it still failed:

2019-05-15 01:52:08 DEBUG (MainThread) [homeassistant.core] Bus:Handling <Event call_service[L]: domain=variable, service=set_variable, service_data=variable=dpd_variable, value_template=delivery, attributes_template={
  "from": "REDACTED",
  "date": "Wed, 15 May 2019 01:51:24 +0200",
  "subject": "Fwd: DPD Kurýr: Dnes Vám doručíme balíček",
}>
2019-05-15 01:52:08 ERROR (MainThread) [homeassistant.core] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/config/custom_components/variable/__init__.py", line 179, in async_set_variable
    attributes = json.loads(attributes_template.async_render({ 'variable': current_state }))
  File "/usr/local/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.7/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 5 column 1 (char 137)
2019-05-15 01:52:08 DEBUG (MainThread) [homeassistant.core] Bus:Handling <Event state_changed[L]: entity_id=automation.dpd_state_change, old_state=<state automation.dpd_state_change=on; last_triggered=None, id=dpd_state_change, friendly_name=DPD state change, hidden=True @ 2019-05-15T01:50:17.586808+02:00>, new_state=<state automation.dpd_state_change=on; last_triggered=2019-05-15T01:52:08.949595+02:00, id=dpd_state_change, friendly_name=DPD state change, hidden=True @ 2019-05-15T01:50:17.586808+02:00>>

[scstraus]

FYI here's my config (after I took out the body attribute setting)

- id: posta_state_change
  alias: 'Posta state change'
  hide_entity: true
  initial_state: 'true'
  trigger:
    platform: state
    entity_id: sensor.cp_packages_coming_today
  condition:
    condition: state
    entity_id: 'sensor.cp_packages_coming_today'
    state: 'delivery'
  action:
    service: variable.set_variable
    data_template:
      variable: posta_variable
      value_template: "{{ states('sensor.cp_packages_coming_today')}}"
      attributes_template: >
        {
          "from": "{{ state_attr('sensor.cp_packages_coming_today', 'from') }}",
          "date": "{{ state_attr('sensor.cp_packages_coming_today', 'date') }}",
          "subject": "{{ state_attr('sensor.cp_packages_coming_today', 'subject') }}",
        } 

#          "body": "{{ state_attr('sensor.cp_packages_coming_today', 'body') }}"
        
- id: dpd_state_change
  alias: 'DPD state change'
  hide_entity: true
  initial_state: 'true'
  trigger:
    platform: state
    entity_id: sensor.dpd_packages_coming_today
  condition:
    condition: state
    entity_id: 'sensor.dpd_packages_coming_today'
    state: 'delivery'
  action:
    service: variable.set_variable
    data_template:
      variable: dpd_variable
      value_template: "{{ states('sensor.dpd_packages_coming_today')}}"
      attributes_template: >
        {
          "from": "{{ state_attr('sensor.dpd_packages_coming_today', 'from') }}",
          "date": "{{ state_attr('sensor.dpd_packages_coming_today', 'date') }}",
          "subject": "{{ state_attr('sensor.dpd_packages_coming_today', 'subject') }}",
        }   
 
#          "body": "{{ state_attr('sensor.dpd_packages_coming_today', 'body') }}"

- id: dpd_variable_reset
  alias: 'DPD Variable Reset'
  hide_entity: true
  initial_state: 'true'
  trigger:
    platform: time
    at: '00:00:01'
  action:
    service: variable.set_variable
    data:
      variable: dpd_variable
      value: no_delivery
      attributes:
        from: ''
        date: ''
        subject: ''
        body: ''
        icon: mdi:delivery

- id: dhl_state_change
  alias: 'DHL state change'
  hide_entity: true
  initial_state: 'true'
  trigger:
    platform: state
    entity_id: sensor.dhl_packages_coming_today
  condition:
    condition: state
    entity_id: 'sensor.dhl_packages_coming_today'
    state: 'delivery'
  action:
    service: variable.set_variable
    data_template:
      variable: dhl_variable
      value_template: "{{ states('sensor.dhl_packages_coming_today')}}"
      attributes_template: >
        {
          "from": "{{ state_attr('sensor.dhl_packages_coming_today', 'from') }}",
          "date": "{{ state_attr('sensor.dhl_packages_coming_today', 'date') }}",
          "subject": "{{ state_attr('sensor.dhl_packages_coming_today', 'subject') }}",
        }   

[scstraus]

So, I've changed the configuration a bit which worked to make the template for the main value of the sensor work when I take out the attributes like this:

- id: posta_state_change
  alias: 'Posta state change'
  hide_entity: true
  initial_state: 'true'
  trigger:
    platform: state
    entity_id: sensor.cp_packages_coming_today
  condition:
    condition: state
    entity_id: 'sensor.cp_packages_coming_today'
    state: 'delivery'
  action:
    service: variable.set_variable
    data:
      variable: posta_variable
    data_template:
      value: >
          {{ states('sensor.cp_packages_coming_today')}}

But when I put the attributes back in like this:

- id: posta_state_change
  alias: 'Posta state change'
  hide_entity: true
  initial_state: 'true'
  trigger:
    platform: state
    entity_id: sensor.cp_packages_coming_today
  condition:
    condition: state
    entity_id: 'sensor.cp_packages_coming_today'
    state: 'delivery'
  action:
    service: variable.set_variable
    data:
      variable: posta_variable
      attributes_template: >
        {
          "from": "{{ state_attr('sensor.cp_packages_coming_today', 'from') }}",
          "date": "{{ state_attr('sensor.cp_packages_coming_today', 'date') }}",
          "subject": "{{ state_attr('sensor.cp_packages_coming_today', 'subject') }}",
          "body": "{{ state_attr('sensor.cp_packages_coming_today', 'body') }}"
        }
    data_template:
      value: >
          {{ states('sensor.cp_packages_coming_today')}}

I get this again:

2019-05-15 23:47:16 ERROR (MainThread) [homeassistant.core] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/config/custom_components/variable/__init__.py", line 179, in async_set_variable
    attributes = json.loads(attributes_template.async_render({ 'variable': current_state }))
  File "/usr/local/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.7/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Invalid control character at: line 5 column 50 (char 188)

[scstraus]

Okay, so the issue is with the size of the email body.. I was able to make it work with the below syntax, but as soon as I add the body on, it crashes just like above.

- id: posta_state_change
  alias: 'Posta state change'
  hide_entity: true
  initial_state: 'true'
  trigger:
    platform: state
    entity_id: sensor.cp_packages_coming_today
  condition:
    condition: state
    entity_id: 'sensor.cp_packages_coming_today'
    state: 'Delivery'
  action:
    service: variable.set_variable
    data:
      variable: posta_variable
      attributes_template: >
        {
          "from": "{{ state_attr('sensor.cp_packages_coming_today', 'from') }}",
          "date": "{{ state_attr('sensor.cp_packages_coming_today', 'date') }}",
          "subject": "{{ state_attr('sensor.cp_packages_coming_today', 'subject') }}"
        }
    data_template:
      value: >
          {{ states('sensor.cp_packages_coming_today')}}

[scstraus]

Also I found a related SQL error.. sensor.dpd_packages_coming_today is the exact same sensor, just with different email content for a different shipping provider.

Error executing query: (MySQLdb._exceptions.DataError) (1406, "Data too long for column 'event_data' at row 1") [SQL: 'INSERT INTO events (event_type, event_data, origin, time_fired, created, context_id, context_user_id) VALUES (%s, %s, %s, %s, %s, %s, %s)'] [parameters: ('state_changed', '{"entity_id": "sensor.dpd_packages_coming_today", "old_state": {"entity_id": "sensor.dpd_packages_coming_today", "state": "Delivery", "attributes": { ... (72419 characters truncated) ... :00", "last_updated": "2019-05-31T06:45:12.660235+00:00", "context": {"id": "3daf7b0e44124eb59eb2f765fa277ae7", "parent_id": null, "user_id": null}}}', 'LOCAL', datetime.datetime(2019, 5, 31, 6, 45, 12, 660283, tzinfo=), datetime.datetime(2019, 5, 31, 6, 45, 12, 670672), '3daf7b0e44124eb59eb2f765fa277ae7', None)] (Background on this error at: http://sqlalche.me/e/9h9h)