Change MQTT Topic structure

[tbnobody]

Due to various shortcomings of the current MQTT Topic structure, I would like to rebuild it.
But since I can't think of all eventualities I wanted to start a discussion here which changes should be included and which structure might not be so optimal.

• Somehow it is required to add the sum of all inverters: Included DTU totals in MqTT publishing #305
• Also it seems to be a requirement to run multiple instances of OpenDTU which write to the same base topic: BREAKING CHANGE: add dtu serial to mqtt status topics #304
• I also read and heard some concerns regarding the structure when writing the values into influx using telegraf
• Currently the topic structure is also limited to just one phase (because it gets channel 0). This is not sufficient for e.g. HMS inverters which might get support in future

When looking at topic 1 compared to topic two, it would only be possible to add the sum values of all inverters which are currently connected to the same ESP.

In the python version of @Sprinterfreak used the following structure:

<basetopic>/<inverter serial>/emeter/<phase id>/<property>
<basetopic>/<inverter serial>/emeter-dc/<string id>/<property>

it needs to be extended by commands, status per inverter and the dtu internal topics like LWT and a topic for e.g. total power of all connected inverters.

I like to make this change only once as it makes all current installations incompatible. Therefor we should first find a structure which suits a lot of people. (It will never suit everybody)

[Setcover]

I don't know what's best but i'd like to present my current MQTT setup and what my thoughts behind it. My preference would be to have system info and pv info separated (comparable to tasmota). Having a separate system-info topic could be implemented in one go together with #304

My current MQTT setup is:

• uptime, ip, hostname, rssi -> gone, has nothing to do with pv performance. If I would need it again I would put it in a separate system-info topic.
• subtopic = inv->name() // not serial. Why? Humanreadable. I need to know what is going on, my raspberry pi doesn't care if the inverter is identified using its serial, name or zodiac sign.
• bootloaderversion, fwbuildversion, fwbuilddatetime, hwpartnumber, hwversion -> gone, same reason as above. In addition should be tramsmittet once + retain, as it does not change all the time there is no need to retransmit it all the time.
• limit_relative, limit_absolute, reachable, producing, timestamp -> without "/status" in front as it is "global" for the inverter and not a channel (subtopic).

the resulting structure is:

<basetopic>/<inverter name>/inverterinformation
<basetopic>/<inverter name>/0/electric_output_information
<basetopic>/<inverter name>/1/electric_input_information
<basetopic>/<inverter name>/2/electric_input_information

My corresponding telegraf config is

[[inputs.mqtt_consumer]]
  name_override = "openDTU"
  servers = ["tcp://mosquitto:1883"]
  topics = ["solar/#"]
  data_format = "value"
  data_type = "float"
  tagexclude = ["host","topic"]
  [[inputs.mqtt_consumer.topic_parsing]]
    topic = "solar/+/+/+"
    tags = "_/name/channel/field"
  [[inputs.mqtt_consumer.topic_parsing]]
    topic = "solar/+/+"
    tags = "_/name/field"
  [[processors.pivot]]
  tag_key = "field"
  value_key = "value"

Grafana visualisation of all power values (input and ouput for all inverters) can be achieved with a single query SELECT mean("power") FROM "openDTU" WHERE $timeFilter GROUP BY time($__interval), "channel", "name" fill(null) and ALIAS $tag_name $tag_channel

I do understand the possible benefits of having every value in a separate topic but in the end i'd prefer a single json message. I don't know if there are impartial reaosns for that or it's just my personal preference.

Regarding "sum of all" discussion: If one needs the sum of all inverters connected to a single openDTU, one would likely need the sum over all openDTU too? Should a "master" esp32 calculate this sum or rather a simple script running on e.g. a raspberry pi?

[Sprinterfreak]

I tried to adopt as much as possible from shellies mqtt layout into my hoymiles python module. The emeter part specifically.
This allowed me to unify Grafana power graphs with shellies [1].

Do not use inverter name in topic's. This would require a 💩 load of sanitization.

<basetopic>/<inverter serial>/name

My previous implementation also prepared for multi input and output phases per inverter and also allows for individual panel aka. "string" naming

<basetopic>/<inverter serial>/name
<basetopic>/<inverter serial>/temperature
<basetopic>/<inverter serial>/uptime
<basetopic>/<inverter serial>/rssi

<basetopic>/<inverter serial>/string/<0-N>/name
<basetopic>/<inverter serial>/string/<0-N>/voltage
<basetopic>/<inverter serial>/string/<0-N>/current

<basetopic>/<inverter serial>/frequency
<basetopic>/<inverter serial>/phase/<0-N>/name
<basetopic>/<inverter serial>/phase/<0-N>/voltage
<basetopic>/<inverter serial>/phase/<0-N>/current
<basetopic>/<inverter serial>/phase/<0-N>/pf

Should we name phases? Maybe. Could be for instance "1F4/1"

I would definitively vote in favor for including the dtu serial into basetopic, as it would enable multi-homed setups to use a central broker. Of course to be able to name dtu's, we would need to put the inverter objects one step further down like so

<basetopic>/dtu serial/name
<basetopic>/dtu serial/uptime
<basetopic>/dtu serial/inverters/<inverter serial>/#

I use flux to query my data in dashboards like

// v.windowPeriod is a variable referring to the current optimized window period (currently: $interval)
from(bucket: "${rp}")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => (
    r["_measurement"] == "${m}" and exists r.string and
    r["location"] =~ /${location:regex}/ and
    r["_field"] == "power"))
  |> group(columns: ["location","string"])
  |> aggregateWindow(every: v.windowPeriod, fn: mean)
  |> yield(name: "mean")

location corresponds to inverter serial. :regex is used to enable multi-select.

[1] https://shelly-api-docs.shelly.cloud/gen1/#shelly-3em-overview

[madmartin]

My 50cent:

• I propose sending systeminfo messages and all the power values in two different sets with two different, separately configurable publish timer intervals.
• The systeminfo messages can be send in a long interval, e.g. every 600 seconds, while the power values can be send e.g. every 30 seconds.
• Sending systeminfo only once and retain (means storing it on the mqtt server forever) is nothing I would prefer.
• I agree to "Do not use inverter name in topic's. This would require a hankey load of sanitization." The current coding as <basetopic>/<inverter serial>/name is IMHO fine.
• I vote for sending the information in two ways, one way as multiple mqtt messages like it's currently done, other way under one topic as json formatted single message. Configurable as on/off for each method. This should not be too hard to implement, and everyone can meet his own taste.
• the json formatted livedata is created for the web api anyway, maybe the code can be shared for json-formatted livedata via mqtt
• the current configurable LWT topic name and online/offline message makes no sense for me. To be clear, LWT topic etc. is fine, I see no sense in having it configurable.
• I also vote for inserting the dtu serial into systeminfo topics like in PR BREAKING CHANGE: add dtu serial to mqtt status topics #304 . It should be possible to operate more than one openDTU in the network.
• About publishing the Sum of all Inverters: yes why not. If in doubt, make it configurable.
• But only for directly configured inverters, a solution that sums up over several openDTUs is IMHO beyond the scope of openDTU itself and should be done somewhere else if necessary.

[tbnobody]

After several discussions and also tests with telegraf and how to subscribe to all important messages as easy as possible we came up with the following structure:

• [basetopic]
  • [dtuserial]
    • status
    • uptime
    • rssi
    • bssid
    • ip
    • hostname
    • git_hash
    • day_mode
    • ac
      • is_valid
      • power
      • yieldtotal
      • yieldday
    • dc
      • is_valid
      • power
      • irradiation
    • inverters
      • [inverter serial]
        • name
        • string
          • [0-N]
            • name
            • voltage
            • current
            • power
            • maxpower
            • yieldday
            • yieldtotal
            • irradiation
        • dc
          • power
          • current
          • irradiation
        • phase
          • [0-N]
            • name
            • voltage
            • current
            • power
            • yieldday
            • yieldtotal
            • reactivepower
            • powerfactor
        • ac
          • current
          • power
          • yieldday
          • yieldtotal
          • frequency
        • temperature
        • uptime
        • status
        • producing
        • limit_relative
        • limit_absolute
        • last_update
        • devinfo
          • bootloaderversion
          • fwbuildversion
          • fwbuilddatetime
          • hwpartnumber
          • hwversion
          • maxpower
        • cmd
          • limit_persistent_relative
          • limit_persistent_absolute
          • limit_nonpersistent_relative
          • limit_nonpersistent_absolute
          • power
          • restart

If you like e.g. receive to all dc voltages, just subscribe to basetopic/+/inverters/+/string/+/voltage.

[roastedelectrons]

As mentioned by @Sprinterfreak you can just subscribe to [basetopic]/+/status to receive all dtus

This only works if the basetopic is known - but since it is freely configurable, I have to assume it is unknown. And just looking for /status will deliver many non-opendtu devices. So if there is a topic wich includes 'opendtu' it would be much easier for auto-dicovery.

[tbnobody]

This only works if the basetopic is known - but since it is freely configurable, I have to assume it is unknown.

But the devices are under your control. Therefor it's your responsibility to set the right base topic or make the base topic in your further software configurable to match your base topic.

[roastedelectrons]

But the devices are under your control. Therefor it's your responsibility to set the right base topic or make the base topic in your further software configurable to match your base topic.

Of course, everything can also be configured manually by the user. But for the end user it would be more convenient if the configuration could still be done automatically by my implementation.

By the way, thanks for your great work.

[avandorp]

I like the structure. I'm just not sure about the inverter serial id. It is a somewhat sensitive value as it allows control over the inverter (when in radio distance). Is it a good idea to send it per default?

[Sprinterfreak]

@avandorp The serial is on the air anyway. In any response from the inverter. You can't do anything about that.
I'ts Your responsibility to secure access to your broker and the *DTU. Since smarthome is a very private and sensitive perimeter, there is no need to hide it internally. The id will stay, there won't be any lookup table to fake it.
@roastedelectrons The basetopic's default is hardcoded to be "opendtu". So you know it unless you changed it yourself and forgot.

[sapiency]

hmm, some questions:

• Is the inverter ID/Number unique over manufactures/types? How do you want to take account of not unique IDs in a centralized database? I suggest to use the ID as default value, but let it open to modify it.
• What is the idea behind saving the database transaction unit (DTU) information? What happens if a DTU is replaced?
• I do not see that you consider of how many panels are connected to a inverter ... ( to distinguish between shadowing, damaged panels, ...)
• Where can I find the list with the abbreviations you have used?

What I've got from the view days I'm running my openDTU with an HM-600:

• entropy of uptime of the DTU: very low during inverter is off, high if uptime is 0 when inverter is "going" online
• entropy of switching on the inverter: high
• entropy of connection loss to inverter: very low during night, irradiationbecause inverter is off, no communication
• It's maybe nice to know the RSSI value you get from the DTU, but can you also get it from the inverter? Or do you expect that all DTU's run with max power level for sending?

Related data:

• inverters.(?.)uptime entropy ? ( openDTU shows me when my inverter switched on, I think the openDTU assumes that the inverter is offline if there is no connection for some time)

btw. it is very difficult to identify the level of any data value if you use the same symbol for all levels. And maybe add some comments / details to the data and explain where you expect to get it from.
BR

[tbnobody]

Is the inverter ID/Number unique over manufactures/types? How do you want to take account of not unique IDs in a centralized

I dont understand the question. this number flashed into the inverters firmware. there is no way to change it.

What is the idea behind saving the database transaction unit (DTU) information? What happens if a DTU is replaced?

It's not database transaction unit. Its Data Transfer Unit. Did you even read the support documents of your inverter?

I do not see that you consider of how many panels are connected to a inverter ... ( to distinguish between shadowing, damaged panels, ...)

Why should a gateway bridge consider connected panels?

Where can I find the list with the abbreviations you have used?

Which one?

entropy of uptime of the DTU: very

What do you mean with entropy?

[sapiency]

danke für die Rückfrage,
ID nicht veränderbar: das ist mir klar, nur wenn man im Ziel eine eindeutige ID erwartet, ist es ungeschickt, wenn diese der Erwartung nicht entspricht. Das ist nur relevant, wenn im Ziel Daten aus mehreren Quellen gesammelt werden, z.B. Community Projekt

Support documents: nur bis zu dem Punkt, der für die Inbetriebnahme erforderlich war, für mehr habe ich mir noch keine Zeit genommen.

Panels: Ich habe aktuell das selbst verschuldete Problem, dass die Panels teilweise abgeschattet werden. Aus der Summe kann ich nur entnehmen, dass es evtl. nicht optimal läuft. Bei einer Vergleichsmöglichkeit der Panels über die Zeit, hätte ich möglicherweise einen Mehrwert.

Abkürzung: DTU, LWT .. und was sonst noch so kommen mag ....

Entropy: Angelehnt an Shannon ..
Meiner Meinung nach können Daten, die in der Quelle berechnet werden und deren Übertragung und Speicherung zusätzlich Aufwand verursachen, auch durchaus im Zielsystem berechnet werden. Ein sparsamer Umgang mit Resourcen ist auch dann angebracht, wenn eine Resource auf den ersten Blick nur wenig in Anpruch genommen wird. Und wenn es nur darum geht mit gutem Beispiel voran zu schreiten.

Grüße und herzlichen Dank für Ihre Mühe.

[avandorp]

@sapiency Your inverter doesn't know about the panels. It knows about its strings (which are in the proposed structure). If you have one panel per string (quite common with the smaller hoymiles inverters) you can map that data to your individual panels. Otherwise you will always only have the combined data of all the panels at one string. Hoymiles has additional monitoring tools (e.g. HRSD Plus-1C) which allow gathering data per panel without needing more inverters/strings; no idea, whether OpenDTU could read them out of the box. I have never seen one out in the wild.

[sapiency]

You are absolutely right, I did not thought about more panels per string. Some time ago I saw a video on heise about self installed solar system. They choosed small inverters (2 x 6) to take care of this problem. That's the main reason why I thought it is helpful to have the details from the different strings (panels).

[Sprinterfreak]

@sapiency
DTU serial: if You change your DTU, the mqtt path must change. This also ensures a relation for absolute addressing between inverter and DTU, enabling to filter by location for instance.
Panels: The inverter does not provide any additional data for multiple panels per string. So OpenDTU can't provide a special case here besides storing a somewhat descriptive static name for the string. That's already planned. In any case it's good advice to put the extra panels onto an additional inverter to gain more accurate mppt.

[DcFamas]

Good day,
Thank you very much for this great software. I definitely wish for a "total" mqtt topic. Maybe you can stop by AhoyDTU, they have already implemented this feature.

Thanks a lot and greetings :)

[Sprinterfreak]

If you guys look closely, it is already in the initial proposal. This value can by design never be accurate. Thats a technical limitation. But it's in there anyways.

[basetopic]/[dtuserial]/ac/power

The only thing holding us back on the topic structure change is, that it forcibly breaks all existing setups entirely.

[Slummi]

Perhaps it would be good to introduce a kind of legacy switch, with which you can keep the old structure and thus no problems with existing configurations occur. The switch should be enabled after an firmware update and the user has to actively switch to the new structure (disable the switch). For new installations of OpenDTU the new structure could be used by default (switch disabled). I think otherwise you'll never be able to implement such breaking changes.

[tbnobody]

A compatibility switch will not be possible due to memory restrictions. And question is, when to remove the old structure? I will mark it bright and shiny in the release and publish it as anything else. Only question is when. Because currently I am working on other stuff.

[Slummi]

Ok, I had not expected that memory is already so exhausted. I hope it won't be too long until the new MQTT structure is implemented. Until then I use the web API to get the missing values.

[Sprinterfreak]

The most accurate way to get the total power at a given time would be and will ever be to sum up the total energy counters difference of all inverters for an interval greater or at least equal than a full poll cycle (all inverters). OpenDTU will never be able to do it that accurate I guess. That way You get an accurate average and it's immune against a statistical error due to taking the state at the wrong time in slope.

[smainz]

Is the structure depicted above agreed upon?

I have a tefegraf connector from the mqtt broker to influxdb set up at home and would be willing to provide:

• a telegraf configuration which takes the current structure and transforms it into the future structure.
• a grafana dashboard using the new structure

This would mean once the migration is done one would only have to change the telegraf config file and is ready to go with no further changes. At least for people starting with this config in the beginning.

Maybee @Kraego is willing to join forces, so we have only one mqtt-telegraf-influxdb-grafana setup

[Kraego]

@smainz of course I can adapt my setup, if you provide the telegraf mapping for the new structure. Maybe I (or someone else) could provide flux queries to migrate the data from the old buckets to the new buckets (new topic mapping).

[smainz]

Maybe someone wants to take a look at this https://github.com/smainz/OpenDTU-MQTT-Telegraf-influxdb-integration
It contains amapping from current topics to the new structure (as far as possible) and a telegraf config which tries to apply the mapping.

Does this go the direction it is envisioned here?

[stefan123t]

@tbnobody thanks for the effort to make this a public discussion. I am late to the party (almost a year tbh) and I would only like to make the request that AhoyDTU and OpenDTU agree on a common tree of topics.
As this agreement has already happened on OpenDTU side I would actually vote for AhoyDTU to adopt the same structure. Though there might be some minor deviations like DTU state, etc. this would allow third party integrations and dashboards but also documentation and zero-export control scripts to use the same, agreed upon commands and set of topics.
This would benefit both projects and our users!

[mimk97]

Hi,
Is it possible to send the string names via MQTT?
As entered in the inverter config...
i see the name will be transmitted.
But how can I set the entity to be called that?

[Sprinterfreak]

This is planned and will be implemented with the topic structure change. This will of course break all existing setups.