Add topic_template for mqtt.publish

[davidjb]

Proposed change

Adds ability to template the topic for the mqtt.publish service, by way of an additional option topic_template; this is implemented in the same vein as payload vs payload_template.

Being able to template the topic allows an MQTT endpoint to be dynamically specified, such as through state, input entities and any other source available in templates. As a worked example, this feature allows me to create an input_select with sensor entity IDs and then publish to MQTT to home/dynamic_entity_id/set; prior to this PR, the topic always needed to be hardcoded.

In terms of implementation, the schema is set to require one of either topic or topic_template and checks validity of rendered output coming from the topic_template to ensure it's suitable for MQTT. Tests have been added to test this and test the rendering of the topic.

Type of change

• Dependency upgrade
• Bugfix (non-breaking change which fixes an issue)
• New integration (thank you!)
• New feature (which adds functionality to an existing integration)
• Breaking change (fix/feature causing existing functionality to break)
• Code quality improvements to existing code or addition of tests

Additional information

• This PR fixes or closes issue: fixes #
• This PR is related to issue:
• Link to documentation pull request: Doc topic_template for mqtt.publish home-assistant.io#18663

Checklist

• The code change is tested and works locally.
• Local tests pass. Your PR cannot be merged unless tests pass
• There is no commented out code in this PR.
• I have followed the development checklist
• The code has been formatted using Black (black --fast homeassistant tests)
• Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

• Documentation added/updated for www.home-assistant.io

If the code communicates with devices, web services, or third-party tools:

• The manifest file has all fields filled out correctly.
  Updated and included derived files by running: python3 -m script.hassfest.
• New or updated dependencies have been added to requirements_all.txt.
  Updated by running python3 -m script.gen_requirements_all.
• Untested files have been added to .coveragerc.

The integration reached or maintains the following Integration Quality Scale:

• No score or internal
• 🥈 Silver
• 🥇 Gold
• 🏆 Platinum

To help with the load of incoming pull requests:

• I have reviewed two other open pull requests in this repository.

[probot-home-assistant]

Hey there @emontnemery, mind taking a look at this pull request as it has been labeled with an integration (mqtt) you are listed as a code owner for? Thanks!
_{^{(message by CodeOwnersMention)}}

[tdejneka]

Adds ability to template the topic for the mqtt.publish service, by way of an additional option topic_template
...
prior to this PR, the topic always needed to be hardcoded

?

Can you explain why a topic_template option is needed when both topic and payload currently support templating?

Here's a simple example I recently posted in the community forum.

- alias: 433 TempChannel Multiplexer
  trigger:
  - platform: mqtt
    topic: home/433
  action:
  - service: mqtt.publish
    data:
      topic: 'home/sensor{{ trigger.payload_json.channel }}'
      payload: '{{ trigger.payload }}'

When tested with the following payload:

{"time":"2021-07-28 21:27:42","protocol":113,"model":"AmbientWeather-WH31E","id":238,"channel":6,"battery_ok":1,"temperature_F":55.58,"humidity":44,"data":"7600000000","mic":"CRC","mod":"FSK","freq1":433.9768,"freq2":433.84298,"rssi":-0.13023,"snr":13.01577,"noise":-13.146}

it correctly re-publishes the payload to the topic home/sensor6.

Unless I have misunderstood the purpose of this PR, it seems to me there's no need for topic_template.

FWIW, payload_template should probably be officially deprecated in favor of payload just like service_template and data_template were superseded by service and data.

[davidjb]

@tdejneka This is a different context; templating is supported within automation actions, where the action's variables get rendered to include the trigger data, like in your example.

Where mqtt.publish is called directly, the service renders the payload_template itself. In my case, I'm calling it from the UI like so:

type: button
name: Set power on behavior
tap_action:
  action: call-service
  service: mqtt.publish
  service_data:
    topic_template: 'zigbee2mqtt/{{ states("input_select.light_entity_id") }}/set'  # Supported by this PR
    payload_template: '{{ states("sensor.hue_power_on_behavior_payload") }}'

and for this to work, I need both topic and payload to be templated.

FWIW, payload_template should probably be officially deprecated in favor of payload just like service_template and data_template were superseded by service and data.

I agree a similar cleanup would be good to see, just have to bear in mind breaking backwards compatibility as someone may have inadvertently have Jinja syntax in their MQTT service call.

Separately, I imagine the current situation around templating could lead to unexpected results if someone tries to use payload_template from within an automation since there are two layers of templating occurring at different times). Any other services or actions that also render templates internally would be affected as well so perhaps some additional documentation might be useful.

[emontnemery]

This looks good overall @davidjb, thanks! 👍
I left a few minor comments.

[emontnemery]

We need to catch homeassistant.exception.TemplateError too

[davidjb]

This looks good overall @davidjb, thanks! 👍
I left a few minor comments.

Thanks @emontnemery - all of your suggestions are merged + added the additional test + am catching both types of template error now. Let me know if anything else's needed to merge. Thanks for the review and feedback!

[emontnemery]

Thanks, @davidjb 👍

[epenet]

Are you aware of the test failures since this was merged in?

[epenet]

See https://github.com/home-assistant/core/runs/4226958170?check_suite_focus=true

Error: homeassistant/components/mqtt/__init__.py:227:39: F821 undefined name 'CONF_TOPIC'
Error: homeassistant/components/mqtt/__init__.py:228:48: F821 undefined name 'CONF_TOPIC'

Error: Process completed with exit code 1.