Use workflows for custom operation handlers

[rina23q]

Is your feature improvement request related to a problem? Please describe.
The custom operation is currently limited to SmartREST input and main device. Since we have workflow feature, mapper can focus on generating a thin-edge command, and move the actual execution of the operation to tedge-agent.

Describe the solution you'd like
Mapper provides a way for users how to map the c8y JSON to thin-edge JSON.

Let's use this custom operation com_cumulocity_model_WebCamDevice as an example.

C8Y JSON input

All operations including custom operations will be delivered to the JSON over MQTT topic c8y/devicecontrol/notifications.

{
   "delivery":{
      "log":[
         
      ],
      "time":"2024-09-02T19:56:01.342Z",
      "status":"PENDING"
   },
   "agentId":"201802315",
   "creationTime":"2024-09-02T19:56:01.313Z",
   "deviceId":"201802315",
   "id":"1800380",
   "status":"PENDING",
   "com_cumulocity_model_WebCamDevice":{
      "name":"take picture",
      "parameters":{
         "duration":"5s",
         "quality":"HD"
      }
   },
   "description":"WebCam picture",
   "externalSource":{
      "externalId":"TST_haul_searing_set",
      "type":"c8y_Serial"
   }
}

Drop 1

File: /etc/tedge/operations/c8y/com_cumulocity_model_WebCamDevice (for main device)

The mapper spawns a child process (= same mechanism as the existing custom operation handler) from the JSON over MQTT input.
The filename is used for declaring supported operation. The keyon_fragment is used to check if the mapping file matches the input payload. If topic is not provided, the device control topic is used by default unless on_message is defined. skip_status_update is optional. If true, mapper doesn't send operation status update messages (501-503/504-506).

[exec]
topic = "c8y/devicecontrol/notifications"
on_fragment = "com_cumulocity_model_WebCamDevice"
command = "/usr/bin/do_something.sh ${.payload.com_cumulocity_model_WebCamDevice.parameters.duration}"
skip_status_update = true

FYI: existing custom operation handler definition

[exec]
topic = "c8y/s/ds"
on_message = "511"
command = "/usr/bin/c8y-command"

Drop 2

Mapping definition by user

The mapper converts the incoming JSON over MQTT message to thin-edge command message as per the definition file.

User is supposed to create a master custom operation definition file with .template extension. e.g. /etc/tedge/operations/c8y/com_cumulocity_model_WebCamDevice.template

[exec]
topic = "c8y/devicecontrol/notifications"
on_fragment = "com_cumulocity_model_WebCamDevice"
command_name = "take_picture" # KEY NAME TBD, as "command" is already taken for executable path
command_payload = "${.payload.com_cumulocity_model_WebCamDevice.parameters}"

To indicate that devices are supporting the operation, the device must publish a command capability message.

tedge mqtt pub -r 'te/device/<name>///cmd/take_picture' '{}'

On receiving, mapper creates a simlink in /etc/tedge/operations/c8y/com_cumulocity_model_WebCamDevice (main), /etc/tedge/operations/c8y/<external_id>/com_cumulocity_model_WebCamDevice (child/service) to the master file.

Mapper detects the change in the /etc/tedge/operations/c8y directory except .template file. When the change is detected, 114 (supported operation) message must be sent to c8y.

Convert to thin-edge command

Topic: te/device/<name>///cmd/take_picture/c8y-mapper-1800380

{
      "status": "init", // Must be added by mapper
      "duration":"5s",
      "quality":"HD"
}

Status update to C8Y

Topic: te/device/<name>///cmd/take_picture/c8y-mapper-1800380

{
      "status": "executing",
      "duration":"5s",
      "quality":"HD"
}

Mapper converts it to SmartREST 504 or 501 for executing status.

504,1800380 
501,com_cumulocity_model_WebCamDevice

The same pattern is applied to successful/failed message, 502/504, and 503/506.

Describe alternatives you've considered
Another option is using command metadata message to define mapping.

tedge mqtt pub -r te/device/<name>///cmd/take_picture`{
   "exec": {
      "topic": "c8y/devicecontrol/notifications"
      "on_fragment": "com_cumulocity_model_WebCamDevice"
      "command_name": "take_picture" # KEY NAME TBD, as "command" is already taken for executable path
      "command_payload": "${.payload.com_cumulocity_model_WebCamDevice.parameters}"
    }
}`

Additional context
The original roadmap ticket is here.

[albinsuresh]

I'd be in favour of defining the mapping rules at the existing location for C8y itself (e.g: /etc/tedge/operations/c8y/com_cumulocity_model_WebCamDevice) so that it is consistent with the existing SmartREST-based custom operation defintions. Proposing the c8y directory itself since the mapping rules defined in it are for Cumulocity itself and not for any other cloud. Just use another toml map name to differentiate this from the existing [exec] blocks.

One other key aspect is the payload manipulation mechanisms available. The proposed sample showcases a simple example where a sub-fragments from the input is passed as-is to the command. But, additional manipulations might be required for more complex operations and hence keep the config format open so that we can accept not just JSON paths but more complex JSON transformation expressions or even an input_script in future.

[albinsuresh]

The proposal for the mapping rules and the file format looks fine. Since the mapping rules defined in the operation file applies to both main and child devices, all devices should be able to re-use the same. And we need to provide a better way for the devices (esp child devices) to declare that they support the given custom operation and reuse the same mapping rules for the same.

In the current proposal, it is not clear to me as to who creates these operation files and when. As far as I understood, the current proposal expects the user to manually create these operation definition files under /etc/tedge/operations/c8y directory and subdirectories for the main device and the expected child devices. Creating those files manually on the main device won't be practical when child devices are connected dynamically. So, we need a mechanism where child devices can declare that they support take_picture command (e.g: empty metadata message published to te/device/<device-id>///cmd/take_picture), and then the mapper should create the appropriate operation file for that device under the /etc/tedge/operations/c8y directory.

So, I'd propose that we define that operation definition file in some c8y-specific but device-neutral location once during the device bootstrapping phase (not directly under /etc/tedge/operations/c8y though, as that implies the "main" device). And then the mapper to create symlinks to it from the device specific directories under /etc/tedge/operations/c8y when a device declares that command capability message.

[rina23q]

Drop 1 is done by #3144