Watcher UI: A watch can be marked with an error status despite having executed fine

[spinscale]

Kibana version: 6.3.2
Elasticsearch version: 6.3.2

Describe the bug:

A watch action that has once failed in the past, can be forever marked as a failure/error in the UI, unless that error goes away. The reason for this is the fields that are being checked to mark sth as a failure.

Steps to reproduce:

Run these in the dev tools console

DELETE .watcher-history-*

DELETE _xpack/watcher/watch/my_watch

POST _xpack/watcher/watch/my_watch
{
  "trigger": {
    "schedule": {
      "interval": "10h"
    }
  },
  "input": {
    "simple": {}
  },
  "actions": {
    "send_email_to_admin": {
      "condition" : {
        "script" : "return ctx.payload.foo == true"
      },
      "email": {
        "to": "foo@example.org",
        "from": "foo@example.org",
        "subject": "subject",
        "body": {
          "text": "body"
        }
      }
    }
  }
}

GET _xpack/watcher/watch/my_watch

# action condition will execute
POST _xpack/watcher/watch/my_watch/_execute
{
  "record_execution": true,
  "alternative_input": {
    "foo": true
  }
}

# action condition will not execute
POST _xpack/watcher/watch/my_watch/_execute
{
  "record_execution": true,
  "alternative_input": {
    "foo": false
  },
  "action_modes": {
    "_all" : "force_execute"
  }
}

GET _xpack/watcher/watch/my_watch

To understand the issue, check the execute watch API calls. The first one will actually try to send an email, but fails because no email server is listening on the remote port. Thus the action is marked as successful: false.

The second watch execution is just not trying to send an email because the condition was not met. The last execution thus is now older than the last run of the watch, but still has the information that this particular execution has failed - which seems to mark the watch as error in the UI.

Expected behavior:

The watch should not be marked as an error in the UI, because the last run was just as expected. And the entry that seems to be checked seems to be older.

Screenshots (if relevant):

When checking the watcher UI

when directly checking the watch history

Note: We can also talk about fixing the status data structure if you think there is a better way to solve this. Ping me for any discussion.

[bmcconaghy]

Thanks for opening @spinscale

[alisonelizabeth]

This seems related to #25134.

[nickofthyme]

@alisonelizabeth Do you know if there is a workaround to show the correct status?

Also is there any update on the status of this issue?

[alisonelizabeth]

@nickofthyme There is no workaround that I am aware of unfortunately.

Also, no updates right now, but myself or someone else on the ES UI team will try to spend some time and look into this.

[alisonelizabeth]

@nickofthyme I have opened a PR with a fix: #67952