[RAC] Update alert documents in lifecycle rule type helper

[dgieselaar]

Closes #100451.

Summary

Create event.kind:signal alert documents, and update those when an alert is/stays active, in addition to the append-only evaluations (which are now event.kind:event). Usage of collapse is also removed for the table/APM app views.

Other changes:

• catch a no_matching_indices exception for the getDynamicIndexPattern function
• make sure an index doesn't accidentally get created when there is no corresponding index template

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[smith]

Works on my machine!

[smith]

This is clever, but it would be cleverer with an explanation of what's going on and a unit test.

[dgieselaar]

OKAY OKAY I GET IT.

[jasonrhodes]

Running this locally, I get this error in the transaction error rate rule type (it's being thrown from the ruleRegistry lifecycle rule type factory):

"action [indices:admin/create] is unauthorized for user [kibana_system_user] with roles [kibana_system] on indices [.alerts-observability-apm-000001], this action is granted by the index privileges [create_index,manage,all]"

Full Error JSON

{
  "reason": "execute",
  "error": {
    "name": "ResponseError",
    "meta": {
      "body": {
        "error": {
          "root_cause": [
            {
              "type": "security_exception",
              "reason": "action [indices:admin/create] is unauthorized for user [kibana_system_user] with roles [kibana_system] on indices [.alerts-observability-apm-000001], this action is granted by the index privileges [create_index,manage,all]"
            }
          ],
          "type": "security_exception",
          "reason": "action [indices:admin/create] is unauthorized for user [kibana_system_user] with roles [kibana_system] on indices [.alerts-observability-apm-000001], this action is granted by the index privileges [create_index,manage,all]"
        },
        "status": 403
      },
      "statusCode": 403,
      "headers": {
        "content-length": "565",
        "content-type": "application/json;charset=utf-8",
        "x-cloud-request-id": "b7SkxMHtRCmqtrHj78PWfQ",
        "x-found-handling-cluster": "b5caf8c576704714a9bb2559bddab987",
        "x-found-handling-instance": "instance-0000000040",
        "date": "Wed, 09 Jun 2021 23:05:39 GMT"
      },
      "meta": {
        "context": null,
        "request": {
          "params": {
            "method": "PUT",
            "path": "/.alerts-observability-apm-000001",
            "body": "{\"aliases\":{\".alerts-observability-apm\":{\"is_write_index\":true}}}",
            "querystring": "",
            "headers": {
              "user-agent": "elasticsearch-js/8.0.0-canary.4 (darwin 18.7.0-x64; Node.js v14.17.0)",
              "x-elastic-product-origin": "kibana",
              "x-elastic-client-meta": "es=8.0.0p,js=14.17.0,t=8.0.0p,hc=14.17.0",
              "content-type": "application/json",
              "content-length": "65"
            },
            "timeout": 30000
          },
          "options": {},
          "id": 309
        },
        "name": "elasticsearch-js",
        "connection": {
          "url": "https://b5caf8c576704714a9bb2559bddab987.us-east-1.aws.staging.foundit.no:9243/",
          "id": "https://b5caf8c576704714a9bb2559bddab987.us-east-1.aws.staging.foundit.no:9243/",
          "headers": {},
          "deadCount": 0,
          "resurrectTimeout": 0,
          "_openRequests": 0,
          "status": "alive",
          "roles": {
            "master": true,
            "data": true,
            "ingest": true,
            "ml": false
          }
        },
        "attempts": 0,
        "aborted": false
      }
    }
  }
}

Currently, Kibana can't parse this error accurately enough to expose the underlying error message, so it just prints:

server    log   [19:01:53.220] [error][alerting][alerting][plugins][plugins] Executing Alert "a57fda90-c96d-11eb-97f6-c3426f999277" has resulted in Error: security_exception, caused by: ""

[dgieselaar]

@jasonrhodes what version of ES are you on? The error you are seeing is supposed to be handled by the ES permission changes in elastic/elasticsearch#72181.

[jasonrhodes]

I'm pointed at the Edge cluster.

[jasonrhodes]

No need to block these changes if we feel confident they're fixed. I'll keep playing with it after the merge.

[dgieselaar]

@jasonrhodes at the time of writing, the edge cluster's ES snapshot is from 26th of April, which predates the privilege changes.

[jasonrhodes]

OK well then let's merge this and I'll test against the Logs and Metrics rule stuff we're currently working on.

At another time, I need a refresher on how to test APM data locally when Edge isn't working, it's been a long time since I've run apm-integration-testing :)

[dgieselaar]

@elasticmachine merge upstream

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: de78d52
• Storybooks Preview

Metrics [docs]

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
ruleRegistry	6	7	+1

History

• 💔 Build #131676 failed ebfeb31
• 💔 Build #131620 failed daab8d4
• 💚 Build #129797 succeeded 7bd4a7d

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[kibanamachine]

💚 Backport successful

Status	Branch	Result
✅	7.x

This backport PR will be merged automatically after passing CI.