[Monitoring] Migrate license expiration alert to Kibana alerting

[chrisronline]

Relates to #42960
Replaces #49219

This is the first PR of a set of PRs aimed to migrate the six watcher-based cluster alerts to use the Kibana alerting framework.

This PR focuses on the license expiration alert, as well as laying out the foundation of how the rest of the alerts will be migrated.

There are three main things we need to do to accomplish this:

1. Disable the (potentially) existing watcher-based cluster alerts
2. Create a UI that allows the user to create kibana-based cluster alerts.
3. Ensure UX parity between watcher-based and kibana-based cluster alerts.

For the first one, the ES team will be introducing an api that will allow us to do that from the UI (pending ticket creation). For testing this PR, let's assume those are disabled properly and we don't need to worry about that now.

For the second one, we want to slowly merge all of this work into master (to avoid one large PR) so we are going to add a constant that will be set to false until all of the alerts are ready. Then, the final PR will remove this constant and all of the new alerts will be available to users.

Finally, for the last one, we want to ensure the messaging is consistent (even though it might be somewhat poor) for this phase.

Screenshots

Testing

1. Flip this switch to true

We need to easily simulate the scenario in which this alert would fire. To do that, we'll leverage the monitoring ingest pipeline to set an expiration date in the near future. See these commands:

# Add the ingest pipeline
PUT _ingest/pipeline/force_license_expiration
{
  "processors": [
    {
      "script": {
        "lang": "painless",
        "source": """
          if (ctx.type == "cluster_stats") {
            ctx.license.expiry_date_in_millis = Instant.ofEpochMilli(new Date().getTime()).plusSeconds(60 * 60 * 24 * 3).getEpochSecond() * 1000;
          }
        """
      }
    }
  ]
}
# Ensure a local exporter is explictly configured so the default pipeline is established
PUT _cluster/settings
{
  "transient": {
    "xpack.monitoring.exporters": { 
      "local": {
        "type": "local",
        "use_ingest": false
      }
    }
  }
}
# Use the created pipeline
PUT .monitoring-es-7-*/_settings
{
  "index.required_pipeline": "force_license_expiration"
}

# This will unset the pipeline (and therefore resolve the alert)
PUT .monitoring-es-7-*/_settings
{
  "index.required_pipeline": null
}

TODO

• Once migrated, all stored alerts in .monitoring-alerts-* should be removed
• Only users with the right permissions should be able to interact with this UI, see [Monitoring] Only elevated permission users should see Setup Mode #50327
• Should we perform CCS queries when searching for monitoring data in the alert?
• Create follow up issues to remove the default_admin_email logic since we aren't using that in the Kibana alerting world
• Ensure alert is set to 1m interval

[chrisronline]

@elasticmachine merge upstream

[igoristic]

Looks good! Really excited about this feature 😃

[chrisronline]

@elasticmachine merge upstream

[tylersmalley]

Looks like there were changes required from the EUI bump in #56228 which were not here, so it failed once it hit master. I am pushing a fix.

[tylersmalley]

@chrisronline, can you confirm 0440ae5 is expected?

[chrisronline]

@tylersmalley Yup, looks expected

[chrisronline]

Backport:

7.x: 1601403

[kibanamachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request
• Commit: 6b82db8

History

• 💚 Build #24046 succeeded 6b82db8
• 💚 Build #23618 succeeded 411186d
• 💚 Build #23017 succeeded 636fed8
• 💔 Build #22867 failed 7ffb67b
• 💔 Build #22836 failed a107da3

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream