[ML] Handle Empty Partition Field Values in Single Metric Viewer

[darnautov]

Summary

Resolves #60286.

Added support for empty partition field values in Single metric viewer. In case the value is blank (empty string) represent it as "" (empty string) in UI controls.

[elasticmachine]

Pinging @elastic/ml-ui (:ml)

[peteharverson]

I don't think all the searches used in the Single Metric Viewer are using the empty partition field value. For example, here the 'context' chart swimlane is showing anomalies from all partitions (the red anomaly is from the US-West partition), and the main focus chart looks like it is showing the metric data from all partitions:

Compared to the Anomaly Explorer chart, which does look correct:

[walterra]

I wonder if "" is obvious enough for users? For us as developers it might be self-explanatory, but I wonder if we should add a text hint or something?

[darnautov]

I wonder if "" is obvious enough for users? For us as developers it might be self-explanatory, but I wonder if we should add a text hint or something?

Indeed @walterra, good point, I was thinking about "_blank" or something similar, but the user might be confused assuming it's a real value. Any suggestions are welcome cc @peteharverson @blaklaybul

[blaklaybul]

I think "" is as good of an option as any, but could lead to confusion since we'd need a way to differentiate between cases where the field doesn't exist and when the exists and is set to "". I'm wondering if we could stylize this text by making it red or italicized and indicating in the docs what this coveys.

[walterra]

Code LGTM

[alvarezmelissa87]

Code LGTM ⚡️

[peteharverson]

The main focus chart here still looks like it is plotting data for all partitions when the blank value is selected:

[peteharverson]

Is it possible to use that same "" (empty string) for the swimlane label and cell tooltip? Nice-to-have - can always be done in a follow-up.

[droberts195]

The commit that caps the time range for determination of field cardinality at 3 months LGTM

(I didn't look at the rest of the PR)

[kibanamachine]

💛 Build succeeded, but was flaky

• continuous-integration/kibana-ci/pull-request
• Commit: 59c0cef
• Flaky suites:
  • xpack-kibana-ciGroup1

---

Test Failures

Kibana Pipeline / kibana-xpack-agent / X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/tests/create_rules·ts.detection engine api security and spaces enabled create_rules creating rules should create a single Machine Learning rule

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

[00:00:00]       │
[00:00:00]         └-: detection engine api security and spaces enabled
[00:00:00]           └-> "before all" hook
[00:00:10]           └-: create_rules
[00:00:10]             └-> "before all" hook
[00:00:10]             └-: creating rules
[00:00:10]               └-> "before all" hook
[00:00:10]               └-> should create a single rule with a rule_id
[00:00:10]                 └-> "before each" hook: global before each
[00:00:10]                 └-> "before each" hook
[00:00:10]                   │ info [o.e.x.i.a.TransportPutLifecycleAction] [kibana-ci-immutable-centos-tests-xl-1585656228912762874] adding index lifecycle policy [.siem-signals-default]
[00:00:10]                   │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-centos-tests-xl-1585656228912762874] adding template [.siem-signals-default] for index patterns [.siem-signals-default-*]
[00:00:10]                   │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-centos-tests-xl-1585656228912762874] [.siem-signals-default-000001] creating index, cause [api], templates [.siem-signals-default], shards [1]/[1], mappings [_doc]
[00:00:10]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xl-1585656228912762874] moving index [.siem-signals-default-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [.siem-signals-default]
[00:00:10]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xl-1585656228912762874] moving index [.siem-signals-default-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] in policy [.siem-signals-default]
[00:00:11]                 │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xl-1585656228912762874] moving index [.siem-signals-default-000001] from [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-follow-shard-tasks"}] in policy [.siem-signals-default]
[00:00:15]                 └- ✓ pass  (4.3s) "detection engine api security and spaces enabled create_rules creating rules should create a single rule with a rule_id"
[00:00:15]               └-> "after each" hook
[00:00:15]                 │ info [o.e.c.m.MetaDataDeleteIndexService] [kibana-ci-immutable-centos-tests-xl-1585656228912762874] [.siem-signals-default-000001/xuYXKX9nTnad23KS1tz6XQ] deleting index
[00:00:15]                 │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-centos-tests-xl-1585656228912762874] removing template [.siem-signals-default]
[00:00:15]               └-> should create a single rule without an input index
[00:00:15]                 └-> "before each" hook: global before each
[00:00:15]                 └-> "before each" hook
[00:00:15]                   │ info [o.e.x.i.a.TransportPutLifecycleAction] [kibana-ci-immutable-centos-tests-xl-1585656228912762874] adding index lifecycle policy [.siem-signals-default]
[00:00:15]                   │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-centos-tests-xl-1585656228912762874] adding template [.siem-signals-default] for index patterns [.siem-signals-default-*]
[00:00:15]                   │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-centos-tests-xl-1585656228912762874] [.siem-signals-default-000001] creating index, cause [api], templates [.siem-signals-default], shards [1]/[1], mappings [_doc]
[00:00:15]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xl-1585656228912762874] moving index [.siem-signals-default-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [.siem-signals-default]
[00:00:15]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xl-1585656228912762874] moving index [.siem-signals-default-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] in policy [.siem-signals-default]
[00:00:15]                 │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xl-1585656228912762874] moving index [.siem-signals-default-000001] from [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-follow-shard-tasks"}] in policy [.siem-signals-default]
[00:00:20]                 │ info [o.e.x.s.a.AuthenticationService] [kibana-ci-immutable-centos-tests-xl-1585656228912762874] Authentication using apikey failed - api key has been invalidated
[00:00:20]                 │ proc [kibana]   log   [13:12:32.625] [error][plugins][siem] An error occurred during rule execution:
[00:00:20]                 │ proc [kibana] message: "[security_exception] missing authentication credentials for REST request [/apm-*-transaction*%2Cauditbeat-*%2Cendgame-*%2Cfilebeat-*%2Cpacketbeat-*%2Cwinlogbeat-*/_search?allow_no_indices=true&size=100&ignore_unavailable=true], with { header={ WWW-Authenticate={ 0="Bearer realm=\"security\"" & 1="ApiKey" & 2="Basic realm=\"security\" charset=\"UTF-8\"" } } }"
[00:00:20]                 │ proc [kibana] name: "Simple Rule Query"
[00:00:20]                 │ proc [kibana] id: "2c5cba21-0484-4feb-a7c0-312ad29cfda9"
[00:00:20]                 │ proc [kibana] rule id: "rule-1"
[00:00:20]                 │ proc [kibana] signals index: ".siem-signals-default"
[00:00:20]                 │ info [o.e.x.s.a.AuthenticationService] [kibana-ci-immutable-centos-tests-xl-1585656228912762874] Authentication using apikey failed - api key has been invalidated
[00:00:20]                 │ proc [kibana]   log   [13:12:32.632] [error][alerting][alerting][plugins][plugins] Executing Alert "2c5cba21-0484-4feb-a7c0-312ad29cfda9" has resulted in Error: [security_exception] missing authentication credentials for REST request [/_security/user/_has_privileges], with { header={ WWW-Authenticate={ 0="Bearer realm=\"security\"" & 1="ApiKey" & 2="Basic realm=\"security\" charset=\"UTF-8\"" } } }
[00:00:20]                 └- ✓ pass  (4.4s) "detection engine api security and spaces enabled create_rules creating rules should create a single rule without an input index"
[00:00:20]               └-> "after each" hook
[00:00:20]                 │ info [o.e.c.m.MetaDataDeleteIndexService] [kibana-ci-immutable-centos-tests-xl-1585656228912762874] [.siem-signals-default-000001/E6c0k3xfTWKaEgMLnHuRkA] deleting index
[00:00:20]                 │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-centos-tests-xl-1585656228912762874] removing template [.siem-signals-default]
[00:00:20]               └-> should create a single rule without a rule_id
[00:00:20]                 └-> "before each" hook: global before each
[00:00:20]                 └-> "before each" hook
[00:00:20]                   │ info [o.e.x.i.a.TransportPutLifecycleAction] [kibana-ci-immutable-centos-tests-xl-1585656228912762874] adding index lifecycle policy [.siem-signals-default]
[00:00:20]                   │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-centos-tests-xl-1585656228912762874] adding template [.siem-signals-default] for index patterns [.siem-signals-default-*]
[00:00:20]                   │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-centos-tests-xl-1585656228912762874] [.siem-signals-default-000001] creating index, cause [api], templates [.siem-signals-default], shards [1]/[1], mappings [_doc]
[00:00:20]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xl-1585656228912762874] moving index [.siem-signals-default-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [.siem-signals-default]
[00:00:20]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xl-1585656228912762874] moving index [.siem-signals-default-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] in policy [.siem-signals-default]
[00:00:20]                 │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xl-1585656228912762874] moving index [.siem-signals-default-000001] from [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-follow-shard-tasks"}] in policy [.siem-signals-default]
[00:00:26]                 │ info [o.e.x.s.a.AuthenticationService] [kibana-ci-immutable-centos-tests-xl-1585656228912762874] Authentication using apikey failed - api key has been invalidated
[00:00:26]                 │ proc [kibana]   log   [13:12:38.685] [error][plugins][siem] An error occurred during rule execution:
[00:00:26]                 │ proc [kibana] message: "[security_exception] missing authentication credentials for REST request [/_security/user/_has_privileges], with { header={ WWW-Authenticate={ 0="Bearer realm=\"security\"" & 1="ApiKey" & 2="Basic realm=\"security\" charset=\"UTF-8\"" } } }"
[00:00:26]                 │ proc [kibana] name: "Simple Rule Query"
[00:00:26]                 │ proc [kibana] id: "39eea052-4366-46f5-891d-2d5ea611d338"
[00:00:26]                 │ proc [kibana] rule id: "f6348a14-ae7b-406e-bab6-3a19d5063905"
[00:00:26]                 │ proc [kibana] signals index: ".siem-signals-default"
[00:00:26]                 │ info [o.e.x.s.a.AuthenticationService] [kibana-ci-immutable-centos-tests-xl-1585656228912762874] Authentication using apikey failed - api key has been invalidated
[00:00:26]                 │ proc [kibana]   log   [13:12:38.692] [error][alerting][alerting][plugins][plugins] Executing Alert "39eea052-4366-46f5-891d-2d5ea611d338" has resulted in Error: [security_exception] missing authentication credentials for REST request [/_security/user/_has_privileges], with { header={ WWW-Authenticate={ 0="Bearer realm=\"security\"" & 1="ApiKey" & 2="Basic realm=\"security\" charset=\"UTF-8\"" } } }
[00:00:27]                 └- ✓ pass  (6.5s) "detection engine api security and spaces enabled create_rules creating rules should create a single rule without a rule_id"
[00:00:27]               └-> "after each" hook
[00:00:27]                 │ info [o.e.c.m.MetaDataDeleteIndexService] [kibana-ci-immutable-centos-tests-xl-1585656228912762874] [.siem-signals-default-000001/8DoqJS_wRfipPJfLs7ossQ] deleting index
[00:00:27]                 │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-centos-tests-xl-1585656228912762874] removing template [.siem-signals-default]
[00:00:27]               └-> should create a single Machine Learning rule
[00:00:27]                 └-> "before each" hook: global before each
[00:00:27]                 └-> "before each" hook
[00:00:27]                   │ info [o.e.x.i.a.TransportPutLifecycleAction] [kibana-ci-immutable-centos-tests-xl-1585656228912762874] adding index lifecycle policy [.siem-signals-default]
[00:00:27]                   │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-centos-tests-xl-1585656228912762874] adding template [.siem-signals-default] for index patterns [.siem-signals-default-*]
[00:00:27]                   │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-centos-tests-xl-1585656228912762874] [.siem-signals-default-000001] creating index, cause [api], templates [.siem-signals-default], shards [1]/[1], mappings [_doc]
[00:00:27]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xl-1585656228912762874] moving index [.siem-signals-default-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [.siem-signals-default]
[00:00:27]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xl-1585656228912762874] moving index [.siem-signals-default-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] in policy [.siem-signals-default]
[00:00:27]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xl-1585656228912762874] moving index [.siem-signals-default-000001] from [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-follow-shard-tasks"}] in policy [.siem-signals-default]
[00:00:32]                 │ proc [kibana]   log   [13:12:44.836] [warning][plugins][siem] Machine learning job is not started:
[00:00:32]                 │ proc [kibana] job id: "some_job_id"
[00:00:32]                 │ proc [kibana] job status: "undefined"
[00:00:32]                 │ proc [kibana] datafeed status: "undefined"
[00:00:32]                 │ proc [kibana] name: "Simple ML Rule"
[00:00:32]                 │ proc [kibana] id: "af842728-52c1-419b-980f-decdfce1ff75"
[00:00:32]                 │ proc [kibana] rule id: "rule-1"
[00:00:32]                 │ proc [kibana] signals index: ".siem-signals-default"
[00:00:34]                 └- ✖ fail: "detection engine api security and spaces enabled create_rules creating rules should create a single Machine Learning rule"
[00:00:34]                 │

Stack Trace

{ Error: expected { created_by: 'elastic',
  description: 'Simple Machine Learning Rule',
  enabled: true,
  false_positives: [],
  from: 'now-6m',
  immutable: false,
  interval: '5m',
  rule_id: 'rule-1',
  output_index: '.siem-signals-default',
  max_signals: 100,
  risk_score: 1,
  name: 'Simple ML Rule',
  references: [],
  severity: 'high',
  updated_by: 'elastic',
  tags: [],
  to: 'now',
  type: 'machine_learning',
  threat: [],
  version: 1,
  lists: [],
  actions: [],
  throttle: 'no_actions',
  last_failure_at: '2020-03-31T13:12:44.852Z',
  last_failure_message: 'Machine learning job is not started:\njob id: "some_job_id"\njob status: "undefined"\ndatafeed status: "undefined"\nname: "Simple ML Rule"\nid: "af842728-52c1-419b-980f-decdfce1ff75"\nrule id: "rule-1"\nsignals index: ".siem-signals-default"',
  anomaly_threshold: 44,
  machine_learning_job_id: 'some_job_id' } to sort of equal { actions: [],
  created_by: 'elastic',
  description: 'Simple Machine Learning Rule',
  enabled: true,
  false_positives: [],
  from: 'now-6m',
  immutable: false,
  interval: '5m',
  rule_id: 'rule-1',
  output_index: '.siem-signals-default',
  max_signals: 100,
  risk_score: 1,
  name: 'Simple ML Rule',
  references: [],
  severity: 'high',
  updated_by: 'elastic',
  tags: [],
  to: 'now',
  type: 'machine_learning',
  threat: [],
  throttle: 'no_actions',
  lists: [],
  version: 1,
  anomaly_threshold: 44,
  machine_learning_job_id: 'some_job_id' }
    at Assertion.assert (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:100:11)
    at Assertion.eql (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:244:8)
    at Context.it (test/detection_engine_api_integration/security_and_spaces/tests/create_rules.ts:101:34)
  actual:
   '{\n  "actions": []\n  "anomaly_threshold": 44\n  "created_by": "elastic"\n  "description": "Simple Machine Learning Rule"\n  "enabled": true\n  "false_positives": []\n  "from": "now-6m"\n  "immutable": false\n  "interval": "5m"\n  "last_failure_at": "2020-03-31T13:12:44.852Z"\n  "last_failure_message": "Machine learning job is not started:\\njob id: \\"some_job_id\\"\\njob status: \\"undefined\\"\\ndatafeed status: \\"undefined\\"\\nname: \\"Simple ML Rule\\"\\nid: \\"af842728-52c1-419b-980f-decdfce1ff75\\"\\nrule id: \\"rule-1\\"\\nsignals index: \\".siem-signals-default\\""\n  "lists": []\n  "machine_learning_job_id": "some_job_id"\n  "max_signals": 100\n  "name": "Simple ML Rule"\n  "output_index": ".siem-signals-default"\n  "references": []\n  "risk_score": 1\n  "rule_id": "rule-1"\n  "severity": "high"\n  "tags": []\n  "threat": []\n  "throttle": "no_actions"\n  "to": "now"\n  "type": "machine_learning"\n  "updated_by": "elastic"\n  "version": 1\n}',
  expected:
   '{\n  "actions": []\n  "anomaly_threshold": 44\n  "created_by": "elastic"\n  "description": "Simple Machine Learning Rule"\n  "enabled": true\n  "false_positives": []\n  "from": "now-6m"\n  "immutable": false\n  "interval": "5m"\n  "lists": []\n  "machine_learning_job_id": "some_job_id"\n  "max_signals": 100\n  "name": "Simple ML Rule"\n  "output_index": ".siem-signals-default"\n  "references": []\n  "risk_score": 1\n  "rule_id": "rule-1"\n  "severity": "high"\n  "tags": []\n  "threat": []\n  "throttle": "no_actions"\n  "to": "now"\n  "type": "machine_learning"\n  "updated_by": "elastic"\n  "version": 1\n}',
  showDiff: true }

---

History

• 💔 Build #37436 failed f3b1e7b
• 💔 Build #37422 failed f5a00e4
• 💚 Build #37148 succeeded 0832ca8
• 💚 Build #37138 succeeded 0c1ee7d
• 💚 Build #37097 succeeded 49f2225

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[peteharverson]

Tested latest edits and LGTM