[Security Solution][Detection Engine] fixes siem-signal update when it was reindexed from v7 to v8

[vitaliidm]

Summary

• addresses https://github.com/elastic/security-team/issues/11440

Testing

1. Create cloud env of 7.17 version, (East US 2 (Virginia) on Azurem where 8.18 snapshot available)
2. Create rule
3. Generate alerts
4. Create cloud env of 8.18 from existing 7.x snapshot (Restore snapshot data option)
5. Connect local Kibana of 8.18 from mirror branch of this one(8.18 mirror of [Security Solution][Detection Engine] fixes siem-signal update when it was reindexed from v7 to v8 #206119 #206120)
6. Add to Kibana dev config following options to enable Upgrade assistant(UA) showing outdated indices

   xpack.upgrade_assistant.featureSet:
     mlSnapshots: true
     migrateDataStreams: true
     migrateSystemIndices: true
     reindexCorrectiveActions: true

7. When Kibana started DO NOT visit Detection rule or any Security page
8. Open KIbana Upgrade Assistant,
9. Got to step 3 - Review deprecated settings and resolve issues
10. Click Elasticsearch section
11. Find outdated .siem-signals-* index
12. Reindex it
13. Visit detection page to ensure index API updated mappings

Visit to that page should initiate POST /api/detection_engine/index, which updates mappings

Subsequent index status check should return:

GET kbn:/api/detection_engine/index

// should return

{
  "name": ".alerts-security.alerts-default",
  "index_mapping_outdated": false
}

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-detection-engine (Team:Detection Engine)

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 990c4f0

Metrics [docs]

✅ unchanged

History

• 💛 Build #267047 was flaky 599930b
• 💔 Build #266431 failed 6f5971c
• 💚 Build #266155 succeeded 4a3d287
• 💔 Build #266132 failed 4fee8a0
• 💔 Build #265763 failed 8a3ca78

cc @vitaliidm

[dhurley14]

I'm approving with a few things that I wanted to point out:

1. When I got to the "click reindex" in upgrade assistant the reindexing process started with the flyout showing up but stayed at 0% for a bit and then it logged me out, so I had to log back in again. Not a big problem but could be scary to users. Not sure if this was seen before.
2. Alerts table is not populated after reindex process but the counts are visible

3. Screenshot that the reindex api worked:

[vitaliidm]

thanks for review @dhurley14

When I got to the "click reindex" in upgrade assistant the reindexing process started with the flyout showing up but stayed at 0% for a bit and then it logged me out, so I had to log back in again. Not a big problem but could be scary to users. Not sure if this was seen before.

Never seen it before (I reindexed probably more than dozen indices while working on thus and similar issues).
Is this happening consistently? If so, maybe worth to file an issue with Kibana team?

Alerts table is not populated after reindex process but the counts are visible

I looked in to it. This affects not reindexed index, but any alert created in 7.17.
Table queries alerts with this filter:

           {
                  "match_phrase": {
                    "kibana.alert.workflow_status": "open"
                  }
                },

But alerts created in 7.17 do not have it.

{
        "_index": ".siem-signals-another-3-000001-r000077",
        "_id": "389a2db0a7ab1b4a99f58a345bf781a0630fcbe2243af7f0253687c4b5dda8cc",
        "_score": 1,
        "_source": {
          "@timestamp": "2025-01-24T10:10:29.875Z",
          "destination.port": 1,
          "source.ip": "127.0.0.1",
          "event": {
            "kind": "signal"
          },
          "destination.ip": "127.0.0.2",
          "signal": {
            "reason": "event with:1, created low alert custom query rule.",
            "parent": {
              "depth": 0,
              "index": "close_alerts",
              "id": "IAPJl5QBKnW_HB3YObWr",
              "type": "event"
            },
            "depth": 1,
            "_meta": {
              "version": 77
            },
            "rule": {
              "throttle": null,
              "severity_mapping": [],
              "references": [],
              "created_at": "2025-01-24T10:10:24.801Z",
              "description": "eee",
              "language": "kuery",
              "output_index": ".siem-signals-another-3",
              "type": "query",
              "enabled": true,
              "exceptions_list": [],
              "updated_at": "2025-01-24T10:10:26.833Z",
              "from": "now-36000018000s",
              "id": "6d13e240-da3b-11ef-83df-57c27462827f",
              "severity": "low",
              "max_signals": 100,
              "risk_score": 21,
              "risk_score_mapping": [],
              "author": [],
              "query": "*",
              "index": [
                "close_alerts"
              ],
              "filters": [],
              "created_by": "942543759",
              "version": 1,
              "tags": [],
              "rule_id": "98c35aa2-ee85-4f63-9066-6670a163c278",
              "license": "",
              "immutable": false,
              "meta": {
                "from": "10000000h",
                "kibana_siem_app_url": "https://to-9-0.kb.eastus2.staging.azure.foundit.no/app/security"
              },
              "name": "custom query rule",
              "updated_by": "942543759",
              "interval": "5h",
              "false_positives": [],
              "threat": [],
              "to": "now",
              "actions": []
            },
            "original_time": "2024-09-27T09:26:30.425Z",
            "ancestors": [
              {
                "depth": 0,
                "index": "close_alerts",
                "id": "IAPJl5QBKnW_HB3YObWr",
                "type": "event"
              }
            ],
            "parents": [
              {
                "depth": 0,
                "index": "close_alerts",
                "id": "IAPJl5QBKnW_HB3YObWr",
                "type": "event"
              }
            ],
            "status": "open"
          }
        }
      }

open status is not covered

[kibanamachine]

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/12948662880

[kibanamachine]

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

[vitaliidm]

@dhurley14

I looked further into it.
Turned out nothing to do with open status. But it is true, it affects alerts index that was not reindexed too.

It was working fine in 8.16. I checked this, and alerts displayed there. Something broke in 8.18.
I will create issue.

I noticed change in filters for 8.18 and 8.16.

In 8.18 there are these filters present:

          {
            "terms": {
              "kibana.alert.rule.consumer": [
                "siem"
              ]
            }
          },
          {
            "terms": {
              "kibana.alert.rule.rule_type_id": [
                "siem.eqlRule",
                "siem.esqlRule",
                "siem.indicatorRule",
                "siem.mlRule",
                "siem.queryRule",
                "siem.savedQueryRule",
                "siem.thresholdRule",
                "siem.newTermsRule"
              ]
            }
          }

I believe they prevent alerts to be displayed in table

[vitaliidm]

issue regarding absent alerts: #208204