[Security Solution] Add support for arrays in the build_ebt_data_views script

[jkelas]

Partially resolves: #140369

Summary

This PR is a follow up for the #234571, where I am introducing telemetry event with array of primitive string values.
In order to display these values in Kibana Lens, a runtime mapping needs to be done in the security-solution-ebt-kibana-server data view. Before that, it was done manually. I am introducing support for arrays, so that manual intervention is not needed for them.

Checklist

• The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

[stephmilovic]

thanks for the script improvement!

[maximpn]

@jkelas Thanks for extending the schema flattening logic to support different array types 🙏

My main concern is maintainability. It might be required to extend the logic in the future so it has to be readable. Unfortunately five folded if's aren't very readable. In fact we have BFS searching for type fields so we could re-write the implementation based on that idea. I've made a code sketch how it could look like in a simpler way but it's it's not tested.

So we could have some unit tests to make sure the implementation works as expected.

[maximpn]

Should it being extracted to a separate function and restructured to avoid deep folded ifs?

[maximpn]

I played around to come up with a simpler version and ended up with the following

function flattenSchema(inputObj: NestedObject): Record<string, string> {
  const result: { [key: string]: string } = {};
  const queue: Array<{ obj: NestedObject; prefix: string }> = [{ obj: inputObj, prefix: '' }];
  while (queue.length > 0) {
    // eslint-disable-next-line @typescript-eslint/no-non-null-assertion
    const { obj, prefix } = queue.shift()!;
    for (const key in obj) {
      if (typeof obj[key] === 'object' && obj[key] !== null) {
        const newKey = `${prefix}${key}`;

        const { childToInspect, nodeType } = inspectSchemaNode(obj[key]);

        if (childToInspect) {
          queue.push({ obj: childToInspect, prefix: `${newKey}.` });
        }

        if (nodeType) {
          result[newKey] = nodeType;
        }
      }
    }
  }

  return result;
}

interface InspectSchemaNodeResult {
  childToInspect?: NestedObject | NestedSchemaNode;
  nodeType?: string;
}

function inspectSchemaNode(node: NestedSchemaNode): InspectSchemaNodeResult {
  if (!node.type) {
    const objectNode = node.properties ?? node;

    return { childToInspect: objectNode };
  }

  if (node.type === 'array' && node.items) {
    return inspectArraySchemaNode(node);
  }

  return { nodeType: String(node.type) };
}

function inspectArraySchemaNode(node: NestedSchemaNode): InspectSchemaNodeResult {
  const itemSchema = node.items;

  if (typeof itemSchema !== 'object' || itemSchema === null) {
    // "items" schema is not defined or not an object, leave it unaffected
    return { nodeType: 'array' };
  }

  if ('type' in itemSchema && itemSchema.type) {
    // array of primitive types
    return { nodeType: String(itemSchema.type) };
  }

  if (itemSchema.properties) {
    // array of objects
    return { childToInspect: itemSchema.properties };
  }

  return {};
}

There is an issue with types. To make it working types should be defined as a union of object and array schemas.

And I haven't tested my code so I could missed an if condition.

[jkelas]

@jkelas Thanks for extending the schema flattening logic to support different array types 🙏

My main concern is maintainability. It might be required to extend the logic in the future so it has to be readable. Unfortunately five folded if's aren't very readable. In fact we have BFS searching for type fields so we could re-write the implementation based on that idea. I've made a code sketch how it could look like in a simpler way but it's it's not tested.

So we could have some unit tests to make sure the implementation works as expected.

Yes, I agree. I was trying to make the changes as little invasive as possible, because there were no unit tests and I tried to push this quickly, but I agree, I can add the tests and then restructure the code.

[maximpn]

@jkelas Thanks for addressing my comments and adding tests 🙏

I've left nit comments to further make the implementation simpler.

[maximpn]

nit: isObjectRecord might be removed in favor of Lodash's isObject.

[jkelas]

Will improve

[maximpn]

nit: Let's move these functions to this file's bottom. These inspect functions are implementation details so it's logical to see the exported members first.

[jkelas]

OK

[maximpn]

To get rid of type casting node schema TS type could be simplified to

interface NestedSchemaNode {
  type?: string;
  properties?: NestedSchemaNode;
  items?: NestedSchemaNode;
  [key: string]: unknown;
}

This simplified type could be used in args and return types. With that you'd need only a single casting in flattenSchema

const { childToInspect, nodeType } = inspectSchemaNode(node as NestedSchemaNode);

[jkelas]

Good idea, thx, will improve

[maximpn]

Instead of any it's possible to use @ts-expect-error above const out = flattenSchema(schema); line.

[kibanamachine]

Starting backport for target branches: 8.18, 8.19, 9.0, 9.1

https://github.com/elastic/kibana/actions/runs/17790809288

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 8678598

Metrics [docs]

Saved Objects .kibana field count

Every field in each saved object type adds overhead to Elasticsearch. Kibana needs to keep the total field count below Elasticsearch's default limit of 1000 fields. Only specify field mappings for the fields you wish to search on or query. See https://www.elastic.co/guide/en/kibana/master/saved-objects-service.html#_mappings

id	before	after	diff
_data_stream_timestamp	1	-	-1
_doc_count	1	-	-1
_ignored_source	1	-	-1
_index_mode	1	-	-1
_inference_fields	1	-	-1
_tier	1	-	-1
apm-custom-dashboards	5	-	-5
apm-server-schema	2	-	-2
apm-service-group	5	-	-5
application_usage_daily	2	-	-2
config	2	-	-2
config-global	2	-	-2
coreMigrationVersion	1	-	-1
created_at	1	-	-1
created_by	1	-	-1
entity-definition	9	-	-9
entity-discovery-api-key	2	-	-2
event_loop_delays_daily	2	-	-2
favorites	4	-	-4
file	11	-	-11
file-upload-usage-collection-telemetry	3	-	-3
fileShare	5	-	-5
infra-custom-dashboards	4	-	-4
infrastructure-monitoring-log-view	2	-	-2
intercept_trigger_record	5	-	-5
legacy-url-alias	7	-	-7
managed	1	-	-1
ml-job	6	-	-6
ml-module	13	-	-13
ml-trained-model	7	-	-7
monitoring-telemetry	2	-	-2
namespace	1	-	-1
namespaces	1	-	-1
observability-onboarding-state	2	-	-2
originId	1	-	-1
product-doc-install-status	7	-	-7
references	4	-	-4
sample-data-telemetry	3	-	-3
security-ai-prompt	8	-	-8
slo	11	-	-11
space	5	-	-5
synthetics-monitor	34	-	-34
synthetics-monitor-multi-space	34	-	-34
tag	4	-	-4
type	1	-	-1
typeMigrationVersion	1	-	-1
ui-metric	2	-	-2
updated_at	1	-	-1
updated_by	1	-	-1
upgrade-assistant-ml-upgrade-operation	3	-	-3
upgrade-assistant-reindex-operation	3	-	-3
uptime-synthetics-api-key	2	-	-2
url	5	-	-5
usage-counters	2	-	-2
total			-246

Unknown metric groups

ESLint disabled in files

id	before	after	diff
securitySolution	102	103	+1

ESLint disabled line counts

id	before	after	diff
securitySolution	677	678	+1

Total ESLint disabled count

id	before	after	diff
securitySolution	779	781	+2

History

• 💛 Build #340076 was flaky 77c5055
• 💚 Build #338829 succeeded 413a065

cc @jkelas

[kibanamachine]

💚 All backports created successfully

Status	Branch	Result
✅	8.18
✅	8.19
✅	9.0
✅	9.1

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation