🌊 Streams: Normalized format for wired streams

[flash1293]

This PR extends streams to allow mixed otel/ecs querying.

Walkthrough (slightly outdated):

otel_streams.mov

To test

• Start up PR, enable streams via POST kbn:/api/streams/_enable
• Redirect all incoming logs to streams:

PUT _ingest/pipeline/logs@custom
{
  "processors": [
    {
      "reroute": {
        "destination": "logs"
      }
    }
  ]
}

• Send some logs
• They will be trandslated to namespaced ECS and can be queried the same way

Some demo actions:

# Make sure trial license is active (because of synthetic source)

# Enable streams
POST kbn:/api/streams/_enable

# Fork some data to be able to edit mapping and stuff
POST kbn:/api/streams/logs/_fork
{
  "stream": { "name": "logs.child" },
  "if": {
    "field": "attributes.custom_field",
    "operator": "exists"
  }
}
# Routing could have been done on "custom_field" as well

# Send some ECS-formatted data
POST logs/_doc
{
  "message": "Hello world",
  "log.level": "warn",
  "host": {
    "name": "my-mbp",
    "ip": "1.2.3.4"
  },
  "custom_field": "value2"
}

# Send some OTel-formatted data
POST logs/_doc
{
  "body": {
    "text": "Hello OTEL world"
  },
  "severity_text": "warn",
  "resource": {
    "attributes": {
      "host.name": "my-mbp",
      "host.ip": "1.2.3.4"
    }
  },
  "attributes": {
    "custom_field": "value"
  }
}

# See how it's stored as namespaced ECS (beware automatic aliasing)
GET logs.child/_search

# See the fields and their aliases
GET kbn:/api/streams/logs.child

# Nested data treatment (objects flattened, arrays kept)
POST logs/_doc
{
  "message": "Hello world",
  "log.level": "warn",
  "host": {
    "name": "my-mbp",
    "ip": "1.2.3.4"
  },
  "custom_field": "value2",
  "nested_custom_field": {
    "deeply": {
      "deeply": {
        "nested": "value"
      }
    }
  },
  "array_values": [
    { "deeply": { "nested": 123 }},
    { "deeply": { "nested": 456 }}
  ]
}

GET logs.child/_search

# Map a field (show alias in app)
PUT kbn:/api/streams/logs.child/_ingest
{
  "ingest": {
    "lifecycle": {
      "inherit": {}
    },
    "processing": [],
    "routing": [],
    "wired": {
      "fields": {
        "attributes.custom_value": {
          "type": "keyword"
        }
      }
    }
  }
}
# Trying to map "custom_value" will result in an error - only attributes.* and resource_attributes.* can be mapped

# End experiment
POST kbn:/api/streams/_disable

Do on separate PRs

• More dedicated integration tests
• Replace the painless implementation of normalization with an Elasticsearch processor (not shipped yet)
• Remove the dot expander processors once Elasticsearch allows accessing flattened field names

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 8230a44

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
streamsApp	468	469	+1

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
@kbn/streams-schema	212	220	+8

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
streamsApp	424.6KB	425.1KB	+490.0B

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
@kbn/streams-schema	17	16	-1

Unknown metric groups

API count

id	before	after	diff
@kbn/streams-schema	221	229	+8

History

• 💔 Build #302387 failed 7e2c3d6
• 💔 Build #302346 failed 89f34fd
• 💔 Build #302331 failed 811f39e
• 💛 Build #302047 was flaky f516471
• 💔 Build #301356 failed a9d54e2

[kibanamachine]

Starting backport for target branches: 8.19

https://github.com/elastic/kibana/actions/runs/15298591946

[kibanamachine]

💔 All backports failed

Status	Branch	Result
❌	8.19	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 205113

Questions ?

Please refer to the Backport tool documentation

[flash1293]

💚 All backports created successfully

Status	Branch	Result
✅	8.19

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

[kibanamachine]

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.
cc: @flash1293

[kibanamachine]

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.
cc: @flash1293