Stop implicitly validating generic.otel for otelcol input packages by teresaromero · Pull Request #3433 · elastic/elastic-package

teresaromero · 2026-04-07T10:45:50Z

What does this PR do?

elastic-package no longer treats generic.otel as a special Fleet stream key or as an extra dataset that system tests must always accept for otelcol input packages. Stream keys follow {package}.{policy_template}; data_stream.dataset comes from user variables, manifest defaults, or the policy template name. Users who need a shared or generic OTel dataset set it explicitly via data_stream.dataset as the base name (see above).

Dataset override: use the base name (important)

For otelcol input packages, data_stream.dataset in Fleet must be the base dataset only—do not include the .otel suffix. Elastic Agent adds .otel at ingest for routing. Fleet and elastic-package’s policy builder do not append it in the package policy payload.

If you set an override that already ends in .otel (for example generic.otel), the agent still appends .otel, and Elasticsearch ends up with a double suffix (generic.otel.otel). System tests in elastic-package mirror the agent by always appending .otel when resolving expected data stream names and document datasets for input packages, so overrides should stay base-only (e.g. generic, zipkinreceiver).

Why policy test goldens look different from sample events: Policy tests snapshot the agent policy YAML Kibana returns (transform set(attributes["data_stream.dataset"], …) lines). Those literals reflect the Fleet variable value—the base name—so they typically do not show .otel. Sample events (and real documents in Elasticsearch) reflect after agent routing, so data_stream.dataset and related fields show the .otel suffix (e.g. zipkinreceiver.otel in sample_event_traces.json). That split is expected, not a bug.

Dependencies

elastic/kibana#262000 should be available at 9.4.0 snapshot; tested manually with local stack using the above Kibana commit.

Background

Related upstream work (Kibana Fleet, ingest-dev) adjusted how OTel dynamic_signal_types and traces packages route data_stream.dataset. elastic-package aligns its policy construction and test expectations with variable-driven datasets rather than hardcoding generic.otel.

What this PR changes in elastic-package

`internal/kibana/packagepolicy.go`

BuildInputPackagePolicy builds the simplified API streams map with key fmt.Sprintf("%s.%s", manifest.Name, policyTemplate.Name). ensureDatasetVar sets data_stream.dataset from (in order): user varValues, existing manifest-backed vars, explicit template var defaults, or the policy template name. There is no branch that forces generic.otel for DynamicSignalTypes. Comments document that for otelcol, the stored value is the base dataset; Elastic Agent applies .otel.

`internal/testrunner/runners/system/tester.go`

BuildDataStreamName — for input-package otelcol, always append .otel to the Fleet data_stream.dataset value when building the Elasticsearch data stream name, matching Elastic Agent routing (Fleet does not add this suffix in the policy payload).
expectedDatasets — for otelcol input packages (pkgManifest.Type == "input"), append .otel the same way for document validation. A policy value that already ends in .otel yields an expected *.otel.otel ingested dataset (misconfiguration).

`AGENTS.md`

Fleet API notes include a bullet on otelcol input packages: base data_stream.dataset, agent .otel append, and how system tests mirror that.

Tests and in-repo fixtures

internal/testrunner/runners/system/tester_test.go — TestBuildDataStreamName and TestExpectedDatasets cover the agent-aligned append and policy values that already include .otel.
internal/kibana/packagepolicy_test.go — TestEnsureDatasetVar expects dynamic_signal_types to fall back to the policy template name like other templates; TestBuildInputPackagePolicy includes otel_dynamic_signal_types_default_dataset against testdata/packages/otel_dynamic_input.
internal/kibana/testdata/ — otel_input_dynamic_signals.json, otel_input_dynamic_signals_legacy.json, and packages/otel_dynamic_input/manifest.yml describe the {package}.{policy_template} / sqlreceiver default pattern instead of generic.otel.
Parallel test packages (e.g. zipkin) — system/policy configs use base dataset overrides where applicable; policy .expected files show transforms without .otel on that literal; sample events show *.otel as ingested.

References

Kibana: elastic/kibana#260385 — Fleet routing for dynamic signal types
Kibana: elastic/kibana#262000 — dependency for this change (see Dependencies above)
Tracking: elastic/ingest-dev#7403 — restore dataset override for OTel input packages

Fleet changed (April 2026) to use "generic.otel" as the stream path/dataset for input packages with dynamic_signal_types: true, instead of the previous <pkgName>.<policyTemplateName> default. Align BuildInputPackagePolicy to send the matching stream key and data_stream.dataset variable in simplified API requests. The test validator's expectedDatasets() was deriving expected document datasets from the policy variable (now "generic.otel"), but indexed documents retain the OTel scope-based dataset (e.g. sqlserverreceiver.otel). For dynamic_signal_types packages, derive the expected dataset from policyTemplate.Name + ".otel" instead. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add policy test config and expected output for sql_server_input_otel (dynamic_signal_types: true), generated against a live stack. Update zipkin_input_otel expected files to reflect Fleet no longer setting data_stream.dataset in OTel trace routing transforms. Add sample event files for both packages. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

jsoriano

LGTM as it seems to mimic current Fleet implementation, but I am still concerned about being able to distinguish generic and non-generic cases for dynamic_signal_types.

jsoriano · 2026-04-07T11:22:04Z

internal/testrunner/runners/system/tester.go

-			// Input packages whose input is `otelcol` must add the `.otel` suffix
-			// Example: httpcheck.metrics.otel
-			expectedDataset += "." + otelSuffixDataset
+			if scenario.policyTemplate.DynamicSignalTypes {


Is it still possible to override the dataset? For example in the case of mysql_input_otel it has dynamic_signal_types: true, but its data should not go to generic data streams.

i've added the case to the test fixtures and used the variable override to prove dataset can be overriden, but i dont see this is happening... i dont see the overriden value as dataset

teresaromero · 2026-04-07T13:35:29Z

test/packages/parallel/mysql_input_otel/manifest.yml

+        title: Dataset name
+        description: |
+          Dataset to write data to. Changing the dataset will send the data to a different index. You can't use `-` in the name of a dataset and only valid characters for [Elasticsearch index names](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html).
+        default: mysql_input_otel.generic


added the default value here, so expected the dataset to be this value. When installing the package and sending events (system test with defer cleanup) i dont see dataset is this value, mysqlreceiver.otel, is this correct?

mrodm · 2026-04-07T14:11:56Z

test/packages/parallel/mysql_input_otel/manifest.yml

+  - opentelemetry
+conditions:
+  kibana:
+    version: "^9.4.0"


This package requires to run a specific kibana/stack version.

It should be created a new file like the one for zipkin:
https://github.com/elastic/elastic-package/blob/main/test/packages/parallel/zipkin_input_otel.stack_version

We would need to review these stack version overrides in case it is needed to bump the minimum kibana version , or remove it if 9.4.0 supports these kind of packages.

mrodm · 2026-04-07T14:14:45Z

test/packages/parallel/mysql_input_otel/_dev/test/system/test-default-config.yml

+service_notify_signal: SIGHUP
+vars:
+  endpoint: "{{Hostname}}:{{Port}}"
+  username: "root"


Tested to add here an override for data_stream.dataset and it is not shown in the sample events:

$ cat test/packages/parallel/mysql_input_otel/_dev/test/system/test-default-config.yml service: mysql_input_otel service_notify_signal: SIGHUP vars: endpoint: "{{Hostname}}:{{Port}}" username: "root" data_stream.dataset: 'mysql.other' assert: min_count: 10

It is still shown mysqlreceiver.otel as dataset.

mrodm · 2026-04-07T14:15:41Z

test/packages/parallel/zipkin_input_otel/_dev/test/policy/test-default-custom-dataset.expected

            - context: span
              statements:
                - set(attributes["data_stream.type"], "traces")
-                - set(attributes["data_stream.dataset"], "zipkin.custom")


It looks like these changes are also related to the failures reported in https://buildkite.com/elastic/elastic-package-test-serverless/builds/783

…ion for dynamic packages BuildDataStreamName and expectedDatasets now apply the .otel suffix conditionally, preventing generic.otel from becoming generic.otel.otel when Fleet stores the real dataset value. The DynamicSignalTypes special case that derived the expected dataset from policyTemplate.Name is removed; scenario.dataStreamDataset is the source of truth for all otelcol input packages. Update policy test fixtures and sample events for mysql_input_otel, sql_server_input_otel, and zipkin_input_otel to reflect Fleet restoring data_stream.dataset in OTel transforms. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Made-with: Cursor

code changes after approval

mrodm · 2026-04-08T15:13:25Z

test/packages/parallel/zipkin_input_otel/sample_event_logs.json

+        }
+    },
+    "data_stream": {
+        "dataset": "generic.otel",


Are those sample event files re-generated after these changes ?

Should this dataset be generic.otel or zipkinreceiver.otel?

the sample_event_logs is coming from the system test with custom dataset. as i used generic as dataset, this is why we get those

internal/testrunner/runners/system/tester.go

mrodm · 2026-04-08T15:15:40Z

internal/testrunner/runners/system/tester.go

+			// The suffix is conditional to avoid double-appending when the stored dataset
+			// already ends in `.otel` (user-configured via data_stream.dataset).
+			if !strings.HasSuffix(expectedDataset, "."+otelSuffixDataset) {
+				expectedDataset += "." + otelSuffixDataset


IIUC, if needed the dataset generic.otel (as it was before), it would be already present in the scenario.datatreamDataset`, is that right?

the agent is setting the suffix. what elastic-package needs is to add the suffix itself to be able to compare results. code changed after last commits

jsoriano · 2026-04-08T16:19:55Z

internal/testrunner/runners/system/tester.go

-		dataset = fmt.Sprintf("%s.%s", dataset, otelSuffixDataset)
+		if !strings.HasSuffix(dataset, "."+otelSuffixDataset) {
+			dataset = dataset + "." + otelSuffixDataset
+		}


Why do we need to double-check if the otel suffix is already there? In what case the suffix would have been already added?

~~when the custom datastet is set using the otel suffix~~

after testing and learning the feature, this wont be possible. the dataset value should be the base name. .otel suffix is added by the agent so anything used as override will have an extra .otel suffix.

i wonder if we should validate this package-spec wise to warn the user when using overrides?

…neric-otel-dynamic-signal-types

…r and simplify logic.

…neric-otel-dynamic-signal-types

…NTS.md to clarify that `data_stream.dataset` should not include a `.otel` suffix, as Elastic Agent appends it during ingestion. Modified packagepolicy.go and tester.go to ensure consistent handling of dataset names, preventing double suffixes in Elasticsearch. Updated system tests to reflect these changes and improve clarity in test case titles.

…hanged default dataset names to 'custom.default' in MySQL input manifest and sample events. Updated timestamps and port numbers in sample event logs and metrics for both MySQL and SQL Server. Adjusted expected dataset names in test policies for consistency. Removed outdated skip reasons from test configurations.

…neric-otel-dynamic-signal-types

teresaromero · 2026-04-13T07:17:06Z

Pipeline errors are expected for the 3 test fixtures (otel inputs) as this change depends on elastic/kibana#262000 being available at 9.4.0 snapshot

elasticmachine · 2026-04-13T07:28:46Z

⏳ Build in-progress, with failures

Buildkite Build
Commit: 09a1ccf

Failed CI Steps

History

💔 Build #7673 failed 4002c43
💔 Build #7672 failed 6c0f63a
💔 Build #7670 failed 8013f74
💔 Build #7658 failed 6e8c589

teresaromero marked this pull request as ready for review April 7, 2026 10:47

teresaromero requested a review from a team April 7, 2026 10:47

teresaromero mentioned this pull request Apr 7, 2026

[updatecli] Update latest snapshot to 9.4.0-88e3ed81 #3430

Closed

jsoriano previously approved these changes Apr 7, 2026

View reviewed changes

teresaromero added 2 commits April 7, 2026 13:57

fix: update example event in README for sql_server_input_otel

a6e6349

add mysql input otel package to test fixtures with dataset override

0a9330a

teresaromero commented Apr 7, 2026

View reviewed changes

mrodm reviewed Apr 7, 2026

View reviewed changes

teresaromero and others added 2 commits April 8, 2026 09:19

add stack version for mysql_input_otel package

6e8c589

teresaromero requested review from jsoriano and mrodm April 8, 2026 14:23

fix: stop implicitly validating generic.otel for otelcol input packages

6c0f63a

Made-with: Cursor

teresaromero changed the title ~~fix: use generic.otel stream key for dynamic_signal_types input packages~~ Stop implicitly validating generic.otel for otelcol input packages Apr 8, 2026

fix: update example event in README for sql_server_input_otel

4002c43

mrodm reviewed Apr 8, 2026

View reviewed changes

jsoriano reviewed Apr 9, 2026

View reviewed changes

jsoriano mentioned this pull request Apr 9, 2026

Skip zipkin and sql_server system tests for Otel inputs #3445

Merged

teresaromero added 6 commits April 10, 2026 11:43

Merge branch 'main' of github.com:elastic/elastic-package into fix/ge…

39f2a51

…neric-otel-dynamic-signal-types

Refactor expectedDatasets function to remove unnecessary nil paramete…

b7943cd

…r and simplify logic.

Merge branch 'main' of github.com:elastic/elastic-package into fix/ge…

d8eacc7

…neric-otel-dynamic-signal-types

Merge branch 'main' of github.com:elastic/elastic-package into fix/ge…

09a1ccf

…neric-otel-dynamic-signal-types

Conversation

teresaromero commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Dataset override: use the base name (important)

Dependencies

Background

What this PR changes in elastic-package

internal/kibana/packagepolicy.go

internal/testrunner/runners/system/tester.go

AGENTS.md

Tests and in-repo fixtures

References

Uh oh!

jsoriano left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

teresaromero Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

teresaromero commented Apr 13, 2026

Uh oh!

elasticmachine commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⏳ Build in-progress, with failures

Failed CI Steps

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

teresaromero commented Apr 7, 2026 •

edited

Loading

`internal/kibana/packagepolicy.go`

`internal/testrunner/runners/system/tester.go`

`AGENTS.md`

teresaromero Apr 10, 2026 •

edited

Loading

elasticmachine commented Apr 13, 2026 •

edited

Loading