From ec5ed454819283d7b65cb6f2e86c43b34ef2efd3 Mon Sep 17 00:00:00 2001 From: "claude[bot]" <41898282+claude[bot]@users.noreply.github.com> Date: Wed, 11 Feb 2026 18:03:47 +0000 Subject: [PATCH 1/6] docs: Add guide for creating OTel Input Packages Add comprehensive documentation for developing OpenTelemetry Input Packages, including: - Package structure and manifest configuration - Handlebars template development patterns - Variable types and configuration options - Testing requirements (policy and system tests) - Documentation guidelines and submission checklist - Links to existing examples (statsd, prometheus, hostmetrics) Also update index.md and integration-definitions.md to reference the new guide. Closes #17381 Co-authored-by: Bill Easton --- docs/extend/index.md | 4 + docs/extend/integration-definitions.md | 2 + docs/extend/otel-input-packages.md | 351 +++++++++++++++++++++++++ 3 files changed, 357 insertions(+) create mode 100644 docs/extend/otel-input-packages.md diff --git a/docs/extend/index.md b/docs/extend/index.md index 125e50becd8..27dbc4c3a32 100644 --- a/docs/extend/index.md +++ b/docs/extend/index.md @@ -15,6 +15,10 @@ Begin by understanding what is an [integration](./what-is-an-integration.md). Dive deep into the technical aspects of building integrations with Elastic products. Our [Building Integrations](./build-new-integration.md) guide covers everything from architecture and design principles to coding best practices and sample projects. +## OTel Input Packages + +Learn how to create [OTel Input Packages](./otel-input-packages.md) that configure OpenTelemetry Collector receivers within EDOT (Elastic Distribution of OpenTelemetry). + ## Testing and Validation Ensure your integrations work seamlessly by following our [Testing and Validation](./testing-validation.md) guidelines. Learn about different testing methodologies, tools, and techniques to validate your integration's performance and reliability. diff --git a/docs/extend/integration-definitions.md b/docs/extend/integration-definitions.md index e535557ebfa..592cc68bc7f 100644 --- a/docs/extend/integration-definitions.md +++ b/docs/extend/integration-definitions.md @@ -36,6 +36,8 @@ Unlike integrations, content packages do not include data streams or Agent polic An input package is a type of package that exposes the raw configuration of an Elastic Agent input for custom use cases. Unlike integrations that provide pre-configured data collection for specific technologies, input packages allow users to configure inputs directly for their unique requirements. +A special type of input package is the **OTel Input Package**, which configures OpenTelemetry Collector receivers within EDOT (Elastic Distribution of OpenTelemetry). See [OTel Input Packages](./otel-input-packages.md) for details on creating these packages. + ## Data stream [_data_stream] diff --git a/docs/extend/otel-input-packages.md b/docs/extend/otel-input-packages.md new file mode 100644 index 00000000000..c7d21180ef2 --- /dev/null +++ b/docs/extend/otel-input-packages.md @@ -0,0 +1,351 @@ +--- +mapped_pages: + - https://www.elastic.co/guide/en/integrations-developer/current/otel-input-packages.html +--- + +# OTel Input Packages [otel-input-packages] + +OTel Input Packages are **input-type packages** (`type: input`) that configure OpenTelemetry Collector receivers within EDOT (Elastic Distribution of OpenTelemetry). Unlike traditional integrations with data streams, these packages define Handlebars templates that generate OTel Collector configuration, allowing data to flow directly through the OTel pipeline to Elasticsearch. + +## When to Create an OTel Input Package [when-to-create] + +Create an OTel Input Package when: +- You want to use an existing [OpenTelemetry Collector receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver) +- No data transformation or field mapping is required (data flows directly to Elasticsearch) +- Users need configurable options exposed through the Fleet UI + +Use a traditional [integration](./what-is-an-integration.md) instead when: +- Custom ingest pipelines or field mappings are required +- You need to define Kibana assets (dashboards, visualizations) +- The data source doesn't have an OTel receiver + +## Package Structure [package-structure] + +``` +packages/_input_otel/ +├── manifest.yml # Package metadata and configuration +├── changelog.yml # Version history +├── LICENSE.txt # Optional: License file +├── docs/ +│ └── README.md # User-facing documentation +├── img/ +│ └── .svg # Package icon (32x32 recommended) +├── agent/ +│ └── input/ +│ └── input.yml.hbs # OTel Collector template +├── sample_event.json # Optional: Example event (generated via tests) +└── _dev/ + ├── deploy/ + │ └── docker/ # Service deployment for tests + │ └── docker-compose.yml + └── test/ + ├── policy/ # Policy tests (required) + │ ├── test-default.yml + │ └── test-default.expected + └── system/ # System tests (recommended) + └── test-default-config.yml +``` + +## Manifest Configuration [manifest-config] + +The `manifest.yml` file defines your package metadata and configurable variables. Here's an annotated example based on the StatsD input package: + +```yaml +format_version: 3.5.0 +name: statsd_input_otel +title: "StatsD OpenTelemetry Input Package" +version: 0.1.0 +source: + license: "Elastic-2.0" +description: "StatsD OpenTelemetry Input Package" +type: input # Must be "input" for OTel packages +categories: + - observability + - opentelemetry # Always include this category + - custom # Add relevant categories +conditions: + kibana: + version: "^9.2.0" # OTel support requires Kibana 9.2.0+ + elastic: + subscription: "basic" +icons: + - src: /img/statsd_otellogo.svg + title: StatsD OTel logo + size: 32x32 + type: image/svg+xml +policy_templates: + - name: statsdreceiver # Usually matches the OTel receiver name + type: metrics # or "logs" depending on data type + title: StatsD OpenTelemetry Input + description: Collect StatsD metrics using OpenTelemetry Collector + input: otelcol # Required: specifies OTel Collector input + template_path: input.yml.hbs + vars: + # Define configurable variables here + - name: endpoint + type: text + required: true + title: Endpoint + description: Address and port to listen on for StatsD metrics. + default: localhost:8125 +owner: + github: elastic/ecosystem # Your team + type: elastic +``` + +### Variable Types [variable-types] + +OTel Input Packages support these variable types: + +| Type | Description | Example | +|------|-------------|---------| +| `text` | Free-form text input | Endpoints, paths | +| `password` | Sensitive text (masked in UI) | Credentials | +| `bool` | Boolean toggle | Feature flags | +| `duration` | Time duration with unit | `60s`, `5m` | +| `select` | Dropdown with predefined options | Protocol selection | +| `yaml` | Multi-line YAML configuration | Advanced settings | + +**Important:** For `password` fields, use `secret: false` as a workaround until [fleet-server#6277](https://github.com/elastic/fleet-server/issues/6277) is resolved. + +### Variable Options [variable-options] + +```yaml +vars: + - name: transport + type: select + required: false + title: Transport Protocol + default: udp + show_user: false # Hide from basic UI + options: + - text: UDP + value: udp + - text: TCP + value: tcp + + - name: aggregation_interval + type: duration # Use duration for time intervals + required: false + title: Aggregation Interval + default: 60s + + - name: targets + type: text + multi: true # Allow multiple values + default: + - localhost:9090 +``` + +## Template Development [template-dev] + +The `input.yml.hbs` file is a Handlebars template that generates OTel Collector configuration. + +### Basic Structure [basic-structure] + +```handlebars +receivers: + : + endpoint: {{endpoint}} + {{#if optional_setting}} + optional_setting: {{optional_setting}} + {{/if}} +processors: + resourcedetection/system: + detectors: ["system"] +service: + pipelines: + metrics: # or "logs" for log data + receivers: [] + processors: [resourcedetection/system] +``` + +### Handlebars Patterns [handlebars-patterns] + +**Conditional fields with defaults:** +```handlebars +{{#if transport}} +transport: {{transport}} +{{/if}} +``` + +**Boolean fields:** Boolean fields with `default: false` should NOT use `{{#if}}` wrappers because Handlebars evaluates `false` as falsy: +```handlebars +# Correct: Always output boolean values directly +enable_metric_type: {{enable_metric_type}} +is_monotonic_counter: {{is_monotonic_counter}} +``` + +**Iterating over multi-value fields:** +```handlebars +static_configs: + - targets: +{{#each targets}} + - {{this}} +{{/each}} +``` + +**Embedding YAML configuration:** +```handlebars +processors: + resourcedetection: + detectors: ["system"] + system: {{system_config}} +``` + +### Resource Detection Processor [resource-detection] + +Always include the `resourcedetection` processor for host enrichment: + +```handlebars +processors: + resourcedetection/system: + detectors: ["system"] +``` + +For more detailed host information, use a YAML variable: + +```handlebars +processors: + resourcedetection: + detectors: ["system"] + system: {{system_config}} +``` + +With a default value in the manifest: +```yaml +- name: system_config + type: yaml + default: | + hostname_sources: ["os"] + resource_attributes: + host.name: + enabled: true + host.arch: + enabled: true +``` + +## Testing [testing] + +OTel Input Packages require policy tests and should include system tests. + +### Policy Tests [policy-tests] + +Policy tests verify that the package configuration generates valid Elastic Agent policies. + +**Required files:** +- `_dev/test/policy/test-default.yml` - Test input values +- `_dev/test/policy/test-default.expected` - Expected output + +**Example `test-default.yml`:** +```yaml +vars: + endpoint: "localhost:8125" + transport: udp + aggregation_interval: 60s + enable_metric_type: false + is_monotonic_counter: false +``` + +**Running policy tests:** +```bash +elastic-package stack up -d --services kibana,elasticsearch +cd packages/ +elastic-package test policy +``` + +For more details, see [Policy Testing](./policy-testing.md). + +### System Tests [system-tests] + +System tests validate the complete data flow from your service to Elasticsearch. + +**Required files:** +- `_dev/deploy/docker/docker-compose.yml` - Service deployment +- `_dev/test/system/test-default-config.yml` - Test configuration + +**Example `test-default-config.yml`:** +```yaml +service_notify_signal: SIGHUP +vars: + endpoint: "0.0.0.0:8125" +assert: + hit_count: 6 # Expected number of documents +``` + +**Running system tests and generating sample events:** +```bash +elastic-package stack up -d +cd packages/ +elastic-package test system --generate +``` + +For more details, see [System Testing](./system-testing.md). + +## Documentation [documentation] + +Create a `docs/README.md` that includes: + +1. **Overview** - What the package does and which OTel receiver it uses +2. **How it works** - Data flow explanation +3. **Configuration** - Link to upstream receiver documentation +4. **Metrics/Logs reference** - Link to upstream documentation + +**Example structure:** +```markdown +# OpenTelemetry Input Package + +## Overview +The OpenTelemetry Input Package enables collection of +using the [](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/). + +## How it works +This package configures the in the EDOT collector, which +forwards data to Elastic Agent for processing and indexing in Elasticsearch. + +## Configuration +For the full list of settings, refer to the upstream +[configuration documentation](https://github.com/open-telemetry/...). + +## Metrics reference +For available metrics, refer to the [ documentation](https://github.com/open-telemetry/...). +``` + +## Package Icon [package-icon] + +Create a combined logo that represents both the product and OpenTelemetry: +- Format: SVG +- Size: 32x32 pixels recommended +- Location: `img/_otellogo.svg` + +## Submission Checklist [submission-checklist] + +Before submitting your OTel Input Package: + +- [ ] `manifest.yml` has `type: input` and `input: otelcol` +- [ ] `manifest.yml` includes `opentelemetry` in categories +- [ ] `manifest.yml` has `conditions.kibana.version: "^9.2.0"` or newer +- [ ] `input.yml.hbs` includes `resourcedetection` processor +- [ ] Policy tests exist and pass (`_dev/test/policy/`) +- [ ] System tests exist and pass (`_dev/test/system/`) +- [ ] `docs/README.md` documents the package +- [ ] `changelog.yml` has an entry for the initial version +- [ ] Package icon exists in `img/` +- [ ] `CODEOWNERS` file includes your team for the package path + +## Examples [examples] + +Reference these existing packages as examples: + +| Package | Complexity | Notable Features | +|---------|------------|------------------| +| [statsd_input_otel](https://github.com/elastic/integrations/tree/main/packages/statsd_input_otel) | Simple | Minimal configuration, good starting point | +| [prometheus_input_otel](https://github.com/elastic/integrations/tree/main/packages/prometheus_input_otel) | Medium | TLS config, authentication, multi-value fields | +| [hostmetrics_input_otel](https://github.com/elastic/integrations/tree/main/packages/hostmetrics_input_otel) | Complex | Conditional scrapers, YAML configuration | + +## Additional Resources [resources] + +- [OpenTelemetry Collector Receivers](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver) +- [elastic-package Policy Testing Guide](https://github.com/elastic/elastic-package/blob/main/docs/howto/policy_testing.md) +- [elastic-package System Testing Guide](https://github.com/elastic/elastic-package/blob/main/docs/howto/system_testing.md) +- [Input Package Definition](./integration-definitions.md#_input_package) From cb1ef46515d6f1edfdce4f68267bbb9759603dbe Mon Sep 17 00:00:00 2001 From: "claude[bot]" <41898282+claude[bot]@users.noreply.github.com> Date: Thu, 12 Feb 2026 13:54:21 +0000 Subject: [PATCH 2/6] Address PR feedback: add toc.yml entry, simplify when-to-use section, add notes - Add otel-input-packages.md to toc.yml to fix docs build - Simplify 'When to Create' section: OTel for Receivers, Beats for traditional - Add note about dataset matching bug in sample files - Add elastic-package format to submission checklist - Add search hint for example packages Co-authored-by: Bill Easton --- docs/extend/otel-input-packages.md | 17 +++++++---------- docs/extend/toc.yml | 1 + 2 files changed, 8 insertions(+), 10 deletions(-) diff --git a/docs/extend/otel-input-packages.md b/docs/extend/otel-input-packages.md index c7d21180ef2..c236dc8517b 100644 --- a/docs/extend/otel-input-packages.md +++ b/docs/extend/otel-input-packages.md @@ -9,15 +9,7 @@ OTel Input Packages are **input-type packages** (`type: input`) that configure O ## When to Create an OTel Input Package [when-to-create] -Create an OTel Input Package when: -- You want to use an existing [OpenTelemetry Collector receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver) -- No data transformation or field mapping is required (data flows directly to Elasticsearch) -- Users need configurable options exposed through the Fleet UI - -Use a traditional [integration](./what-is-an-integration.md) instead when: -- Custom ingest pipelines or field mappings are required -- You need to define Kibana assets (dashboards, visualizations) -- The data source doesn't have an OTel receiver +Use **OTel Input Packages** for [OpenTelemetry Collector receivers](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver). Use **traditional [integrations](./what-is-an-integration.md)** for Beats-based data collection. ## Package Structure [package-structure] @@ -280,6 +272,10 @@ cd packages/ elastic-package test system --generate ``` +::::{note} +There is currently a known bug where Datasets in generated sample files do not match with the expected files in test policies. This mismatch is expected behavior for now. +:::: + For more details, see [System Testing](./system-testing.md). ## Documentation [documentation] @@ -332,10 +328,11 @@ Before submitting your OTel Input Package: - [ ] `changelog.yml` has an entry for the initial version - [ ] Package icon exists in `img/` - [ ] `CODEOWNERS` file includes your team for the package path +- [ ] Run `elastic-package format` to ensure all YAML files are formatted as expected ## Examples [examples] -Reference these existing packages as examples: +Reference these existing packages as examples, or search for packages containing `_input_otel` in the [packages directory](https://github.com/elastic/integrations/tree/main/packages): | Package | Complexity | Notable Features | |---------|------------|------------------| diff --git a/docs/extend/toc.yml b/docs/extend/toc.yml index a2feec4d4fa..1007f49aeeb 100644 --- a/docs/extend/toc.yml +++ b/docs/extend/toc.yml @@ -3,6 +3,7 @@ toc: - file: what-is-an-integration.md children: - file: integration-definitions.md + - file: otel-input-packages.md - file: quick-start.md - file: build-new-integration.md children: From c7e9ca639e932886b7ef6af6891cc68230d9a571 Mon Sep 17 00:00:00 2001 From: "claude[bot]" <41898282+claude[bot]@users.noreply.github.com> Date: Thu, 12 Feb 2026 14:11:40 +0000 Subject: [PATCH 3/6] Add cross-references to existing documentation - Link to integration-definitions.md for input package concepts - Link to manifest-spec.md for complete manifest specification - Link to testing.md overview and individual policy/system testing guides - Link to documentation-guidelines.md for doc standards - Apply jlind23's suggestion to add bug link for dataset matching issue - Remove duplicated running commands (now in linked docs) - Reorganize Additional Resources into internal and external sections Co-authored-by: Bill Easton --- docs/extend/otel-input-packages.md | 46 ++++++++++++------------------ 1 file changed, 19 insertions(+), 27 deletions(-) diff --git a/docs/extend/otel-input-packages.md b/docs/extend/otel-input-packages.md index c236dc8517b..9467d3b47f5 100644 --- a/docs/extend/otel-input-packages.md +++ b/docs/extend/otel-input-packages.md @@ -13,6 +13,8 @@ Use **OTel Input Packages** for [OpenTelemetry Collector receivers](https://gith ## Package Structure [package-structure] +OTel Input Packages follow the standard [input package](./integration-definitions.md#_input_package) structure with specific naming and configuration requirements. + ``` packages/_input_otel/ ├── manifest.yml # Package metadata and configuration @@ -40,7 +42,7 @@ packages/_input_otel/ ## Manifest Configuration [manifest-config] -The `manifest.yml` file defines your package metadata and configurable variables. Here's an annotated example based on the StatsD input package: +The `manifest.yml` file defines your package metadata and configurable variables. For the complete manifest specification, refer to [manifest.yml](./manifest-spec.md). Here's an annotated example based on the StatsD input package: ```yaml format_version: 3.5.0 @@ -219,11 +221,11 @@ With a default value in the manifest: ## Testing [testing] -OTel Input Packages require policy tests and should include system tests. +OTel Input Packages require policy tests and should include system tests. For a complete overview of all test types, refer to [Test an integration](./testing.md). ### Policy Tests [policy-tests] -Policy tests verify that the package configuration generates valid Elastic Agent policies. +Policy tests verify that the package configuration generates valid Elastic Agent policies. For complete details on policy testing, refer to [Policy testing](./policy-testing.md). **Required files:** - `_dev/test/policy/test-default.yml` - Test input values @@ -239,18 +241,9 @@ vars: is_monotonic_counter: false ``` -**Running policy tests:** -```bash -elastic-package stack up -d --services kibana,elasticsearch -cd packages/ -elastic-package test policy -``` - -For more details, see [Policy Testing](./policy-testing.md). - ### System Tests [system-tests] -System tests validate the complete data flow from your service to Elasticsearch. +System tests validate the complete data flow from your service to Elasticsearch. For complete details on system testing, including deployment methods and troubleshooting, refer to [System Testing Guide](./system-testing.md). **Required files:** - `_dev/deploy/docker/docker-compose.yml` - Service deployment @@ -265,22 +258,13 @@ assert: hit_count: 6 # Expected number of documents ``` -**Running system tests and generating sample events:** -```bash -elastic-package stack up -d -cd packages/ -elastic-package test system --generate -``` - ::::{note} -There is currently a known bug where Datasets in generated sample files do not match with the expected files in test policies. This mismatch is expected behavior for now. +There is currently a known [bug](https://github.com/elastic/elastic-package/issues/3275) where Datasets in generated sample files do not match with the expected files in test policies. This mismatch is expected behavior for now. :::: -For more details, see [System Testing](./system-testing.md). - ## Documentation [documentation] -Create a `docs/README.md` that includes: +Create a `docs/README.md` following the [Documentation guidelines](./documentation-guidelines.md). For OTel Input Packages, include: 1. **Overview** - What the package does and which OTel receiver it uses 2. **How it works** - Data flow explanation @@ -342,7 +326,15 @@ Reference these existing packages as examples, or search for packages containing ## Additional Resources [resources] +**Internal documentation:** +- [Test an integration](./testing.md) - Overview of all test types +- [Policy testing](./policy-testing.md) - Policy test details +- [System Testing Guide](./system-testing.md) - System test details +- [Documentation guidelines](./documentation-guidelines.md) - Writing integration docs +- [manifest.yml](./manifest-spec.md) - Complete manifest specification +- [Input Package Definition](./integration-definitions.md#_input_package) - Input package concepts + +**External resources:** - [OpenTelemetry Collector Receivers](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver) -- [elastic-package Policy Testing Guide](https://github.com/elastic/elastic-package/blob/main/docs/howto/policy_testing.md) -- [elastic-package System Testing Guide](https://github.com/elastic/elastic-package/blob/main/docs/howto/system_testing.md) -- [Input Package Definition](./integration-definitions.md#_input_package) +- [elastic-package Policy Testing HOWTO](https://github.com/elastic/elastic-package/blob/main/docs/howto/policy_testing.md) +- [elastic-package System Testing HOWTO](https://github.com/elastic/elastic-package/blob/main/docs/howto/system_testing.md) From 2ddd966f3faac0196c21d167a704972fd40ff89b Mon Sep 17 00:00:00 2001 From: "claude[bot]" <41898282+claude[bot]@users.noreply.github.com> Date: Thu, 12 Feb 2026 14:20:53 +0000 Subject: [PATCH 4/6] Add step-by-step development workflow guide for OTel Input Packages Adds a comprehensive Development Workflow section with 6 steps: 1. Research the Upstream Receiver - how to find and understand OTel receiver docs 2. Identify Configurable Variables - mapping upstream types to package variables 3. Create the Input Template - building the Handlebars template 4. Write the Documentation - what to include in README.md 5. Test the Package - policy and system test guidance 6. Finalize and Submit - pre-submission checklist This helps ensure consistent, high-quality packages that align with existing patterns. Co-authored-by: Bill Easton --- docs/extend/otel-input-packages.md | 110 +++++++++++++++++++++++++++++ 1 file changed, 110 insertions(+) diff --git a/docs/extend/otel-input-packages.md b/docs/extend/otel-input-packages.md index 9467d3b47f5..d6dbec46001 100644 --- a/docs/extend/otel-input-packages.md +++ b/docs/extend/otel-input-packages.md @@ -11,6 +11,116 @@ OTel Input Packages are **input-type packages** (`type: input`) that configure O Use **OTel Input Packages** for [OpenTelemetry Collector receivers](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver). Use **traditional [integrations](./what-is-an-integration.md)** for Beats-based data collection. +## Development Workflow [development-workflow] + +Follow this step-by-step workflow when creating a new OTel Input Package. This ensures consistent, high-quality packages that align with existing patterns. + +### Step 1: Research the Upstream Receiver [step-1-research] + +Before writing any code, thoroughly understand the OTel receiver you're wrapping: + +1. **Find the receiver documentation**: Navigate to the [opentelemetry-collector-contrib receivers directory](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver) and locate your receiver (e.g., `statsdreceiver`, `prometheusreceiver`). + +2. **Read the README.md**: Review the receiver's README to understand: + - What data types it collects (metrics, logs, traces) + - Required vs optional configuration options + - Default values for all settings + - Any special behaviors or limitations + +3. **Examine the config.go file**: Look at the receiver's `config.go` for the complete configuration structure. This reveals all possible options and their Go types. + +4. **Check for related documentation**: Some receivers have additional docs (e.g., metric documentation files listing all available metrics). + +### Step 2: Identify Configurable Variables [step-2-variables] + +From your research, determine which configuration options should be exposed to users: + +1. **Always expose essential settings**: Options like `endpoint`, collection intervals, or authentication are typically required. + +2. **Evaluate optional settings**: For each optional setting, ask: + - Does a typical user need to change this? + - Is the default appropriate for most use cases? + - Does exposing this add unnecessary complexity? + +3. **Group by user visibility**: + - `show_user: true` — Common settings users frequently configure + - `show_user: false` — Advanced settings most users can ignore + +4. **Map upstream types to package variable types**: + + | Upstream Go Type | Package Variable Type | Notes | + |------------------|----------------------|-------| + | `string` | `text` | For endpoints, paths, names | + | `string` (secret) | `password` | For credentials (use `secret: false` for now) | + | `bool` | `bool` | For feature toggles | + | `time.Duration` | `duration` | For intervals like `60s`, `5m` | + | `[]string` | `text` with `multi: true` | For lists of values | + | Complex structs | `yaml` | For advanced nested configuration | + | Enum/limited values | `select` | For protocol choices, modes | + +### Step 3: Create the Input Template [step-3-template] + +Build your `input.yml.hbs` template based on the upstream receiver configuration: + +1. **Start with the receiver section**: Mirror the upstream YAML structure exactly. + +2. **Add conditional wrappers**: Use `{{#if variable}}` for optional fields that shouldn't appear when empty. + +3. **Handle booleans carefully**: Emit boolean values directly (not wrapped in `{{#if}}`) to ensure `false` values are included. + +4. **Match upstream defaults**: When a variable has a default in your manifest that matches the upstream default, consider whether you need to emit it. + +5. **Include resource detection**: Add the `resourcedetection` processor to enrich data with host information. + +6. **Define the service pipeline**: Connect receivers and processors in the appropriate pipeline (metrics, logs, or traces). + +### Step 4: Write the Documentation [step-4-docs] + +Create user-facing documentation that helps users understand and configure the package: + +1. **Overview section**: Explain what the package does and link to the upstream receiver. + +2. **How it works**: Describe the data flow from source through OTel to Elasticsearch. + +3. **Configuration reference**: Either document key settings inline or link directly to upstream documentation for the complete list. + +4. **Data reference**: Link to upstream docs for metrics/logs documentation rather than duplicating. + +5. **Troubleshooting**: Add common issues specific to the receiver (e.g., platform requirements, firewall considerations). + +### Step 5: Test the Package [step-5-test] + +Validate your package works correctly: + +1. **Create policy tests**: Add test cases in `_dev/test/policy/` covering: + - Default configuration + - Non-default values for key settings + - Edge cases (empty optional fields, multiple values) + +2. **Set up system tests**: Create `_dev/deploy/docker/` infrastructure to test live data flow. If the receiver only works on specific platforms (e.g., Windows), document this limitation. + +3. **Run tests locally**: + ```bash + elastic-package test policy -v + elastic-package test system -v + ``` + +4. **Generate expected files**: Use `elastic-package test policy --generate` to create `.expected` files after verifying output is correct. + +### Step 6: Finalize and Submit [step-6-submit] + +Complete the package and prepare for submission: + +1. **Format all files**: Run `elastic-package format` to ensure consistent YAML formatting. + +2. **Add CODEOWNERS entry**: Add your team to `.github/CODEOWNERS` for the package path. + +3. **Create package icon**: Design an SVG that represents both the source technology and OpenTelemetry. + +4. **Review the submission checklist**: Verify all items in the [Submission Checklist](#submission-checklist) are complete. + +5. **Validate the package**: Run `elastic-package check` to catch any specification violations. + ## Package Structure [package-structure] OTel Input Packages follow the standard [input package](./integration-definitions.md#_input_package) structure with specific naming and configuration requirements. From b81cd61179bc6944eb0071136fd664f8cf8b17d6 Mon Sep 17 00:00:00 2001 From: "claude[bot]" <41898282+claude[bot]@users.noreply.github.com> Date: Thu, 12 Feb 2026 14:32:55 +0000 Subject: [PATCH 5/6] Condense Development Workflow section for conciseness MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Reduce workflow from ~110 lines to ~48 lines - Link to existing docs (policy-testing.md, system-testing.md, documentation-guidelines.md) instead of duplicating content - Keep type mapping table as quick reference - Fix Vale linter suggestions: 'see' → 'refer to', 'Simple' → 'Basic' - Validated workflow with InfluxDB receiver dry run Co-authored-by: Bill Easton --- docs/extend/otel-input-packages.md | 123 ++++++++--------------------- 1 file changed, 31 insertions(+), 92 deletions(-) diff --git a/docs/extend/otel-input-packages.md b/docs/extend/otel-input-packages.md index d6dbec46001..df055807712 100644 --- a/docs/extend/otel-input-packages.md +++ b/docs/extend/otel-input-packages.md @@ -13,113 +13,52 @@ Use **OTel Input Packages** for [OpenTelemetry Collector receivers](https://gith ## Development Workflow [development-workflow] -Follow this step-by-step workflow when creating a new OTel Input Package. This ensures consistent, high-quality packages that align with existing patterns. +Follow this workflow when creating a new OTel Input Package: -### Step 1: Research the Upstream Receiver [step-1-research] +### 1. Research the Upstream Receiver -Before writing any code, thoroughly understand the OTel receiver you're wrapping: +Find your receiver in the [opentelemetry-collector-contrib receivers](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver): -1. **Find the receiver documentation**: Navigate to the [opentelemetry-collector-contrib receivers directory](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver) and locate your receiver (e.g., `statsdreceiver`, `prometheusreceiver`). +- **README.md** — Configuration options, defaults, data types collected +- **config.go** — Complete configuration structure and Go types +- **metadata.yaml** — Available metrics (if applicable) -2. **Read the README.md**: Review the receiver's README to understand: - - What data types it collects (metrics, logs, traces) - - Required vs optional configuration options - - Default values for all settings - - Any special behaviors or limitations +### 2. Define Variables -3. **Examine the config.go file**: Look at the receiver's `config.go` for the complete configuration structure. This reveals all possible options and their Go types. +Map upstream configuration to [manifest variables](#variable-types): -4. **Check for related documentation**: Some receivers have additional docs (e.g., metric documentation files listing all available metrics). +| Upstream Go Type | Package Variable Type | +|------------------|----------------------| +| `string` | `text` | +| `string` (secret) | `password` (with `secret: false`) | +| `bool` | `bool` | +| `time.Duration` | `duration` | +| `[]string` | `text` with `multi: true` | +| Complex structs | `yaml` | +| Enum values | `select` | -### Step 2: Identify Configurable Variables [step-2-variables] +Set `show_user: true` for commonly configured options, `show_user: false` for advanced settings. -From your research, determine which configuration options should be exposed to users: +### 3. Build the Template -1. **Always expose essential settings**: Options like `endpoint`, collection intervals, or authentication are typically required. +Create `agent/input/input.yml.hbs` following the [Template Development](#template-dev) patterns: -2. **Evaluate optional settings**: For each optional setting, ask: - - Does a typical user need to change this? - - Is the default appropriate for most use cases? - - Does exposing this add unnecessary complexity? +- Mirror the upstream YAML structure +- Use `{{#if var}}` for optional fields, but emit booleans directly +- Include the `resourcedetection/system` processor -3. **Group by user visibility**: - - `show_user: true` — Common settings users frequently configure - - `show_user: false` — Advanced settings most users can ignore +### 4. Write Documentation -4. **Map upstream types to package variable types**: +Follow the [Documentation guidelines](./documentation-guidelines.md). For OTel packages, link to upstream receiver docs rather than duplicating configuration details. - | Upstream Go Type | Package Variable Type | Notes | - |------------------|----------------------|-------| - | `string` | `text` | For endpoints, paths, names | - | `string` (secret) | `password` | For credentials (use `secret: false` for now) | - | `bool` | `bool` | For feature toggles | - | `time.Duration` | `duration` | For intervals like `60s`, `5m` | - | `[]string` | `text` with `multi: true` | For lists of values | - | Complex structs | `yaml` | For advanced nested configuration | - | Enum/limited values | `select` | For protocol choices, modes | +### 5. Add Tests -### Step 3: Create the Input Template [step-3-template] +- **Policy tests** — Refer to [Policy testing](./policy-testing.md). Cover default and non-default configurations. +- **System tests** — Refer to [System Testing Guide](./system-testing.md). Set up Docker infrastructure for live data flow testing. -Build your `input.yml.hbs` template based on the upstream receiver configuration: +### 6. Finalize -1. **Start with the receiver section**: Mirror the upstream YAML structure exactly. - -2. **Add conditional wrappers**: Use `{{#if variable}}` for optional fields that shouldn't appear when empty. - -3. **Handle booleans carefully**: Emit boolean values directly (not wrapped in `{{#if}}`) to ensure `false` values are included. - -4. **Match upstream defaults**: When a variable has a default in your manifest that matches the upstream default, consider whether you need to emit it. - -5. **Include resource detection**: Add the `resourcedetection` processor to enrich data with host information. - -6. **Define the service pipeline**: Connect receivers and processors in the appropriate pipeline (metrics, logs, or traces). - -### Step 4: Write the Documentation [step-4-docs] - -Create user-facing documentation that helps users understand and configure the package: - -1. **Overview section**: Explain what the package does and link to the upstream receiver. - -2. **How it works**: Describe the data flow from source through OTel to Elasticsearch. - -3. **Configuration reference**: Either document key settings inline or link directly to upstream documentation for the complete list. - -4. **Data reference**: Link to upstream docs for metrics/logs documentation rather than duplicating. - -5. **Troubleshooting**: Add common issues specific to the receiver (e.g., platform requirements, firewall considerations). - -### Step 5: Test the Package [step-5-test] - -Validate your package works correctly: - -1. **Create policy tests**: Add test cases in `_dev/test/policy/` covering: - - Default configuration - - Non-default values for key settings - - Edge cases (empty optional fields, multiple values) - -2. **Set up system tests**: Create `_dev/deploy/docker/` infrastructure to test live data flow. If the receiver only works on specific platforms (e.g., Windows), document this limitation. - -3. **Run tests locally**: - ```bash - elastic-package test policy -v - elastic-package test system -v - ``` - -4. **Generate expected files**: Use `elastic-package test policy --generate` to create `.expected` files after verifying output is correct. - -### Step 6: Finalize and Submit [step-6-submit] - -Complete the package and prepare for submission: - -1. **Format all files**: Run `elastic-package format` to ensure consistent YAML formatting. - -2. **Add CODEOWNERS entry**: Add your team to `.github/CODEOWNERS` for the package path. - -3. **Create package icon**: Design an SVG that represents both the source technology and OpenTelemetry. - -4. **Review the submission checklist**: Verify all items in the [Submission Checklist](#submission-checklist) are complete. - -5. **Validate the package**: Run `elastic-package check` to catch any specification violations. +Run `elastic-package format` and `elastic-package check`, then verify the [Submission Checklist](#submission-checklist). ## Package Structure [package-structure] @@ -430,7 +369,7 @@ Reference these existing packages as examples, or search for packages containing | Package | Complexity | Notable Features | |---------|------------|------------------| -| [statsd_input_otel](https://github.com/elastic/integrations/tree/main/packages/statsd_input_otel) | Simple | Minimal configuration, good starting point | +| [statsd_input_otel](https://github.com/elastic/integrations/tree/main/packages/statsd_input_otel) | Basic | Minimal configuration, good starting point | | [prometheus_input_otel](https://github.com/elastic/integrations/tree/main/packages/prometheus_input_otel) | Medium | TLS config, authentication, multi-value fields | | [hostmetrics_input_otel](https://github.com/elastic/integrations/tree/main/packages/hostmetrics_input_otel) | Complex | Conditional scrapers, YAML configuration | From 5c01e00bf18bafab8d3696dbd4a9703f03f549bc Mon Sep 17 00:00:00 2001 From: "claude[bot]" <41898282+claude[bot]@users.noreply.github.com> Date: Thu, 12 Feb 2026 17:44:16 +0000 Subject: [PATCH 6/6] Address PR feedback: add icon examples and sample_event.json checklist item - Add links to existing OTel package icons as examples (hostmetrics, iis) - Add checklist item for generating sample_event.json from system tests Co-authored-by: Bill Easton --- docs/extend/otel-input-packages.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/extend/otel-input-packages.md b/docs/extend/otel-input-packages.md index df055807712..1460222f2e7 100644 --- a/docs/extend/otel-input-packages.md +++ b/docs/extend/otel-input-packages.md @@ -347,6 +347,10 @@ Create a combined logo that represents both the product and OpenTelemetry: - Size: 32x32 pixels recommended - Location: `img/_otellogo.svg` +Refer to these existing icons as examples: +- [hostmetrics_input_otel/img/system-otel.svg](https://github.com/elastic/integrations/blob/main/packages/hostmetrics_input_otel/img/system-otel.svg) +- [iis_input_otel/img/iis_otellogo.svg](https://github.com/elastic/integrations/blob/main/packages/iis_input_otel/img/iis_otellogo.svg) + ## Submission Checklist [submission-checklist] Before submitting your OTel Input Package: @@ -357,6 +361,7 @@ Before submitting your OTel Input Package: - [ ] `input.yml.hbs` includes `resourcedetection` processor - [ ] Policy tests exist and pass (`_dev/test/policy/`) - [ ] System tests exist and pass (`_dev/test/system/`) +- [ ] `sample_event.json` is generated from system tests (use `elastic-package test system --generate`) - [ ] `docs/README.md` documents the package - [ ] `changelog.yml` has an entry for the initial version - [ ] Package icon exists in `img/`