Customize attribute that can be use for OpenTelemetry.

[fadianzkaba]

Hi All

I have been reviewing the telemetry for the Benthos pipeline as described here, but I was unable to find a way to extract custom attributes.

For example, I have a configuration similar to the one shown below. From my understanding, under the hood, payload.go is responsible for extracting the configuration and presenting it in the telemetry data.

Currently, the telemetry output only includes the following attributes:

• name: output_pipeline_name
• name: output_cloud_storage

However, I would like to extract additional details from the pipeline_name section, such as dsn , max_in_flight, and organize them under a separate custom attribute.

My goal is to ensure that these attributes are explicitly available in the telemetry data for easier monitoring and debugging. Is there a recommended approach to achieve this? If Redpanda's telemetry pipeline does not support custom attributes by default, are there any workarounds, such as modifying the payload.go implementation or using an alternative configuration method?

Any guidance or best practices on this would be greatly appreciated!

output:
  switch:
    retry_until_success: false
    strict_mode: false
    cases:
      - check: 'metadata("test123") == "abcd"'
        output:
          broker:
            copies: 1
            outputs:
              - pipeline_name:
                  dsn: destination
                  max_in_flight: 50
                  service_account: services_account_id
              - cloud_storage:
                  bucket: bucket_name
                  path: path.json
                  batching:
                    count: 100000
                    period: 5s
                    byte_size: 1000000
                    processors:
                      - bloblang: |
                          root = {}
                          root.message = content().decode("base64")
                          root.AttributeMap = metadata()
                      - archive:
                          format: json_array

And here is the trace configuration

tracer:
  open_telemetry_collector:
    grpc: [{ address: "endpoint", secure: true }]
    tags:
      service.name: the_name_of_services
      deployment.environment: test
    sampling:
      enabled: false