Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[OTLP] index_label from log_attributes #15128

Open
a-patos opened this issue Nov 26, 2024 · 2 comments · May be fixed by #15293
Open

[OTLP] index_label from log_attributes #15128

a-patos opened this issue Nov 26, 2024 · 2 comments · May be fixed by #15293
Labels
OTEL/OLTP type/feature Something new we should do

Comments

@a-patos
Copy link

a-patos commented Nov 26, 2024

Describe the bug
We collect our logs from our instances by using an OtterCol agent to retrieve all files contained in /var/log/**/*.
We then receive this stream on our OTLP endpoint where we would like to index the log_file_path contained in the logs context of events.

In the documentation, we saw that it was possible to index the attribute: https://grafana.com/docs/loki/latest/send-data/otel/

However, it seems to be limited to resource_attributes.
Our problem is that the definition of a resource_attribute at the OtterCol agent level requires a transformation step that is not compatible with the retrieval of multiple files :(

To Reproduce
Steps to reproduce the behavior:
loki CONFIG :

overrides:
  trash:
    otlp_config:
      resource_attributes:
        ignore_defaults: true
        log_attributes:
          - action: index_label
            regex: application
          - action: index_label
            regex: deployment.environment.name
          - action: index_label
            regex: service.name
          - action: index_label
            regex: host.name
          - action: index_label
            regex: log.file.path

Expected behavior

level=error ts=2024-11-26T16:03:53.256584508Z caller=loki.go:560 msg="module failed" module=distributor error="failed to start distributor, because it depends on module runtime-config, which has failed: invalid service state: Failed, expected: Running, failure: starting module runtime-config: invalid service state: Failed, expected: Running, failure: failed to load runtime config: load file: invalid override for tenant default: index_label action is only supported for resource_attributes"

Environment:

  • Infrastructure: EC2
  • Deployment tool: RPM

Screenshots, Promtail config, or terminal output

  1. OtelCOL config :
receivers:
  filelog/access:
    # Get Nginx logs
    include_file_name: true
    include_file_path: true
    include:
      - /var/log/nginx/access.log
      - /var/log/nginx/error.log
    retry_on_failure:
      enabled: true
    start_at: end
    max_batches: 1
    
processors:
  # Add Compagnie attribute for identify service
  attributes:
    actions:
      - action: upsert
        key: application
        value: "${env:ENV_ST}"
      - action: upsert
        key: deployment.environment.name
        value: "${env:ENV_NAME}"
      - action: upsert
        key: service.name
        value: "${env:COMPOSANT_NAME}"
      - action: upsert
        key: host.name
        value: "${env:HOSTNAME}"

transform:
    error_mode: ignore
    log_statements:
      - context: log
      # Convert Attribute to Ressource Attribute  for Loki Indexation (Attribute can't be index on loki backend).
        statements:
          - set(resource.attributes["application"], attributes["application"]) where attributes["application"] != nil
          - set(resource.attributes["deployment.environment.name"], attributes["deployment.environment.name"]) where attributes["deployment.environment.name"] != nil
          - set(resource.attributes["service.name"], attributes["service.name"]) where attributes["service.name"] != nil
          - set(resource.attributes["host.name"], attributes["host.name"]) where attributes["host.name"] != nil

exporters:
  # output to loki backend
  otlphttp/logs:
    endpoint: "https://loki.prod.local.fr:8443/otlp"
    headers:
      "X-Scope-OrgID": "trash" 


service:
  pipelines:
    logs/nginx:
          receivers:
            - filelog/access
          processors:
            - attributes
            - transform
          exporters:
            - otlphttp/logs
@JStickler JStickler added type/feature Something new we should do OTEL/OLTP labels Dec 2, 2024
@samuelarogbonlo
Copy link

Looks like an interesting issue - will jump on it now

@samuelarogbonlo
Copy link

@JStickler I have worked on this and sent out a PR here.

Let me know what you or the team thinks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
OTEL/OLTP type/feature Something new we should do
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants