Should we have API for Logs in Open-telemetry?

[scheler]

What are you trying to achieve?
For modeling RUM events, we have chosen to use LogRecord as the underlying data model and hence proposing to have an API based on LogRecords for the purpose of creating Events. However, it is not clear if API should be extended to both logs and events use-cases or just events.

What did you expect to see?

We have a document detailing why we need an API for creating Events and an initial version of the API interface, it has a section on the concerns for having Logs API in Open-telemetry. This issue is created to invite folks to review the concerns and provide thoughts on the idea of having Logs API. Feel free to add comments within the google doc.

https://docs.google.com/document/d/1C9m3rctxFoHRiUsG3lSrOKZexsliNgl8D_6G6963Llo/edit?usp=sharing

cc: @tigrannajaryan @jack-berg

Additional context.

Add any other context about the problem here. If you followed an existing documentation, please share the link to it.

[jkwatson]

I would suggest that we should not have a full-fledged logging API in OpenTelemetry, unless there is a language that doesn't already have a plethora of logging libraries & APIs to choose from where it might make sense to define one.

But, I think having an Events API that is backed by the logs SDK and data model sounds like a fantastic idea to me, since I'm not aware of any such thing in Java (aside from those proprietary APIs implemented by observability vendors).

[reyang]

YES iff (if-and-only-if) a programming language doesn't have a well-established logging API (e.g. C++).

For languages that already have established logging API, even the API might not be "ideal", creating yet another API is likely to make the situation even worse.

[tigrannajaryan]

@jkwatson @reyang We have been actively discussing the API in the Log SIG for the last few weeks. We intend to make it very clear that the API is not intended to be used as a general-purpose logging API.

Instead we intend to advertise 2 distinct use cases:

• Otel Log SDK as a tool to build appenders for existing logging libraries to emit Otel-compliant logs (essentially what this says: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/logging-library-sdk.md)
• API to emit structured events to be used by various domains that use LogRecords to represent the events (e.g. client-side events or profiling events).

[reyang]

Regarding OTel Log SDK with the LogEmitter, totally.

Regarding API to emit structured events - many programming languages have already established good way to do it, on the other side - there are also languages which do not have a good logging API at all (at least not a well established one).

[tigrannajaryan]

Regarding API to emit structured events - many programming languages have already established good way to do it

Can you post some links to these?

[scheler]

@jack-berg thinks, in this comment, that the Log API should include both Events API and Log Appender API. Comments?

There's consensus that we don't want a log API to compete with existing frameworks. The log record API looks remarkably similar to a log API, so why include it instead of just an event API? Well, we need an API to bridge the existing log APIs into OpenTelemetry. That API has a lot of conceptual overlap with an event API. We could add a separate event API and log appender API, but why have all the code and conceptual duplication? Seems better to have a single API that serves both purposes, and to indicate using documentation that the log appender API is only meant for appenders, and NOT for instrumentation authors.

In otel java, we've bridged two popular logging frameworks (Log4j and Logback) to otel, but needed to introduce an appender API to decouple the appenders from taking a dependency on the otel java log SDK. The appender API is pretty silly - it just duplicates all the APIs of the otel java log SDK and delegates to a log SDK instance.

If the API proposed in this otep had a mechanism for producing log records, the otel java appender API would be unnecessary.

Additionally, a single API for producing events as well as log records produces a API / SDK that is very symmetric to the design of tracing / metrics. There is no precedent in tracing or metrics for a setup where two APIs (events and log appender) delegate to one (or two SDKs) which share the same data model. This symmetry reduces maintenance and cognitive overhead.

[tigrannajaryan]

I think we have a path forward and we can close this issue, since we have the answer: "yes, in a specific form defined by OTEL202".

From yesterday’s Log SIG meeting notes:

• Log SIG discussed 2 approaches for having logs and events API. It was not possible to achieve a consensus on which approach to use and Log SIG voted to have the TC member (Tigran Najaryan) to make the call. The TC member decided to go ahead with one Events and Logs API.
• OTEP202 will be updated to reflect the one Events and Logs API understanding. We will have a LoggerProvider/Logger, LogRecordBuilder/EventBuilder concepts. @scheler will update the OTEP.
• The Events and Logs API will be proposed as Experimental and is subject to change based on feedback from language SIGs. If new major arguments are discovered we will consider splitting the API into 2 separate ones.
• We agreed to adopt the following terminology when speaking about logs and events: “To log an event” and “event logging” are valid ways to speak about creating events represented as LogRecord instances. “To log a record” and “log recording” are valid ways to speak about creating arbitrary LogRecord instances.
• @MSNev will look into the possibility to have consistent definition of attributes for all signals, by removing the nesting limitations that are currently in the spec for Resource, Span and Metric attributes.