Move towards Log API/SDK stabilization

[alanwest]

OpenTelemetry's vision for supporting logs is clear:

We embrace existing logging solutions and make sure OpenTelemetry works nicely with existing logging libraries, log collection and processing solutions.

A number of prototypes exist demonstrating the Log API's utility in implementing log appenders for existing logging libraries.

• Java Log4j2 appender
• Java Logback appender
• .NET Serilog appender
• Possibly others in Python or Javascript maybe? (share links if anyone knows)

We have also received end user feedback suggesting a demand for a stable release of these log appenders.

I believe the API and SDK components as specified are nearly sufficient and we should consider making them stable. The LoggerProvider and Logger APIs are necessary for developing log appenders. The SDK components (i.e., log processors, exporters, etc) nicely mirrors the trace specification.

Below I've attempted to capture a list of remaining issues. If folks have additional things to add, please share!

Remaining issues

• Java: Log API and SDK prototype implementation
• Java: Log4J appender implementation. An example using Java agent is here.
• .Net: Log API and SDK prototype implementation
• .Net: Serilog appender implementation. .Net maintainers to link to example also showing trace context injection
• Python: Log API and SDK prototype implementation
• Python: logging handler implementation and usage example.
• Split logging functionality into API and SDK like other signals opentelemetry-python#3134
• Split out Event API from Log API #2941
• Define LogRecord batching processor defaults #3002
• Reorganize specification related to OTLP file exporter #3031
• Make context available in log record processor onEmit #2927
• Define data ownership for LogRecordProcessor #2969
• Add log attribute limit configuration #2861
• Settle on a name (Logs API vs SPI vs Bride API, etc)
• Temporarily remove "Events" from logs README #3247
• Present to Specification SIG the plans to stabilize logs API #3022
• Present in Maintainers meeting the plans to stabilize logs API #3115
• Proofread Logs Bridge API and SDK in preparation to mark it Stable #3268
• Make sure the current Logs Bridge API/SDK design doesn’t prevent future addition of end-user facing log API #3301
• Should "Emit a LogRecord" API set Timestamp if unspecified? #3386
• Drop log API include_trace_context parameter #3397
• Clarify how ObservedTimestamp field is set if unspecified #3385
• Consider removing include_trace_context from "Get a Logger" API #3394
• Clarify "Emit a LogRecord" API #3383
• Mark Logs API and SDK stable

[tigrannajaryan]

I support this. Let's discuss in the SIG meeting today.

The event_domain parameter of the Get a Logger API should remain experimental.

I think this means the entire Get a Logger needs to be marked experimental for now. At least in some languages adding/removing a parameter is a breaking change.

SDK spec

I think there is less pressure for stabilizing the SDK spec. Do need to decide on for example batching processor defaults right now? Seems like a low priority decision.

I would rather spend our efforts on figuring out the Events API first since this is where most of the unknowns are.

[tigrannajaryan]

One thing we need to decide now: in a hypothetical scenario where we eventually come to a decision that we don't need an "Events API" at all would the current structure of LoggerProvider/Logger API work fine on its own even if we delete everything related to "Events" from the API?

We may decide to support "Events" using some other form of helpers instead of making it a sibling API of Logger so we need to be prepared for such deletion.

[alanwest]

I think this means the entire Get a Logger needs to be marked experimental for now. At least in some languages adding/removing a parameter is a breaking change.

Marking Get a Logger stable is core to the premise of decoupling logs from events. I think this topic warrants a separate issue/PR, but my initial thought is the event_domain parameter just needs to be removed from the Get a Logger API. That would effectively decouple the Get a Logger API from anything event related. A new experimental API would need to be proposed in support of events.

would the current structure of LoggerProvider/Logger API work fine on its own even if we delete everything related to "Events" from the API?

Yes, this is my hope.

I think there is less pressure for stabilizing the SDK spec. Do need to decide on for example batching processor defaults right now? Seems like a low priority decision.

Ideally we move towards stabilizing the SDK to the extent possible as well. A stable SDK spec allows for a stable release of OTLP log exporters.

[jack-berg]

I've opened #2955 to discuss issue 5 in this list:

Consider the mutability of the ReadWriteLogRecord interface: Is everything mutable or only some things?

[tigrannajaryan]

@jack-berg I added 2 items for Java in the Remaining issues in the description. Please check the boxes when/if it is done.

[tigrannajaryan]

@open-telemetry/dotnet-maintainers I added 2 items for .Net in the Remaining issues in the description. Please check the boxes when/if it is done.

[tigrannajaryan]

@open-telemetry/python-maintainers I added 2 items for Python in the Remaining issues in the description. Please check the boxes when/if it is done.

[jack-berg]

Java maintainers to link to example also showing trace context injection

@tigrannajaryan could you elaborate on this? There's test code that validates trace context inject. Not sure if you're looking for something different / more specific.

[tigrannajaryan]

Java maintainers to link to example also showing trace context injection

@tigrannajaryan could you elaborate on this? There's test code that validates trace context inject. Not sure if you're looking for something different / more specific.

@jack-berg ideally we would want a minimal standalone example, but test code is probably also good enough. I just wanted to make sure we clearly demonstrate that trace context injection works with the current design. Please feel free to tick the boxes for Java.

[srikanthccv]

Python: logging handler implementation. Python maintainers to link to example also showing trace context injection

Handler implementation here. Where do we link the example? Here is an example that demonstrates https://github.com/open-telemetry/opentelemetry-python/tree/main/docs/examples/logs

Python: Log API and SDK prototype implementation

API and SDK prototype implementation. The above example shows how to use API + SDK + Handler + Exporter to glue everything up to send data to the collector

I think both items can be ticked.

[srikanthccv]

@open-telemetry/python-maintainers I added 2 items for Python in the Remaining issues in the description. Please check the boxes when/if it is done.

@tigrannajaryan I can't tick the boxes because of a permissions issue. Please tick the two items for Python.

[tigrannajaryan]

@open-telemetry/python-maintainers I added 2 items for Python in the Remaining issues in the description. Please check the boxes when/if it is done.

@tigrannajaryan I can't tick the boxes because of a permissions issue. Please tick the two items for Python.

I have just tried the latest Python example and can confirm it works as expected, including the trace context in logs. I marked Python items done. Thanks.

[jack-berg]

ideally we would want a minimal standalone example, but test code is probably also good enough. I just wanted to make sure we clearly demonstrate that trace context injection works with the current design.

There's a working example here as well that uses the same instrumentation, but applies it automatically using the java agent. It exports spans/metrics/logs to a collector running with the logging exporter. If you exercise the the app by invoking its endpoint, it will log a message in the context of a span, which you can see in the logs of the collector.

I'm going to check the boxes because while its always good to have more examples, I think the docs and the test code are a good enough description.

[tigrannajaryan]

.Net: Log API and SDK prototype implementation
.Net: Serilog appender implementation. .Net maintainers to link to example also showing trace context injection

@alanwest can you please tell if any of these are done? If not I will create separate issues for tracking.

[tigrannajaryan]

@CodeBlanch please ping me after you check .net prototype so that I can mark the relevant checkboxes.

[CodeBlanch]

@tigrannajaryan Here is the .NET prototype (main-logs branch):

• API: https://github.com/open-telemetry/opentelemetry-dotnet/tree/main-logs/src/OpenTelemetry.Api/Logs
• SDK: https://github.com/open-telemetry/opentelemetry-dotnet/tree/main-logs/src/OpenTelemetry/Logs
• Serilog appender: https://github.com/open-telemetry/opentelemetry-dotnet/tree/main-logs/src/OpenTelemetry.Extensions.Serilog
• EventSource appender: https://github.com/open-telemetry/opentelemetry-dotnet/tree/main-logs/src/OpenTelemetry.Extensions.EventSource

I did those a while back before the split out of the Events API but as far as LoggerProvider/Logger/LogRecord it is all there and the spec met our needs for the prototype 👍

[tigrannajaryan]

I am adding #3301 as a pre-requisite.

[trask]

does Log API/SDK stabilization include Trace Context in Legacy Formats?

if so, can you add consideration of #3303 as a pre-requisite?

[jack-berg]

@trask that's a good callout. Thinking about it more, this seems like its more appropriate to define as part of semantic conventions, or perhaps as a section under compatibility. I don't think we should gate the Logs Bridge API / SDK stability on this.

[tigrannajaryan]

Added #3386 as a requirement.

[tigrannajaryan]

Added #3387 to the checklist.

[tigrannajaryan]

Added #3385 to the checklist.

[tigrannajaryan]

Added #3394 to the checklist.

[jack-berg]

Replaced #3387 with #3397. Marked #3397 and #3385 as completed.

[jack-berg]

@open-telemetry/dotnet-maintainers, @open-telemetry/python-maintainers, @open-telemetry/javascript-maintainers, @open-telemetry/cpp-maintainers: a couple of changes have been made recently to the log bridge API / SDK spec. Can you review #3386, #3397, #3385 and ensure that they are implementable, and confirm by commenting on this issue?

• cpp
• dotnet
• java
• javascript
• php
• python

[srikanthccv]

Can you review #3386, #3397, #3385 and ensure that they are implementable, and confirm by commenting on this issue?

These changes are implementable in Python.

[alanwest]

Can you review #3386, #3397, #3385 and ensure that they are implementable, and confirm by commenting on this issue?

Looks good from .NET's perspective.

[brettmc]

Can you review #3386, #3397, #3385 and ensure that they are implementable, and confirm by commenting on this issue?

No problems for the PHP implementation.

[tigrannajaryan]

Need to also merge #3383

[pichlermarc]

Can you review #3386, #3397, #3385 and ensure that they are implementable, and confirm by commenting on this issue?

As far as I can tell this should be no problem for the JS implementation.
cc-ing @martinkuba for confirmation 🙂

[dyladan]

Looks fine to me as well

[lalitb]

Can you review #3386, #3397, #3385 and ensure that they are implementable, and confirm by commenting on this issue?

Looks good from C++ prospective.

[jack-berg]

Completed in #3376!