MicrosoftExtensionsLogging: Add each log argument/attribute to the context data

[tylerohlsen]

Feature Description

Currently, with MEL, only log scopes are included in the context data (using logger.BeginScope(...)). The log attributes that are passed as arguments to the standard logger.Log* methods are not in the context data. They are included in the formatted message, but then they lose a lot of value because they need to be parsed back out during ingestion.

For example, explicit calls to the logger (or logger extension methods) like this do not have MyData as a context data value:

logger.LogInformation("My log message with some data: {MyData}", myData);

or via the Source Generator (generated) calls like this do not have the RouteData, MethodInfo, etc as context data values:

[LoggerMessage(102, LogLevel.Information, "Route matched with {RouteData}. Executing controller action with signature {MethodInfo} on controller {Controller} ({AssemblyName}).", EventName = "ControllerActionExecuting", SkipEnabledCheck = true)]
private static partial void ControllerActionExecuting(ILogger logger, string routeData, MethodInfo methodInfo, string controller, string? assemblyName);

Describe Alternatives

None considered

Additional context

I believe this already works as described with other loggers (e.g. Serilog and NLog), but I could be wrong.

Priority

Really Want

[workato-integration]

https://new-relic.atlassian.net/browse/NR-276392

[nrcventura]

@tylerohlsen thank you for submitting this feature request. I passed the information along to other members of the team so that it can be further reviewed.

[dolivanu]

Very useful to get the message attributes parsed by New Relic.

[tylerohlsen]

@nrcventura I'm not seeing a linked PR nor release notes of a related change that would include what was requested here. Could you point me to the PR or release that included this request? Or is there a different reason why this was closed?

[nrcventura]

Unfortunately, the tooling that product management uses for managing feature requests does not update github with meaningful information. From what I can tell in the other system, the feature request went from the "gathering interest" step to the "closed" step with a "resolution" of future consideration. I did not see any more information about it.

[tylerohlsen]

Well that's disappointing. I can't recommend using NewRelic without this feature. Too much very important contextual information is lost without it.

[dolivanu]

I agree with @tylerohlsen, this feature is already present in Python APM Agent and is the basis of Structured/Semantic Logging. Some projects use only Microsoft.Extensions.Logging without a logging framework or implementation like Serilog/NLog/log4net due to not requiring persisting/storing/forwarding the logs to any system except NR1. This approach is lighter and removes one dependency and some configuration effort.

[chynesNR]

Hi @tylerohlsen (and @dolivanu ) -- apologies for the lack of communication. We did some restructuring of our internal Jira board and some Feature Requests auto-closed, and I missed that this needed an update.

Unfortunately this is one of those issues that sounds simple but isn't in practice. Unlike e.g. Serilog, or even Serilog-backed-MEL, there's no practical place for us to grab the attributes (And even just the context data was a challenge!). If we had a path forward we would add this support but at the moment we don't.

Thank you for communicating that this is still important to you. I'll reopen this Feature Request and we'll take another stab at it.

[dolivanu]

Thanks @chynesNR, would be great if can provide extra clarifications:

1. To achieve the desired behaviour with NR, in your opinion, the best approach is to use MEL as Log API and Serilog as Log Framework, so the dependency with Serilog is minimized across the project.
2. When using MEL backed with Serilog, NR is able to capture the log attributes from the Serilog in order to add them to the context data. As Serilog is Open Source, could you review the source code of Serilog to get the path to capture the log attributes from MEL directly?

[chynesNR]

Hi @dolivanu -- It sounds like you might be better off using one of our other log forwarding options so you don't have to rely on in-Agent forwarding. If you do want to stick with in-Agent forwarding, I believe using Serilog.Extensions.Logging should work for you (i.e. giving you MEL and structured logging).
On your second point: yes, exactly. We are able to identify and pull out the necessary context data from Serilog (as well as NLog), but because MEL allows many implementations of ILogger we don't usually know where the context data is kept.