Structured variables not shown when Serilog used

[vladopandzic]

I get logs to cloudwatch however variables are not recognized inside aws cloudwatch.
This is my configuration:

"serilog": {
    "Using": [
      "AWS.Logger.SeriLog"
    ],
    "LogGroup": "rimacSD-alecs-api-production",
    "IncludeLogLevel": true,
    "IncludeCategory": true,
    "IncludeNewline": true,
    "IncludeException": true,
    "IncludeEventId": true,
    "IncludeScopes": true,
    "Region": "eu-west-1",
    "MinimumLevel": {
      "Default": "Information",
      "Override": {
        "Microsoft": "Warning",
        "System": "Warning",
        "Microsoft.Hosting.Lifetime": "Information"
      }
    },
    "Enrich": [ "FromLogContext", "WithMachineName", "WithEnvironmentUserName", "FromLogContext" ],
    "WriteTo": [
      {
        "Name": "AWSSeriLog",
        
      }
    ]
  },

If I use alternative Serilog.Sinks.AwsCloudWatch then I get variables inside cloudwatch but I can't figure out how to remove logs
from Microsoft and would rather use this official library.

Inide Program.cs I have this:

        public static IHostBuilder CreateHostBuilder(string[] args) =>
            Host.CreateDefaultBuilder(args)
                .UseSerilog((hostingContext, services, loggerConfiguration) =>
                {
                    _ = loggerConfiguration.ReadFrom.Configuration(hostingContext.Configuration)
                                        .Enrich.WithProperty("Application", hostingContext.HostingEnvironment.ApplicationName)
                                        .Enrich.WithProperty("Environment", hostingContext.HostingEnvironment.EnvironmentName);
                })
                .ConfigureWebHostDefaults(webBuilder =>
                {
                    _ = webBuilder.UseStartup<Startup>();
                });
    }

[ashishdhingra]

Hi @vladop,

Good morning.

Thanks for posting the issue. Could you please elaborate what variables you are referring to and provide some code sample to investigate the issue? Any logs would also be helpful. Are you referring to enrichers FromLogContext, WithMachineName and WithEnvironmentUserName?

Preferred format for sharing above information:

Expected Behavior

Current Behavior

Possible Solution

Steps to Reproduce (for bugs)

Context

Your Environment

• AWSSDK.Core version used:
• Service assembly and version used:
• Operating System and version:
• Visual Studio version:
• Targeted .NET platform:

.NET Core Info

• .NET Core version used for development:
• .NET Core version installed in the environment where application runs:
• Output of dotnet --info:
• Contents of project.json/project.csproj:

Thanks,
Ashish

[github-actions]

This issue has not received a response in a week. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled.

[ghost]

I'm also seeing an issue with structured logging variables not showing up with the .Net Core API, so I don't think it's Serilog specific. I have the following configuration:

builder.Services.AddLogging(config =>
{
    config.AddAWSProvider(builder.Configuration.GetAWSLoggingConfigSection());
    config.SetMinimumLevel(LogLevel.Trace);
});

Here is my appSettings.json:

  "AWS.Logging": {
    "Region": "us-east-2",
    "LogGroup": "WidgetService",
    "LogLevel": {
      "Default": "Information",
      "System": "Information",
      "Microsoft": "Information"
    }
  },

I'm logging the following structured log:

_logger.LogError("error test message {Test} {Field1} {Field2}", "Text", "1", 42);

I would expect this to be sent as a structured log to CloudWatch, but it appears to send an interpolated string:

Here, I would expect to see the variables:
@test Text
@field1 1
@field2 42

It appears the library doesn't preserve the message. I'm guessing this is what the OP was expressing, that when they use Serilog with Serilog's AWS sink that it correctly preserves the variables, but when using the "official" microsoft library for logging to CloudWatch with Serilog that it isn't preserving the variables. Seems like it's because the core library just doesn't properly support structured logging.

[ghost]

This is a follow-up to my previous comment.

My cloud experience to date has primarily been in Azure. Upon seeing AWS CloudWatch's interface, I presumed it was facilitating discrete custom fields the way Azure AppInsights does. Using Microsoft's AppInsights logging provider, it takes the structured logging data and maps it to the custom properties for the log entry. After learning a bit more about CloudWatch, I'm now thinking that it doesn't quite work that way. It seems it has the notion of standard fields and "discovered" fields, but it it doesn't seem to deal with custom fields in quite the same way as AppInsights. If this is indeed the way it works, it would seem that the CloudWatch provider really has no other recourse but to use.Net Core's structured logging data for string interpolation, in which case it isn't a bug in this library so much as just a limitation of CloudWatch.

[ashishdhingra]

Needs review with team to investigate structured logging and how it could be supported by AWS logging libraries.

[wahmedswl]

This is really a problem because if we need to push message to Standard Log-group then we are doomed. We are using Serilog console sink and it works pretty well but it logs to the lambda group.
We need to push messages to standard log-group so we implemented using AWS.Logger.SeriLog but its pushing messages as raw. This is the event different between Serilog console sink vs AWS.Logger.SeriLog.

If json is written to message just like serilog console sink is doing, everything will work but AWS.Logger.SeriLog is doing text transform on the log message but all context fields are lost. Only thing pushed to logs is formatted message, Information("Serilog with Lambda logs to CloudWatch with Level = Information worked on Iteration {Iteration}", i + 1);

[wahmedswl]

There might be something OFF the way normal implementation works but this article gives the right way to implement structured logging for Cloudwatch.

https://www.rahulpnath.com/blog/amazon-cloudwatch-logs-dotnet/

[louisobasohan]

There might be something OFF the way normal implementation works but this article gives the right way to implement structured logging for Cloudwatch.

https://www.rahulpnath.com/blog/amazon-cloudwatch-logs-dotnet/

The steps in the article do not still produce structured logs. Does anyone have an idea on how to address this issue? When I run the application locally and log in to cloud watch, the structured logs are shown correctly, but when the lambda is deployed and runs, the logs show in plaintext, and all the enrich information are lost.

[wahmedswl]

@louisobasohan You can switch to Serilog.Sinks.AwsCloudWatch for proper structured logging.

[louisobasohan]

@louisobasohan You can switch to Serilog.Sinks.AwsCloudWatch for proper structured logging.

Serilog.Sinks.AwsCloudWatch is also giving me the same output. Also, I'm using .NET6

[wahmedswl]

Log.Logger = new LoggerConfiguration()
                .Destructure.JsonNetTypes()
                .ReadFrom.Configuration(configurationRoot)
                .WriteTo.Console(new CompactJsonFormatter())
                .WriteTo.Logger(x => x.Filter.ByIncludingOnly(y => y.Level <= LogEventLevel.Information).WriteTo.AmazonCloudWatch(new CloudWatchSinkOptions { LogGroupName = "ServerlessApp/Info", TextFormatter = new CompactJsonFormatter(), Period = TimeSpan.FromMilliseconds(100) }, new AmazonCloudWatchLogsClient(regionEndpoint)))
                .WriteTo.Logger(x => x.Filter.ByIncludingOnly(y => y.Level > LogEventLevel.Information).WriteTo.AmazonCloudWatch(new CloudWatchSinkOptions { LogGroupName = "ServerlessApp/Error", TextFormatter = new CompactJsonFormatter(), Period = TimeSpan.FromMilliseconds(100) }, new AmazonCloudWatchLogsClient(regionEndpoint)))
                .CreateLogger();

I guess you are missing the Formatter.

[louisobasohan]

Log.Logger = new LoggerConfiguration()
                .Destructure.JsonNetTypes()
                .ReadFrom.Configuration(configurationRoot)
                .WriteTo.Console(new CompactJsonFormatter())
                .WriteTo.Logger(x => x.Filter.ByIncludingOnly(y => y.Level <= LogEventLevel.Information).WriteTo.AmazonCloudWatch(new CloudWatchSinkOptions { LogGroupName = "ServerlessApp/Info", TextFormatter = new CompactJsonFormatter(), Period = TimeSpan.FromMilliseconds(100) }, new AmazonCloudWatchLogsClient(regionEndpoint)))
                .WriteTo.Logger(x => x.Filter.ByIncludingOnly(y => y.Level > LogEventLevel.Information).WriteTo.AmazonCloudWatch(new CloudWatchSinkOptions { LogGroupName = "ServerlessApp/Error", TextFormatter = new CompactJsonFormatter(), Period = TimeSpan.FromMilliseconds(100) }, new AmazonCloudWatchLogsClient(regionEndpoint)))
                .CreateLogger();

I guess you are missing the Formatter.

Thanks Wahmedswl, writing to console with the JsonFormatter solved the issue. I didnt need to add the additional filters.

Log.Logger = new LoggerConfiguration()
                .ReadFrom.Configuration(configurationRoot)
                .WriteTo.Console(new CompactJsonFormatter())
                .CreateLogger();

[wahmedswl]

@louisobasohan Yes, if you writing to the log group of Lambda, you just need Serilog.Sinks.Console. If you need to write to other/centralized log group, then you need to use Serilog.Sinks.AwsCloudWatch.