Support access logging with Common Log Format/Extended Log Format

[muratg]

From @CesarBS on February 3, 2017 18:6

Every web server does this:

http://httpd.apache.org/docs/current/logs.html#accesslog

http://redmine.lighttpd.net/projects/1/wiki/Docs_ModAccessLog

http://nginx.org/en/docs/http/ngx_http_log_module.html#access_log

https://www.iis.net/configreference/system.applicationhost/log/centralw3clogfile

There are tools that analyze logs using those formats.

For reference:

https://en.wikipedia.org/wiki/Common_Log_Format

https://www.w3.org/TR/WD-logfile.html

Copied from original issue: aspnet/KestrelHttpServer#1339

[muratg]

From @Tratcher on February 3, 2017 18:13

Would this be better as middleware? Is there any information middleware wouldn't have access to?

[muratg]

From @CesarBS on February 3, 2017 18:14

With Kestrel at least, middleware wouldn't log bad requests rejected before the app runs.

[muratg]

@shirhatti @DamianEdwards thoughts?

[muratg]

From @DamianEdwards on February 3, 2017 21:16

This is interesting. I wasn't aware the log file format was a standard. Do we think we'd just do this internally in Kestrel or would we try to use ILoggerFactory somehow, e.g. have Kestrel support using its own ILoggerFactory instance which would have an ILoggerProvider configured for this format by default, but could add any other provider to it too via API.

[muratg]

@DamianEdwards Yeah, this is a simple format and there are a bunch of analysis tools since forever that can parse this format and provide information.

We discussed different options for implementing this (just doing it internally and writing a file, providing a stream that user can do whatever they want with, using the existing logging abstractions) -- each has pros and cons.

[muratg]

From @DamianEdwards on February 3, 2017 21:31

Would be good to capture the different approaches and their pros and cons.

[muratg]

From @CesarBS on February 3, 2017 21:35

@DamianEdwards The advantage of using existing log infrastructure is the flexibility and if people use Serilog they get log file rotation and such. The downside is that setting it up is more involved than just offering an option in KestrelServerOptions where they set the file name or Stream to write the logs to.

[muratg]

From @DamianEdwards on February 3, 2017 21:50

How would it work using ILogger? Would Kestrel do all the formatting itself internally and just log the messages through the current ILoggerFactory to whatever providers were configured? Is that enough to get the correct output?

[muratg]

From @CesarBS on February 17, 2017 0:10

Yeah, Kestrel would have to do the formatting. If a logger were to intercept the messages and decorate it in any way, that would make the message useless since it couldn't be parsed by tools that look into these logs.

[muratg]

From @halter73 on February 17, 2017 0:31

Decorated messages wouldn't exactly be useless, but they wouldn't fit the common log format. Presumably if you're using a logger that decorates messages, you aren't to interested in the common log format to begin with. There's also nothing stopping developers from configuring multiple logger providers: one that decorates and another that doesn't for common logs.

I think that common logs should have a unique source so providers can be configured to filter out everything else. Being able to use existing providers to get features like rolling log files, logging to some other persistence layer, etc. certainly would be nice.

[muratg]

From @benaadams on February 17, 2017 0:42

Is it an external component though? The while the layout is standard the output is configurable

For example IIS's options for W3C logging

The fields would have to be identifiable so they can be switched on and off

[muratg]

From @CesarBS on February 17, 2017 1:16

@benaadams Yeah, that's the extended format. The first W3C log format had fixed fields, but modern servers also support Extended Log Format which allows you to specify which fields you want logged per access.

We'd have to add a ServerOptions.AccessLogFormat option, like Apache's LogFormat directive: http://httpd.apache.org/docs/current/logs.html#accesslog

[muratg]

@shirhatti Needs design (though lower pri for 2.0.0)

[cwe1ss]

How about doing this with a DiagnosticListener and one of the existing DiagnosticSource events (EndRequest, HttpRequestIn.Stop)? Requests rejected by the server (e.g. Kestrel) should call some method on Hosting that then raises another DiagnosticSource event - which would be interesting for many other use cases as well anyway.

[davidfowl]

@cwe1ss That's the idea. The one thing we need to figure out is rejected requests that don't make it into hosting.