One of the most common types of log input is tailing a file. The in_tail input plugin allows you to read from a text log file as though you were running the tail -f command. Full documentation on this plugin can be found here.
<source>
@type tail
<parse>
@type none
</parse>
path /home/logs/*
pos_file /home/logs/backend.application.pos
path_key filename
tag backend.application
</source>
In this tail example, we are declaring that the logs should not be parsed by seeting @type none. We are also adding a tag that will control routing. By setting tag backend.application we can specify filter and match blocks that will only process the logs from this one source. More details on how routing works in Fluentd can be found here.
Two other parameters are used here. Path_key is a value that the filepath of the log file data is gathered from will be stored into. So in this case, the log that appears in New Relic Logs will have an attribute called "filename" with the value of the log file data was tailed from. Pos_file is a database file that is created by Fluentd and keeps track of what log data has been tailed and successfully sent to the output. This helps to ensure that the all data from the log is read.
Another very common source of logs is syslog, This example will bind to all addresses and listen on the specified port for syslog messages.
<source>
@type syslog
port 5140
tag syslog.messages
</source>
In order to make previewing the logging solution easier, you can configure output using the out_copy plugin to wrap multiple output types, copying one log to both outputs.
<match **>
@type copy
<store>
@type file
path /var/log/testlog/testlog
</store>
<store>
@type newrelic
api_key blahBlahBlaHHABlablahabla
</store>
</match>
Sometimes you will have logs which you wish to parse. There is a set of built-in parsers listed here which can be applied. Some of the parsers like the nginx parser understand a common log format and can parse it "automatically." Others like the regexp parser are used to declare custom parsing logic. There is also a very commonly used 3rd party parser for grok that provides a set of regex macros to simplify parsing.
This next example is showing how we could parse a standard NGINX log we get from file using the in_tail plugin.
<source>
@type tail
<parse>
@type nginx
</parse>
path /var/log/nginx/error.log
tag nginx.error
</source>
Notice that we have chosen to tag these logs as nginx.error to help route them to a specific output and filter plugin after. If we wanted to apply custom parsing the grok filter would be an excellent way of doing it. In this next example, a series of grok patterns are used. The first pattern is %{SYSLOGTIMESTAMP:timestamp} which pulls out a timestamp assuming the standard syslog timestamp format is used. The next pattern grabs the log level and the final one grabs the remaining unnmatched txt. Each substring matched becomes an attribute in the log event stored in New Relic. This makes it possible to do more advanced monitoring and alerting later by using those attributes to filter, search and facet.
<source>
@type tail
<parse>
@type grok
<grok>
pattern %{SYSLOGTIMESTAMP:timestamp} %{LOGLEVEL:loglevel}: %{GREEDYDATA:message}
</grok>
</parse>
path /home/log/test.log
tag custom.application
</source>
Some logs have single entries which span multiple lines. Typically one log entry is the equivalent of one log line; but what if you have a stack trace or other long message which is made up of multiple lines but is logically all one piece? In that case you can use a multiline parser with a regex that indicates where to start a new log entry. A common start would be a timestamp; whenever the line begins with a timestamp treat that as the start of a new log entry. If the next line begins with something else, continue appending it to the previous log entry.
<filter backend.application>
@type parser
<parse>
@type multiline_grok
grok_failure_key grokfailure
multiline_start_regex ^abc
<grok>
pattern %{GREEDYDATA:message}
</grok>
</parse>
</filter>
The above example uses multiline_grok to parse the log line; another common parse filter would be the standard multiline parser. This is also the first example of using a . Multiple filters can be applied before matching and outputting the results. In the example, any line which begins with "abc" will be considered the start of a log entry; any line beginning with something else will be appended.
It is possible to add data to a log entry before shipping it. In Fluentd entries are called "fields" while in NRDB they are referred to as the attributes of an event. Different names in different systems for the same data. One important field for organizing your logs is the service_name field. This is a reserved field name in New Relic along with message.
<source>
@type syslog
port 5140
tag backend.application
</source>
<filter backend.application>
@type record_transformer
<record>
service_name ${tag}
hostname ${hostname}
</record>
</filter>
This example makes use of the record_transformer filter. It allows you to change the contents of the log entry (the record) as it passes through the pipeline. The field name is service_name and the value is a variable ${tag} that references the tag value the filter matched on. The tag value of backend.application set in the
Hostname is also added here using a variable. This syntax will only work in the record_transformer filter. If you are trying to set the hostname in another place such as a source block, use the following:
hostname "#{Socket.gethostname}"
The module filter_grep can be used to filter data in or out based on a match against the tag or a record value.
<filter backend.application>
@type grep
<regexp>
key service_name
pattern /backend.application/
</regexp>
</filter>
<filter backend.application>
@type grep
<regexp>
key sample_field
pattern /some_other_value/
</regexp>
</filter>
This example would only collect logs that matched the filter criteria for service_name. Multiple filters that all match to the same tag will be evaluated in the order they are declared. So in this example, logs which matched a service_name of backend.application_ and a sample_field value of some_other_value would be included.
<source>
@type tail
<parse>
@type none
</parse>
path /var/log/*
tag sample.tag
</source>
<filter sample.tag>
@type record_transformer
<record>
service_name ${tag}
</record>
</filter>
<match **>
@type newrelic
api_key <your key goes here>
</match>