Configurable line terminator

[kvch]

This PR add a new input configuration option named line_terminator:

# Characters which separate the lines. Valid values: auto, line_feed, vertical_tab, form_feed,
# carriage_return, carriage_return_line_feed, next_line, line_separator, paragraph_separator.
#line_terminator: auto

The option auto tells Filebeat to use our current hybrid new line finder approach. Thus, we can avoid introducing a breaking change.

It also contains a minor refactoring in readfile package. I have created a new type Config which stores the configuration of the readers of the package. This eliminates a long list of parameters in the constructors of EncodeReader and LineReader.

Closes #5500

[kvch]

jenkins test this

[kvch]

jenkins test this

[kvch]

jenkins test this

[kvch]

jenkins test this

[urso]

Looks like a breaking change. Old behavior did support \r\n and \n based line terminators. It used to split on \n first and remove trailing \r\n or \n characters from the line.

This change seems to make it 'less' dynamic and this sense. What will be the default for Windows where \r\n is more common?

[kvch]

It is indeed less dynamic. This PR requires the input file to use one kind of newline. Previously the following input file was readable:

hello\n
hello\r\n

Also if someone has different files with different newlines, e.g.

hello\n
hello\n

and

hello\r\n
hello\r\n

he/she was able to add it to the same input.

By default on Windows the configured newline is carriage_return_line_feed, on other OSes the default is line_feed.

To make the configuration backward compatible, a new option auto (for a lack of a better name) could be added which is basically our current newline handling. WDYT?

[urso]

SGTM.

Alternatively we can have 2 patterns. One 'split' pattern and one 'right-trim' pattern. This way we're not limited to newlines anymore, but could support more complex patterns for split based multiline.