in_tail: add option to skip long lines

[kenhys]

Which issue(s) this PR fixes:

Fixes #

What this PR does / why we need it:

There is a case that processing unexpected long line causes a
BufferChunkOverflow exception.

With #3562 (Fluentd v1.14.3), it can just ignore only problematic long
record, but it always raise an exception. Thus it has performance
regression.

In this fix, skip_long_lines option enables to just skip processing
event further more.

Docs Changes:

It should be documented in in_tail.

Release Note:

[fujimotos]

Overall, I'm okay with this patch. I added some comment inline, plese check them out.

[kenhys]

I've missed the additional comment.
Hmm, skip_long_lines is explicitly set by system admin, so logging info is redundant,
in the contrast to it, it is also reasonable user want to know 'which line is skipped'.

[kenhys]

Revised.

[ashie]

2 tests are failed:

2021-11-30T09:51:10.4473858Z ===============================================================================
2021-11-30T09:51:10.9880823Z F
2021-11-30T09:51:10.9882009Z ===============================================================================
2021-11-30T09:51:10.9892254Z Failure: test: skip_long_lines[small](TailInputTest::receive_lines)
2021-11-30T09:51:10.9908987Z /home/runner/work/fluentd/fluentd/test/plugin/test_in_tail.rb:1733:in `block (2 levels) in <class:TailInputTest>'
2021-11-30T09:51:10.9909875Z      1730:       t = Time.parse("Mon Nov 29 11:22:33 JST 2021")
2021-11-30T09:51:10.9910401Z      1731:       stub(Time).now { t }
2021-11-30T09:51:10.9910877Z      1732:       d.run(expect_records: 2)
2021-11-30T09:51:10.9911456Z   => 1733:       assert_equal([
2021-11-30T09:51:10.9911928Z      1734:                      [{"message" => "foo"},{"message" => "bar"}],
2021-11-30T09:51:10.9912367Z      1735:                      [
2021-11-30T09:51:10.9913266Z      1736:                        "2021-11-29 11:22:33 +0900 [warn]: received line length is longer than #{size}\n",
2021-11-30T09:51:10.9915271Z <[[{"message"=>"foo"}, {"message"=>"bar"}],
2021-11-30T09:51:10.9916426Z  ["2021-11-29 11:22:33 +0900 [warn]: received line length is longer than 128\n",
2021-11-30T09:51:10.9921979Z   "2021-11-29 11:22:33 +0900 [debug]: skipped line: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n"]]> expected but was
2021-11-30T09:51:10.9927250Z <[[{"message"=>"foo"}, {"message"=>"bar"}],
2021-11-30T09:51:10.9928138Z  ["2021-11-29 11:22:33 +0000 [warn]: received line length is longer than 128\n",
2021-11-30T09:51:10.9933730Z   "2021-11-29 11:22:33 +0000 [debug]: skipped line: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n"]]>
2021-11-30T09:51:10.9938701Z 
2021-11-30T09:51:10.9939079Z diff:
2021-11-30T09:51:10.9939539Z   [[{"message"=>"foo"}, {"message"=>"bar"}],
2021-11-30T09:51:11.0219476Z ?  ["2021-11-29 11:22:33 +09000 [warn]: received line length is longer than 128\n",
2021-11-30T09:51:11.0370814Z ?   "2021-11-29 11:22:33 +09000 [debug]: skipped line: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n"]]
2021-11-30T09:51:11.0375617Z ===============================================================================
2021-11-30T09:51:11.1507446Z F
2021-11-30T09:51:11.1508058Z ===============================================================================
2021-11-30T09:51:11.1509017Z Failure: test: skip_long_lines[KiB](TailInputTest::receive_lines)
2021-11-30T09:51:11.1530474Z /home/runner/work/fluentd/fluentd/test/plugin/test_in_tail.rb:1733:in `block (2 levels) in <class:TailInputTest>'
2021-11-30T09:51:11.1531922Z      1730:       t = Time.parse("Mon Nov 29 11:22:33 JST 2021")
2021-11-30T09:51:11.1532857Z      1731:       stub(Time).now { t }
2021-11-30T09:51:11.1534219Z      1732:       d.run(expect_records: 2)
2021-11-30T09:51:11.1534993Z   => 1733:       assert_equal([
2021-11-30T09:51:11.1535856Z      1734:                      [{"message" => "foo"},{"message" => "bar"}],
2021-11-30T09:51:11.1536625Z      1735:                      [
2021-11-30T09:51:11.1538066Z      1736:                        "2021-11-29 11:22:33 +0900 [warn]: received line length is longer than #{size}\n",
2021-11-30T09:51:11.1539127Z <[[{"message"=>"foo"}, {"message"=>"bar"}],
2021-11-30T09:51:11.1540489Z  ["2021-11-29 11:22:33 +0900 [warn]: received line length is longer than 1024\n",
2021-11-30T09:51:11.1859463Z   "2021-11-29 11:22:33 +0900 [debug]: skipped line: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n"]]> expected but was
2021-11-30T09:51:11.2111918Z <[[{"message"=>"foo"}, {"message"=>"bar"}],
2021-11-30T09:51:11.2113295Z  ["2021-11-29 11:22:33 +0000 [warn]: received line length is longer than 1024\n",
2021-11-30T09:51:11.2368502Z   "2021-11-29 11:22:33 +0000 [debug]: skipped line: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n"]]>

[cosmo0920]

I'm OK for this PR patch but parameter name should be changed as different name that represents this situation.

[ashie]

LGTM