Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

parser_syslog has a naming conflict on parser_type #3296

Closed
nbertram opened this issue Mar 16, 2021 · 2 comments · Fixed by #3302
Closed

parser_syslog has a naming conflict on parser_type #3296

nbertram opened this issue Mar 16, 2021 · 2 comments · Fixed by #3302
Assignees
@ashie
Labels
bug Something isn't working

Comments

@nbertram
Copy link

nbertram commented Mar 16, 2021 •
edited
Loading

Describe the bug
The parser_syslog plugin accepts a parameter parser_type (equal to string or regexp):

fluentd/lib/fluent/plugin/parser_syslog.rb

Line 59 in 436c7da

config_param :parser_type, :enum, list: [:regexp, :string], default: :regexp

This configuration key has a naming conflict with a method that's also called parser_type defined here:

fluentd/lib/fluent/plugin/parser.rb

Line 104 in 436c7da

def parser_type

When used in conjunction with an input plugin that changes how it calls parse() depending on the value of parser_type(), it receives a value that's outside PARSER_TYPES, namely string or regexp rather than say :text_per_line

I reproduced this with input_exec input plugin, which falls back to calling parse() with the entire input buffer if the parser plugin doesn't return :text_per_line from parser_type(). This causes the parser to get confused (all lines end up in the message) if the exec exits, or seems to completely not parse anything if the exec'd process is long-running (presumably because the fallback buffer relies on the exec'd command exiting before it passes the buffer over).

To Reproduce

This config standalone is enough to reproduce the issue:

<source>
    @type exec
    command printf '%bMar 16 21:28:01 hostname CRON[6477]: blah\nMar 16 21:28:11 hostname puppet-agent[1388]: foo'
    tag test
    <parse>
        @type syslog
        with_priority false
        parser_type string
        message_format rfc3164
    </parse>
</source>
<match *>
    @type stdout
</match>

Expected behavior
The above config should output:

2021-03-16 21:28:01.000000000 +0000 test: {"host":"hostname","ident":"CRON","pid":"6477","message":"blah"}
2021-03-16 21:28:11.000000000 +0000 test: {"host":"hostname","ident":"puppet-agent","pid":"1388","message":"foo"}

Instead it outputs:

1970-01-01 00:00:00.000000000 +0000 test: {"host":"hostname","ident":"CRON","pid":"6477","message":"blah\nMar 16 21:28:11 hostname puppet-agent[1388]: foo"}

Your Environment

  • Fluentd or td-agent version: td-agent 4.1.0 fluentd 1.12.1 (e3effa3)
  • Operating system: Debian Buster
  • Kernel version: 4.19.118

Your Configuration
Similar to the above, though with a different command that's long-running and streams syslog.

Your Error Log

2021-03-16 09:16:37 +0000 [info]: fluent/log.rb:329:info: parsing config file is succeeded path="/etc/td-agent/td-agent.conf"
2021-03-16 09:16:37 +0000 [info]: fluent/log.rb:329:info: gem 'fluent-plugin-elasticsearch' version '4.3.3'
2021-03-16 09:16:37 +0000 [info]: fluent/log.rb:329:info: gem 'fluent-plugin-flowcounter-simple' version '0.1.0'
2021-03-16 09:16:37 +0000 [info]: fluent/log.rb:329:info: gem 'fluent-plugin-kafka' version '0.16.0'
2021-03-16 09:16:37 +0000 [info]: fluent/log.rb:329:info: gem 'fluent-plugin-prometheus' version '1.8.5'
2021-03-16 09:16:37 +0000 [info]: fluent/log.rb:329:info: gem 'fluent-plugin-prometheus_pushgateway' version '0.0.2'
2021-03-16 09:16:37 +0000 [info]: fluent/log.rb:329:info: gem 'fluent-plugin-record-modifier' version '2.1.0'
2021-03-16 09:16:37 +0000 [info]: fluent/log.rb:329:info: gem 'fluent-plugin-rewrite-tag-filter' version '2.3.0'
2021-03-16 09:16:37 +0000 [info]: fluent/log.rb:329:info: gem 'fluent-plugin-s3' version '1.5.1'
2021-03-16 09:16:37 +0000 [info]: fluent/log.rb:329:info: gem 'fluent-plugin-sd-dns' version '0.1.0'
2021-03-16 09:16:37 +0000 [info]: fluent/log.rb:329:info: gem 'fluent-plugin-stdin' version '0.1.2'
2021-03-16 09:16:37 +0000 [info]: fluent/log.rb:329:info: gem 'fluent-plugin-systemd' version '1.0.2'
2021-03-16 09:16:37 +0000 [info]: fluent/log.rb:329:info: gem 'fluent-plugin-td' version '1.1.0'
2021-03-16 09:16:37 +0000 [info]: fluent/log.rb:329:info: gem 'fluent-plugin-webhdfs' version '1.4.0'
2021-03-16 09:16:37 +0000 [info]: fluent/log.rb:329:info: gem 'fluentd' version '1.12.1'
2021-03-16 09:16:37 +0000 [trace]: fluent/log.rb:286:trace: registered output plugin 'stdout'
2021-03-16 09:16:38 +0000 [trace]: fluent/log.rb:286:trace: registered buffer plugin 'memory'
2021-03-16 09:16:38 +0000 [trace]: fluent/log.rb:286:trace: registered formatter plugin 'stdout'
2021-03-16 09:16:38 +0000 [trace]: fluent/log.rb:286:trace: registered formatter plugin 'json'
2021-03-16 09:16:39 +0000 [trace]: fluent/log.rb:286:trace: registered input plugin 'exec'
2021-03-16 09:16:39 +0000 [trace]: fluent/log.rb:286:trace: registered parser plugin 'syslog'
2021-03-16 09:16:39 +0000 [debug]: fluent/log.rb:308:debug: No fluent logger for internal event
2021-03-16 09:16:39 +0000 [info]: fluent/log.rb:329:info: using configuration file: <ROOT>
  <source>
    @type exec
    command "printf \'%bMar 16 21:28:01 hostname CRON[6477]: blah\\nMar 16 21:28:11 hostname puppet-agent[1388]: foo\'"
    tag "test"
    <parse>
      @type "syslog"
      with_priority false
      parser_type string
      message_format rfc3164
    </parse>
  </source>
  <match *>
    @type stdout
  </match>
</ROOT>
2021-03-16 09:16:39 +0000 [info]: fluent/log.rb:329:info: starting fluentd-1.12.1 pid=3683 ruby="2.7.2"
2021-03-16 09:16:39 +0000 [info]: fluent/log.rb:329:info: spawn command to main:  cmdline=["/opt/td-agent/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/sbin/td-agent", "-vv", "--under-supervisor"]
2021-03-16 09:16:50 +0000 [info]: fluent/log.rb:329:info: adding match pattern="*" type="stdout"
2021-03-16 09:16:51 +0000 [trace]: #0 fluent/log.rb:286:trace: registered output plugin 'stdout'
2021-03-16 09:16:51 +0000 [trace]: #0 fluent/log.rb:286:trace: registered buffer plugin 'memory'
2021-03-16 09:16:51 +0000 [trace]: #0 fluent/log.rb:286:trace: registered formatter plugin 'stdout'
2021-03-16 09:16:51 +0000 [trace]: #0 fluent/log.rb:286:trace: registered formatter plugin 'json'
2021-03-16 09:16:51 +0000 [info]: fluent/log.rb:329:info: adding source type="exec"
2021-03-16 09:16:51 +0000 [trace]: #0 fluent/log.rb:286:trace: registered input plugin 'exec'
2021-03-16 09:16:51 +0000 [trace]: #0 fluent/log.rb:286:trace: registered parser plugin 'syslog'
2021-03-16 09:16:51 +0000 [debug]: #0 fluent/log.rb:308:debug: No fluent logger for internal event
2021-03-16 09:16:51 +0000 [info]: #0 fluent/log.rb:329:info: starting fluentd worker pid=3688 ppid=3683 worker=0
2021-03-16 09:16:51 +0000 [debug]: #0 fluent/log.rb:308:debug: Executing command title=:exec_input spawn=[{}, "printf '%bMar 16 21:28:01 hostname CRON[6477]: blah\\nMar 16 21:28:11 hostname puppet-agent[1388]: foo'"] mode=[:read] stderr=:discard
2021-03-16 09:16:51 +0000 [info]: #0 fluent/log.rb:329:info: fluentd worker is now running worker=0
1970-01-01 00:00:00.000000000 +0000 test: {"host":"hostname","ident":"CRON","pid":"6477","message":"blah\nMar 16 21:28:11 hostname puppet-agent[1388]: foo"}
@ashie ashie added the bug Something isn't working label Mar 16, 2021
@ashie
Copy link
Member

ashie commented Mar 16, 2021

parser_csv plugin seems have same issue.

@ashie
Copy link
Member

ashie commented Mar 17, 2021

Strategy to fix it:

  • rename the config name to other name like parser_format
  • make parser_type an alias for it to keep compatibility
    • aliases doesn't seem override parent's method

@ashie ashie self-assigned this Mar 17, 2021
ashie added a commit to ashie/fluentd that referenced this issue Mar 30, 2021
@ashie
parser_csv, parser_syslog: Fix a naming conflict on parser_type
5c84037 
These plugins accept a parameter `plugin_type` but it hides the parent
class's method `Fluent::Plugin::Parser#parser_type`, it causes
unexpected behavior on a plugin which uses this method such as in_exec.

Fix fluent#3296

Signed-off-by: Takuro Ashie <[email protected]>
@ashie ashie mentioned this issue Mar 30, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ashie ashie

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

parser_csv, parser_syslog: Fix a naming conflict on parser_type ashie/fluentd
2 participants
@nbertram @ashie