Parser doesn't work with fluent-plugin-multi-format-parser

[geekofalltrades]

When this parser is configured as a subparser under the multi-format parser plugin, it doesn't work. Adding -v and -vv to fluentd produces no additional logs indicating why.

I am running fluentd 1.4 (from the official Docker container fluent/fluentd:v1.4-debian) and adding the two plugins like so:

FROM fluent/fluentd:v1.4-debian

USER root

RUN apt update && apt install -y build-essential

RUN fluent-gem install fluent-plugin-multi-format-parser
RUN fluent-gem install fluent-plugin-keyvalue-parser

Here's an example of the key-value format I'm trying to parse:

# key=value pairs, space-separated
this=should totally=work

# values may contain whitespace, in which case they are quoted
field="contains quoted whitespace"

This log format works when I pass it to the keyvalue parser directly with this configuration:

<filter **>
    @type parser
    key_name message
    hash_value_field parsed
    reserve_data true

    <parse>
        @type keyvalue
        pair_delimiter " "
        key_value_seperator "="
    </parse>
</filter>

However, this is only one of several formats that log messages from this <source> may have, so I am trying to nest this parser as one of several under the multi-format parser, and that's not working. I have tried with two different configurations, which break in different ways.

<filter **>
    @type parser
    key_name message
    hash_value_field parsed
    reserve_data true

    <parse>
        @type multi_format
        <pattern>
            format keyvalue
            pair_delimiter " "
            key_value_seperator "="
        </pattern>
        <pattern>
            format json
        </pattern>
        <pattern>
            format none
        </pattern>
    </parse>
</filter>

In this configuration, messages always pass through the keyvalue parser directly to the JSON parser, even if they plainly match. There are no log messages from fluentd about why this is happening.

<filter **>
    @type parser
    key_name message
    hash_value_field parsed
    reserve_data true

    <parse>
        @type multi_format
        <pattern>
            format keyvalue
            pair_delimiter " "
            key_value_seperator "="
        </pattern>
    </parse>
</filter>

Here, I cut out the other parsers to see if I could get more information about why the keyvalue parser is failing, and I do: the message doesn't match, and fluentd logs a pattern not match error message, even though the event plainly matches.

I'm not sure whether the problem is with this plugin, the multi-format plugin, or just my configuration.

[geekofalltrades]

An AWS CloudWatch engineer we've been working on this with discovered the source of the problem:

I found the reason why the kv plugin you were using is not compatible with the multi_format plugin.

https://github.com/repeatedly/fluent-plugin-multi-format-parser/blob/master/lib/fluent/plugin/parser_multi_format.rb#L31

just change “&&” to “||” it works because this kv plugin doesn’t handle the time variable.

Adding a link to a tagged version so that the file doesn't change out from under the previous link: https://github.com/repeatedly/fluent-plugin-multi-format-parser/blob/v1.0.0/lib/fluent/plugin/parser_multi_format.rb#L31