Grok's "break_on_match => false" option does not work

[AlexMonson]

Logstash v1.4.2, testing on Windows.
Unless I'm misunderstanding what's meant to happen, Grok's break_on_match => false does not work correctly. The second pattern is never matched when the first one is.

CONFIG FILE

input { stdin{} }
filter {
    grok {
        break_on_match => false
        match => [
            "message", "%{GREEDYDATA:name1}beard",
            "message", "tree%{GREEDYDATA:name2}"
        ]
    }
}
output { stdout { codec => rubydebug } }

INPUT1

treebranch

OUTPUT1

{
       "message" => "treebranch\r",
      "@version" => "1",
    "@timestamp" => "2014-07-17T15:31:52.488Z",
          "host" => "hostyhost",
         "name2" => "branch\r"
}

VERDICT1
That was as expected. The first pattern was not matched but the second one was, so the field name2 is created.

INPUT2

bushbeard

OUTPUT2

{
       "message" => "bushbeard\r",
      "@version" => "1",
    "@timestamp" => "2014-07-17T15:33:27.792Z",
          "host" => "hostyhost",
         "name1" => "bush"
}

VERDICT2
That was as expected. The first pattern was matched but the second one was not, so the field name1 is created.

INPUT3

treebeard

OUTPUT3

{
       "message" => "treebeard\r",
      "@version" => "1",
    "@timestamp" => "2014-07-17T15:33:50.543Z",
          "host" => "hostyhost",
         "name1" => "tree"
}

VERDICT3
That was not as expected. The first pattern was matched, so name1 was created properly but then the second pattern was never attempted or failed so name2 is missing.

This is what was expected:

{
       "message" => "treebeard\r",
      "@version" => "1",
    "@timestamp" => "2014-07-17T15:33:50.543Z",
          "host" => "hostyhost",
         "name1" => "tree"
         "name2" => "beard"
}

[coolacid]

In my investigation into this, the problem seems to stem from here: https://github.com/elasticsearch/logstash/blob/master/lib/logstash/filters/grok.rb#L281

In this case, the do loop only loops once because patterns is not a list of patterns to try, even though: Grok compile {:field=>"message", :patterns=>["%{GREEDYDATA:name1}beard", "tree%{GREEDYDATA:name2}"], :level=>:info, :file=>"logstash/filters/grok.rb", :line=>"264"}

The problem stems from:
https://github.com/elasticsearch/logstash/blob/master/lib/logstash/filters/grok.rb#L268

This creates the @patterns array - alas, it creates it against the field, not the pattern. As such, because in the example above there is only one field referenced, it will create an array of one pattern compiled with both patterns in it (which is why 1 and 2 work as expected).

[AlexMonson]

Excellent, thanks for looking into it. Do you think you know how to fix it?

I'm surprised no one else has ever spotted that this doesn't work, unless it was from a recent change.

[suyograo]

@coolacid Thanks for this. I am looking into Grok these days and can take a stab at it

[coolacid]

@suyograo Hit me up on IRC for more details if you want them.

[AlexMonson]

@coolacid So I should expect to see this in the next release?
@suyograo thanks for your work!