Invalid multibyte character

I am getting this error in logstash.log. Logstash is not sending any logs to the elasticsearch nodes (tcpdump shows no traffic to them).

I'm using the redis module to pull data in, index it, and send it to elasticsearch nodes.

{:timestamp=>"2015-06-23T17:09:04.145000+0000", :message=>"+---------------------------------------------------------+\n| An unexpected error occurred. This is probably a bug. |\n| You can find help with this problem in a few places: |\n| |\n| * chat: #logstash IRC channel on freenode irc. |\n| IRC via the web: http://goo.gl/TI4Ro |\n| * email: logstash-users@googlegroups.com |\n| * bug system: https://logstash.jira.com/ |\n| |\n+---------------------------------------------------------+\nThe error reported is: \n invalid multibyte character: /(?.?) (?\b\w+\b) (?(?\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))(.?|\b)))-(?\b\w+\b): (?(?(?>\d\d){1,2})./-./-- ) [(?([Aa]lert|ALERT|[Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nfo|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?))] (?\b(?:[1-9][0-9])\b)#(?(?:(?(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:.[0-9]+)?)|(?:.[0-9]+)))))): (?.)(?:, client: (?(?(?:(?((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))(%.+)?)|(?(?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})...)(?![0-9]))))|(?\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))(.?|\b))))(?:, server: (?(?:(?\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))(.?|\b))|(?(?:(?((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))(%.+)?)|(?(?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})...)(?![0-9])))))))(?:, request: (?(?(?>(?<!\)(?>"(?>\.|[^\\\"]+)+"|""|(?>'(?>\.|[^\\']+)+')|''|(?>(?>\\\\.|[^\\\\]+)+)|``)))))?(?:, host: (?<a28>(?<a29>(?>(?<!\\\\)(?>\"(?>\\\\.|[^\\\\\"]+)+\"|\"\"|(?>'(?>\\\\.|[^\\\\']+)+')|''|(?>(?>\.|[^\]+)+)|``)))))?(?:, referrer: \â\u0080\u009D(?(?[A-Za-z]+(+[A-Za-z+]+)?)://(?:(?(?[a-zA-Z0-9.-]+))(?::[^@])?@)?(?:(?(?(?:(?\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))(.?|\b))|(?(?:(?((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))(%.+)?)|(?(?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})...)(?![0-9]))))))(?::(?\b(?:[1-9][0-9])\b))?))?(?:(?(?(?:/[A-Za-z0-9$.+!'(){},~:;=@#%-])+)(?:(??[A-Za-z0-9$.+!'|(){},~@#%&/=:;_?-[]]*))?))?))/"}

[suyograo]

@tdm4 can you paste your config here and sample input data?

/etc/logstash/conf.d/indexer.conf:

input {
redis {
host => "(hostname of REDIS server)"
data_type => "list"
key => "logstash"
threads => 2
}
}

filter {

# Handle Magento's monstrous logs
if "system.log" in [tags] {
  grok {
        match   => ["message", "%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{WORD:severity}%{SPACE}\S+%{INT:slevel:int}\S+%{SPACE}%{WORD:logtype}{1,2}\S+%{GREEDYDATA:message}"]
        overwrite => [ "message" ]
  }

} else if "exception.log" in [tags] {
  grok {
        match   => ["message", "(?m)%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{WORD:severity}%{SPACE}\S+%{INT:slevel:int}\S+\s+\n[Ee]xception\s+'%{WORD:excepttype}'\s+with\s+message\s+%{GREEDYDATA:message}\nStack\s+[Tt]race\S+\n%{GREEDYDATA:stack}"]
        overwrite => [ "message" ]
  }

  grok {
        match => ["message", "'%{DATA:errmessage}[:']%{GREEDYDATA}"]
  }
    }

    if "web_nginx_access" in [tags] {

      grok {
            match => [
              "message", "%{IPORHOST:http_host} %{IPORHOST:clientip} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{NUMBER:request_time:float} %{NUMBER:upstream_time:float}",
              "message", "%{IPORHOST:http_host} %{IPORHOST:clientip} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{NUMBER:request_time:float}"
            ]
      }
      date {
            match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
      }
      geoip {
            source => "clientip"
      }
    }

    if "web_nginx_error" in [tags] {

      grok {
        match => [
          "message" , "%{DATA} %{WORD:webserver} %{HOST:myhost}\-%{WORD:class}\: (?%{YEAR}[./-]%{MONTHNUM}[./-]%{MONTHDAY}[- ]%{TIME}) \[%{LOGLEVEL:severity}\] %{POSINT:pid}#%{NUMBER}: %{GREEDYDATA:errormessage}(?:, client: (?%{IP}|%{HOSTNAME}))(?:, server: %{IPORHOST:server})(?:, request: %{QS:request})?(?:, host: %{QS:host})?(?:, referrer: \”%{URI:referrer})"
        ]
      }
    }

}

output {
stdout { codec => rubydebug }

elasticsearch {
host => "(IP of internal load balancer going to 2 elasticsearch nodes)"
protocol => "http"
}
}

I am collecting nginx error logs, Magento system.log and exception.log. Examples:

system.log:
2015-06-24T08:13:59+00:00 ERR (3): Warning: array_key_exists() expects parameter 2 to be array, null given in /var/www/somedomain/site/app/code/core/Mage/Captcha/Model/Observer.php on line 166

exception.log:
2015-06-24T05:30:41+00:00 ERR (3):
exception 'Mage_Core_Exception' with message 'Invalid block type: Mage_Page_Block_Html_CookieNotice' in /var/www/somedomain/site/app/Mage.php:594
Stack trace:
#0 /var/www/somedomain/site/app/code/core/Mage/Core/Model/Layout.php(495): Mage::throwException('Invalid block t...')
#1 /var/www/somedomain/site/app/code/core/Mage/Core/Model/Layout.php(437): Mage_Core_Model_Layout->_getBlockInstance('page/html_cooki...', Array)
#2 /var/www/somedomain/site/app/code/core/Mage/Core/Model/Layout.php(472): Mage_Core_Model_Layout->createBlock('page/html_cooki...', 'global_cookie_n...')
#3 /var/www/somedomain/site/app/code/core/Mage/Core/Model/Layout.php(239): Mage_Core_Model_Layout->addBlock('page/html_cooki...', 'global_cookie_n...')
#4 /var/www/somedomain/site/app/code/core/Mage/Core/Model/Layout.php(205): Mage_Core_Model_Layout->_generateBlock(Object(Mage_Core_Model_Layout_Element), Object(Mage_Core_Model_Layout_Element))
#5 /var/www/somedomain/site/app/code/core/Mage/Core/Model/Layout.php(206): Mage_Core_Model_Layout->generateBlocks(Object(Mage_Core_Model_Layout_Element))
#6 /var/www/somedomain/site/app/code/core/Mage/Core/Model/Layout.php(206): Mage_Core_Model_Layout->generateBlocks(Object(Mage_Core_Model_Layout_Element))
#7 /var/www/somedomain/site/app/code/core/Mage/Core/Controller/Varien/Action.php(344): Mage_Core_Model_Layout->generateBlocks()
#8 /var/www/somedomain/site/app/code/core/Mage/Catalog/controllers/CategoryController.php(146): Mage_Core_Controller_Varien_Action->generateLayoutBlocks()
#9 /var/www/somedomain/site/app/code/core/Mage/Core/Controller/Varien/Action.php(419): Mage_Catalog_CategoryController->viewAction()
#10 /var/www/somedomain/site/app/code/core/Mage/Core/Controller/Varien/Router/Standard.php(254): Mage_Core_Controller_Varien_Action->dispatch('view')
#11 /var/www/somedomain/site/app/code/core/Mage/Core/Controller/Varien/Front.php(176): Mage_Core_Controller_Varien_Router_Standard->match(Object(Mage_Core_Controller_Request_Http))
#12 /var/www/somedomain/site/app/code/core/Mage/Core/Model/App.php(354): Mage_Core_Controller_Varien_Front->dispatch()
#13 /var/www/somedomain/site/app/Mage.php(683): Mage_Core_Model_App->run(Array)
#14 /var/www/somedomain/site/index.php(102): Mage::run('somedomain', 'website')
#15 {main}

nginx error.log:
2015/06/24 09:15:05 [error] 22116#0: *6129340 directory index of "/var/www/local/" is forbidden, client: 127.0.0.1, server: , request: "GET / HTTP/1.1", host: "localhost"
2015/06/24 09:16:59 [error] 22116#0: *6129661 open() "/var/www/somedomain/site/apple-touch-icon.png" failed (2: No such file or directory), client: 149.254.180.73, server: www.somedomain.co.uk, request: "GET /apple-touch-icon.png HTTP/1.1", host: "www.somedomain.co.uk"
2015/06/24 09:17:09 [error] 22116#0: *6129686 open() "/var/www/somedomain/site/apple-touch-icon-precomposed.png" failed (2: No such file or directory), client: 149.254.180.73, server: www.somedomain.co.uk, request: "GET /apple-touch-icon-precomposed.png HTTP/1.1", host: "www.somedomain.co.uk"

I tried turning off various sources.. and adding them back in one by one and it doesn't matter which one.. as soon as beaver ships to logstash, that error occurs with the multibyte.

OK I narrowed it down to something in the nginx error.log config above. I basically took this config out of indexer.conf and it now seems to index now.

Basically, this block in indexer.conf is problematic:

if "web_nginx_error" in [tags] {

  grok {
    match => [
      "message" , "%{DATA} %{WORD:webserver} %{HOST:myhost}\-%{WORD:class}\: (?%{YEAR}[./-]%{MONTHNUM}[./-]%{MONTHDAY}[- ]%{TIME}) \[%{LOGLEVEL:severity}\] %{POSINT:pid}#%{NUMBER}: %{GREEDYDATA:errormessage}(?:, client: (?%{IP}|%{HOSTNAME}))(?:, server: %{IPORHOST:server})(?:, request: %{QS:request})?(?:, host: %{QS:host})?(?:, referrer: \”%{URI:referrer})"
    ]
  }
}

Anyone know how I can fix this?

I tried to use the nginx error.log grok from https://logstash.jira.com/browse/LOGSTASH-1663 but it seems broken now.

Turns out it was a bad paste. It pasted a funky UTF-8 left double quote and that really screwed up logstash when it parsed the indexer.conf file.