filter/xml fix for LOGSTASH-2246: extract non-ascii content with xpath

[wiibaa]

As reported in LOGSTASH-2246, xml filter xpath fails with non-ascii content,
here is a test case and fix.

It is due to Nokogiri to_s calling to_xhml or to_xml without encoding parameters.
using the to_str method extract the content differently (hidden in the C or Java implementation) and do not suffer encoding issue

[elasticsearch-release]

Can one of the admins verify this patch?

[jordansissel]

I can confirm that without the patch to filters/xml.rb, the test you added fails (as expected):

  1) LogStash::Filters::Xml parse correctly non ascii content with xpath "{"xmldata":"<foo><key>Français</key></foo>"}" when processed
     Failure/Error: pipeline.filter(e) {|new_event| results << new_event }
     Encoding::UndefinedConversionError:
       ""\xC3\xA7"" from UTF-8 to US-ASCII

[jordansissel]

I am reviewing to see if the mysterious to_str is the correct way to do this.

[jordansissel]

Nokogiri::XML is shorthand for Nokogiri::XML::Document.parse(), which has this method signature:

parse(string_or_io, url = nil, encoding = nil, options = ParseOptions::DEFAULT_XML, &block) Show Source
(docs here)

[jordansissel]

>> t = Nokogiri::XML("<foo><key>Français</key></foo>").xpath("/foo/key/text()")
>> t.to_s
Encoding::UndefinedConversionError: ""\xC3\xA7"" from UTF-8 to US-ASCII

Definitely fails! Nasty.

I think the parse encoding is the way to fix this:

>> t = Nokogiri::XML("<foo><key>Français</key></foo>", nil, "UTF-8").xpath("/foo")[0].to_s
=> "<foo><key>Français</key></foo>"
>> t = Nokogiri::XML("<foo><key>Français</key></foo>", nil, "UTF-8").xpath("/foo/key/text()")[0].to_s
=> "Français"

[jordansissel]

jordansissel@d5dfc5a is my proposal to fix this. It appears to work and in testing this still passes your new specs. Woo!

[jordansissel]

Closing in favor of #1803

[jordansissel]

@wiibaa thank you for figuring out the problem and solving it! <3

[wiibaa]

@jordansissel thanks you but the credits should go to the original reporter in jira.
While you are toying with xml filter filter, any chance to look at #1255 ;)

[samoin]

thanks you. I've tried to submit it to #1804.

------------------ 原始邮件 ------------------
发件人: "Wiibaa";[email protected];
发送时间: 2014年9月30日(星期二) 中午11:32
收件人: "elasticsearch/logstash"[email protected];

主题: Re: [logstash] filter/xml fix for LOGSTASH-2246: extract non-asciicontent with xpath (#1790)

@jordansissel thanks you but the credits should go to the original reporter in jira.
While you are toying with xml filter filter, any chance to look at #1255 ;)

—
Reply to this email directly or view it on GitHub.