Tail plugin does not parse the path if path contains folder with multibyte characters (e.g. Chinese) on windows while it works on fluentd image in linux docker.

[deepaksb2]

Describe the bug

We have a scenario where log folder path contains the folder with multibyte characters (let’s say Chinese character 日志), the tail plugin fails to parse the path and logs are not collected.

This behavior we have only seen in windows environment (Td-agent version 4.3.1) which it just works fine with docker image of fluentd on Linux environment.

To Reproduce

Configure td-agent.conf with source tail plugin with path input which contains folder with multibyte character (C:\logs\日志文件夹\0_gen_apache.log). This only happens with Windows fluentD.

Expected behavior

FluentD should start tailing the logs when path contains multibyte characters on windows environment

Your Environment

- Fluentd version: 1.14.6 
- TD Agent version: 4.3.1
- Operating system: Windows Server 2012 R2 and 2019
- Kernel version:

Your Configuration

<system>
    <log>
      rotate_age 30
    </log>
  </system>
  <source>
    @type tail
    path "C:\logs\日志文件夹\0_gen_apache.log"
    pos_file "C:\logs\日志文件夹\0_gen_apache.log.pos"
    tag "TSO.gridlog"
    <parse>
      @type "apache2"
      unmatched_lines 
    </parse>
  </source>
  <match TSO.gridlog>
    @type stdout
    <format>
      @type "json"
    </format>
  </match>
  <match debug.**>
    @type stdout
    @id output_stdout
  </match>

Your Error Log

2022-05-29 22:59:20 -1200 [info]: starting fluentd-1.14.6 pid=7780 ruby="2.7.6"
2022-05-29 22:59:20 -1200 [info]: spawn command to main:  cmdline=["C:/opt/td-agent/bin/ruby.exe", "-Eascii-8bit:ascii-8bit", "C:/opt/td-agent/bin/fluentd", "-c", "C:\\opt\\td-agent\\etc\\td-agent\\td-agent.conf", "-o", "C:\\opt\\td-agent\\td-agent.log", "-x", "fluentdwinsvc", "--under-supervisor"]
2022-05-29 22:59:23 -1200 [info]: adding match pattern="TSO.gridlog" type="stdout"
2022-05-29 22:59:23 -1200 [info]: adding match pattern="debug.**" type="stdout"
2022-05-29 22:59:23 -1200 [info]: adding source type="tail"
2022-05-29 22:59:23 -1200 [info]: adding source type="forward"
2022-05-29 22:59:24 -1200 [info]: adding source type="http"
2022-05-29 22:59:24 -1200 [info]: adding source type="debug_agent"
2022-05-29 22:59:24 -1200 [info]: #0 starting fluentd worker pid=2368 ppid=7780 worker=0
2022-05-29 22:59:24 -1200 [info]: #0 [input_debug_agent] listening dRuby uri="druby://127.0.0.1:24230" object="Fluent::Engine" worker=0
2022-05-29 22:59:24 -1200 [info]: #0 [input_forward] listening port port=24224 bind="0.0.0.0"
2022-05-29 22:59:24 -1200 [warn]: #0 expand_paths: stat() for C:\logs\日志文件夹\0_gen_apache.log failed with Errno::ENOENT. Skip file.
2022-05-29 22:59:24 -1200 [warn]: #0 expand_paths: stat() for C:\logs\日志文件夹\0_gen_apache.log failed with Errno::ENOENT. Skip file.
2022-05-29 22:59:24 -1200 [info]: #0 fluentd worker is now running worker=0
2022-05-29 23:00:24 -1200 [warn]: #0 expand_paths: stat() for C:\logs\日志文件夹\0_gen_apache.log failed with Errno::ENOENT. Skip file.
2022-05-29 23:01:24 -1200 [warn]: #0 expand_paths: stat() for C:\logs\日志文件夹\0_gen_apache.log failed with Errno::ENOENT. Skip file.
2022-05-29 23:02:24 -1200 [warn]: #0 expand_paths: stat() for C:\logs\日志文件夹\0_gen_apache.log failed with Errno::ENOENT. Skip file.
2022-05-29 23:03:24 -1200 [warn]: #0 expand_paths: stat() for C:\logs\日志文件夹\0_gen_apache.log failed with Errno::ENOENT. Skip file.
2022-05-29 23:04:24 -1200 [warn]: #0 expand_paths: stat() for C:\logs\日志文件夹\0_gen_apache.log failed with Errno::ENOENT. Skip file.
2022-05-29 23:05:24 -1200 [warn]: #0 expand_paths: stat() for C:\logs\日志文件夹\0_gen_apache.log failed with Errno::ENOENT. Skip file. 
-------------------------------------------------------------------------------------------------------
if we have the custom configuration then it throws below error (i am not sure why it is different)
2022-05-25 21:11:31 +0800 [error]: #3 unexpected error error_class=Fluent::Win32Error error="Fluent::Win32Error: code: 123, The filename, directory name, or volume label syntax is incorrect. - C:/\u65E5\u5FD7\u6587\u4EF6\u5939/catalina.log"
  2022-05-25 21:11:31 +0800 [error]: #3 C:/connector0524/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/file_wrapper.rb:126:in `initialize'
  2022-05-25 21:11:31 +0800 [error]: #3 C:/connector0524/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/file_wrapper.rb:31:in `new'
  2022-05-25 21:11:31 +0800 [error]: #3 C:/connector0524/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/file_wrapper.rb:31:in `stat'
  2022-05-25 21:11:31 +0800 [error]: #3 C:/connector0524/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:342:in `block in expand_paths'
  2022-05-25 21:11:31 +0800 [error]: #3 C:/connector0524/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:338:in `each'
  2022-05-25 21:11:31 +0800 [error]: #3 C:/connector0524/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:338:in `expand_paths'
  2022-05-25 21:11:31 +0800 [error]: #3 C:/connector0524/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:246:in `start'
  2022-05-25 21:11:31 +0800 [error]: #3 C:/connector0524/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:203:in `block in start'
  2022-05-25 21:11:31 +0800 [error]: #3 C:/connector0524/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:192:in `block (2 levels) in lifecycle'
  2022-05-25 21:11:31 +0800 [error]: #3 C:/connector0524/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:191:in `each'
  2022-05-25 21:11:31 +0800 [error]: #3 C:/connector0524/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:191:in `block in lifecycle'
  2022-05-25 21:11:31 +0800 [error]: #3 C:/connector0524/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:178:in `each'
  2022-05-25 21:11:31 +0800 [error]: #3 C:/connector0524/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:178:in `lifecycle'
  2022-05-25 21:11:31 +0800 [error]: #3 C:/connector0524/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:202:in `start'
  2022-05-25 21:11:31 +0800 [error]: #3 C:/connector0524/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/engine.rb:248:in `start'
  2022-05-25 21:11:31 +0800 [error]: #3 C:/connector0524/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/engine.rb:147:in `run'
  2022-05-25 21:11:31 +0800 [error]: #3 C:/connector0524/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:720:in `block in run_worker'
  2022-05-25 21:11:31 +0800 [error]: #3 C:/connector0524/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:971:in `main_process'
  2022-05-25 21:11:31 +0800 [error]: #3 C:/connector0524/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:711:in `run_worker'
  2022-05-25 21:11:31 +0800 [error]: #3 C:/connector0524/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/command/fluentd.rb:376:in `<top (required)>'
  2022-05-25 21:11:31 +0800 [error]: #3 C:/connector0524/opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
  2022-05-25 21:11:31 +0800 [error]: #3 C:/connector0524/opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
  2022-05-25 21:11:31 +0800 [error]: #3 C:/connector0524/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/bin/fluentd:15:in `<top (required)>'
  2022-05-25 21:11:31 +0800 [error]: #3 C:/connector0524/opt/td-agent/bin/fluentd:23:in `load'
  2022-05-25 21:11:31 +0800 [error]: #3 C:/connector0524/opt/td-agent/bin/fluentd:23:in `<main>'
2022-05-25 21:11:31 +0800 [error]: #3 unexpected error error_class=Fluent::Win32Error error="Fluent::Win32Error: code: 123, The filename, directory name, or volume label syntax is incorrect. - C:/\u65E5\u5FD7\u6587\u4EF6\u5939/catalina.log"

Additional context

This failure only occurs in Windows based td-agent and works as expected on Linux (Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux")

[ashie]

Thanks for your report.

Ah, yes, it seems in_tail's bug.
It passes UTF-8 encoded path string to CreateFile:

fluentd/lib/fluent/plugin/file_wrapper.rb

Line 137 in 12c15f5

@file_handle = CreateFile.call(@path, access, sharemode,