Td-Agent wrongly interprets character coding

[jiribaloun]

Hello there,

Describe the bug

I'm having issues with encoding characters coming from Docker daemon, running on MS Windows 2019 Core Edition and TD-Agent, running on Centos 7.8

To Reproduce

td-agent.conf:

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

<filter **>
  @type record_modifier
  char_encoding utf-8
</filter>

<filter docker.**>
  @type parser
  format json
  key_name log
  reserve_data true
</filter>

<filter docker.**>
  @type concat
  key log
  separator ""
  stream_identity_key container_id
  multiline_start_regexp /^-e:2:in `\/'/
  multiline_end_regexp /^-e:4:in/
</filter>

Docker run:

• docker run --log-driver=fluentd --log-opt fluentd-address=10.170.163.172:24224 -ti mcr.microsoft.com/windows/nanoserver:1809 cmd

Output of logs:

2020-07-16 13:27:22.000000000 +0000 5976659d20bc: {"container_name":"/dreamy_perlman","source":"stdout","log":"\u001b[165X\u001b[165C\u001b[4;5H\u001b[?25h\u001b[?25lh \b\u001b[?25h\u001b[HM\u001b[4;6H\u001b[?25lo \b\u001b[?25h\u001b[HM\u001b[4;7H\u001b[?25ls \b\u001b[?25h\u001b[HM\u001b[4;8H\u001b[?25lt \b\u001b[?25h\u001b[HM\u001b[4;9H\u001b[?25ln \b\u001b[?25h\u001b[HM\u001b[4;10H\u001b[?25la \b\u001b[?25h\u001b[HM\u001b[4;11H\u001b[?25lm \b\u001b[?25h\u001b[HM\u001b[4;12H\u001b[?25le \b\u001b[?25h\u001b[HM\u001b[4;13H\u001b[?25l\u001b[HM\u001b[5;1H\u001b[?25h\u001b[?25l\u001b[HMicrosoft Windows [Version 10.0.17763.1339]\u001b[122X\u001b[122C\r","container_id":"5976659d20bc9b2d201f4cc01a287c2fe2f03464abcf0d3a12a4770fe298799f"}
2020-07-16 13:27:22.000000000 +0000 5976659d20bc: {"source":"stdout","log":"(c) 2018 Microsoft Corporation. All rights reserved.\u001b[113X\u001b[113C\r","container_id":"5976659d20bc9b2d201f4cc01a287c2fe2f03464abcf0d3a12a4770fe298799f","container_name":"/dreamy_perlman"}

Expected behavior

Log lines captured by td-agent, recoded correctly to utf-8, ready to forward to ELK.

Your Environment

• td-agent 1.11.1
• NAME="CentOS Linux"
  VERSION="7 (Core)"
  ID="centos"
  ID_LIKE="rhel fedora"
  VERSION_ID="7"
  PRETTY_NAME="CentOS Linux 7 (Core)"
  ANSI_COLOR="0;31"
  CPE_NAME="cpe:/o:centos:centos:7"
  HOME_URL="https://www.centos.org/"
  BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

• Linux 3.10.0-1127.13.1.el7.x86_64 force fluentd/fluent-gem scripts to use installed ruby interpreter  #1 SMP Tue Jun 23 15:46:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux