[Q&A] error_class=Encoding::UndefinedConversionError ... from ASCII-8BIT to UTF-8

[MarekUniq]

What is a problem?

I want that incoming logs are treated as UTF-8.

I get error message:

2024-12-10 14:56:25 +0000 [error]: #0 unexpected error on reading data host="192.168.112.1" port=49374 error_class=Encoding::UndefinedConversionError error="\"\\xC4\" from ASCII-8BIT to UTF-8"

Another docker container is sending logs to fluentd container using fluentd driver:

...
    logging:
      driver: fluentd
      options:
        fluentd-address: 127.0.0.1:24224
        tag: fluent-test

Simple log lines reach destination as expected. Example of log line:

{ "severity": "INFO", "message": "Example successful log message" }

Log entries which contain non-ascii symbols fail (not expected). Example of log line:

{ "severity": "ERROR", "message": "Example of failing message (Česká republika)" }

I could impact ruby (Encoding.default_external) thru environment variable LANG=C.UTF-8 but based on the logs I can see that encoding ascii-8bit looks like hard-coded for Encoding.default_external and Encoding.default_internal. Take a look at the argument "-Eascii-8bit:ascii-8bit" below:

2024-12-10 14:56:19 +0000 [info]: starting fluentd-1.18.0 pid=8 ruby="3.2.6"
2024-12-10 14:56:19 +0000 [info]: spawn command to main:  cmdline=["/usr/local/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/local/bundle/bin/fluentd", "--config", "/fluentd/etc/fluent.conf", "--plugin", "/fluentd/plugins", "--under-supervisor"]

Describe the configuration of Fluentd

Fluentd docker image is started:

docker-compose --file ./fluentd.yml up -d

File fluentd.yml:

services:
  fluentd:
    image: fluent-custom-image:latest
    container_name: fluentd
    ports:
      - 127.0.0.1:24224:24224
    volumes:
      - type: bind
        source: /home/user/mount
        target: /fluentd

File fluent.conf:

<source>
  @type forward
</source>

<filter *>
  @type docker
</filter>

<match *>
  @type google_cloud_logging_on_prem
  project_id myproject
</match>

Dockerfile to generate image: fluent-custom-image:latest:

FROM fluent/fluentd:edge-debian
USER root
RUN apt update -y  &&  buildDeps="make gcc g++ libc-dev ruby-dev"  \
    &&  apt install -y --no-install-recommends ${buildDeps} \
    &&  gem install fluent-plugin-google-cloud-logging-on-prem \
    &&  gem install fluent-plugin-docker \
    &&  gem sources --clear-all \
    &&  apt purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false ${buildDeps} \
    &&  rm -rf /var/lib/apt/lists/*
ENV GOOGLE_APPLICATION_CREDENTIALS=/fluentd/etc/application_default_credentials.json

Describe the logs of Fluentd

2024-12-10 14:56:19 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2024-12-10 14:56:19 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2024-12-10 14:56:19 +0000 [info]: gem 'fluentd' version '1.18.0'
2024-12-10 14:56:19 +0000 [info]: gem 'fluent-plugin-docker' version '0.1.0'
2024-12-10 14:56:19 +0000 [info]: gem 'fluent-plugin-google-cloud-logging-on-prem' version '0.1.2'
2024-12-10 14:56:19 +0000 [info]: gem 'fluentd' version '1.16.6'
2024-12-10 14:56:19 +0000 [info]: using configuration file: <ROOT>
  <source>
    @type forward
  </source>
  <filter *>
    @type docker
  </filter>
  <match *>
    @type google_cloud_logging_on_prem
    project_id "myproject"
  </match>
</ROOT>
2024-12-10 14:56:19 +0000 [info]: starting fluentd-1.18.0 pid=8 ruby="3.2.6"
2024-12-10 14:56:19 +0000 [info]: spawn command to main:  cmdline=["/usr/local/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/local/bundle/bin/fluentd", "--config", "/fluentd/etc/fluent.conf", "--plugin", "/fluentd/plugins", "--under-supervisor"]
2024-12-10 14:56:20 +0000 [info]: #0 init worker0 logger path=nil rotate_age=nil rotate_size=nil
2024-12-10 14:56:20 +0000 [info]: adding filter pattern="*" type="docker"
2024-12-10 14:56:20 +0000 [info]: adding match pattern="*" type="google_cloud_logging_on_prem"
2024-12-10 14:56:20 +0000 [info]: adding source type="forward"
2024-12-10 14:56:20 +0000 [info]: #0 starting fluentd worker pid=17 ppid=8 worker=0
2024-12-10 14:56:20 +0000 [info]: #0 listening port port=24224 bind="0.0.0.0"
2024-12-10 14:56:20 +0000 [info]: #0 fluentd worker is now running worker=0
2024-12-10 14:56:25 +0000 [warn]: #0 emit transaction failed: error_class=Encoding::UndefinedConversionError error="\"\\xC4\" from ASCII-8BIT to UTF-8" location="/usr/local/bundle/gems/google-cloud-logging-2.3.3/lib/google/cloud/logging/convert.rb:47:in `initialize'" tag="fluent-test"
  2024-12-10 14:56:25 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.18.0/lib/fluent/event_router.rb:198:in `rescue in emit_events'
  2024-12-10 14:56:25 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.18.0/lib/fluent/event_router.rb:195:in `emit_events'
  2024-12-10 14:56:25 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.18.0/lib/fluent/event_router.rb:115:in `emit_stream'
  2024-12-10 14:56:25 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.18.0/lib/fluent/event_router.rb:106:in `emit'
  2024-12-10 14:56:25 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.18.0/lib/fluent/plugin/in_forward.rb:355:in `on_message'
  2024-12-10 14:56:25 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.18.0/lib/fluent/plugin/in_forward.rb:226:in `block in handle_connection'
  2024-12-10 14:56:25 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.18.0/lib/fluent/plugin/in_forward.rb:263:in `block (3 levels) in read_messages'
  2024-12-10 14:56:25 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.18.0/lib/fluent/plugin/in_forward.rb:262:in `feed_each'
  2024-12-10 14:56:25 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.18.0/lib/fluent/plugin/in_forward.rb:262:in `block (2 levels) in read_messages'
  2024-12-10 14:56:25 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.18.0/lib/fluent/plugin/in_forward.rb:271:in `block in read_messages'
  2024-12-10 14:56:25 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.18.0/lib/fluent/plugin_helper/server.rb:649:in `on_read_without_connection'
  2024-12-10 14:56:25 +0000 [warn]: #0 /usr/local/bundle/gems/cool.io-1.9.0/lib/cool.io/io.rb:123:in `on_readable'
  2024-12-10 14:56:25 +0000 [warn]: #0 /usr/local/bundle/gems/cool.io-1.9.0/lib/cool.io/io.rb:186:in `on_readable'
  2024-12-10 14:56:25 +0000 [warn]: #0 /usr/local/bundle/gems/cool.io-1.9.0/lib/cool.io/loop.rb:88:in `run_once'
  2024-12-10 14:56:25 +0000 [warn]: #0 /usr/local/bundle/gems/cool.io-1.9.0/lib/cool.io/loop.rb:88:in `run'
  2024-12-10 14:56:25 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.18.0/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
  2024-12-10 14:56:25 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.18.0/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2024-12-10 14:56:25 +0000 [error]: #0 unexpected error on reading data host="192.168.112.1" port=49374 error_class=Encoding::UndefinedConversionError error="\"\\xC4\" from ASCII-8BIT to UTF-8"
  2024-12-10 14:56:25 +0000 [error]: #0 /usr/local/bundle/gems/google-cloud-logging-2.3.3/lib/google/cloud/logging/convert.rb:47:in `initialize'
  2024-12-10 14:56:25 +0000 [error]: #0 /usr/local/bundle/gems/google-cloud-logging-2.3.3/lib/google/cloud/logging/convert.rb:47:in `new'
  2024-12-10 14:56:25 +0000 [error]: #0 /usr/local/bundle/gems/google-cloud-logging-2.3.3/lib/google/cloud/logging/convert.rb:47:in `object_to_value'
  2024-12-10 14:56:25 +0000 [error]: #0 /usr/local/bundle/gems/google-cloud-logging-2.3.3/lib/google/cloud/logging/convert.rb:27:in `block in hash_to_struct'
  2024-12-10 14:56:25 +0000 [error]: #0 /usr/local/bundle/gems/google-cloud-logging-2.3.3/lib/google/cloud/logging/convert.rb:27:in `to_h'
  2024-12-10 14:56:25 +0000 [error]: #0 /usr/local/bundle/gems/google-cloud-logging-2.3.3/lib/google/cloud/logging/convert.rb:27:in `hash_to_struct'
  2024-12-10 14:56:25 +0000 [error]: #0 /usr/local/bundle/gems/google-cloud-logging-2.3.3/lib/google/cloud/logging/convert.rb:53:in `object_to_value'
  2024-12-10 14:56:25 +0000 [error]: #0 /usr/local/bundle/gems/google-cloud-logging-2.3.3/lib/google/cloud/logging/convert.rb:27:in `block in hash_to_struct'
  2024-12-10 14:56:25 +0000 [error]: #0 /usr/local/bundle/gems/google-cloud-logging-2.3.3/lib/google/cloud/logging/convert.rb:27:in `to_h'
  2024-12-10 14:56:25 +0000 [error]: #0 /usr/local/bundle/gems/google-cloud-logging-2.3.3/lib/google/cloud/logging/convert.rb:27:in `hash_to_struct'
  2024-12-10 14:56:25 +0000 [error]: #0 /usr/local/bundle/gems/google-cloud-logging-2.3.3/lib/google/cloud/logging/entry.rb:507:in `append_payload'
  2024-12-10 14:56:25 +0000 [error]: #0 /usr/local/bundle/gems/google-cloud-logging-2.3.3/lib/google/cloud/logging/entry.rb:442:in `to_grpc'
  2024-12-10 14:56:25 +0000 [error]: #0 /usr/local/bundle/gems/google-cloud-logging-2.3.3/lib/google/cloud/logging/project.rb:302:in `map'
  2024-12-10 14:56:25 +0000 [error]: #0 /usr/local/bundle/gems/google-cloud-logging-2.3.3/lib/google/cloud/logging/project.rb:302:in `write_entries'
  2024-12-10 14:56:25 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-google-cloud-logging-on-prem-0.1.2/lib/fluent/plugin/out_google_cloud_logging_on_prem.rb:28:in `block in process'
  2024-12-10 14:56:25 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.18.0/lib/fluent/event.rb:196:in `block in each'
  2024-12-10 14:56:25 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.18.0/lib/fluent/event.rb:195:in `each'
  2024-12-10 14:56:25 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.18.0/lib/fluent/event.rb:195:in `each'
  2024-12-10 14:56:25 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-google-cloud-logging-on-prem-0.1.2/lib/fluent/plugin/out_google_cloud_logging_on_prem.rb:19:in `process'
  2024-12-10 14:56:25 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.18.0/lib/fluent/plugin/output.rb:885:in `emit_sync'
  2024-12-10 14:56:25 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.18.0/lib/fluent/event_router.rb:196:in `emit_events'
  2024-12-10 14:56:25 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.18.0/lib/fluent/event_router.rb:115:in `emit_stream'
  2024-12-10 14:56:25 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.18.0/lib/fluent/event_router.rb:106:in `emit'
  2024-12-10 14:56:25 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.18.0/lib/fluent/plugin/in_forward.rb:355:in `on_message'
  2024-12-10 14:56:25 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.18.0/lib/fluent/plugin/in_forward.rb:226:in `block in handle_connection'
  2024-12-10 14:56:25 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.18.0/lib/fluent/plugin/in_forward.rb:263:in `block (3 levels) in read_messages'
  2024-12-10 14:56:25 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.18.0/lib/fluent/plugin/in_forward.rb:262:in `feed_each'
  2024-12-10 14:56:25 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.18.0/lib/fluent/plugin/in_forward.rb:262:in `block (2 levels) in read_messages'
  2024-12-10 14:56:25 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.18.0/lib/fluent/plugin/in_forward.rb:271:in `block in read_messages'
  2024-12-10 14:56:25 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.18.0/lib/fluent/plugin_helper/server.rb:649:in `on_read_without_connection'
  2024-12-10 14:56:25 +0000 [error]: #0 /usr/local/bundle/gems/cool.io-1.9.0/lib/cool.io/io.rb:123:in `on_readable'
  2024-12-10 14:56:25 +0000 [error]: #0 /usr/local/bundle/gems/cool.io-1.9.0/lib/cool.io/io.rb:186:in `on_readable'
  2024-12-10 14:56:25 +0000 [error]: #0 /usr/local/bundle/gems/cool.io-1.9.0/lib/cool.io/loop.rb:88:in `run_once'
  2024-12-10 14:56:25 +0000 [error]: #0 /usr/local/bundle/gems/cool.io-1.9.0/lib/cool.io/loop.rb:88:in `run'
  2024-12-10 14:56:25 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.18.0/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
  2024-12-10 14:56:25 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.18.0/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'

Environment

- Fluentd version: fluentd 1.18.0
- TD Agent version: <not relevant?>
- Fluent Package version: 1.18.0?
- Docker image (tag): edge-debian
- Operating system: Debian GNU/Linux 12 (bookworm)
- Kernel version: 6.1.0-25-amd64