Filbeat stops shipping journald logs when encountered "failed to read message field: bad message" error

[snjoetw]

Filebeat seems to stop sending journald logs when encountered "failed to read message field: bad message" error

journald input:

- type: journald
  id: appdaemon_logs
  paths:
    - /var/log/journal
  include_matches.match:
    - syslog.priority=6

Logs:

{
	"log.level": "error",
	"@timestamp": "2022-08-23T00:31:06.768-0700",
	"log.logger": "input.journald",
	"log.origin": {
		"file.name": "compat/compat.go",
		"file.line": 124
	},
	"message": "Input 'journald' failed with: input.go:130: input appdaemon_logs failed (id=appdaemon_logs)\n\tfailed to read message field: bad message",
	"service.name": "filebeat",
	"id": "appdaemon_logs",
	"ecs.version": "1.6.0"
}

Filebeat 8.3.3

Similar bug is also reported in Loki, not sure if the fix is similar or not:
Bug Report: grafana/loki#2812
PR: grafana/loki#2928

[jsoriano]

Could be related to the issues discussed in #23627

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[trauta]

I'm having the same problem with filebeat filebeat version 7.17.8 (amd64) on Ubuntu 20.04.5 LTS.

Here is the journald input config:

- type: journald
  id: syslog
  seek: cursor
  fields_under_root: false
  tags:
    - journald
    - syslog

From the filebeat log:

2023-01-11T12:19:13.844+0100	ERROR	[input.journald]	compat/compat.go:124	Input 'journald' failed with: input.go:130: input syslog failed (id=syslog)
	failed to iterate journal: bad message	{"id": "syslog"}

After this error message the journald input completey stops, there are no new journald events transmitted.
When restarting filebeat via systemd the missing journal events get transmitted without any problems.

This is a very irritating behavior.

Is it possible to implement a fix so that in case of such an error the 'bad mesage' is skipped and the input continues to parse the other messages?

[trauta]

Hi, any updates on this?

[cmacknz]

Sorry, we haven't been able to prioritize this issue yet.

[nicklausbrown]

Any chance this will be prioritized soon, or a timeline on promoting journald out of tech preview? Thanks!

[georgivalentinov]

Seems like the underlying issue is with systemd - they seem to have introduced a regression and have trouble finding and fixing it.
That being said, this is a major issue as people are moving away from log files and the use of journald is prevalent these days.

Any chance such records (or entire journald files) can be skipped by Filebeat and just continue with the rest? As is today it blocks and silently stops processing any log entries upon stumbling into a problematic spot.

[groegeorg]

I'm also desperately hoping for a fix on this issue. I second georgivalentinov's statement: "Any chance such records (or entire journald files) can be skipped by Filebeat and just continue with the rest?"

[cmacknz]

We have taking journald out of tech preview as a priority (see #37086) and this needs to be fixed as part of that work.