Cleanup a malformed log record which blocks elasticsearch ingestion

[averatio]

Hi All,
We use Microk8s 1.21 with Fluentd add-on, the cluster works perfect.
We have a malformed log record which fails on Elasticsearch ingestion (elastic search complains that the json of the log is malformed, see the error log below). Fluentd retries indefinitely the record submission. Eventually it blocks the logging process on Elasticsearch.

We have difficulties to find:

• How to cleanup this faulty record so that Elasticsearch ingestion can work again
• How to find which pod created the malformed record
• How to troubleshoot the issue in general.

By the way, disable and enable Fluentd addon has no effect.

Any advice and clue is welcome
Thanks and regards
AFV

error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error=\"400 - Rejected by Elasticsearch\" location=nil tag=\"kubernetes.var.log.containers.fluentd-es-v3.1.0-njw8w_kube-system_fluentd-es-3e4483e5f318c89b7f50942263e181599ca917c34420406b2a5d7542657d5b99.log\" time=2022-03-18 15:28:50.463671132 +0000 record={\"stream\"=>\"stdout\", \"docker\"=>{\"container_id\"=>\"3e4483e5f318c89b7f50942263e181599ca917c34420406b2a5d7542657d5b99\"}, \"kubernetes\"=>{\"container_name\"=>\"fluentd-es\", \"namespace_name\"=>\"kube-system\", \"pod_name\"=>\"fluentd-es-v3.1.0-njw8w\", \"container_image\"=>\"quay.io/fluentd_elasticsearch/fluentd:v3.1.0\", \"container_image_id\"=>\"quay.io/fluentd_elasticsearch/fluentd@sha256:4883be463f4cdded2f66cdd6c12ba0a11f702b11d6db8f444b66faaf4a36d7d7\", \"pod_id\"=>\"cf62625e-3e75-498a-918f-1d04a92c2d8f\", \"host\"=>\"tstcopservices01blue\", \"labels\"=>{\"controller-revision-hash\"=>\"97795957f\2022-03-18T15:31:03.272520145Z ", \"k8s-app\"=>\"fluentd-es\", \"pod-template-generation\"=>\"1\", \"version\"=>\"v3.1.0\"}, \"master_url\"=>\"[https://10.152.183.1:443/api\"](https://10.152.183.1/api/%22), \"namespace_id\"=>\"ebbc5fd1-d50c-43e2-b394-e52d42c68672\", \"namespace_labels\"=>{\"kubernetes_io/metadata_name\"=>\"kube-system\"}}, \"message\"=>\"2022-03-18 15:28:49.851402403 +0000 fluent.warn: {\\\"error\\\":\\\"#<Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError: 400 - Rejected by

[jalsanishant]

how did this issue solved? I have the same issue:

2024-10-01 13:11:18 +0000 [warn]: #0 send an error event to @error: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch" location=nil tag="kubernetes.var.log.containers.ddb.log" time=2024-10-01 13:11:15.329969734 +0000

Thanks in advance.