[8.17](backport #45128) [kafka] Handle configuration errors

[mergify]

Proposed commit message

This PR handles kafka configuration errors by dropping the message. Since configuration errors are of permanent error type and should not be retried.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Bring kafka server up using following command

cd filebeat
mage docker:composeUp

Use following filebeat.yml

filebeat.inputs:
  - type: filestream
    id: filestream-input-id
    enabled: true
    paths:
      - ./test.log
    file_identity.native: ~
    prospector.scanner.fingerprint.enabled: false      

output:
  kafka:
    hosts: ["localhost:9092", "localhost:9094"]
    topic: beats
    max_message_bytes: 10

and you can see this error

{
  "log.level": "error",
  "@timestamp": "2025-07-02T15:28:04.373+0530",
  "log.logger": "kafka",
  "log.origin": {
    "function": "github.com/elastic/beats/v7/libbeat/outputs/kafka.(*msgRef).fail",
    "file.name": "kafka/client.go",
    "file.line": 396
  },
  "message": "Kafka (topic=beats): dropping message due to invalid configuration kafka: invalid configuration (Attempt to produce message larger than configured Producer.MaxMessageBytes: 851 > 10)",
  "service.name": "filebeat",
  "ecs.version": "1.6.0"
}

Related issues

• Closes Kafka output retries 'too large messages' (message len > max_message_bytes) instead of dropping it (>= v8.18.0) #44619

---

This is an automatic backport of pull request #45128 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of 23f4491 has failed:

On branch mergify/bp/8.17/pr-45128
Your branch is up to date with 'origin/8.17'.

You are currently cherry-picking commit 23f4491cc.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   libbeat/outputs/kafka/config.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   libbeat/outputs/kafka/client.go
	both modified:   libbeat/outputs/kafka/kafka_integration_test.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[botelastic]

This pull request doesn't have a Team:<team> label.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b mergify/bp/8.17/pr-45128 upstream/mergify/bp/8.17/pr-45128
git merge upstream/8.17
git push upstream mergify/bp/8.17/pr-45128

[khushijain21]

This bug was introduced after 8.17. Closing this backport