filebeat fails to use lz4 with kafka 1.0

[danielmotaleite]

• Version: filebeat 6.2.2
• Operating System: debian stretch
• Steps to Reproduce:

I have a kafka server 1.0.0 and in filebeat i use something like this:

output:
  kafka:
    hosts: ["172.26.55.191:9092","172.26.21.90:9092","172.26.33.68:9092"]
    topic: '%{[fields.color]}'
    partition.round_robin:
      reachable_only: false
    required_acks: 1
    compression: lz4
    max_message_bytes: 104857600
    version: "0.11.0.0"
    keep_alive: 60

version is set to 0.11.0.0 as it is the highest that filebeat reports as acceptable parameter

When i start filebeat, i get this:

 filebeat[17613]: Exiting: error initializing publisher: kafka: invalid configuration (lz4 compression requires Version >= V0_10_0_0)

even trying 0.10.0.0 in version, it still reports the same error.
switching the compression to gzip, it can start and work correctly

So either there is a bug in kafka output and version check, or filebeat needs to be updated to work correctly with kafka 1.0

[kaplun]

I confirm this. I also have tried using different versions such as 0.10.0.0 and it fails in an identical way. Putting 1.0.0 fails with unknown/unsupported kafka version '1.0.0'.

[urso]

Recent update (#7025) should fix the issue.