Output kafka avro: schema not found

[lzwaan]

Bug Report

Not sure if this a known bug, but couldn't fine any solution here...

Describe the bug
When sent messages to kafka with avro enabled, no schema is created, but shows some generated id?

To Reproduce
Fluentbit config (output) and there were 2 messages sent, see screenshot...

[OUTPUT]
    Name                                kafka
    Match                               lease
    Brokers                             kafka1:10091,kafka2:10092
    topics                              input.fluentbit.b2b.dhcp.lease
    Format                              avro
    Schema_str                          {"namespace":"com.github.lzwaan.avro","type":"record","name":"dhcp_lease","fields":[{"name":"TIMESTAMP","type":"string"},{"name":"CID","type":"string"},{"name":"LEASE","type":"string"},{"name":"TOPIC","type":"string"}]}
    Schema_id                           12345
    timestamp_key                       TIMESTAMP
    timestamp_format                    iso8601
    message_key_field                   CID
    rdkafka.client.id                   fluentbit_dhcp_clientid
    rdkafka.debug                       All
    rdkafka.security.protocol           plaintext
    rdkafka.request.required.acks       1
    rdkafka.log.connection.close        false
    rdkafka.log_level                   7
    rdkafka.metadata.broker.list        kafka1:10091,kafka2:10092

Screenshots from AKHQ

Your Environment
Debian

[lzwaan]

Is there anybody who has used fluentbit succesfull wih avro enabled inside docker containers?

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.

[flarno11]

Is there anybody who has used fluentbit succesfull wih avro enabled inside docker containers?

yes works here. by manually compiling with -DFLB_AVRO_ENCODER=On and the fix from #5435

[shuishou1983]

Hi @flarno11, could you share me which code base did you use for customized build? I tried 2.0.9 but got error with avro format in kafka output. TIA.

[flarno11]

@shuishou1983 I don't remember the exact version, but it was already on the 2.x branch, cherry-picked the fix and recompiled

[ppenpraze]

I ended up using the HTTP output plugin and made a Kafka/Avro bridge in Python, works well.

[shuishou1983]

Thank your comments, @flarno11 @ppenpraze. After applying the fix from #5435 and rebuilt it, now my fluent-bit could output to kafka with avro format without issue.