schema-registry-kafkaconnect-converter artifact is huge

[TomBillietKlarrio]

The schema-registry-kafkaconnect-converter artifact is a whopping 55Mb on mavencentral. I see it's a shaded fat-jar.
Any specific reason for this? The schema-registry-serde is packaged nicely.

[blacktooth]

Kafka converters are packaged as ZIPs or fat JARs to be able to install it on connectors. Reference

[TomBillietKlarrio]

I agree with you.
However this is a library right, not a connector itself?
We're embedding this library into our own connector. We're packaging this connector ourselves as per the documentation. But this would be way more efficient if the glue converter would be a regular library and not a fat jar.

Or did I miss something and you can deploy this converter separately and still reference it from another connector?

[blacktooth]

We package it as a converter to provide convenience to users who don't want to do it separately. However, I do see value in separating the library and the converter part of it to support use-cases where users want to package it themselves. Thanks for highlighting this issue!

Does it help if we separate the converter code into a separate module? I will also explore if there are simpler alternatives.

[TomBillietKlarrio]

I think that would solve our use-case yes. Thanks for considering.

[L-Applin]

I do have a similar use case, we have imported the schema-registry-kafkaconnect-converter into a connector and package it ourself for ease of use. Having the converter as a library would indeed make more sense for this use case.

[OneCricketeer]

embedding this library into our own connector.

Ideally, Connectors only work with Schema and Struct types directly within the Connect API, and not import converters. Can you explain the use case more? If you need other POJO types, then those should be mapped to Struct objects, not a specific binary format

[TomBillietKlarrio]

embedding this library into our own connector.

Ideally, Connectors only work with Schema and Struct types directly within the Connect API, and not import converters. Can you explain the use case more? If you need other POJO types, then those should be mapped to Struct objects, not a specific binary format

This was the easiest workflow in our deployment model. Hence why we opted for this.
In the meanwhile we're no longer using Glue as schema-registry due to other issues/requirements.

[ryber]

I'll also add to this that amazon-kinesis-producer has also pulled this into itself as a dependency and has made it so this library cannot be easily excluded (both IKinesisProducer and KinesisProducer have direct dependencies and the only way to exclude the Glue stuff is to stub out the classes).

awslabs/amazon-kinesis-producer#436