Change kafka library to kafka-go

[sansmoraxz]

Proposal

Switch the kafka client library to use Kafka-go in place of sarama

Use-Case

Kafka-go has support for MSK IAM which can help address #3431

In addition to this it has much cleaner api, better documented and much better resource utilization.

Is this a feature you are interested in implementing yourself?

Maybe

Anything else?

kafka-go doesn't provide mock objects so any tests that need to be done has to be made through a kafka server (perhaps a seperate container can be provisioned for this).

In addition to this the supported auth types include:

• Plain
• SCRAM
• MSK

I guess Oauthbearer and GSSAPI is as of yet not supported in kafka-go.

[sbdtu5498]

LGTM! Will do this. /assign

[sansmoraxz]

I was working on this. Perhaps we can collaborate.

[sansmoraxz]

Just to put it out there, kafka-go does provide an interface for various sasl authentication mechanisms, you can check out here: https://github.com/segmentio/kafka-go/tree/main/sasl

Implementing any new sasl based authentication mechanisms will be easier as we don't need to wait for the library vendor (segment) to add support for them.

[zroubalik]

Cool, @sansmoraxz are you able to tell me if we can use the same configuration & settings with this library?

    lagThreshold: '5'
    activationLagThreshold: '3'
    offsetResetPolicy: latest
    allowIdleConsumers: false
    scaleToZeroOnInvalidOffset: false
    excludePersistentLag: false
    version: 1.0.0
    partitionLimitation: '1,2,10-20,31'

[sansmoraxz]

I am not sure if the version's something that's really required. The methods in it does have error as return types for api calls that can result in such. Others should work without issues.

[sbdtu5498]

I was working on this.

Extremely sorry, I had no idea you were working on it.

Just to put it out there, kafka-go does provide an interface for various sasl authentication mechanisms, you can check out here: https://github.com/segmentio/kafka-go/tree/main/sasl

Implementing any new sasl based authentication mechanisms will be easier as we don't need to wait for the library vendor (segment) to add support for them.

Sure that's a great advantage!

Perhaps we can collaborate.

Sure thing! If you are currently working on the implementation part or migrating current implementation from Sarama to Kafka-go. Let me know once you are done, I will add auth support for other SASL mechanism as well. Further I will take a look at Kafka-go, they have many auth providers missing. So maybe it'd be a good idea to add some in them too.

[sansmoraxz]

My bad. Didn't mention it here before.

I will add auth support for other SASL mechanism as well. Further I will take a look at Kafka-go, they have many auth providers missing. So maybe it'd be a good idea to add some in them too.

Sure, perhaps you can take a dig at Oauth.

[sansmoraxz]

Quick update: I have implemented the core code changes.

I have a question regarding the unit tests, viz for checking lag? How are we going to approach it?

Can we use embeded docker container for this or is it gonna break something? For example: https://github.com/orlangure/gnomock

[zroubalik]

@sansmoraxz I think we can give it a try, as discussed on Slack. Btw have you changed the scaler completely? I think that the safest option for us is to provide an alternative implementation and don't drop Sarama implementation (at least initially).

[sansmoraxz]

Btw @sansmoraxz I think we can give it a try, as discussed on Slack. Btw have you changed the scaler completely? I think that the safest option for us is to provide an alternative implementation and don't drop Sarama implementation (at least initially).

Oops that I did. Need to rollback those. Yes it makes sense, to keep them seperate.
Should I implement this as a seperate scaler?

[dttung2905]

Should we create a separate scaler i.e kafka scaler v2 for example? Then we can slowly deprecate the current one once the new one is stable

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[sansmoraxz]

/unstale