I think the root of problem lies in OffsetCommit response error handling, we're probably getting a ERR_ILLEGAL_GENERATION back from the coordinator at this point, which causes us to rejoin the group - thus the rebalance.
Even if we add a rebalancing check to commit() it is possible for us to receive this error anyway, due to timing, so I think it might be better to also improve the error handling here and not trigger a rejoin if the group is rebalancing.

Agreed, improving the error handling would make a great difference as well. I haven't been able to confirm this yet, but I believe a new rebalance is also triggered when the error is ERR_REBALANCE_IN_PROGRESS

It would be great if this behaviour were documented somewhere in the wiki or in the config description. We spent some efforts to understand what the heck is going on and why the cooperative-sticky rebalance is not working as expected and leads to endless rebalances.

We use confluent_kafka for python. I tried to implement a workaround using on_assign/on_revoke callbacks and prevent manual commits for some time when these callbacks are triggered. But noticed the group generation id can be changed even before these callbacks are triggered. As I understand currently with confluent_kafka python there is now way to know for sure the rebalance is in progress and we can't use manual commits with cooperative-sticky. Also I noticed when auto commit enabled librdkafka can send OffsetCommitRequests when rebalance is in progress and it doesn't lead to invalid group generation id error. I see this offset commit happens between the on_revoke callback triggered in one consumer and on_assign in another.

so I think it might be better to also improve the error handling here and not trigger a rejoin if the group is rebalancing.

Is not this contradicts to the behavior described in this KIP document? It says:

If received UNKNOWN_MEMBER_ID or ILLEGAL_GENERATION from join-group / sync-group / commit / heartbeat response: reset generation / clear member-id correspondingly, call rebalance listener's onPartitionsLost for all the partition and then re-join group with empty assigned partition.

Just adding an additional data point that we see this exact behavior with our setup in kafka. I can try to take a crack at fixing this in my off time, but wondering if there is anyone on Confluent side working on this @milindl @emasab since if you're using manual offset commits, you cannot use Cooperative rebalancing today in any context.

is there any ongoing work for this?

@scanterog see #4220 , but in short, I think that this issue has dropped off the radar. Unfortunately the fix that @roxelo created (and I attempted to get merged) broke other behavior in librdkafka. @milindl said the librdkafka team was looking into this internally, but unfortunately from my side there are few things I can do as my place of work doesn't pay for confluent support and therefore we have no sway on their roadmap. If anyone who is interested in getting this more attention on the confluent side and is a paying confluent customer I highly recommend they push via their TAMs to get traction on this.

Same issue here. Hopefully this will be prioritized by Confluent because it makes cooperative-stiky unusable with manual commits.

Commits are possible during a rebalance, before a partition is revoked the user can commit offsets for the revoked partitions in the rebalance callback, that is possible in Java client too.
What isn't needed is to avoid a re-join when the ILLEGAL_GENERATION error happens, as proposed by Magnus.

Commits are possible during a rebalance, before a partition is revoked the user can commit offsets for the revoked partitions in the rebalance callback, that is possible in Java client too. What isn't needed is to avoid a re-join when the ILLEGAL_GENERATION error happens, as proposed by Magnus.

Thanks for looking into this @emasab. Any hope to see this sorted out soon?

Thank you!

@massimeddu-sj at the moment there's a strategy that should prevent this error during commit and a subsequent rebalance.
It requires you have a rebalance callback set, but that is often used when using manual offset management. The consumer should always commit in the consume loop and only when a message or an EOF is returned.
Given between the first and second phase of the cooperative rebalance the revoke callback is called, it returns from the consume call after calling it with no messages. Then second rejoin is started that increases the generation id, there is a second assignment callback with no message returning from the consume call and finally messages start coming again. If you commit manually only when you have a new message or receive a partition EOF, that error should be avoided.

@massimeddu-sj at the moment there's a strategy that should prevent this error during commit and a subsequent rebalance. It requires you have a rebalance callback set, but that is often used when using manual offset management. The consumer should always commit in the consume loop and only when a message or an EOF is returned. Given between the first and second phase of the cooperative rebalance the revoke callback is called, it returns from the consume call after calling it with no messages. Then second rejoin is started that increases the generation id, there is a second assignment callback with no message returning from the consume call and finally messages start coming again. If you commit manually only when you have a new message or receive a partition EOF, that error should be avoided.

Thank you very much for the additional information @emasab . Unfortunately I'm not too familiar with the rebalance/revoke callbacks, and I'm not too confident on overriding the default implementation. It would be great if you are able to share any snippet or implementation example.

My current implementation is actually quite simple:

consumer_conf = {
    [...]
    'enable.auto.commit': False,
    'partition.assignment.strategy': 'cooperative-sticky'
}

kafka_consumer = Consumer(consumer_conf, logger=logger)

while True:
    message = kafka_consumer.poll()

    if message is None: continue

    if message.error():
        if message.error().code() == KafkaError._PARTITION_EOF:
            # End of partition event
            logger.error('%% %s [%d] reached end at offset %d\n' %
                        (message.topic(), message.partition(), message.offset()))
        else:
            raise KafkaConsumerException(Exception(message.error()))

    message_handler(message)

    kafka_consumer.commit(asynchronous=False)

Hi team, for anyone still looking for a workaround. It seems like we have been able to get around this issue by using Kafka autocommit, as suggested, however we still want control over when offsets are committed, so we used store_offsets() to mark them ready for commit. This requires setting enable.auto.offset.store to False and enable.auto.commit to True. In our code, anywhere we would use commit() we now use store_offsets([offsets]) and let Kafka commit automatically using auto.commit.interval.ms.

This way, the cooperative-sticky assignment does not trigger endless rebalances and we also have control over which offsets to commit. Hope this helps!

If you commit manually only when you have a new message or receive a partition EOF, that error should be avoided.

TL;DR: manual commits during a rebalance will be sent but they won't cause an additional rebalance or losing the assignment.

It can happen that the consumer receives a message after the first rejoin of the cooperative incremental assignment, in that case the partitions are resumed, and only later the second rejoin is done to redistribute the revoked partitions.

The message can be committed manually, causing an ILLEGAL_GENERATION error.
That is possible, but must not cause it to lose the assignment while rebalance is in progress, as in this Java client code.

Generation id was incorrectly reset to -1 on OffsetCommit ILLEGAL_GENERATION error and, immediately after that, the second SyncGroup fails with same error because of the wrong generation id, this time causing lost partitions.

Differences from Java client: it avoids sending the commit and the exception is a RebalanceInProgressException instead.
As users are already expecting an ILLEGAL_GENERATION error, in librdkafka we leave this change to the error code for 3.x.