[Bug]: Current offset is higher than last offset

[vector-mj]

Bug Description

After a network incident(without any PVC issue or data loss at disk level) in our data center, we see that the current offset(consumer group offset) of some partitions in some topics is higher than the last offset(last record offset in the partition) of that partition.
I read relevant Kafka documents about this issue, then I noticed that this issue never should happen because when you try to fetch or commit an offset that doesn't exist(higher or lower than Kafka offset range), Kafka returns an OffsetOutOfRangeException
also when there is no committed position in consumer group Kafka will use auto.offset.reset configuration.

Questions:

• Can Strimzi modify these current or last offset values for our consumer groups or topics?
• Can this issue be from our consumer/producer library(Golang/Sarama)?

Steps to reproduce

I tried to reproduce this issue by changing the consumer group offset but I can't.

Expected behavior

This problem, in my opinion, should never arise after a network incident.

Strimzi version

0.33.2

Kubernetes version

K8s: v1.20.2

Installation method

YAML files

Infrastructure

Bare-metal

Configuration files and logs

No response

Additional context

No response

[scholzj]

Strimzi does not modify any offsets in your Kafka cluster. That is all up to Kafka.

I'm definitely not the biggest Kafka expert in the world. But I'm not sure this is something that cannot happen during some incident assuming some particular configurations. E.g. when the client commits the offsets but the messages themself are not yet written to disk and get lost during the incident etc.

[vector-mj]

Hi @scholzj
It's good that Strimzi doesn't modify offsets. I don't have any problem if I miss some records in writing to the disk process, but how can it happen if Kafka also doesn't allow to commit an offset number out of offset range? I know that some consumers can shift committed offset forward and backward to jump over or re-fetch some records but also in this scenario committed offset should be in our actual offset range.

So I should move my questions to Kafka project issues..., Thank you @scholzj for your response.

[scholzj]

Honestly, I do not know how can it happen. You did not provide any details, any logs, any reproducers, or anything else really. So I think anything else would be pure speculation. I guess the Kafka project might give you a better answer as to the generic causes of this error. But I doubt you will get the exact cause of the issue without providing any details.

[vector-mj]

Yeah, I don't have any logs for this issue, and we have some limitations to reproducing this issue in our shared K8s environment because generating network issues can affect other teams' services on the cluster too.
I will share details here if I reproduce it again.