Pekko-based Application Pods Idle at Startup, Not Consuming Kafka Messages Due to Consumer Group Creation Behavior

[dahiyahimanshu]

Issue Description:
During the initial startup of Pekko-based streaming application pods, when the input Kafka topic is created for the first time, and the consumer group is automatically generated by the application, the pods remain idle and do not consume messages from all partitions. This issue does not occur if we manually create the same consumer group using Kafka CLI before starting the application.

Steps to Reproduce:

1. Create a new Kafka topic with multiple partitions using the CLI.
2. Start the Pekko-based application, allowing it to create a consumer group automatically.
3. Observe the logs: the application does not consume messages, and the consumer group has no active members initially.

Observed Behavior:
When the application creates the consumer group itself, the CURRENT-OFFSET remains - initially, and no messages are processed. However, if we manually create the consumer group beforehand, the CURRENT-OFFSET is set to 0, and the application starts consuming messages as expected.

Workaround:
Creating the consumer group manually using Kafka CLI before starting the application resolves this issue, enabling the application to begin consuming messages from all partitions as intended.

Configuration Details:
The application sets the following properties on the Kafka consumer:

AUTO_OFFSET_RESET_CONFIG set to "earliest"
ENABLE_AUTO_COMMIT_CONFIG set to "false"

Additional Information:
We are using Apache Pekko with Scala for streaming applications.
The issue appears specific to when the consumer group is created by the application itself, and not when created manually via CLI.

Expected Behavior:
The application should automatically start consuming messages from all partitions when it creates the consumer group upon startup, without needing the manual step of creating the consumer group through Kafka CLI.

Status:
A workaround exists, but the root cause remains unclear. We need to understand why the consumer group creation by the application results in an idle state, whereas a manually created consumer group does not cause this issue.