Unable to backup kafka data from persistent volumes

[shreyasarani23]

I have created one storage class [kafka-storage-class-> name of the storage class]. And the storage class name is specified in kafka deployment file. I have given three replicas for kafka and zookeeper so it is creating 6 persistent volumes and 6 pvc for kafka and zookeeper pods. Basically creating 6 azure disks. When i delete the kafka cluster and redploy it, the data in the topics is lost. But the persistent volumes remain as it is, because the reclaim policy is given as true for storage class. But I didn't know how to restore kafka data from PV's. I have referred this link https://strimzi.io/docs/operators/latest/full/using.html#cluster-recovery_str. But couldn't restore the data.
Can you please specify the steps in detail to restore kafka data from persistent voulmes? I have attached my kafka deployment file please help me to restore kafka data from persistent volumes
kafka.yaml.txt

[scholzj]

What version of Strimzi do you use?

[scholzj]

No. You have to delete the two internal topic first, then enable the topic operator and that way your my-topic will not be deleted. That is exactly what the documentation says.

[shreyasarani23]

But unable to delete the two internal topics
shreyas@Azure:~$ kubectl run kafka-admin -ti --image=quay.io/strimzi/kafka:latest-kafka-2.7.0 --rm=true --restart=Never -- ./bin/kafka-topics.sh --bootstrap-server my-kafka-cluster-kafka-bootstrap.kafka.svc:9092 --topic __strimzi-topic-operator-kstreams-topic-store-changelog-0 --delete && ./bin/kafka-topics.sh --bootstrap-server my-kafka-cluster-kafka-bootstrap.kafka.svc:9092 --topic __strimzi_store_topic-0 --delete
If you don't see a command prompt, try pressing enter.
Error while executing topic command : Topic '__strimzi-topic-operator-kstreams-topic-store-changelog-0' does not exist as expected
[2021-05-06 08:46:36,447] ERROR java.lang.IllegalArgumentException: Topic '__strimzi-topic-operator-kstreams-topic-store-changelog-0' does not exist as expected
at kafka.admin.TopicCommand$.kafka$admin$TopicCommand$$ensureTopicExists(TopicCommand.scala:539)
at kafka.admin.TopicCommand$AdminClientTopicService.deleteTopic(TopicCommand.scala:357)
at kafka.admin.TopicCommand$.main(TopicCommand.scala:72)
at kafka.admin.TopicCommand.main(TopicCommand.scala)
(kafka.admin.TopicCommand$)
pod "kafka-admin" deleted
pod default/kafka-admin terminated (Error)

I don't know why I am facing this issue.

[shreyasarani23]

What happens if we deploy topic operator without deleting these two internal topics?

[shreyasarani23]

Hi @scholzj i was able to delete the internal topic by entering into the pod with the following command
[kafka@my-kafka-cluster-kafka-0 kafka]$ bin/kafka-topics.sh --bootstrap-server my-kafka-cluster-kafka-bootstrap:9092 --topic __strimzi_store_topic --delete
[kafka@my-kafka-cluster-kafka-0 kafka]$ bin/kafka-topics.sh --list --bootstrap-server my-kafka-cluster-kafka-bootstrap:9092
__consumer_offsets
my-topic

We can see that __Strimzi_store_topic is deleted. And I deployed my kafka cluster with topic operator. I was able to get the topic using the command kubectl get KafkaTopic. To check whether the data is present in the topic or not I executed the consumer pod and I was able to consume the records.

Thanks @scholzj for assisting me. Looking forward to work more on Strimzi.

[scholzj]

👍