Kafka Connect Persistent Storage

[zeyaddeeb]

Can I add persistent storage for the kafka connect resource?

I'm currently creating connectors via REST API since access to secrets is sometimes tricky

  annotations:
    strimzi.io/use-connector-resources: "false"

But when the pod restarts, all the connectors created via REST API disappear.

Any guidance would be much appreciated.

[scholzj]

You cannot add persistent storage to the Connect deployment. But the connectors and their configurations are not stored locally, but in Kafka topics. So I do not think the storage has anything to do with it.

[scholzj]

Everything is stored in Kafka topics. Even the connector configurations. The only thing stored locally is the Connect configuration file itself. But that is generated at startup by Strimzi. So there is no need to store it locally.

[scholzj]

Note: Just to be clear, Strimzi runs Kafka Connect n the distributed mode.

[zeyaddeeb]

How come when the pod restarts all existing configuration created via REST are wiped, is there a way to bring them backup automatically?

[scholzj]

Well, they are for sure not wiped for me. So I'm afraid I do not have an answer why is it happening to you. But it is not related to persistent storage.

[zeyaddeeb]

Good point, the underlying node actually gets restarted. Which might wipe kafka offsets since I have ephemeral storage. Thanks for the quick response