Skip to content
This repository was archived by the owner on Aug 23, 2023. It is now read-only.

Remove offest tracking in Kafka plugins #1016

Closed
Dieterbe opened this issue Aug 27, 2018 · 1 comment · Fixed by #1110
Closed

Remove offest tracking in Kafka plugins #1016

Dieterbe opened this issue Aug 27, 2018 · 1 comment · Fixed by #1110
Milestone

Comments

@Dieterbe
Copy link
Contributor

We don't use it anymore

@woodsaj
Copy link
Member

woodsaj commented Aug 29, 2018

the offset tracking is only needed when the "offset" for the kafka-mdm-in plugin or kafka-cluster plugin are set to "last".

This config setting never makes sense. Data is internally buffered until a "chunk" is filled up, so data will be lost if MT doesn't replay old data at startup (offset=oldest, or offset=<time duration>) ensuring all un-written chunks are restored in their entirety.

If the user can tolerate the lost data, due to having multiple replicas, then they are better off just using "offset=newest"

So, lets just get rid of "offset=last" support and get rid of the offset tracking.

Dieterbe added a commit that referenced this issue Oct 24, 2018
there is no use case for it.
see #1016
Dieterbe added a commit that referenced this issue Oct 24, 2018
there is no use case for it.
see #1016
@Dieterbe Dieterbe added this to the 0.11.0 milestone Dec 12, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants