Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow consumers to control start from beginning behavior #352

Closed
wants to merge 2 commits into from
Closed

Allow consumers to control start from beginning behavior #352

wants to merge 2 commits into from

Conversation

alisaifee
Copy link
Contributor

Clients of the multi-process consumer currently have no way of enforcing that the consumer resumes from the last committed offset. SimpleConsumer automatically receives this behaviour when auto_commit is enabled and can explicitly request the offset using fetch_last_known_offsets.

To ensure backward compatibility a start_from_beginning parameter has been added to the base consumer which defaults to True.

Reference: #347

mp consumer does not resume from the last comitted
offset.
If auto_commit is set on the mp consumers or an explicit
start_from_beginning is set to False, the child workers will
request the last known offsets as well.
@alisaifee
Copy link
Contributor Author

The failing test is unrelated. Theres a new PR #354 which pretty much does the same thing (with inverted logic for the default value of the kwarg for SimpleConsumer). I'm ok with closing this if #354 is preferred.

@vshlapakov
Copy link
Contributor

Oops, my apologies @alisaifee, I missed your PR (searched by word MultiProcessConsumer and found earlier hanging PR #252). They both do practically the same and I'll be fine too if your PR is preferred, it's be just great to add this functionality asap.

@dpkp
Copy link
Owner

dpkp commented Mar 30, 2015

I think there's a better approach to this -- see #356 . I cherry-picked your test commits :)

@alisaifee
Copy link
Contributor Author

👍
closing this PR.
Thanks!

@alisaifee alisaifee closed this Mar 31, 2015
@alisaifee alisaifee deleted the mp_offset_resume branch March 31, 2015 05:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants