Merge #1
Merged
Merge #1
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 2 commits
January 7, 2018 14:11
**New functionality:**
- (Experimental) Implemented Java-based in-memory data storage which provides some new functionality. Right now this is lightly implemented into the front-end, but over time will replace the current implementation. This allowed me to implement some new functionality:
- Report if consumer-group is currently active
- This will eventually allow us to report on inactive consumer-groups
- (Experimental) Report Burrow-like consumer-group status calculation via REST endpoint (/consumergroup), while updating Burrows rules a bit. The rules I implemnted here are:
- Evaluate per consumer-group topic-partition:
- Rule 0: If there are no committed offsets, then there is nothing to calculate and the period is OK.
- Rule 1: If the difference between now and the last offset timestamp is greater than the difference between the last and first offset timestamps, the consumer has stopped committing offsets for that partition (error)
- Rule 2: If the consumer offset decreases from one interval to the next the partition is marked as a rewind (error)
- Rule 3: If over the stored period, the lag is ever zero for the partition, the period is OK
- Rule 4: If the consumer offset does not change, and the lag is non-zero, it's an error (partition is stalled)
- Rule 5: If the consumer offsets are moving, but the lag is consistently increasing, it's a warning (consumer is slow)
- Roll-up all consumer-group topic-partitions per consumer-group and report a consumer-group status:
- Set consumer-group status to ERROR if any topic-partition status is STOP
- Set consumer-group status to ERROR if any topic-partition status is REWIND
- Set consumer-group status to ERROR if any topic-partition status is STALL
- Set consumer-group status to WARN if any topic-partition status is WARN
- Set consumer-group status to OK if none of the above rules match
**Of course some of the bugs you were seeing were fixed as well:**
- Synchronizing around all SQLite DB activity. SQLite only allows one operation at a time with the DB file.
- This fixed all DB create/update/delete issues at the expense of sometimes blocking DB operations while another DB operation is taking place. This is unavoidable using SQLite. Long term fix will be to replace SQLite with a more appropriate DB engine.
- Fixed an issue where LogEndOffset and Lag can display incorrect values.
- Added retry logic around building the ZkUtils object. This fixed the issue where we would not re-connect to Zookeeper if the zk service went down and then was restored.
- Updated some dependency versions.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.