Skip to content

[LI-HOTFIX] Prevent ReplicaFetcherThread from throwing UnknownTopicOrPartitionException upon topic creation and deletion. - #80

Merged
efeg merged 1 commit into
linkedin:2.3-li-1from
efeg:fix/UnknownTopicOrPartitionExceptionInLogs
Apr 29, 2020
Merged

[LI-HOTFIX] Prevent ReplicaFetcherThread from throwing UnknownTopicOrPartitionException upon topic creation and deletion.#80
efeg merged 1 commit into
linkedin:2.3-li-1from
efeg:fix/UnknownTopicOrPartitionExceptionInLogs

Conversation

@efeg

@efeg efeg commented Apr 29, 2020

Copy link
Copy Markdown

TICKET =
LI_DESCRIPTION =
When does UnknownTopicOrPartitionException typically occur?

  • Upon a topic creation, a follower broker of a new partition starts replica fetcher before the prospective leader broker of the new partition receives the leadership information from the controller. Apache Kafka has a an open issue about this (see KAFKA-6221)
  • Upon a topic deletion, a follower broker of a to-be-deleted partition starts replica fetcher after the leader broker of the to-be-deleted partition processes the deletion information from the controller.
  • As expected, clusters with frequent topic creation and deletion report UnknownTopicOrPartitionException with relatively higher frequency.

What is the impact?

  • Exception tracking systems identify the error logs with UnknownTopicOrPartitionException as an exception. This results in a lot of noise for a transient issue that is expected to recover by itself and a natural process in Kafka due to its asynchronous state propagation.

Why not move it to a lower than warn-level log?

  • Despite typically being a transient issue, UnknownTopicOrPartitionException may also indicate real issues if it doesn't fix itself after a short period of time. To ensure detection of such scenarios, this PR sets the log level to warn.

EXIT_CRITERIA = TICKET [KAFKA-6221]

More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.

Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.

Committer Checklist (excluded from commit message)

  • Verify design and implementation
  • Verify test coverage and CI build status
  • Verify documentation (including upgrade notes)

…PartitionException upon topic creation and deletion.

TICKET =
LI_DESCRIPTION =
When does UnknownTopicOrPartitionException typically occur?
* Upon a topic creation, a follower broker of a new partition starts replica fetcher before the prospective leader broker of the new partition receives the leadership information from the controller. Apache Kafka has a an open issue about this (see KAFKA-6221)
* Upon a topic deletion, a follower broker of a to-be-deleted partition starts replica fetcher after the leader broker of the to-be-deleted partition processes the deletion information from the controller.
* As expected, clusters with frequent topic creation and deletion report UnknownTopicOrPartitionException with relatively higher frequency.

What is the impact?
* Exception tracking systems identify the error logs with UnknownTopicOrPartitionException as an exception. This results in a lot of noise for a transient issue that is expected to recover by itself and a natural process in Kafka due to its asynchronous state propagation.

Why not move it to a lower than warn-level log?
* Despite typically being a transient issue, UnknownTopicOrPartitionException may also indicate real issues if it doesn't fix itself after a short period of time. To ensure detection of such scenarios, this PR sets the log level to warn.

EXIT_CRITERIA = TICKET [KAFKA-6221]
@efeg
efeg requested review from gitlw and radai-rosenblatt April 29, 2020 00:36

@gitlw gitlw left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Do you plan to submit this change to upstream?

@efeg

efeg commented Apr 29, 2020

Copy link
Copy Markdown
Author

@gitlw Thanks for the review! Yes, I will create a PR on Apache Kafka, as well.

@efeg
efeg merged commit 0efd268 into linkedin:2.3-li-1 Apr 29, 2020
@efeg
efeg deleted the fix/UnknownTopicOrPartitionExceptionInLogs branch April 29, 2020 01:37
gitlw pushed a commit that referenced this pull request Jun 13, 2020
…PartitionException upon topic creation and deletion. (#80)

TICKET =
LI_DESCRIPTION =
When does UnknownTopicOrPartitionException typically occur?
* Upon a topic creation, a follower broker of a new partition starts replica fetcher before the prospective leader broker of the new partition receives the leadership information from the controller. Apache Kafka has a an open issue about this (see KAFKA-6221)
* Upon a topic deletion, a follower broker of a to-be-deleted partition starts replica fetcher after the leader broker of the to-be-deleted partition processes the deletion information from the controller.
* As expected, clusters with frequent topic creation and deletion report UnknownTopicOrPartitionException with relatively higher frequency.

What is the impact?
* Exception tracking systems identify the error logs with UnknownTopicOrPartitionException as an exception. This results in a lot of noise for a transient issue that is expected to recover by itself and a natural process in Kafka due to its asynchronous state propagation.

Why not move it to a lower than warn-level log?
* Despite typically being a transient issue, UnknownTopicOrPartitionException may also indicate real issues if it doesn't fix itself after a short period of time. To ensure detection of such scenarios, this PR sets the log level to warn.

EXIT_CRITERIA = TICKET [KAFKA-6221]
gitlw pushed a commit that referenced this pull request Jun 13, 2020
…PartitionException upon topic creation and deletion. (#80)

TICKET =
LI_DESCRIPTION =
When does UnknownTopicOrPartitionException typically occur?
* Upon a topic creation, a follower broker of a new partition starts replica fetcher before the prospective leader broker of the new partition receives the leadership information from the controller. Apache Kafka has a an open issue about this (see KAFKA-6221)
* Upon a topic deletion, a follower broker of a to-be-deleted partition starts replica fetcher after the leader broker of the to-be-deleted partition processes the deletion information from the controller.
* As expected, clusters with frequent topic creation and deletion report UnknownTopicOrPartitionException with relatively higher frequency.

What is the impact?
* Exception tracking systems identify the error logs with UnknownTopicOrPartitionException as an exception. This results in a lot of noise for a transient issue that is expected to recover by itself and a natural process in Kafka due to its asynchronous state propagation.

Why not move it to a lower than warn-level log?
* Despite typically being a transient issue, UnknownTopicOrPartitionException may also indicate real issues if it doesn't fix itself after a short period of time. To ensure detection of such scenarios, this PR sets the log level to warn.

EXIT_CRITERIA = TICKET [KAFKA-6221]
gitlw pushed a commit that referenced this pull request Jun 17, 2020
…PartitionException upon topic creation and deletion. (#80)

TICKET =
LI_DESCRIPTION =
When does UnknownTopicOrPartitionException typically occur?
* Upon a topic creation, a follower broker of a new partition starts replica fetcher before the prospective leader broker of the new partition receives the leadership information from the controller. Apache Kafka has a an open issue about this (see KAFKA-6221)
* Upon a topic deletion, a follower broker of a to-be-deleted partition starts replica fetcher after the leader broker of the to-be-deleted partition processes the deletion information from the controller.
* As expected, clusters with frequent topic creation and deletion report UnknownTopicOrPartitionException with relatively higher frequency.

What is the impact?
* Exception tracking systems identify the error logs with UnknownTopicOrPartitionException as an exception. This results in a lot of noise for a transient issue that is expected to recover by itself and a natural process in Kafka due to its asynchronous state propagation.

Why not move it to a lower than warn-level log?
* Despite typically being a transient issue, UnknownTopicOrPartitionException may also indicate real issues if it doesn't fix itself after a short period of time. To ensure detection of such scenarios, this PR sets the log level to warn.

EXIT_CRITERIA = TICKET [KAFKA-6221]
gitlw pushed a commit that referenced this pull request Jun 17, 2020
…PartitionException upon topic creation and deletion. (#80)

TICKET =
LI_DESCRIPTION =
When does UnknownTopicOrPartitionException typically occur?
* Upon a topic creation, a follower broker of a new partition starts replica fetcher before the prospective leader broker of the new partition receives the leadership information from the controller. Apache Kafka has a an open issue about this (see KAFKA-6221)
* Upon a topic deletion, a follower broker of a to-be-deleted partition starts replica fetcher after the leader broker of the to-be-deleted partition processes the deletion information from the controller.
* As expected, clusters with frequent topic creation and deletion report UnknownTopicOrPartitionException with relatively higher frequency.

What is the impact?
* Exception tracking systems identify the error logs with UnknownTopicOrPartitionException as an exception. This results in a lot of noise for a transient issue that is expected to recover by itself and a natural process in Kafka due to its asynchronous state propagation.

Why not move it to a lower than warn-level log?
* Despite typically being a transient issue, UnknownTopicOrPartitionException may also indicate real issues if it doesn't fix itself after a short period of time. To ensure detection of such scenarios, this PR sets the log level to warn.

EXIT_CRITERIA = TICKET [KAFKA-6221]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants