Skip to content

Not log retriable exceptions as errors#5904

Merged
hachikuji merged 3 commits into
apache:trunkfrom
t3hnar:trunk
Apr 25, 2019
Merged

Not log retriable exceptions as errors#5904
hachikuji merged 3 commits into
apache:trunkfrom
t3hnar:trunk

Conversation

@t3hnar

@t3hnar t3hnar commented Nov 12, 2018

Copy link
Copy Markdown
Contributor

Background: I've spotted tons of kafka related errors in logs, after investigation I found out that those are harmless as being retried.
Hence I propose to not log retriable exceptions as errors.

Examples of what I've see in logs:

  • Offset commit failed on partition .. at offset ..: The request timed out.
  • Offset commit failed on partition .. at offset ..: The coordinator is loading and hence can't process requests.
  • Offset commit failed on partition .. at offset ..: This is not the correct coordinator.
  • Offset commit failed on partition .. at offset ..: This server does not host this topic-partition.

@t3hnar

t3hnar commented Nov 13, 2018 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Hi @mjsax, @ijuma and @hachikuji, this is really a very simple pull request, which makes troubleshooting much simpler. Could you please take a look?

@bvoss

bvoss commented Jan 7, 2019

Copy link
Copy Markdown

Hi @t3hnar
any feedback here?
And is there a related issue in https://issues.apache.org/jira/projects/KAFKA/

I have the same problem to get to a clean error log policy and just ignored ConsumerCoordinator.class in my logging config. Which is bad because errors in ConsumerRebalanceListener may be lost.

So +1 for this

@t3hnar

t3hnar commented Jan 7, 2019

Copy link
Copy Markdown
Contributor Author

@bvoss no feedback yet. Looks like this PR is being ignored :(

@bvoss

bvoss commented Jan 7, 2019

Copy link
Copy Markdown

@t3hnar I didn't found a related issue. Did you open one? May this is needed.

@t3hnar

t3hnar commented Jan 7, 2019

Copy link
Copy Markdown
Contributor Author

@bvoss nope, I did not.

@t3hnar

t3hnar commented Jan 7, 2019

Copy link
Copy Markdown
Contributor Author

@bvoss I've just created one https://issues.apache.org/jira/browse/KAFKA-7791

@mjsax

mjsax commented Jan 11, 2019

Copy link
Copy Markdown
Member

@hachikuji LGTM. But this is client code, so you should merge it if you think it makes sense.

@mjsax

mjsax commented Feb 13, 2019

Copy link
Copy Markdown
Member

@hachikuji Should we get this into 2.2 ? If yes, please have a quick look (if you thinks it makes sense--I am not 100% sure) -- it's an one-liner, so should not take long. Just pinging again because code freeze is end of the week.

@MenschNestor

MenschNestor commented Apr 1, 2019

Copy link
Copy Markdown

@mjsax @hachikuji Any updates on this? It's cluttering our error log as well.

@mjsax
mjsax requested review from cmccabe, gwenshap, hachikuji, ijuma and rajinisivaram and removed request for cmccabe, gwenshap, hachikuji, ijuma and rajinisivaram April 13, 2019 23:13
@mjsax

mjsax commented Apr 13, 2019

Copy link
Copy Markdown
Member

@gwenshap Maybe you can have a look?

hachikuji
hachikuji approved these changes Apr 25, 2019
View reviewed changes

@hachikuji hachikuji left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the delay. LGTM. Thanks for the patch!

@hachikuji
hachikuji merged commit f6c59df into apache:trunk Apr 25, 2019
hachikuji pushed a commit that referenced this pull request Apr 25, 2019
@t3hnar @hachikuji
MINOR: Do not log retriable offset commit exceptions as errors (#5904)
725e184 
Reviewers: Jason Gustafson <jason@confluent.io>
hachikuji pushed a commit that referenced this pull request Apr 25, 2019
@t3hnar @hachikuji
MINOR: Do not log retriable offset commit exceptions as errors (#5904)
c7c5932 
Reviewers: Jason Gustafson <jason@confluent.io>
hachikuji pushed a commit that referenced this pull request Apr 25, 2019
@t3hnar @hachikuji
MINOR: Do not log retriable offset commit exceptions as errors (#5904)
1ff4d74 
Reviewers: Jason Gustafson <jason@confluent.io>
dhruvilshah3 added a commit to confluentinc/kafka that referenced this pull request Apr 29, 2019
@dhruvilshah3
Merge remote-tracking branch 'ak/trunk' into master-sync
f678af8 
* ak/trunk: (42 commits)
  KAFKA-8134: `linger.ms` must be a long
  KAFKA-7779; Avoid unnecessary loop iteration in leastLoadedNode (apache#6081)
  MINOR: Update Gradle to 5.4.1 and update its plugins  (apache#6436)
  MINOR: improve Session expiration notice (apache#6618)
  KAFKA-8029: In memory session store (apache#6525)
  MINOR: In-memory stores cleanup (apache#6595)
  KAFKA-7862 & KIP-345 part-one: Add static membership logic to JoinGroup protocol (apache#6177)
  KAFKA-8254: Pass Changelog as Topic in Suppress Serdes (apache#6602)
  KAFKA-7903: automatically generate OffsetCommitRequest (apache#6583)
  KAFKA-8291 : System test fix (apache#6637)
  MINOR: Do not log retriable offset commit exceptions as errors (apache#5904)
  MINOR: Fix log message error of loadTransactionMetadata (apache#6571)
  MINOR: Fix 404 security features links (apache#6634)
  MINOR: Remove an unnecessary character from broker's startup log
  MINOR: Make LogCleaner.shouldRetainRecord more readable (apache#6590)
  MINOR: Remove implicit return statement (apache#6629)
  KAFKA-8237; Untangle TopicDeleteManager and add test cases (apache#6588)
  KAFKA-8227 DOCS Fixed missing links duality of streams tables (apache#6625)
  MINOR: reformat settings.gradle to be more readable (apache#6621)
  MINOR: Correct RestServerTest formatting
  ...

 Conflicts:
	build.gradle
	settings.gradle
pengxiaolong pushed a commit to pengxiaolong/kafka that referenced this pull request Jun 14, 2019
@t3hnar @pengxiaolong
MINOR: Do not log retriable offset commit exceptions as errors (apach…
774cc75 
…e#5904)

Reviewers: Jason Gustafson <jason@confluent.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hachikuji hachikuji hachikuji approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@t3hnar @bvoss @mjsax @MenschNestor @hachikuji