Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

java.lang.OutOfMemoryError: GC overhead limit exceeded' is observed when creating producers on a non-existent topic #6838

Closed
sijie opened this issue Apr 29, 2020 · 3 comments · Fixed by #6879 or #7120
Closed

java.lang.OutOfMemoryError: GC overhead limit exceeded' is observed when creating producers on a non-existent topic #6838

sijie opened this issue Apr 29, 2020 · 3 comments · Fixed by #6879 or #7120
Assignees
@codelipenghui
Labels
type/bug The PR fixed a bug or issue reported a bug

Comments

@sijie
Copy link
Member

sijie commented Apr 29, 2020

Describe the bug

if we try to create producer for a non-existent topic the ProducerImpl object is hanging in the dump. Although producer creation fails but we see that ProducerImpl object corresponding to each producing action hanging in the dump. This behavior will lead to OOM in microservice which by mistake tries to produce consistently to a non-existent topic.
@sijie sijie added the type/bug The PR fixed a bug or issue reported a bug label Apr 29, 2020
@sijie sijie mentioned this issue Apr 29, 2020
Closed
@codelipenghui codelipenghui mentioned this issue May 5, 2020
Merged
codelipenghui added a commit that referenced this issue May 11, 2020
@codelipenghui
Close producer when the topic does not exists. (#6879)
6eed217 
Fixes #6838

### Motivation

Close producer when the topic does not exists.

### Modifications

1. Fix exception handle for the topic does not exist.
2. Change state to Close when producer got TopicDoesNotExists exception so that the producer can close the cnx and will no longer add send timeout tasks to HashedWheelTimer.
jiazhai pushed a commit that referenced this issue May 12, 2020
@codelipenghui @jiazhai
Close producer when the topic does not exists. (#6879)
2b1a1ed 
Fixes #6838

### Motivation

Close producer when the topic does not exists.

### Modifications

1. Fix exception handle for the topic does not exist.
2. Change state to Close when producer got TopicDoesNotExists exception so that the producer can close the cnx and will no longer add send timeout tasks to HashedWheelTimer.
(cherry picked from commit 6eed217)
@devprashantksinha
Copy link

Hi, we are consuming your 2.5.2 release we still observe this issue. PulsarProducerImp is hanging in the dump as reported previously.

Huanli-Meng pushed a commit to Huanli-Meng/pulsar that referenced this issue May 27, 2020
@codelipenghui @Huanli-Meng
Close producer when the topic does not exists. (apache#6879)
2bd677a 
Fixes apache#6838

### Motivation

Close producer when the topic does not exists.

### Modifications

1. Fix exception handle for the topic does not exist.
2. Change state to Close when producer got TopicDoesNotExists exception so that the producer can close the cnx and will no longer add send timeout tasks to HashedWheelTimer.
@codelipenghui
Copy link
Contributor

@devprashantksinha Did you upgrade the Pulsar broker? #6879 makes some changes on the broker side.

@devprashantksinha
Copy link

@codelipenghui : yes we have upgraded both client and pulsar chart components including broker to 2.5.2 . These are the steps I followed to reproduce :

  1. Produce to a non existent topic from one of the application using pulsar-client 2.5.2.
  2. Take heap dump after producing we see a lot of instances of ProducerImpl in the dump being the top memory consumer. This might lead to OOM after some time.

Ideally, we should not see any ProducerImpl for a topic that doesn't exist. Right ?

Please let me know if you need more info about the setup.

@codelipenghui codelipenghui mentioned this issue Jun 1, 2020
Merged
codelipenghui added a commit that referenced this issue Jun 3, 2020
@codelipenghui
Close producer without state check while the topic does not exists. (#…
842996e 
…7120)

Fixes #6838 related to #6879

### Motivation

Close producer without state check while the topic does not exists. The problem is the `State` of the producer failed to pass the check condition so that the producer can't be closed. If TopicDoesNotExsitsException happens on a producer, we can directly close this producer without state checking for the producer.
addisonj pushed a commit to instructure/pulsar that referenced this issue Jun 12, 2020
@codelipenghui
Close producer when the topic does not exists. (apache#6879)
aab9245 
Fixes apache#6838

### Motivation

Close producer when the topic does not exists.

### Modifications

1. Fix exception handle for the topic does not exist.
2. Change state to Close when producer got TopicDoesNotExists exception so that the producer can close the cnx and will no longer add send timeout tasks to HashedWheelTimer.
(cherry picked from commit 6eed217)
huangdx0726 pushed a commit to huangdx0726/pulsar that referenced this issue Aug 24, 2020
@codelipenghui @huangdx0726
Close producer when the topic does not exists. (apache#6879)
4fd986e 
Fixes apache#6838

### Motivation

Close producer when the topic does not exists.

### Modifications

1. Fix exception handle for the topic does not exist.
2. Change state to Close when producer got TopicDoesNotExists exception so that the producer can close the cnx and will no longer add send timeout tasks to HashedWheelTimer.
huangdx0726 pushed a commit to huangdx0726/pulsar that referenced this issue Aug 24, 2020
@codelipenghui @huangdx0726
Close producer without state check while the topic does not exists. (a…
18ac4a8 
…pache#7120)

Fixes apache#6838 related to apache#6879

### Motivation

Close producer without state check while the topic does not exists. The problem is the `State` of the producer failed to pass the check condition so that the producer can't be closed. If TopicDoesNotExsitsException happens on a producer, we can directly close this producer without state checking for the producer.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@codelipenghui codelipenghui

Labels
type/bug The PR fixed a bug or issue reported a bug
Projects
None yet
Milestone
No milestone
4 participants
@sijie @jiazhai @codelipenghui @devprashantksinha