[Bug] Resource Leak when do health check.

[thetumbled]

Search before asking

• I searched in the issues and found nothing similar.

Version

master

Minimal reproduce step

We will do the health check for the cluster by rest api org.apache.pulsar.broker.admin.impl.BrokersBase#healthCheck every 1 min. But recently we meet a case that the health check fail.

bin/pulsar-admin --admin-url http://XXX brokers healthcheck

There is no any error log in the broker log, the random message has been produced to the health-check topic, but the reader fail to read it out, so the logic stuck at

org.apache.pulsar.broker.admin.impl.BrokersBase#healthCheckRecursiveReadNext

, which result into the created producers and readers can't be closed, and the scheduled async tasks can't be completed as well, and the number of these objects increase as time goes by.
For example:
bin/pulsar-admin topics stats persistent://pulsar/XXX/healthcheck

What did you expect to see?

1

What did you see instead?

1

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!