[improve] [client] Add api to get producer/consumer stats for partition topic #18212

rdhabalia · 2022-10-26T19:20:21Z

Motivation

Right now, client is not able to fetch producer/consumer stats for individual partition if topic is partitioned topic and it returns non-set value 0 or some aggregate value across all partitions which is not useful if client wants to track individual partition in case if any specific partition is having issue at broker side and individual partition metrics are useful for debugging issue specially various latency metrics.

Modifications

Add API to fetch producer/consumer stats for partition topics.

Verifying this change

Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

Added integration tests for end-to-end deployment with large payloads (10MB)
Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

Documentation

doc
doc-required
doc-not-needed
doc-complete

Matching PR in forked repository

PR in forked repository:

codecov-commenter · 2022-10-26T20:01:14Z

Codecov Report

Merging #18212 (b25ee34) into master (6c65ca0) will increase coverage by 3.73%.
The diff coverage is 34.64%.

@@             Coverage Diff              @@
##             master   #18212      +/-   ##
============================================
+ Coverage     34.91%   38.64%   +3.73%     
- Complexity     5707     8248    +2541     
============================================
  Files           607      685      +78     
  Lines         53396    67351   +13955     
  Branches       5712     7216    +1504     
============================================
+ Hits          18644    26029    +7385     
- Misses        32119    38330    +6211     
- Partials       2633     2992     +359

Flag	Coverage Δ
unittests	`38.64% <34.64%> (+3.73%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...org/apache/bookkeeper/mledger/LedgerOffloader.java	`0.00% <ø> (ø)`
...che/bookkeeper/mledger/LedgerOffloaderFactory.java	`0.00% <ø> (ø)`
...pache/bookkeeper/mledger/LedgerOffloaderStats.java	`0.00% <ø> (ø)`
...ookkeeper/mledger/LedgerOffloaderStatsDisable.java	`0.00% <ø> (ø)`
...a/org/apache/bookkeeper/mledger/ManagedCursor.java	`85.71% <ø> (ø)`
...che/bookkeeper/mledger/ManagedLedgerException.java	`41.17% <ø> (ø)`
...bookkeeper/mledger/ManagedLedgerFactoryConfig.java	`86.66% <ø> (ø)`
...g/apache/bookkeeper/mledger/ManagedLedgerInfo.java	`22.22% <ø> (ø)`
...he/bookkeeper/mledger/OffloadedLedgerMetadata.java	`0.00% <ø> (ø)`
...ava/org/apache/bookkeeper/mledger/ScanOutcome.java	`0.00% <ø> (ø)`
... and 726 more

Technoboy- · 2022-10-27T03:29:49Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/PartitionedProducerImpl.java

@@ -353,7 +353,8 @@ public synchronized ProducerStatsRecorderImpl getStats() {
            return null;
        }
        stats.reset();
-        producers.values().forEach(p -> stats.updateCumulativeStats(p.getStats()));
+        producers.forEach(
+                (partition, producer) -> stats.updateCumulativeStats(producer.getTopic(), producer.getStats()));


producer.getTopic() -> partition ?

here, we are keeping map of <topicName, stats> to maintain consistency with consumer-stats.
so, code change is correct where we are not using partition but we are using actual topic name of the partition.

AnonHxy · 2022-10-27T06:32:07Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerStatsDisabled.java

@@ -145,7 +146,12 @@ public void reset() {
    }

    @Override
-    public void updateCumulativeStats(ConsumerStats stats) {
+    public void updateCumulativeStats(String partition, ConsumerStats stats) {


If we change the method singuture, I'm not sure will it bring some compatible problem�, because this method is public.

The name of this method is updateCumulativeStats, doing some partitioned stats work is not cumulative I think. How about add a new method like updatePartitionedStats() :)

I have removed the api change.

…on topic

codelipenghui · 2022-10-28T01:52:41Z

pulsar-client-api/src/main/java/org/apache/pulsar/client/api/ConsumerStats.java

+    /**
+     * @return stats for each partition if topic is partitioned topic
+     */
+    Map<String, ConsumerStats> getPartitionStats();


Here is not only about partition stats. Users can provide topic lists and topic patterns.
Is it better to use a new interface MultiTopicConsumerStats extends ConsumerStats?

For the producer side, we can have a new interface PartitionedTopicProducerStats extends ProducerStats

So that we can add new methods to the MultiTopicConsumerStats and PartitionedTopicProducerStats such getStatsForInternalTopics and getStatsForPartitions

yes, sure, we can have separate interface for partitioned topic with more clear code.
However, the current implementation with combined stats-recorder and stats-interface implementation has complicated things though they are two separate entities (recorder can't become stats but recorder should record the stats ). combining both entities incorrectly complicates the inheritance.
We also should still keep the Partitioned topic abstraction from user. Producer/Consumer::getStats() should return one interface with all required API and user doesn't have to cast based on topic type. Because Partitioned-topic-stats are just a collection of stats so, we should have getPartitionedTopicStats() API with in Stats interface and it should return empty response if topic is not partitioned topic.
Atleast this will be a good start to creating the abstraction and without having existing API changes in this PR.
I will create a separate PR with further improvement.

This PR has addressed all comments.

rdhabalia · 2022-11-01T16:23:59Z

@codelipenghui @AnonHxy comments are addressed. can you PTAL.

eolivelli

Lgtm

…on topic (#18212) * [improve] [client] Add api to get producer/consumer stats for partition topic * introduce partition topic stats interface

rdhabalia added area/client doc-not-needed Your PR changes do not impact docs labels Oct 26, 2022

rdhabalia added this to the 2.11.0 milestone Oct 26, 2022

rdhabalia requested review from merlimat, lhotari, sijie, jiazhai, dlg99, eolivelli, 315157973, AnonHxy, codelipenghui and BewareMyPower October 26, 2022 19:20

rdhabalia self-assigned this Oct 26, 2022

rdhabalia added the ready-to-test label Oct 26, 2022

Technoboy- reviewed Oct 27, 2022

View reviewed changes

AnonHxy reviewed Oct 27, 2022

View reviewed changes

[improve] [client] Add api to get producer/consumer stats for partiti…

4aabf19

…on topic

rdhabalia force-pushed the part_npe_stats branch from 0427832 to 4aabf19 Compare October 27, 2022 21:54

codelipenghui reviewed Oct 28, 2022

View reviewed changes

rdhabalia force-pushed the part_npe_stats branch from 5309454 to 1e4111d Compare October 28, 2022 23:28

introduce partition topic stats interface

b25ee34

rdhabalia force-pushed the part_npe_stats branch from 1e4111d to b25ee34 Compare October 28, 2022 23:45

eolivelli approved these changes Nov 1, 2022

View reviewed changes

rdhabalia merged commit 26f9ffa into apache:master Nov 1, 2022

rdhabalia mentioned this pull request Nov 2, 2022

[improve][client] Add partitioned-stats api to avoid partition abstraction #18308

Merged

14 tasks

Technoboy- added the cherry-picked/branch-2.11 label Nov 3, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[improve] [client] Add api to get producer/consumer stats for partition topic #18212

[improve] [client] Add api to get producer/consumer stats for partition topic #18212

rdhabalia commented Oct 26, 2022 •

edited

Loading

codecov-commenter commented Oct 26, 2022 •

edited

Loading

Technoboy- Oct 27, 2022

rdhabalia Oct 27, 2022

Technoboy- Oct 27, 2022

AnonHxy Oct 27, 2022

rdhabalia Nov 1, 2022

codelipenghui Oct 28, 2022

rdhabalia Oct 28, 2022

rdhabalia commented Nov 1, 2022

eolivelli left a comment

[improve] [client] Add api to get producer/consumer stats for partition topic #18212

[improve] [client] Add api to get producer/consumer stats for partition topic #18212

Conversation

rdhabalia commented Oct 26, 2022 • edited Loading

Motivation

Modifications

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Matching PR in forked repository

codecov-commenter commented Oct 26, 2022 • edited Loading

Codecov Report

Technoboy- Oct 27, 2022

Choose a reason for hiding this comment

rdhabalia Oct 27, 2022

Choose a reason for hiding this comment

Technoboy- Oct 27, 2022

Choose a reason for hiding this comment

AnonHxy Oct 27, 2022

Choose a reason for hiding this comment

rdhabalia Nov 1, 2022

Choose a reason for hiding this comment

codelipenghui Oct 28, 2022

Choose a reason for hiding this comment

rdhabalia Oct 28, 2022

Choose a reason for hiding this comment

rdhabalia commented Nov 1, 2022

eolivelli left a comment

Choose a reason for hiding this comment

rdhabalia commented Oct 26, 2022 •

edited

Loading

codecov-commenter commented Oct 26, 2022 •

edited

Loading