Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Issue 6793][broker] Add config of maxNumPartitionsPerPartitionedTopic #6794

Merged
merged 3 commits into from
Apr 29, 2020

Conversation

equanz
Copy link
Contributor

@equanz equanz commented Apr 22, 2020

Fixes #6793

Motivation

Now broker, can't set max number of partitions per partitioned topic. So we can't prevent to create partitioned topic by many partitions from server side.
In this PR, introduce limit of max partitions to broker and to be able to control it.

Modifications

Add maxNumPartitionsPerPartitionedTopic config to broker and compare with numPartitions at create or update partitioned topic.
If the config is set to 0(is default), then disable the check.

Verifying this change

  • Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:

  • Unit tests for maxNumPartitionsPerPartitionedTopic config

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

  • Dependencies (does it add or upgrade a dependency): (no)
  • The public API: (no)
  • The schema: (no)
  • The default values of configurations: (yes)
  • The wire protocol: (no)
  • The rest endpoints: (no)
  • The admin cli options: (no)
  • Anything that affects deployment: (no)

Documentation

  • Does this pull request introduce a new feature? (yes)
  • If yes, how is the feature documented? (docs)

@equanz
Copy link
Contributor Author

equanz commented Apr 22, 2020

/pulsarbot run-failure-checks

@codelipenghui codelipenghui requested review from sijie, codelipenghui and jiazhai and removed request for codelipenghui April 22, 2020 09:40
@codelipenghui codelipenghui added the type/feature The PR added a new feature or issue requested a new feature label Apr 22, 2020
@codelipenghui codelipenghui added this to the 2.6.0 milestone Apr 22, 2020
Copy link
Contributor

@codelipenghui codelipenghui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change looks good to me. Could you please help add some checks for partitioned topic auto-creation? If users enable the partitioned topic auto-creation, the maxNumPartitionsPerPartitionedTopic should greater than defaultNumPartitions.

@equanz
Copy link
Contributor Author

equanz commented Apr 23, 2020

@codelipenghui
Thank you for your review!

Could you please help add some checks for partitioned topic auto-creation?

Is this here or should I add some unit tests?

private CompletableFuture<PartitionedTopicMetadata> createDefaultPartitionedTopicAsync(TopicName topicName) {
final int defaultNumPartitions = pulsar.getBrokerService().getDefaultNumPartitions(topicName);
final int maxPartitions = pulsar().getConfig().getMaxNumPartitionsPerPartitionedTopic();
checkArgument(defaultNumPartitions > 0, "Default number of partitions should be more than 0");
checkArgument(maxPartitions <= 0 || defaultNumPartitions <= maxPartitions, "Number of partitions should be less than or equal to " + maxPartitions);

@equanz
Copy link
Contributor Author

equanz commented Apr 25, 2020

/pulsarbot run-failure-checks

1 similar comment
@nkurihar
Copy link
Contributor

/pulsarbot run-failure-checks

@sijie sijie merged commit 9030e84 into apache:master Apr 29, 2020
gaoran10 pushed a commit to gaoran10/pulsar that referenced this pull request May 5, 2020
apache#6794)

Fixes apache#6793

### Motivation
Now broker, can't set max number of partitions per partitioned topic. So we can't prevent to create partitioned topic by many partitions from server side.
In this PR, introduce limit of max partitions to broker and to be able to control it.

### Modifications
Add `maxNumPartitionsPerPartitionedTopic` config to broker and compare with numPartitions at create or update partitioned topic.
If the config is set to `0`(is default), then disable the check.
gaoran10 pushed a commit to gaoran10/pulsar that referenced this pull request May 5, 2020
apache#6794)

Fixes apache#6793

### Motivation
Now broker, can't set max number of partitions per partitioned topic. So we can't prevent to create partitioned topic by many partitions from server side.
In this PR, introduce limit of max partitions to broker and to be able to control it.

### Modifications
Add `maxNumPartitionsPerPartitionedTopic` config to broker and compare with numPartitions at create or update partitioned topic.
If the config is set to `0`(is default), then disable the check.
Huanli-Meng pushed a commit to Huanli-Meng/pulsar that referenced this pull request May 27, 2020
apache#6794)

Fixes apache#6793

### Motivation
Now broker, can't set max number of partitions per partitioned topic. So we can't prevent to create partitioned topic by many partitions from server side.
In this PR, introduce limit of max partitions to broker and to be able to control it.

### Modifications
Add `maxNumPartitionsPerPartitionedTopic` config to broker and compare with numPartitions at create or update partitioned topic.
If the config is set to `0`(is default), then disable the check.
huangdx0726 pushed a commit to huangdx0726/pulsar that referenced this pull request Aug 24, 2020
apache#6794)

Fixes apache#6793

### Motivation
Now broker, can't set max number of partitions per partitioned topic. So we can't prevent to create partitioned topic by many partitions from server side.
In this PR, introduce limit of max partitions to broker and to be able to control it.

### Modifications
Add `maxNumPartitionsPerPartitionedTopic` config to broker and compare with numPartitions at create or update partitioned topic.
If the config is set to `0`(is default), then disable the check.
@equanz equanz deleted the add_max_number_of_partitions branch December 2, 2020 05:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/feature The PR added a new feature or issue requested a new feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add limit of max partitions per partitioned topic
4 participants