[improve][pip] PIP-321 Introduce allowed-cluster at the namespace level

pip/pip-321.md

@@ -0,0 +1,251 @@
+# PIP-321: Split the responsibilities of namespace replication-clusters
+
+# Background knowledge
+
+Pulsar's geo-replication mechanism is typically used for disaster recovery, enabling the replication of persistently stored message data across multiple data centers. For instance, your application publishes data in one region, and you would like to process it for consumption in other regions. With Pulsar's geo-replication mechanism, messages can be produced and consumed in different geo-replication. See the introduction of geo-replication to get more information.[1]
+A client can set replication clusters for a namespace, and the pulsar broker internally manages replication to all the clusters.
+A namespace has multiple topics. Once a namespace is configured with replication clusters, all the topics under this namespace will enable replication in these clusters.
+An allowed-cluster for a namespace is a cluster where the topics under the namespace can be loaded or created.
+This concept is introduced with PIP–8. It does not formally propose this concept, but the implementation uses replication-clusters as allowed-clusters. See more information about this in PIP-8. [2]
+Namespace Policy is a configuration in the namespace level, that is stored in the metadata store, e.g. zookeeper, and this configuration can not be accessed across multiple clusters with different metadata stores.
+Topic Policy refers to the configurations at the topic level that are stored in the bookkeeper ledger. Users can specify whether it is global. If it is global, the topic policy will be replicated to the multiple clusters. The clusters are configured by replication clusters now, and in the following, we called it Topic-Policy-Synchronize-Clusters.
+See more information about global topic policy in PIP-92 [3]
+
+[1] https://pulsar.apache.org/docs/3.1.x/concepts-replication
+[2] https://github.com/apache/pulsar/pull/903
+[3] https://github.com/apache/pulsar/wiki/PIP-92%3A-Topic-policy-across-multiple-clusters
+
+# Motivation
+Replication clusters, allowed-Clusters, and the topic-policy-synchronized-clusters are different in definition, and users don’t always want them to keep the same configuration. But in the current implementation, the replication-clusters, allowed-Clusters, and the topic-policy-synchronized-clusters are all configured by specifying the replication clusters. This will make the topic unable to perform some more flexible configuration.
+A specific example is enabling geo-replication at the topic level. NamespaceA enables geo-replication in cluster1 and cluster2. The topics under the namespace have their own geo-replication configurations.
+```
+NamespaceA | [cluster1, cluster2]
+{topic1 | [cluster1], topic2, topic3 | [cluster1, cluster2, cluster3]}
+```
+In this case, topic1 is the only pub/sub in cluster1 without replication. This works as expected. topic2 follows the replication configuration of the namespaceA, replicating messages between cluster1 and cluster1. That's no problem, either. Topic3 expects to replicate messages at cluster1, cluster2, and cluster3. A problem occurred. Topic3 can not be created in cluster3 and the global topic policies will only be replicated between cluster1 and cluster2 regardless of the geo-replication configuration at the topic level.
+The root cause is that the geo-replication clusters configured in the namespace level also represent the clusters where the topics under the namespace can be created or loaded, and the clusters of the global topic policies synchronize.
+
+To support a more flexible configuration at the topic level, we must make the cluster configuration at the namespace level more clearly.
+Splitting the responsibilities of replication clusters into three configurations:
+
+* ``Replication-Clusters``: the default replication clusters for the topics under the namespace.
+* ``Allowed-Clusters``: the allowed clusters for the topics under the namespace. Topics under this namespace can load in these clusters. It`s not related to the replication.
+* ``Topic-Policy-Synchronized-Clusters``: the clusters of the global topic policies synchronize.
+
+**notes:**
+The scope of allowed-clusters should contain replication-clusters and Topic-Policy-Synchronized-Clusters.
+If a namespace does not set the allowed-clusters, the allowed-clusters will be kept the same as the replication clusters.
+If a namespace does not set the allowed-clusters and replication-clusters, the topics under the namespace can only be loaded in the local cluster.
+If a namespace does not set the topic-policy-synchronized-clusters, the topic-policy-synchronized-clusters will be kept the same as the replication clusters.
+If a namespace does not set the topic-policy-synchronized-clusters and replication-clusters, the global topic policy of the topics under this namespace can only be sent to the local cluster.
+
+## Geo-Replication at Topic Level: Problems and Solutions
+
+Let's take a look at the examples to understand the problem of geo-replication at the topic level.
+And figure out how these new namespace policies can resolve this problem.
+
+### Problem 1: Geo-replication Not Working as Expected
+
+When the namespace disables geo-replication and the clusters share the same metadata store, 
+geo-replication at the topic level does not work as expected.
+
+**Example 1:**
+
+- Environment:
+  - cluster1 and cluster2 in different regions sharing the same Zookeeper cluster.
+
+- Replication clusters configuration: 
+  - Set namespace `ns` replication clusters : cluster1 (local cluster)
+  - Set topic `ns/topic1` replication clusters : cluster1, cluster2.
+
+- Expected: 
+  - Topic `ns/topic1` can replicate between cluster1 and cluster2.
+
+- Actual: 
+  - Topic cannot be created at cluster2.
+
+**Example 2:**
+
+- Environment: 
+  - cluster1, cluster2, and cluster3 in different regions using different Zookeeper clusters.
+
+- Replication clusters configuration: 
+  - Set namespace `ns` replication clusters:  cluster1, cluster2, cluster3 (local cluster);
+  - Set topic `ns/topic1` replication clusters: cluster1, cluster2, cluster3; 
+  - Set topic `ns/topic2` replication clusters: null;
+
+- Expected: 
+  - Topic `ns/topic1` can replicate in cluster1, cluster2 and cluster3.
+
+- Actual: 
+  - As expected.
+
+### Solution 1: Introduce 'Allowed-Clusters'
+
+For Problem1, we introduce `allowed-clusters` to configure the clusters where the topics under the namespace can be loaded or created.
+
+**Example:**
+
+- Environment: 
+  - cluster1 and cluster2 in different regions sharing the same Zookeeper cluster.
+
+- Replication clusters configuration: 
+  - Set namespace `ns` replication clusters : cluster1 (local cluster); 
+  - Set topic `ns/topic1` replication clusters : cluster1, cluster2.
+  - Set `allowed-clusters` : cluster1, cluster2
+
+- Expected: 
+  - Topic `ns/topic1` can replicate between cluster1 and cluster2.
+
+- Actual: 
+  - Topic `ns/topic1` can be created at cluster2. Replication works as expected.
+
+### Problem 2: Global Topic Policies Not Replicating as Expected
+
+The global topic policies cannot be replicated to the other clusters as expected, regardless of whether the clusters share the same metadata store.
+
+**Example:**
+
+- Environment: 
+  - cluster1, cluster2, and cluster3 in different regions using different Zookeeper clusters.
+
+- Replication clusters configuration: 
+  - Set namespace `ns` replication clusters : cluster1, cluster2, cluster3 (local cluster); 
+  - Set topic `ns/topic1` replication clusters :  cluster1, cluster2, Cluster3; 
+  - Set topic `ns/topic2` replication clusters :  null;
+
+- Expected: 
+  - Topic `ns/change_event` can replication in cluster1, cluster2 and cluster3.
+
+- Actual: 
+  - Topic `ns/change_event` does not have a replication configuration, so the topic policies will not be replicated.
+
+### Solution 2: Introduce 'Topic-Policy-Synchronized-Clusters'
+
+For Problem2, we introduce 'topic-policy-synchronized-clusters' to configure the clusters where the topics under the namespace can replicate topic policies to.
+
+**Example:**
+
+- Environment: 
+  - cluster1, cluster2, and cluster3 in different regions using different Zookeeper clusters.
+
+- Replication clusters configuration: 
+  - Set namespace 'ns' replication clusters :  cluster1, cluster2, cluster3 (local cluster);
+  - Set topic `ns/topic1` replication clusters : cluster1, cluster2, cluster3;
+  - Set topic `ns/topic2` replication clusters :  null;
+  - Set `topic-policy-synchronized-clusters`:  cluster1, cluster2, cluster3;
+
+- Expected: 
+  - Topic `ns/change_event` can replicate in cluster1, cluster2 and cluster3.
+
+- Actual: 
+  - Topic policies can work as expected.
+
+**Note:** This is an optional operation. You can choose not to configure the 'Topic-Policy-Synchronized-Clusters' and define different topic policies in the different clusters.
+
+# Goals
+
+## In Scope
+
+The namespace has a more detailed configuration for clusters. Users can use `replication-clusters`, `allowed-clusters`, and `topic-policy-synchronized-clusters` to configure the different behavioral logic of topics in the multiple clusters, rather than just using `replication-clusters` to control all behaviors.
+
+## Out of Scope
+
+This proposal can be used to solve the problem of topic-level geo-replication can not work as expected. It is the initial motivation for this proposal, but this proposal does not involve modifications to geo-replication.
+
+# High Level Design
+
+Add two new namespace policies `allowed_clusters`, and `topic_policy_synchronized_clusters`.
+Replace the function of  `replication_clusters` with `replication_clusters`, `allowed_clusters`, and `topic_policy_synchronized_clusters`.
+
+# Detailed Design
+
+## Public-facing Changes
+
+### Public API
+Add a new endpoint `setNamespaceAllowedClusters` to the REST API.
+```
+Type: POST
+Path: /namespaces/{tenant}/{namespace}/allowedcluster
+Query parameters: tenant, namespace
+HTTP body parameters: List<String> clusterIds
+Response code:
+  * 400 Bad request, Allowed-clusters should contain all replication-clusters
+  * 403 Forbidden, Don't have admin permission
+  * 404 Not Found, Tenant or cluster or namespace doesn't exist.
+  * 409 Conflict, Peer-cluster can't be part of allowed-cluster.
+  * 412 Precondition Failed, Namespace is not global or invalid cluster ids.
+Explanation for 409 Conflict: This follows the behavior of namespace replication clusters. In the PIP-8, quoted in background knowledge, it introduce the concept of peer-cluster and peer-cluster can't be part of replication-cluster. For the allowed-clusters, users could enable replication at the topic level, so the peer-cluster can't be part of allowed-clusters too.
+```
+
+Add a new endpoint `getNamespaceAllowedClusters` to the REST API.
+```
+type: GET
+Path: /namespaces/{tenant}/{namespace}/allowedcluster
+Query parameters: tenant, namespace
+Response code:
+  * 403 Forbidden, Don't have admin permission
+  * 404 Not Found, Tenant or cluster or namespace doesn't exist.
+  * 412 Precondition Failed, Namespace is not global.
+```
+
+Add a new endpoint `setNamespaceTopicPolicySynchronizedClusters` to the REST API.
+```
+Type: POST
+Path: /namespaces/{tenant}/{namespace}/topicpolicycluster
+Query parameters: tenant, namespace
+HTTP body parameters: List<String> clusterIds
+Response code:
+  * 400 Bad request, Allowed-clusters should contain all topic-policy-synchronized-clusters
+  * 403 Forbidden, Don't have admin permission
+  * 404 Not Found, Tenant or cluster or namespace doesn't exist.
+  * 412 Precondition Failed, Namespace is not global or invalid cluster ids.
+```
+
+Add a new endpoint `getNamespaceTopicPolicySynchronizedClusters` to the REST API.
+```
+Type: GET
+Path: /namespaces/{tenant}/{namespace}/topicpolicycluster
+Query parameters: tenant, namespace
+Response code:
+  * 403 Forbidden, Don't have admin permission
+  * 404 Not Found, Tenant or cluster or namespace doesn't exist.
+  * 412 Precondition Failed, Namespace is not global.
+```
+
+### Binary protocol
+
+### Configuration
+
+### CLI
+
+### Metrics
+None.
+
+# Monitoring
+None.
+# Security Considerations
+If the broker enables authentication, then this configuration can only be set by the client who was authenticated.
+
+# Backward & Forward Compatibility
+
+## Revert
+This can not be reverted if they enable geo-replication at the topic level. Otherwise, the replicator of these topics will be closed.
+
+## Upgrade
+There are no additional operations that need to be performed. The replication-clusters will be the default value of allowed-clusters and topic-policy-synchronized-clusters.
+
+# Alternatives
+
+Add a configuration `enableReplicationInTopicLevel` to make the topic able to be loaded in all clusters.
+
+
+# General Notes
+
+# Links
+
+<!--
+Updated afterwards
+-->
+* Mailing List discussion thread:
+* Mailing List voting thread: