[improve][pip] PIP-321 Introduce allowed-cluster at the namespace level

pip/pip-321.md

@@ -0,0 +1,233 @@
+# PIP-321: Split the responsibilities of namespace replication-clusters
+
+# Background knowledge
+
+Pulsar's geo-replication mechanism is typically used for disaster recovery, enabling the replication of persistently stored message data across multiple data centers. For instance, your application publishes data in one region, and you would like to process it for consumption in other regions. With Pulsar's geo-replication mechanism, messages can be produced and consumed in different geo-replicated regions. See the introduction of geo-replication to get more information.[1]
+
+A client can set allowed clusters for a tenant. The allowed-cluster for the tenant is a cluster that the tenant can access.
+
+A client can set replication clusters for a namespace, and the pulsar broker internally manages replication to all the replication clusters. And the replication clusters for a namespace must be a subgroup of the tenant's allowed clusters.
+
+A client can not set allowed clusters at the namespace level, although PIP-8 [2] essentially achieves a similar functionality. It does not formally propose this configuration, but the implementation uses replication-clusters as allowed-clusters for a namespace. It introduces peer cluster for global namespace redirection and fails `PartitionedMetadata-Lookup` request if global namespace's replication-clusters doesn't contain current/peer-clusters. See more information about this in PIP-8. [2]
+
+A namespace has multiple topics. Once a namespace is configured with replication clusters, all the topics under this namespace will enable replication in these clusters.
+
+Namespace Policy is a configuration in the namespace level, that is stored in the metadata store, e.g. zookeeper, and this configuration can not be accessed across multiple clusters with different metadata stores.
+
+Topic Policy refers to the configurations at the topic level, which are stored in the BookKeeper ledger. Users can specify whether it is global. If it is global, the topic policy will be replicated to the multiple clusters. The clusters are configured by replication clusters now. In fact, these are the clusters specified in the replication clusters of the `topic policy` topic. See more information about global topic policy in PIP-92 [3]
+
+Replication clusters can be configured at the message level. Pulsar support setting replication clusters when send messages. 
+```java
+producer.newMessage().replicationClusters(List.of("cluster1", "cluster2")).send();
+```
+
+[1] https://pulsar.apache.org/docs/3.1.x/concepts-replication
+[2] https://github.com/apache/pulsar/pull/903
+[3] https://github.com/apache/pulsar/wiki/PIP-92%3A-Topic-policy-across-multiple-clusters
+
+# Motivation
+
+Geo-replication at the topic level and message level can't work as expected when geo-replication is disabled at the namespace level and the clusters share the same metadata store.
+Let's see an example:
+
+**Example For Topic Level:**
+
+- Environment:
+  - cluster1 and cluster2 in different regions sharing the same Zookeeper cluster.
+
+- Replication clusters configuration:
+  - Set namespace `ns` replication clusters : cluster1 (local cluster)
+  - Set topic `ns/topic1` replication clusters : cluster1, cluster2.
+
+- Expected:
+  - Topic `ns/topic1` can replicate between cluster1 and cluster2.
+
+- Actual:
+  - Topic cannot be created at cluster2.
+
+**Example For Message Level**
+
+- Environment:
+  - cluster1 and cluster2 in different regions sharing the same Zookeeper cluster.
+
+- Replication clusters configuration:
+  - Set namespace `ns` replication clusters : cluster1 (local cluster)
+  - Set replication clusters when send message1: cluster1, cluster2.
+
+- Expected:
+  - Message1 can replicate between cluster1 and cluster2.
+
+- Actual:
+  - Topic cannot be created at cluster2, and so the message1 can not be replicated to cluster2.
+
+The root cause of these issues is that topics cannot access clusters that are not included in the replication-clusters of the namespace policy. 
+The replication clusters and allowed clusters are different in the definition, and users don’t always want them to keep the same configuration. 
+But in the current implementation, the replication-clusters and allowed-Clusters are all configured by specifying the replication clusters.
+This will make the topic unable to have its replication policies.
+
+To support geo-replication policies at the topic level and the message level, we must make the cluster configuration at the namespace level more clearly.
+Introduce `allowed-clusters` at the namespace level and make `replication-clusters` only the default replication clusters for the topics under the namespace.
+
+# Goals
+
+## In Scope
+
+The namespace will have a more detailed configuration for clusters. Users can use `replication-clusters` and `allowed-clusters` to configure the different behavioral logic of topics in the multiple clusters, rather than just using `replication-clusters` to control all behaviors.
+
+## Out of Scope
+
+This proposal can be used to solve the problem of topic-level and message level geo-replication can not work as expected. It is the initial motivation for this proposal, but this proposal does not involve modifications to geo-replication.
+
+# High Level Design
+Add a new namespace policy `allowed_clusters`.
+
+# Detailed Design
+
+## Public-facing Changes
+
+### Public API
+
+#### `setNamespaceAllowedClusters` Endpoint
+
+This new endpoint allows setting the list of allowed clusters for a specific namespace.
+
+**Method:**
+```
+POST
+```
+
+**Path:**
+```
+/namespaces/{tenant}/{namespace}/allowedclusters
+```
+
+**Query Parameters:**
+
+- `tenant`: The tenant within which the namespace resides.
+- `namespace`: The namespace for which you are setting the allowed clusters.
+
+**HTTP Body Parameters:**
+
+- `clusterIds`: A list of cluster IDs.
+
+**Response Codes:**
+
+- `400 Bad Request`: The list of allowed clusters should include all replication clusters.
+- `403 Forbidden`: The requester does not have admin permissions.
+- `404 Not Found`: The specified tenant, cluster, or namespace does not exist.
+- `409 Conflict`: A peer-cluster cannot be part of an allowed-cluster.
+- `412 Precondition Failed`: The namespace is not global or the provided cluster IDs are invalid.
+
+**Explanation for 409 Conflict:** This follows the behavior of namespace replication clusters. As per PIP-8, a peer-cluster cannot be part of a replication-cluster. Similarly, for allowed-clusters, users could enable replication at the topic level, hence a peer-cluster cannot be part of allowed-clusters as well.
+
+#### `getNamespaceAllowedClusters` Endpoint
+
+This new endpoint allows retrieving the list of allowed clusters for a specific namespace.
+
+**Method:**
+```
+GET
+```
+
+**Path:**
+```
+/namespaces/{tenant}/{namespace}/allowedclusters
+```
+
+**Query Parameters:**
+
+- `tenant`: The tenant within which the namespace resides.
+- `namespace`: The namespace for which you are retrieving the allowed clusters.
+
+**Response Codes:**
+
+- `403 Forbidden`: The requester does not have admin permissions.
+- `404 Not Found`: The specified tenant, cluster, or namespace does not exist.
+- `412 Precondition Failed`: The namespace is not global.
+
+### Binary protocol
+
+### Configuration
+
+### CLI
+
+#### `setNamespaceAllowedClusters` Command
+
+This command allows you to set the list of allowed clusters for a specific namespace.
+
+**Usage:**
+
+```
+$ pulsar admin namespaces set-allowed-clusters --clusters <cluster-ids> <tenant>/<namespace>
+```
+
+**Parameters:**
+
+- `<cluster-ids>`: A comma-separated list of cluster IDs.
+- `<tenant>`: The tenant within which the namespace resides.
+- `<namespace>`: The namespace for which you are setting the allowed clusters.
+
+**Response Codes:**
+
+- `400 Bad Request`: The allowed clusters should contain all replication clusters.
+- `403 Forbidden`: You do not have admin permission.
+- `404 Not Found`: The tenant, cluster, or namespace does not exist.
+- `409 Conflict`: A peer-cluster cannot be part of an allowed-cluster.
+- `412 Precondition Failed`: The namespace is not global or the cluster IDs are invalid.
+
+**Explanation for 409 Conflict:** This follows the behavior of namespace replication clusters. In PIP-8, it introduced the concept of a peer-cluster, which cannot be part of a replication-cluster. For the allowed-clusters, users could enable replication at the topic level, so the peer-cluster cannot be part of the allowed-clusters too.
+
+#### `getNamespaceAllowedClusters` Command
+
+This command allows you to retrieve the list of allowed clusters for a specific namespace.
+
+**Usage:**
+
+```
+$ pulsar admin namespaces get-allowed-clusters <tenant>/<namespace>
+```
+
+**Parameters:**
+
+- `<tenant>`: The tenant within which the namespace resides.
+- `<namespace>`: The namespace for which you are getting the allowed clusters.
+
+**Response Codes:**
+
+- `403 Forbidden`: You do not have admin permission.
+- `404 Not Found`: The tenant, cluster, or namespace does not exist.
+- `412 Precondition Failed`: The namespace is not global.
+
+### Metrics
+None.
+
+# Monitoring
+None.
+# Security Considerations
+If the broker enables authentication, then this configuration can only be set by the client who was authenticated.
+Only the superuser and tenant admin is allowed to access the newly added API.
+
+# Backward & Forward Compatibility
+The new namespace policy will not impact the behavior of existing systems.
+If users do not utilize the new feature, no operation should be executed during an upgrade or revert
+
+## Revert
+To revert, simply switch back to the old version of Pulsar. However, note that topics will be removed from those clusters that are not included in the replication clusters configured at the namespace level.
+For example, replication clusters at the topic level, for topic1, is `cluster1, cluster2, cluster3`. Replication clusters at the namespace level is `cluster1, cluster2`. Allowed clusters at the namespace level is `cluster1, cluster2, cluster3`. After revert pulsar version to the old one, the topic1 will be deleted at the cluster3.
+
+## Upgrade
+No additional operations need to be performed. The replication-clusters will be the default value of allowed-clusters.
+
+# Alternatives
+
+None.
+
+# General Notes
+
+# Links
+
+<!--
+Updated afterwards
+-->
+* Mailing List discussion thread:https://lists.apache.org/thread/87qfp8ht5s0fvw2y4t3j9yzgfmdzmcnz
+* Mailing List voting thread: