-
Notifications
You must be signed in to change notification settings - Fork 4
DOC-1457 separate Dedicate PSC docs #334
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
micheleRP
merged 6 commits into
main
from
DOC-1457-Dedicated-private-service-connect-docs-say-create-a-BYOVPC-cluster-instead-of-a-dedicated-cluster
Jun 23, 2025
Merged
Changes from 3 commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
a17cf2c
DOC-1457 separate Dedicate PSC docs
micheleRP 22435d4
remove redundant line
micheleRP 6d0bb2f
add link to using API for Dedicated
micheleRP b3a9e29
Update modules/networking/partials/psc-ui.adoc
micheleRP 54a33a0
incorporate Paul's feedback
micheleRP d94caee
update AWS privatelink for Dedicated
micheleRP File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
107 changes: 2 additions & 105 deletions
107
modules/networking/pages/configure-private-service-connect-in-cloud-ui.adoc
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,109 +1,6 @@ | ||
| = Configure GCP Private Service Connect in the Cloud UI | ||
| :description: Set up GCP Private Service Connect in the Redpanda Cloud UI. | ||
| :page-aliases: deploy:deployment-option/cloud/configure-private-service-connect-in-cloud-ui.adoc | ||
| :env-byoc: true | ||
|
|
||
| [NOTE] | ||
| ==== | ||
|
|
||
| * This guide is for configuring GCP Private Service Connect using the Redpanda Cloud UI. To configure and manage Private Service on an existing public cluster, you must use the xref:networking:gcp-private-service-connect.adoc[Redpanda Cloud API]. | ||
| * The latest version of Redpanda GCP Private Service Connect (available March, 2025) supports AZ affinity. This allows requests from Private Service Connect endpoints to stay within the same availability zone, avoiding additional networking costs. | ||
| * DEPRECATION: The original Redpanda GCP Private Service Connect is deprecated and will be removed in a future release. For more information, see xref:manage:maintenance.adoc#deprecated-features[Deprecated features]. | ||
| ==== | ||
|
|
||
|
|
||
| The Redpanda GCP Private Service Connect service provides secure access to Redpanda Cloud from your own VPC. Traffic over Private Service Connect does not go through the public internet because these connections are treated as their own private GCP service. While your VPC has access to the Redpanda VPC, Redpanda cannot access your VPC. | ||
|
|
||
| Consider using the endpoint services if you have multiple VPCs and could benefit from a more simplified approach to network management. | ||
|
|
||
| [NOTE] | ||
| ==== | ||
| * Each client VPC can have one endpoint connected to Private Service Connect. | ||
| * Private Service Connect allows overlapping xref:networking:cidr-ranges.adoc[CIDR ranges] in VPC networks. | ||
| * The number of connections is limited only by your Redpanda usage tier. Private Service Connect does not add extra connection limits. | ||
| * You control from which GCP projects connections are allowed. | ||
| ==== | ||
|
|
||
| == Requirements | ||
|
|
||
| * Use the https://cloud.google.com/sdk/docs/install[gcloud^] command-line interface (CLI) to create the consumer-side resources, such as a client VPC and forwarding rule, or to modify existing resources to use the Private Service Connect attachment created for your cluster. | ||
| * The client VPC must be in the same region as your Redpanda cluster. | ||
|
|
||
| == Enable endpoint service for existing clusters | ||
|
|
||
| . In the Redpanda Cloud UI, open your https://cloud.redpanda.com/clusters[cluster^], and click **Cluster settings**. | ||
| . Under Private Service Connect, click **Enable**. | ||
| . For xref:get-started:cluster-types/byoc/gcp/vpc-byo-gcp.adoc[BYOVPC clusters], you need a NAT subnet with `purpose` set to `PRIVATE_SERVICE_CONNECT`. You also need to create VPC firewall rules to allow Private Service Connect traffic. You can use the `gcloud` CLI: | ||
| + | ||
| NOTE: The firewall rules support up to 20 Redpanda brokers. If you have more than 20 brokers, or for help enabling Private Service Connect, contact https://support.redpanda.com/hc/en-us/requests/new[Redpanda support^]. | ||
| + | ||
| [,bash] | ||
| ---- | ||
| gcloud compute networks subnets create <subnet-name> \ | ||
| --project=<host-project-id> \ | ||
| --network=<shared-vpc-name> \ | ||
| --region=<region> \ | ||
| --range=<subnet-range> \ | ||
| --purpose=PRIVATE_SERVICE_CONNECT | ||
| ---- | ||
| + | ||
| [,bash] | ||
| ---- | ||
| gcloud compute firewall-rules create redpanda-psc \ | ||
| --description="Allow access to Redpanda PSC endpoints" \ | ||
| --network="<shared-vpc-name>" \ | ||
| --project="<host-project-id>" \ | ||
| --direction="INGRESS" \ | ||
| --target-tags="redpanda-node" \ | ||
| --source-ranges="10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,100.64.0.0/10" \ | ||
| --allow="tcp:30181,tcp:30282,tcp:30292,tcp:31004,tcp:31082-31101,tcp:31182-31201,tcp:31282-31301,tcp:32092-32111,tcp:32192-32211,tcp:32292-32311" | ||
| ---- | ||
| + | ||
| Provide your values for the following placeholders: | ||
| + | ||
| - `<subnet-name>`: The name of the NAT subnet. | ||
| - `<host-project-id>`: The host GCP project ID. | ||
| - `<shared-vpc-name>`: The name of the VPC being used for your Redpanda Cloud cluster. | ||
| - `<region>`: The region of the Redpanda Cloud cluster. | ||
| - `<subnet-range>`: The CIDR range of the subnet. The mask should be at least `/29`. Each Private Service Connect connection takes up one IP address from the NAT subnet, so the CIDR must be able to accommodate all projects from which connections to the service attachment will be issued. | ||
| + | ||
| See the GCP documentation for https://cloud.google.com/vpc/docs/configure-private-service-connect-producer#add-subnet-psc[creating a subnet for Private Service Connect^]. | ||
| . For the accepted consumers list, you need the GCP project IDs from which incoming connections will be accepted. | ||
| . It may take several minutes for your cluster to update. When the update is complete, the Private Service Connect status in **Cluster settings** changes from **In progress** to **Enabled**. | ||
|
|
||
| === Deploy consumer-side resources | ||
|
|
||
| For each VPC network, you must complete the following steps to successfully connect to the service and use Kafka API and other Redpanda services such as HTTP Proxy. | ||
|
|
||
| . In **Cluster settings**, copy the **Service attachment URL** under **Private Service Connect**. Use this URL to create the Private Service Connect endpoint in GCP. | ||
|
|
||
| . Create a private DNS zone. Use the cluster **DNS zone** value as the DNS name. | ||
| + | ||
| [,bash] | ||
| ---- | ||
| gcloud dns --project=<gcp-project-id> managed-zones create <dns-zone-name> --description="<description>" --dns-name="<dns-zone-from-the-ui>" --visibility="private" --networks="<list-of-fully-qualified-names-of-networks-where-the-dns-zone-will-be-visible>" | ||
| ---- | ||
|
|
||
| . In the newly-created DNS zone, create a wildcard DNS record using the cluster **DNS record** value. | ||
| + | ||
| [,bash] | ||
| ---- | ||
| gcloud dns --project=<gcp-project-id> record-sets create '*.<dns-zone-from-the-ui>' --zone="<dns-zone-name>" --type="A" --ttl="300" --rrdatas="<psc-endpoint-ip>" | ||
| ---- | ||
|
|
||
| . Confirm that your GCP VPC firewall allows traffic to and from the Private Service Connect forwarding rule IP address, on the expected ports. | ||
|
|
||
| == Access Redpanda services through VPC endpoint | ||
|
|
||
| After you have enabled Private Service Connect for your cluster, your connection URLs are available in the *How to Connect* section of the cluster overview in the Redpanda Cloud UI. | ||
|
|
||
| include::networking:partial$private-links-access-rp-services-through-vpc.adoc[] | ||
|
|
||
| == Test the connection | ||
|
|
||
| You can test the connection to the endpoint service from any VM or container in the consumer VPC. If configuring a client isn't possible right away, you can do these checks using `rpk` or cURL: | ||
|
|
||
| include::networking:partial$private-links-test-connection.adoc[] | ||
|
|
||
| == Disable endpoint service | ||
|
|
||
| In **Cluster settings**, click **Disable**. Existing connections are closed after GCP Private Service Connect is disabled. To connect using Private Service Connect again, you must re-enable the service. | ||
| include::networking:partial$psc-ui.adoc[] |
157 changes: 157 additions & 0 deletions
157
modules/networking/pages/dedicated/gcp/configure-psc-in-api.adoc
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,157 @@ | ||
| = Configure GCP Private Service Connect with the Cloud API | ||
| :description: Set up GCP Private Service Connect to securely access Redpanda Cloud. | ||
| :env-dedicated: true | ||
|
|
||
| include::networking:partial$psc-api.adoc[] | ||
|
|
||
| == Create a new cluster with Private Service Connect | ||
|
|
||
| . In the https://cloud.redpanda.com/[Redpanda Cloud UI], go to **Resource groups** and select the resource group in which you want to create a cluster. | ||
| + | ||
| Copy and store the resource group ID (UUID) from the URL in the browser. | ||
| + | ||
| [,bash] | ||
| ---- | ||
| export RESOURCE_GROUP_ID=<uuid> | ||
| ---- | ||
|
|
||
| . Create VPC firewall rules to allow Private Service Connect traffic. Use the `gcloud` CLI to create the firewall rules: | ||
| + | ||
| NOTE: The firewall rules support up to 20 Redpanda brokers. If you have more than 20 brokers, or for help enabling Private Service Connect, contact https://support.redpanda.com/hc/en-us/requests/new[Redpanda support^]. | ||
| + | ||
| ``` | ||
| gcloud compute firewall-rules create redpanda-psc \ | ||
| --description="Allow access to Redpanda PSC endpoints" \ | ||
| --network="<shared-vpc-name>" \ | ||
| --project="<host-project-id>" \ | ||
| --direction="INGRESS" \ | ||
| --target-tags="redpanda-node" \ | ||
| --source-ranges="10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,100.64.0.0/10" \ | ||
| --allow="tcp:30181,tcp:30282,tcp:30292,tcp:31004,tcp:31082-31101,tcp:31182-31201,tcp:31282-31301,tcp:32092-32111,tcp:32192-32211,tcp:32292-32311" | ||
| ``` | ||
|
|
||
| . Make a request to the xref:api:ROOT:cloud-controlplane-api.adoc#post-/v1/networks[`POST /v1/networks`] endpoint to create a network. | ||
| + | ||
| [,bash] | ||
| ---- | ||
| network_post_body=`cat << EOF | ||
| { | ||
| "cloud_provider": "CLOUD_PROVIDER_GCP", | ||
| "cluster_type": "DEDICATED", | ||
| "name": "<shared-vpc-name>", | ||
| "resource_group_id": "$RESOURCE_GROUP_ID", | ||
| "region": "<region>", | ||
| "customer_managed_resources": { | ||
| "gcp": { | ||
| "network_name": "<network-name>", | ||
| "network_project_id": "<network-gcp-project-id>", | ||
| "management_bucket": { "name" : "<management-bucket>" } | ||
| } | ||
| } | ||
| } | ||
| EOF` | ||
|
|
||
| curl -vv -X POST \ | ||
| -H "Content-Type: application/json" \ | ||
| -H "Authorization: Bearer $AUTH_TOKEN" \ | ||
| -d "$network_post_body" $PUBLIC_API_ENDPOINT/v1/networks | ||
| ---- | ||
| + | ||
| Replace the following placeholder variables for the request body: | ||
| + | ||
| -- | ||
| - `<shared-vpc-name>`: The name for the network. | ||
| - `<region>`: The GCP region where the network will be created. | ||
| - `<network-gcp-project-id>`: The ID of the GCP project where your VPC is created. | ||
| - `<network-name>`: The name of your VPC. | ||
| - `<management-bucket>`: The name of the Google Storage bucket you created for the cluster. | ||
| -- | ||
|
|
||
|
|
||
| . Store the network ID (`metadata.network_id`) returned in the response to the Create Network request. | ||
| + | ||
| [,bash] | ||
| ---- | ||
| export NETWORK_ID=<metadata.network_id> | ||
| ---- | ||
|
|
||
| . Make a request to the xref:api:ROOT:cloud-controlplane-api.adoc#post-/v1/clusters[`POST /v1/clusters`] endpoint to create a Redpanda Cloud cluster with Private Service Connect enabled. | ||
| + | ||
| [,bash] | ||
| ---- | ||
| export CLUSTER_POST_BODY=`cat << EOF | ||
| { | ||
| "cloud_provider": "CLOUD_PROVIDER_GCP", | ||
| "connection_type": "CONNECTION_TYPE_PRIVATE", | ||
| "type": "DEDICATED", | ||
| "name": "<cluster-name>", | ||
| "resource_group_id": "$RESOURCE_GROUP_ID", | ||
| "network_id": "$NETWORK_ID", | ||
| "region": "<region>", | ||
| "zones": <zones>, | ||
| "throughput_tier": "<throughput-tier>", | ||
| "redpanda_version": "<redpanda-version>", | ||
| "gcp_private_service_connect": { | ||
| "enabled": true, | ||
| "consumer_accept_list": <consumer-accept-list> | ||
| }, | ||
| } | ||
| EOF` | ||
|
|
||
| curl -vv -X POST \ | ||
| -H "Content-Type: application/json" \ | ||
| -H "Authorization: Bearer $AUTH_TOKEN" \ | ||
| -d "$CLUSTER_POST_BODY" $PUBLIC_API_ENDPOINT/v1/clusters | ||
| ---- | ||
| + | ||
| -- | ||
| - `<cluster-name>`: Provide a name for the new cluster. | ||
| - `<region>`: Choose a GCP region where the network will be created. | ||
| - `<zones>`: Provide the list of GCP zones where the brokers will be deployed. Format: `["<zone 1>", "<zone 2>", "<zone N>"]` | ||
| - `<throughput-tier>`: Choose a Redpanda Cloud cluster tier. For example, `tier-1-gcp-v2-x86`. | ||
| - `<redpanda-version>`: Choose the Redpanda Cloud version. | ||
| - `<consumer-accept-list>`: The list of IDs of GCP projects from which Private Service Connect connection requests are accepted. Format: `[{"source": "<GCP-project-ID-1>"}, {"source": "<GCP-project-I-2>"}, {"source": "<GCP-project-ID-N>"}]` | ||
| -- | ||
|
|
||
| == Enable Private Service Connect on an existing cluster | ||
|
|
||
| CAUTION: As soon as Private Service Connect is available on your VPC, all communication on existing Redpanda bootstrap server and broker ports is interrupted due to the change on the private DNS resolution. Make sure all applications running in your VPC are ready to start using the corresponding Private Service Connect ports. | ||
|
|
||
| . In the Redpanda Cloud UI, go to the cluster overview and copy the cluster ID from the **Details** section. | ||
| + | ||
| [,bash] | ||
| ---- | ||
| export CLUSTER_ID=<cluster-id> | ||
| ---- | ||
|
|
||
|
|
||
| . Make a xref:api:ROOT:cloud-controlplane-api.adoc#patch-/v1/clusters/-cluster.id-[`PATCH /v1/clusters/{cluster.id}`] request to update the cluster to enable Private Service Connect. | ||
| + | ||
| [,bash] | ||
| ---- | ||
| CLUSTER_PATCH_BODY=`cat << EOF | ||
| { | ||
| "gcp_private_service_connect": { | ||
| "enabled": true, | ||
| "consumer_accept_list": <accept-list> | ||
| } | ||
| } | ||
| EOF` | ||
| curl -v -X PATCH \ | ||
| -H "Content-Type: application/json" \ | ||
| -H "Authorization: Bearer $AUTH_TOKEN" \ | ||
| -d "$CLUSTER_PATCH_BODY" $PUBLIC_API_ENDPOINT/v1/clusters/$CLUSTER_ID | ||
| ---- | ||
| + | ||
| Replace the following placeholder: | ||
| + | ||
| `<accept-list>`: a JSON list specifying the projects from which incoming connections will be accepted. All other sources. For example, `[{"source": "consumer-project-ID-1"},{"source": "consumer-project-ID-2"}]`. | ||
| + | ||
| Wait for the cluster to apply the new configuration (around 15 minutes). The Private Service Connect attachment is available when the cluster update is complete. To monitor the service attachment creation, run the following `gcloud` command with the project ID: | ||
| + | ||
| [,bash] | ||
| ---- | ||
| gcloud compute service-attachments list --project '<service-project-id>' | ||
| ---- | ||
|
|
||
paulzhang97 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| include::networking:partial$psc-api2.adoc[] | ||
5 changes: 5 additions & 0 deletions
5
modules/networking/pages/dedicated/gcp/configure-psc-in-ui.adoc
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| = Configure GCP Private Service Connect in the Cloud UI | ||
| :description: Set up GCP Private Service Connect in the Redpanda Cloud UI. | ||
| :env-dedicated: true | ||
|
|
||
| include::networking:partial$psc-ui.adoc[] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.