From 0aa3e363423f0a90e860842954c5f9f23828029a Mon Sep 17 00:00:00 2001 From: Gavin Frazar Date: Wed, 7 Feb 2024 16:27:42 -0800 Subject: [PATCH 1/2] backport #37606 to branch/v14. --- .../guides/cockroachdb-self-hosted.mdx | 114 +++++++++++++--- .../guides/mongodb-self-hosted.mdx | 41 ++---- .../database-access/guides/redis-cluster.mdx | 122 ++++++------------ docs/pages/database-access/guides/redis.mdx | 8 +- .../includes/database-access/custom-db-ca.mdx | 15 +++ .../database-access/split-db-ca-details.mdx | 21 +++ 6 files changed, 180 insertions(+), 141 deletions(-) create mode 100644 docs/pages/includes/database-access/custom-db-ca.mdx create mode 100644 docs/pages/includes/database-access/split-db-ca-details.mdx diff --git a/docs/pages/database-access/guides/cockroachdb-self-hosted.mdx b/docs/pages/database-access/guides/cockroachdb-self-hosted.mdx index e6fdb1fafea57..47b52206df222 100644 --- a/docs/pages/database-access/guides/cockroachdb-self-hosted.mdx +++ b/docs/pages/database-access/guides/cockroachdb-self-hosted.mdx @@ -20,17 +20,16 @@ description: How to configure Teleport database access with self-hosted Cockroac (!docs/pages/includes/edition-prereqs-tabs.mdx!) - CockroachDB cluster. + - A host, e.g., an Amazon EC2 instance, where you will run the Teleport Database Service. + - (!docs/pages/includes/tctl.mdx!) + - A certificate authority to issue CockroachDB certificates for nodes in your CockroachDB cluster. - Teleport uses a split CA architecture for database access, and requires that - you create your own CA for node-to-node mTLS communication. - See [cockroach cert](https://www.cockroachlabs.com/docs/stable/cockroach-cert) - for instructions on creating a CA and issuing a client certificate and key - for each node. The certificate and key should be named `client.node.crt` and - `client.node.key`. + + (!docs/pages/includes/database-access/split-db-ca-details.mdx db="CockroachDB"!) ## Step 1/4. Set up the Teleport Database Service @@ -81,49 +80,128 @@ in the CockroachDB documentation for more information. ### Set up mutual TLS -(!docs/pages/includes/database-access/tctl-auth-sign.mdx!) - To set up mutual TLS authentication, you need to make sure that: - Teleport trusts certificates presented by CockroachDB nodes. -- CockroachDB trusts client certificates signed by Teleport. +- CockroachDB nodes trust client certificates signed by both your CockroachDB CA + and your Teleport cluster's `db_client` CA. + +CockroachDB nodes need to trust the Teleport `db_client` CA so that Teleport +users can authenticate to your CockroachDB cluster as clients. + +The CockroachDB CA needs to be trusted by each CockroachDB node so that nodes +can authenticate themselves as clients to other nodes in the CockroachDB +cluster. This is because CockroachDB uses mTLS for node-to-node communication. + + + +In this configuration, your CockroachDB CA will be used to issue the server cert +`node.crt` for each CockroachDB node. -Generate the secrets by running the following `tctl` command against your -Teleport cluster: +This configuration is simpler to set up, because an existing CockroachDB cluster +already has `node.crt` issued for each node and you only need to configure the +CockroachDB nodes to trust your Teleport `db_client` CA. +Another benefit is that your CockroachDB nodes will continue to serve the same +CockroachDB CA-issued cert, rather than serving a new cert signed by Teleport's +`db` CA, so you don't have to configure other clients to trust a new CA. + +Copy your CockroachDB CA cert to `ca-client.crt` in the certs directory of +each CockroachDB node: + +```code +$ CERTS_DIR= +$ cp "${CERTS_DIR}/ca.crt" "${CERTS_DIR}/ca-client.crt" +``` + +Next, for each CockroachDB node, export Teleport's `db_client` CA using `tctl` +(or export it once and copy it to each node) and append the certificate to +`ca-client.crt`: + +```code +$ tctl auth export --type=db_client >> /ca-client.crt +``` + +(!docs/pages/includes/database-access/custom-db-ca.mdx db="CockroachDB" protocol="cockroachdb" port="26257"!) + + +In this configuration, Teleport's CA will be used to issue the server cert, +`node.crt`, and your own custom CA will be used to issue the client certificate, +`client.node.crt`, for each CockroachDB node. + +(!docs/pages/includes/database-access/tctl-auth-sign.mdx!) + +Generate secrets for a CockroachDB node using `tctl`: ```code $ tctl auth sign \ --format=cockroachdb \ --host=roach.example.com \ - --out=/path/to/cockroach/certs/dir/ \ + --out= \ --ttl=2190h ``` +(!docs/pages/includes/database-access/ttl-note.mdx!) + The command will produce 4 files: - `ca.crt` with Teleport's `db` certificate authority - `ca-client.crt` with Teleport's `db_client` certificate authority - `node.crt` / `node.key` with the node's certificate and key. + +You can specify multiple comma-separated addresses e.g. +`--host=roach,node-1,192.168.1.1`. +However, you must include the hostname that Teleport will use to connect to the +database. + + Do not rename these files as this is how CockroachDB expects them to be named. See [Node key and certificates](https://www.cockroachlabs.com/docs/v21.1/create-security-certificates-custom-ca#node-key-and-certificates) for details. -Generate the secrets for each cluster node and make sure to use the hostname -Teleport will be using to connect to the nodes in the `--host` flag. +Prepend your CockroachDB CA's certificate to `ca-client.crt`. +Now issue a client certificate for the node using your CockroachDB CA: - - You can specify multiple comma-separated addresses e.g. `--host=roach,node-1,192.168.1.1`. - +```code +$ cockroach cert create-client node \ + --certs-dir= \ + --ca-key=ca-secrets/ca-client.key +``` + +
+If you see an error message like: `tls: private key does not match public key`, +it likely means you did not prepend your CockroachDB CA cert to `ca-client.crt` +earlier. + +`cockroach cert create-client` expects the first certificate in `ca-client.crt` +(in the `--certs-dir` specified) to be the certificate signed by `--ca-key`. +Ensure that your CockroachDB CA certificate is the first certificate in +`ca-client.crt`. +
+ +Now copy `` to the CockroachDB +node and repeat these steps for all of your other CockroachDB nodes. + + Restart your CockroachDB nodes, passing them the directory with generated secrets via the `--certs-dir` flag: ```code $ cockroach start \ - --certs-dir=/path/to/cockroachdb/certs/dir/ \ + --certs-dir= \ # other flags... ``` +Alternatively, if the nodes were already started with +`--certs-dir=`, you can send a +`SIGHUP` signal to the `cockroach` process to reload certificates without +restarting the node. You must send `SIGHUP` as the same user that started the +`cockroach` process: + +```code +$ pkill -SIGHUP -x cockroach +``` + ## Step 4/4. Connect Log in to your Teleport cluster. Your CockroachDB cluster should appear in the diff --git a/docs/pages/database-access/guides/mongodb-self-hosted.mdx b/docs/pages/database-access/guides/mongodb-self-hosted.mdx index b501e6fe2efc7..f935153173cd2 100644 --- a/docs/pages/database-access/guides/mongodb-self-hosted.mdx +++ b/docs/pages/database-access/guides/mongodb-self-hosted.mdx @@ -22,27 +22,18 @@ videoBanner: 6lgVObxoLkc - MongoDB cluster (standalone or replica set) version `(=mongodb.min_version=)` or newer. - + Teleport database access supports MongoDB `(=mongodb.min_version=)` and newer. Older versions have not been tested and are not guaranteed to work. MongoDB `(=mongodb.min_version=)` was released in November 2017 and reached EOL in April 2021 so if you're still using an older version, consider upgrading. - + - (!docs/pages/includes/tctl.mdx!) - A certificate authority for MongoDB Replica Set, and the public certificate - for that CA, in PEM format, e.g. `/path/to/your/ca.crt`. + for that CA, in PEM format: ``. - - A MongoDB replica set requires that each MongoDB server is configured with a - certificate that has extendedKeyUsage `clientAuth, serverAuth`. - - Teleport uses a split CA architecture for better security: - the Teleport Database Server CA issues certs with `serverAuth` - and the Teleport Database Client CA issues certs with `clientAuth`. - Therefore, you must provide each MongoDB server with a certificate issued by - your own CA with extendedKeyUsage `clientAuth, serverAuth`. - + (!docs/pages/includes/database-access/split-db-ca-details.mdx db="MongoDB Replica Set"!) ## Step 1/3. Install and configure Teleport @@ -124,12 +115,10 @@ accordingly to grant the user appropriate database permissions. ### Set up mutual TLS -(!docs/pages/includes/database-access/tctl-auth-sign.mdx!) - -Create the secrets: - + (!docs/pages/includes/database-access/tctl-auth-sign.mdx!) + When connecting to standalone MongoDB, sign the certificate for the hostname over which Teleport will be connecting to it. @@ -152,28 +141,14 @@ Create the secrets: ```code $ tctl auth export --type=db_client > db-client-ca.crt - $ cat /path/to/your/ca.crt db-client-ca.crt > /etc/certs/mongo.cas + $ cat db-client-ca.crt > /etc/certs/mongo.cas ``` When MongoDB is configured to trust these CAs, it will trust the Teleport Database Client CA and allow access via Teleport, while still allowing MongoDB replication over TLS using your CA's certs for peer verification. - Next, modify your Teleport Database Service to trust your MongoDB Replica - Set CA: - ```yaml - databases: - - name: "example-mongo" - protocol: "mongodb" - uri: "mongo.example.com:27017" - static_labels: - "env": "dev" - tls: - ca_cert_file: "/path/to/your/ca.crt" - ``` - - Now the Teleport Database Service will trust certificates presented by your - MongoDB Replica Set. + (!docs/pages/includes/database-access/custom-db-ca.mdx db="MongoDB Replica Set" protocol="mongodb" port="27017"!) diff --git a/docs/pages/database-access/guides/redis-cluster.mdx b/docs/pages/database-access/guides/redis-cluster.mdx index 86dbc1e229a05..0a5eb6eb3bebf 100644 --- a/docs/pages/database-access/guides/redis-cluster.mdx +++ b/docs/pages/database-access/guides/redis-cluster.mdx @@ -22,12 +22,19 @@ If you want to configure Redis Standalone, please read [Database Access with Red (!docs/pages/includes/edition-prereqs-tabs.mdx!) - Redis version `6.0` or newer. + + + RESP3 (REdis Serialization Protocol) is currently not supported. + + - `redis-cli` version `6.2` or newer installed and added to your system's `PATH` environment variable. + - A host where you will run the Teleport Database Service. - - Redis `7.0` and RESP3 (REdis Serialization Protocol) are currently not supported. - +- A certificate authority to issue server certificates for nodes in your Redis + Cluster. + + (!docs/pages/includes/database-access/split-db-ca-details.mdx db="Redis Cluster"!) - (!docs/pages/includes/tctl.mdx!) @@ -66,43 +73,16 @@ Install and configure Teleport where you will run the Teleport Database Service: ## Step 4/6. Set up mutual TLS -(!docs/pages/includes/database-access/tctl-auth-sign.mdx!) - -We will show you how to use the `tctl auth sign` command below. - -When connecting to Redis Cluster, sign certificates for each member -using their hostnames and IP addresses. -For example, if the first member is accessible at `redis1.example.com` with IP `10.0.0.1` and -the second at `redis2.example.com` with IP `10.0.0.2`, run: +Export your Teleport cluster's `db_client` CA cert and concatenate it with your Redis +Cluster's CA cert (in PEM format): ```code -$ tctl auth sign --format=redis --host=redis1.example.com,10.0.0.1 --out=redis1 --ttl=2190h -$ tctl auth sign --format=redis --host=redis2.example.com,10.0.0.2 --out=redis2 --ttl=2190h +$ tctl auth export --type=db_client > db-client-ca.crt +$ cat db-client-ca.crt > pem-bundle.cas ``` -(!docs/pages/includes/database-access/ttl-note.mdx!) - -The command will create three files: -- `out.cas` with Teleport's database client certificate authority -- `out.key` with a generated private key -- `out.crt` with a generated certificate issued by Teleport's database server -certificate authority - -You will need these files to enable mutual TLS on your Redis server. - - - -If you wish to enable intra-cluster TLS communication in Redis, you will need -to generate and manage a CA to issue certificates for Redis Cluster nodes, -and configure your Teleport Database Service to trust that CA. - -Concatenate your CA's cert (in PEM format) with Teleport's database client -certificate authority: -```sh -cat /path/to/your/ca.crt out.cas > pem-bundle.cas -``` - -Using your CA, issue `server.crt` for your Redis Cluster node and enable mutual -TLS in your `redis.conf` configuration file, then restart the database: +Using your Redis Cluster's CA, issue `server.crt` for each of your Redis Cluster +nodes and enable mutual TLS in your `redis.conf` configuration file, then +restart each node: ```ini tls-port 7001 @@ -119,43 +99,6 @@ tls-ca-cert-file /usr/local/etc/redis/certs/pem-bundle.cas tls-protocols "TLSv1.2 TLSv1.3" ``` -Modify your Teleport Database Service to trust your Redis Cluster CA: -```yaml - databases: - - name: "example-redis" - protocol: "redis" - uri: "rediss://redis.example.com:6379?mode=cluster" - static_labels: - "env": "dev" - tls: - ca_cert_file: "/path/to/your/ca.crt" -``` - - -Use the generated secrets to enable mutual TLS in your `redis.conf` configuration -file and restart the database: - -```ini -tls-port 7001 -port 0 -cluster-enabled yes -tls-replication no -tls-cluster no -aclfile /path/to/users.acl -masterauth GENERATED_STRONG_PASSWORD -masteruser replica-user -tls-cert-file /usr/local/etc/redis/certs/server.crt -tls-key-file /usr/local/etc/redis/certs/server.key -tls-ca-cert-file /usr/local/etc/redis/certs/server.cas -tls-protocols "TLSv1.2 TLSv1.3" -``` - -With this configuration, communication between user clients and the Redis -Cluster will still use TLS, but Redis nodes will not use TLS to communicate with -each other. - - - Once mutual TLS has been enabled, you will no longer be able to connect to the cluster without providing a valid client certificate. You can use the `tls-auth-clients optional` setting to allow connections @@ -164,21 +107,28 @@ from clients that do not present a certificate. See [TLS Support](https://redis.io/topics/encryption) in the Redis documentation for more details. +(!docs/pages/includes/database-access/custom-db-ca.mdx db="Redis Cluster" protocol="redis" scheme="rediss://" port="6379" query="?mode=cluster"!) ## Step 5/6. Create a cluster +To create the cluster after mutual TLS is enabled, you will need to use a +certificate that the Redis nodes trust for client authentication. +You can use a certificate that you already issued for one of the nodes, i.e. +`server.crt` or you can issue a new client certificate using your Redis Cluster +CA. + Use the following command to create the cluster. Please note `redis-cli --cluster create` accepts only IP addresses. -```sh -export REDISCLI_AUTH=STRONG_GENERATED_PASSWORD -export CERTS_DIR=/path/to/certs/ -export IP1=10.0.0.1 # update with the real node 1 IP -export IP2=10.0.0.2 # update with the real node 2 IP -export IP3=10.0.0.3 # update with the real node 3 IP -export IP4=10.0.0.4 # update with the real node 4 IP -export IP5=10.0.0.5 # update with the real node 5 IP -export IP6=10.0.0.6 # update with the real node 6 IP -redis-cli --user alice --cluster-replicas 1 --tls --cluster-yes \ ---cluster create ${IP1}:7001 ${IP2}:7002 ${IP3}:7003 ${IP4}:7004 ${IP5}:7005 ${IP6}:7006 \ ---cacert ${CERTS_DIR}/server.cas --key ${CERTS_DIR}/server.key --cert ${CERTS_DIR}/server.crt +```code +$ export REDISCLI_AUTH=STRONG_GENERATED_PASSWORD +$ export CERTS_DIR=/path/to/certs/ +$ export IP1=10.0.0.1 # update with the real node 1 IP +$ export IP2=10.0.0.2 # update with the real node 2 IP +$ export IP3=10.0.0.3 # update with the real node 3 IP +$ export IP4=10.0.0.4 # update with the real node 4 IP +$ export IP5=10.0.0.5 # update with the real node 5 IP +$ export IP6=10.0.0.6 # update with the real node 6 IP +$ redis-cli --user alice --cluster-replicas 1 --tls --cluster-yes \ + --cluster create ${IP1}:7001 ${IP2}:7002 ${IP3}:7003 ${IP4}:7004 ${IP5}:7005 ${IP6}:7006 \ + --cacert ${CERTS_DIR}/ca.crt --key ${CERTS_DIR}/server.key --cert ${CERTS_DIR}/server.crt ``` ## Step 6/6. Connect diff --git a/docs/pages/database-access/guides/redis.mdx b/docs/pages/database-access/guides/redis.mdx index 04dd0be020ce6..a6f50feff7856 100644 --- a/docs/pages/database-access/guides/redis.mdx +++ b/docs/pages/database-access/guides/redis.mdx @@ -23,16 +23,16 @@ If you want to configure Redis Cluster, please read [Database Access with Redis - Redis version `6.0` or newer. + + RESP3 (REdis Serialization Protocol) is currently not supported. + + - `redis-cli` version `6.2` or newer installed and added to your system's `PATH` environment variable. - A host where you will run the Teleport Database Service. See [Installation](../../installation.mdx) for details. - - Redis `7.0` and RESP3 (REdis Serialization Protocol) are currently not supported. - - - (!docs/pages/includes/tctl.mdx!) ## Step 1/5. Set up the Teleport Database Service diff --git a/docs/pages/includes/database-access/custom-db-ca.mdx b/docs/pages/includes/database-access/custom-db-ca.mdx new file mode 100644 index 0000000000000..96ed9057512b9 --- /dev/null +++ b/docs/pages/includes/database-access/custom-db-ca.mdx @@ -0,0 +1,15 @@ +{{ scheme="" query="" }} +Modify the Teleport Database Service to trust your {{ db }} CA: + ```yaml + databases: + - name: "example-{{ protocol }}" + protocol: "{{ protocol }}" + uri: "{{ scheme }}{{ protocol }}.example.com:{{ port }}{{ query }}" + static_labels: + "env": "example" + tls: + ca_cert_file: "" + ``` + +Now the Teleport Database Service will trust certificates presented by your +{{ db }}. diff --git a/docs/pages/includes/database-access/split-db-ca-details.mdx b/docs/pages/includes/database-access/split-db-ca-details.mdx new file mode 100644 index 0000000000000..1605a45b7127b --- /dev/null +++ b/docs/pages/includes/database-access/split-db-ca-details.mdx @@ -0,0 +1,21 @@ +
+Distributed databases like {{ db }} use mTLS for node-to-node communication. +Teleport requires that you have your own CA to issue certificates for +node-to-node mTLS communication. + +Teleport uses a split-CA architecture for database access. The Teleport `db` CA +issues server certificates and the `db_client` CA issues client certificates. + +Databases are configured to trust the Teleport `db_client` CA for client +authentication, but not the `db` CA. +Additionally, Teleport only issues *ephemeral* `db_client` CA certificates. + +When a {{ db }} node connects to another {{ db }} node, it must present a +certificate that the other node trusts for client authentication. +Since Teleport does not issue long-lived `db_client` certificates, the node +needs to have a long-lived certificate issued by another CA that its peer node +trusts. + +The split `db` and `db_client` CA architecture was introduced as a security fix +in Teleport `15`. +
From 96329377cecfcac2838ab387024aa78c29963402 Mon Sep 17 00:00:00 2001 From: Gavin Frazar Date: Tue, 20 Feb 2024 17:49:47 -0800 Subject: [PATCH 2/2] backport #38137 to branch/v14 --- docs/config.json | 12 +- docs/cspell.json | 1 + .../reference/configuration.mdx | 21 +- docs/pages/management/operations.mdx | 2 + .../management/operations/ca-rotation.mdx | 29 ++- .../operations/db-ca-migrations.mdx | 176 ++++++++++++++ .../management/operations/db-ca-rotation.mdx | 228 ++++++++++++++++++ 7 files changed, 450 insertions(+), 19 deletions(-) create mode 100644 docs/pages/management/operations/db-ca-migrations.mdx create mode 100644 docs/pages/management/operations/db-ca-rotation.mdx diff --git a/docs/config.json b/docs/config.json index a7dfe3a3b4740..dd86ba930dd51 100644 --- a/docs/config.json +++ b/docs/config.json @@ -893,6 +893,10 @@ "title": "Cert Authority Rotation", "slug": "/management/operations/ca-rotation/" }, + { + "title": "Database Cert Authority Rotation", + "slug": "/management/operations/db-ca-rotation/" + }, { "title": "TLS Routing Migration", "slug": "/management/operations/tls-routing/", @@ -904,9 +908,11 @@ { "title": "Proxy Peering Migration", "slug": "/management/operations/proxy-peering/", - "forScopes": [ - "enterprise" - ] + "forScopes": ["enterprise"] + }, + { + "title": "Database CA Migrations", + "slug": "/management/operations/db-ca-migrations/" } ] }, diff --git a/docs/cspell.json b/docs/cspell.json index 8c8c9ed8fc50a..5da57f61615d6 100644 --- a/docs/cspell.json +++ b/docs/cspell.json @@ -620,6 +620,7 @@ "nodename", "nohup", "nologin", + "noout", "noprompt", "nosql", "nowait", diff --git a/docs/pages/database-access/reference/configuration.mdx b/docs/pages/database-access/reference/configuration.mdx index 748d61f06f014..de8fafd156fa6 100644 --- a/docs/pages/database-access/reference/configuration.mdx +++ b/docs/pages/database-access/reference/configuration.mdx @@ -110,11 +110,22 @@ spec: # Database connection endpoint. uri: "localhost:5432" - # Optional CA for validating the database certificate. - ca_cert: | - -----BEGIN CERTIFICATE----- - ... - -----END CERTIFICATE----- + # Optional TLS configuration. + tls: + # TLS verification mode. Valid options are: + # 'verify-full' - performs full certificate validation (default). + # 'verify-ca' - the same as `verify-full`, but skips the server name validation. + # 'insecure' - accepts any certificate provided by database (not recommended). + mode: verify-full + # Optional database DNS server name. It allows to override the DNS name on + # a client certificate when connecting to a database. + # Use only with 'verify-full' mode. + server_name: db.example.com + # Optional CA for validating the database certificate. + ca_cert: | + -----BEGIN CERTIFICATE----- + ... + -----END CERTIFICATE----- # Database admin user for automatic user provisioning. admin_user: diff --git a/docs/pages/management/operations.mdx b/docs/pages/management/operations.mdx index 4d6a3f80a184c..602159573383b 100644 --- a/docs/pages/management/operations.mdx +++ b/docs/pages/management/operations.mdx @@ -13,5 +13,7 @@ the [Cluster Administration Guides](./admin.mdx) section. - [Scaling](./operations/scaling.mdx): How to configure Teleport for large-scale deployments. - [Backup and Restore](./operations/backup-restore.mdx): Backing up and restoring the cluster. - [CA Rotation](./operations/ca-rotation.mdx): Rotating Teleport certificate authorities. +- [Database CA Rotation](./operations/db-ca-rotation.mdx): Rotating Teleport's `db` or `db_client` certificate authorities. - [TLS Routing Migration](./operations/tls-routing.mdx): Migrating your Teleport cluster to single-port TLS routing mode. - [Proxy Peering Migration](./operations/proxy-peering.mdx): Migrating your Teleport cluster to Proxy Peering mode. +- [Database CA Migrations](./operations/db-ca-migrations.mdx): Completing Teleport's Database CA migrations. diff --git a/docs/pages/management/operations/ca-rotation.mdx b/docs/pages/management/operations/ca-rotation.mdx index 2f1fa3e1e72c5..54001014d9d40 100644 --- a/docs/pages/management/operations/ca-rotation.mdx +++ b/docs/pages/management/operations/ca-rotation.mdx @@ -16,13 +16,20 @@ This section will show you how to rotate Teleport's certificate authority. If you are joining Teleport processes to a cluster via the Teleport Auth Service using a [join token](../../agents/join-services-to-your-cluster/join-token.mdx), each Teleport process will need a CA pin to trust the Auth Service. The CA pin will -change after each CA rotation. Make sure you use the *new* CA pin when adding -Teleport services after rotation. +change after each `host` CA rotation. Make sure you use the *new* CA pin when adding +Teleport services after `host` CA rotation. - -Teleport signs Windows Desktop certificates with the user certificate authority. -If the user CA is rotated, the new CA certificate must be exported and + +Teleport signs Windows Desktop client certificates with the `user` certificate +authority. +If the `user` CA is rotated, the new CA certificate must be exported and configured in group policy. + +Teleport signs self-hosted database host certificates with the `db` certificate +authority and signs database client certificates with the `db_client` CA. +If either of these CAs is rotated, then self-hosted databases must be +reconfigured. +Refer to [the Database CA Rotation Guide](./db-ca-rotation.mdx). ### Rotation phases @@ -54,7 +61,7 @@ There are two kinds of certificate rotations: - **Semi-automatic:** Teleport automatically transitions between phases of the rotation after some amount of time (known as a *grace period*) elapses. -For both types of rotations, the cluster goes through the phases in the +For both types of rotation, the cluster goes through the phases in the following order: - `standby` -> `init` -> `update_clients` -> `update_servers` -> `standby` @@ -63,7 +70,7 @@ Administrators can abort the rotation and revert all changes any time before the rotation is completed by entering the `rollback` phase. ```code -$ tctl auth rotate --phase=rollback --type= --manual +$ tctl auth rotate --manual --type= --phase=rollback ``` For example, if an admin has detected that some nodes failed to upgrade during @@ -90,7 +97,7 @@ of the cluster. Initiate the manual rotation of host certificate authorities: ```code -$ tctl auth rotate --phase=init --type= --manual +$ tctl auth rotate --manual --type= --phase=init Updated rotation phase to "init". To check status use 'tctl status' ``` @@ -134,7 +141,7 @@ transitions. Execute the transition from `init` to `update_clients`: ```code -$ tctl auth rotate --phase=update_clients --type= --manual +$ tctl auth rotate --manual --type= --phase=update_clients # Updated rotation phase to "update_clients". To check status use 'tctl status' $ tctl status # Cluster acme.cluster @@ -164,7 +171,7 @@ Now that all nodes have caught up, execute the transition from `update_clients` to `update_servers`: ```code -$ tctl auth rotate --phase=update_servers --type= --manual +$ tctl auth rotate --manual --type= --phase=update_servers # Updated rotation phase to "update_servers". To check status use 'tctl status' $ tctl status @@ -205,7 +212,7 @@ $ tsh ssh hello@terminal ```code -$ tctl auth rotate --phase=standby --type= --manual +$ tctl auth rotate --manual --type= --phase=standby ``` Verify that the rotation has completed with `tctl`: diff --git a/docs/pages/management/operations/db-ca-migrations.mdx b/docs/pages/management/operations/db-ca-migrations.mdx new file mode 100644 index 0000000000000..805a0c96a9238 --- /dev/null +++ b/docs/pages/management/operations/db-ca-migrations.mdx @@ -0,0 +1,176 @@ +--- +title: Database CA Migrations +description: How to complete Teleport Database CA migrations. +--- + +In Teleport, self-hosted databases must be configured with certificates to +enable mTLS authentication via the Teleport Database Service. + +Teleport 15 introduced a new `db_client` certificate authority (CA) to split the +responsibilities of the Teleport `db` CA, which was acting as both host and +client CA for Teleport self-hosted database access. + +Teleport's host/client database CA split is intended to limit the potential for +lateral movement to other resources in the event that a database instance's +private key is compromised. + +The `db` and `db_client` CAs were both introduced as an automatic migration +after upgrading to Teleport 10 and Teleport 15, respectively. + +This guide will explain why these CAs were created and how to determine if your +CA(s) should be rotated to complete the migration process. + +## Prerequisites + +(!docs/pages/includes/edition-prereqs-tabs.mdx!) + +- (!docs/pages/includes/tctl.mdx!) +- A Teleport cluster that was created prior to Teleport 15. + If your Teleport cluster was created with Teleport 15+, then this guide does + not apply to your cluster, because your `db` and `db_client` CAs were not + migrated. + +## Teleport `db` CA migration + +Your Teleport cluster's `db` CA can be used to issue certificates to self-hosted +databases. +This is convenient, because the Teleport Database Service trusts certificates +issued by the `db` CA by default, so there is no additional TLS configuration +in Teleport required. + +Alternatively, you can issue certificates to your self-hosted databases using +an external CA - you just need to configure the Teleport Database Service to +trust that CA when connecting to your database(s). + +
+For a static database defined in your Teleport Database Service `teleport.yaml` +configuration file, set `tls.ca_cert_file` to a file containing your CA's root +certificate. + +For a dynamic database, put your CA's root certificate in `spec.tls.ca_cert`. + +For examples and more information, consult the +[Database Access Configuration Reference](../../database-access/reference/configuration.mdx). +
+ +Prior to Teleport 10, the Teleport `host` CA was used to issue certificates to +self-hosted databases (via `tctl auth sign`). +The Teleport `db` CA was introduced to decouple self-hosted database CA rotation +from the rest of your Teleport cluster. +The idea is that you should be able to rotate the CA used for self-hosted +databases without affecting other resources connected to your cluster. +Likewise, when you rotate your cluster's `host` CA, you should not have to worry +about affecting self-hosted databases. + +To avoid breaking database access after upgrading to Teleport 10, Teleport +clusters are automatically migrated to create the `db` CA as a copy of +the `host` CA. + +If your cluster was upgraded to Teleport 10 and you use Teleport to issue +certificates to your self-hosted databases, then you should ensure that you have +completed the `db` CA migration. +Otherwise, if you later rotate just one CA for any reason, a copy of the old CA +will still exist. +While this does not necessarily lead to a vulnerability in your cluster, it is +bad security practice to keep an old CA around after rotating it. + +To complete the `db` CA migration: +- we recommend rotating your `host` CA +- we **strongly recommend** rotating your `db` CA + +## Teleport `db_client` CA migration + +The Teleport Database Service needs to authenticate itself to self-hosted +database(s) using a client certificate, which requires that you configure your +database(s) to trust Teleport's `db_client` CA. +Prior to the introduction of the `db_client` CA in Teleport 15, self-hosted +had to be configured to trust the Teleport `db` CA for client authentication. + +With the old approach - trusting the `db` CA for client connections - if a +database's private key is compromised, and a `db` certificate was issued for +that key, then it could be used to gain access to other databases. + +Not all self-hosted databases are vulnerable to lateral movement after a private +key compromise. +For example, MySQL and PostgreSQL both verify that a client's certificate +subject matches the client's database user. +Other databases only verify that a client's certificate is trusted, but do not +match the certificate subject to the database username. +For example, Cassandra, ScyllaDB, and Redis do not verify the client cert +subject. +All of these databases can be configured to require password authentication +after a successful mTLS handshake. +However, for defense in depth, these databases should only mTLS handshake with +a client that presents a `db_client` CA-issued certificate. + +If your Teleport cluster was upgraded to Teleport 15, then you should ensure +that you have completed the `db_client` migration. +To complete the `db_client` CA migration: +- we recommend rotating your `db` CA +- we **strongly recommend** rotating your `db_client` CA. +- we **strongly recommend** reconfiguring your databases' certificates after + you complete the `db_client` CA rotation. + + + If you use `tctl auth sign` to reconfigure a database's certificates during + a `db_client` CA rotation, then the trusted certificate output will include + both the old and the new CA certificates. + To complete the migration, you should reconfigure those databases again after + the rotation - that way they will only trust the new CA. + + If you don't want to reconfigure each database both during and after the + `db_client` CA rotation, and you do not mind temporarily losing connectivity + to your databases via Teleport, then you can just complete the `db_client` CA + rotation and reconfigure your databases afterward. + + +## 1/2. Check for Teleport CA migrations + +If you upgraded your cluster to Teleport 10 and you have never rotated your +`host` or `db` CAs, then you should complete the `db` CA migration. + +If you upgraded your cluster to Teleport 15 and you have never rotated your +`db` or `db_client` CAs, then you should complete the `db_client` CA migration. + +If you are unsure whether you need to complete the migration for either the `db` +or `db_client` CAs, you can check for duplicated CAs. +Use these commands to print the X.509 certificate serial number for your `host`, +`db`, and `db_client` CAs (in that order): + +```code +$ tctl auth export --type=tls-host | openssl x509 -noout -serial +$ tctl auth export --type=db | openssl x509 -noout -serial +$ tctl auth export --type=db-client | openssl x509 -noout -serial +``` + +If the `db` CA serial number matches the `host` CA serial number, then you +need to complete the `db` CA migration. + +If the `db_client` CA serial number matches the `db` CA serial number, then you +need to complete the `db_client` CA migration. + +## 2/2. Rotate CAs + +If you need to complete both the `db` and `db_client` migrations, then a single +rotation of each of the `host`, `db`, and `db_client` CAs is enough: you do not +need to rotate the `db` CA twice. + +If you need to rotate the `host` CA, we recommend completing that rotation +before starting either of the `db` or `db_client` CA rotations: do not rotate +other CAs in parallel with a `host` CA rotation. +For information about `host` CA rotation, refer to the +[CA Rotation Guide](./ca-rotation.mdx). + +Database CA rotations are a little different, because they involve configuring +external resources (self-hosted databases) with new certificates during the +rotation. +You can (and should) rotate the `db` and `db_client` CAs at the same time to +avoid repeating the database certificate reconfiguration steps. + +For details on rotating the `db` or `db_client` CA, refer to the +[Database CA Rotation Guide](./db-ca-rotation.mdx). + +## Further reading + +- How the [Teleport Certificate Authority](../../architecture/authentication.mdx) works. +- How [Teleport Database Access](../../database-access/architecture.mdx) works. diff --git a/docs/pages/management/operations/db-ca-rotation.mdx b/docs/pages/management/operations/db-ca-rotation.mdx new file mode 100644 index 0000000000000..0454968fcd205 --- /dev/null +++ b/docs/pages/management/operations/db-ca-rotation.mdx @@ -0,0 +1,228 @@ +--- +title: Database Cert Authority Rotation +description: How to rotate Teleport database certificate authorities +--- + +Teleport connects to self-hosted databases using mutual TLS, or mTLS for short. +To enable mTLS connections to your self-hosted database via Teleport, you have +to configure your database and Teleport so that they can complete an mTLS +handshake: essentially, they need to trust each other. +To that end, your Teleport cluster manages its own internal database certificate +authorities. +This guide will explain how to rotate your Teleport cluster's database CAs. + +## Prerequisites + +(!docs/pages/includes/edition-prereqs-tabs.mdx!) + +- (!docs/pages/includes/tctl.mdx!) + +## Database CA rotation + +Teleport uses two certificate authorities for self-hosted database access: + +- The `db` certificate authority is used to issue server certificates to + self-hosted databases. +- The `db_client` certificate authority is used to issue an ephemeral + client certificate to the Teleport Database Service when it connects to a + self-hosted database. + +Certificate authority rotation consists of several rotation phases. +The rotation phases have different effects depending on which CA is rotated. +This section will explain the differences and show you how to rotate your +database CAs. + +### `db` CA Rotation phases + +- `standby`: All operations have completed or haven't started yet. +- `init`: All Teleport cluster services are notified of the rotation. + A new `db` certificate authority is created in your cluster. + Remote trusted clusters must fetch the new CA. + `tctl auth sign` will issue database server certs with the new CA. +- `update_clients`: Ephemeral client certificates issued to the Teleport Proxy + Service will be issued by the new CA. +- `update_servers`: No effect during `db` CA rotation. +- `rollback`: The rotation was aborted and is rolling back to the old + certificate authority. + +### `db_client` CA Rotation phases + +- `standby`: All operations have completed or haven't started yet. +- `init`: All Teleport cluster services are notified of the rotation. + A new `db_client` certificate authority is created in your cluster. + `tctl auth sign` will output both the old and new certificate authorities + in its trusted CA output. +- `update_clients`: Ephemeral client certificates issued to the Teleport + Database Service will be issued by the new CA. +- `update_servers`: No effect during `db_client` CA rotation. +- `rollback`: The rotation was aborted and is rolling back to the old + certificate authority. + +### Rotation types + +There are two types of certificate authority rotation processes: + +- **Manual:** it is the cluster administrator's responsibility to transition + between each phase of the rotation while monitoring database connectivity. + Manual rotations are performed by providing the `--manual` flag and the + desired phase using the `--phase` flag to the `tctl auth rotate` command. +- **Semi-automatic:** Teleport automatically transitions between phases of the + rotation after some amount of time (known as a *grace period*) elapses. + +For both types of rotation, the cluster goes through the phases in the +following order: + +- `standby` -> `init` -> `update_clients` -> `update_servers` -> `standby` + +It does not make sense to use semi-automatic rotation when rotating the `db` +or `db_client` CA, because self-hosted databases are not automatically +reconfigured by the semi-automatic process. +Therefore this guide will only explain manual rotation. + +In manual mode we transition between phases while monitoring database +connectivity, and reconfigure databases with new CA certificates as needed. + +You will need to reconfigure any given self-hosted database only once during the +rotation: it does not need to be done at each step. +You can reconfigure your databases during any phase, although it's best to do +so during the `init` phase to avoid losing database access later in the rotation. + +If you lose connectivity to your databases during any phase prior to +`standby`, you can [roll back](#rollback) to the old certificate authority to +restore connectivity. + +## Step 1/4. Start the rotation + +Initiate manual rotation of the `db` or `db_client` certificate authority: +
+You should not rotate the `db` or `db_client` CA in parallel with other CA +rotations. +Rotating in parallel with other CA types, especially the `host` CA, is more +complicated, more likely to cause problems, and does not save you any effort. + +However, if you want to rotate both the `db` and `db_client` CA, then you can +(and should) save yourself some effort by rotating both of them at the same +time. + +You can rotate both `db` and `db_client` certificate authorities in parallel +by transitioning both CAs through each phase step. +Just use the `tctl auth rotate` command for each step with `--type=db` and +then `--type=db_client`. +
+ +```code +$ tctl auth rotate --manual --type= --phase=init +Updated rotation phase to "init". To check status use 'tctl status' +``` + +Use `tctl` to confirm that there is an active rotation in progress: +```code +$ tctl status +``` + + +If you are rotating the `db_client` CA and you do not want to lose access to +your self-hosted databases in the next phase, you should reconfigure your +databases now. +You do not need to reconfigure databases at this point if you are rotating only +the `db` CA, although there is no harm in doing so. + +Consult the appropriate +[Teleport Database Access Guide](../../database-access/guides) for your +databases before proceeding to the `update_clients` rotation phase. + + +## Step 2/4. Update clients + +Execute the transition from `init` to `update_clients`: + +```code +$ tctl auth rotate --manual --type= --phase=update_clients +# Updated rotation phase to "update_clients". To check status use 'tctl status' +``` + +If you are rotating the `db_client` CA, this is the phase where the Teleport +Database Service will start using client certificates issued by the new CA to +connect to databases. +You should verify that you can still access your databases after transitioning +to this phase. + +If you lose connectivity to your databases, you can reconfigure your +databases to restore access. +If that does not restore access or you are unable to reconfigure a database, +then [roll back](#rollback) to the old certificate authority. + +## Step 3/4. Update servers + +Execute the transition from `update_clients` to `update_servers`: + +```code +$ tctl auth rotate --manual --type= --phase=update_servers +# Updated rotation phase to "update_servers". To check status use 'tctl status' +``` + +This phase does not actually do anything for `db` or `db_client` CA rotation, +but it's required as part of the generalized Teleport CA rotation process before +we can finish the rotation. + +## Step 4/4. Finish the rotation + +If any of your self-hosted databases have not yet been reconfigured, you **will +lose access** to those databases after transitioning to the `standby` phase in +this final step. + +To avoid down time, consult the appropriate +[Teleport Database Access Guide](../../database-access/guides) and reconfigure +your databases before proceeding. +Otherwise, access may still be restored by reconfiguring your self-hosted +databases after this step. + +Execute the transition from `update_servers` to `standby`: + +```code +$ tctl auth rotate --manual --type= --phase=standby +# Updated rotation phase to "standby". To check status use 'tctl status' +``` + +## Rollback + +Rollback can only be performed before the rotation is completed, meaning before +the rotation transitions to `standby` from `update_servers`. + +The most common reason you would want to rollback if is if you cannot +reconfigure your databases. +If you have connectivity issues after reconfiguring a database, it's likely that +you simply misconfigured the database. +If you roll back the rotation, then you will have to manually revert any +database reconfigurations and start from step 1 to reattempt rotating your +certificate authorities, so consider this a method of last resort. + +First, enter the rollback phase with a manual phase transition: + +```code +$ tctl auth rotate --manual --type= --phase=rollback +# Updated rotation phase to "rollback". To check status use 'tctl status' +``` + +If you reconfigured any of your databases during the rotation, you will need +to reconfigure them again before transitioning to `standby` from the `rollback` +phase. + +Enter the standby phase manually: + +```code +$ tctl auth rotate --manual --type= --phase=standby +# Updated rotation phase to "standby". To check status use 'tctl status' +``` + +It is very unlikely that you will lose database connectivity during a `db` CA +rotation and need to rollback. +However, when rotating your `db_client` CA you will definitely lose access to +any database that was not reconfigured during the `init` rotation phase. +If you are unable to reconfigure those databases, then rolling back should +restore access to them immediately. + +## Further reading + +- How the [Teleport Certificate Authority](../../architecture/authentication.mdx) works. +- How [Teleport Database Access](../../database-access/architecture.mdx) works.