From 85089b5a4cc97f8418f329b5c20a652356551612 Mon Sep 17 00:00:00 2001 From: Tim Buckley Date: Tue, 8 Jul 2025 21:59:16 -0600 Subject: [PATCH 01/17] MWI: Add documentation for bound keypair joining This adds documentation for bound keypair joining, including a user-facing deployment guide, an admin guide (with architecture information), and the usual provision token details. --- .../provision-token/bound-keypair-spec.mdx | 62 ++ .../machine-id/deployment/bound-keypair.mdx | 132 +++++ docs/pages/reference/join-methods.mdx | 13 + .../reference/machine-id/bound-keypair.mdx | 557 ++++++++++++++++++ .../pages/reference/machine-id/machine-id.mdx | 1 + 5 files changed, 765 insertions(+) create mode 100644 docs/pages/includes/provision-token/bound-keypair-spec.mdx create mode 100644 docs/pages/machine-workload-identity/machine-id/deployment/bound-keypair.mdx create mode 100644 docs/pages/reference/machine-id/bound-keypair.mdx diff --git a/docs/pages/includes/provision-token/bound-keypair-spec.mdx b/docs/pages/includes/provision-token/bound-keypair-spec.mdx new file mode 100644 index 0000000000000..c39dc4d7df22f --- /dev/null +++ b/docs/pages/includes/provision-token/bound-keypair-spec.mdx @@ -0,0 +1,62 @@ +```yaml +kind: token +version: v2 +metadata: + name: example-token +spec: + roles: [Bot] + join_method: bound_keypair + bot_name: example + + # Fields related to the bound keypair joining process. + bound_keypair: + # Fields related to the initial join attempt. + onboarding: + # If set to a public key in SSH authorized_keys format, the + # joining client must have the corresponding private key to join. This + # keypair may be created using `tbot keypair create`. + initial_public_key: "" + + # If set to a secret string value, a client may use this secret to perform + # the first join without pre-registering a public key in + # `initial_public_key`. + registration_secret: "" + + # If set to an RFC 3339 timestamp, attempts to register via + # `registration_secret` will be denied once the timestamp has elapsed. If + # more time is needed, this field can be edited to extend the registration + # period. + must_register_before: "" + + # Fields related to recovery after certificates have expired. + recovery: + # The maximum number of allowed recovery attempts. This value may + # be raised or lowered after creation to allow additional recovery + # attempts should the initial limit be exhausted. If `mode` is set to + # `standard`, recovery attempts will only be allowed if + # `.status.bound_keypair.recovery_count` is less than this limit. This + # limit is not enforced if `mode` is set to `relaxed` or `insecure`. This + # value must be at least 1 to allow for the initial join during + # onboarding, which counts as a recovery. + limit: 1 + + # The recovery rule enforcement mode. Valid values: + # - standard (or unset): all configured rules enforced. The recovery limit + # and client join state are required and verified. This is the most + # secure recovery mode. + # - relaxed: recovery limit is not enforced, but client join state is + # still required. This effectively allows unlimited recovery attempts, + # but client join state still helps mitigate stolen credentials. + # - insecure: neither the recovery limit nor client join state are + # enforced. This allows any client with the private key to join freely. + # This is less secure, but can be useful in certain situations, like in + # otherwise unsupported CI/CD providers. This mode should be used with + # care, and RBAC rules should be configured to heavily restrict which + # resources this identity can access. + mode: "standard" + + # If set to an RFC 3339 timestamp, once elapsed, a keypair rotation will be + # forced on next join if it has not already been rotated. The most recent + # rotation is recorded in `.status.bound_keypair.last_rotated_at`. + rotate_after: "" +``` \ No newline at end of file diff --git a/docs/pages/machine-workload-identity/machine-id/deployment/bound-keypair.mdx b/docs/pages/machine-workload-identity/machine-id/deployment/bound-keypair.mdx new file mode 100644 index 0000000000000..c28eb6bbbc31a --- /dev/null +++ b/docs/pages/machine-workload-identity/machine-id/deployment/bound-keypair.mdx @@ -0,0 +1,132 @@ +--- +title: Deploying Machine ID on Bare-metal with Bound Keypair +description: "How to install and configure Machine ID on a bare-metal server with Bound Keypair joining" +--- + +In this guide, you will install Machine ID's agent, `tbot`, on an arbitrary +node. This includes bare-metal machines, VMs, containers, or other hosts with +persistent storage. + +Note that this is a generic join method. A platform-specific join method may be +available that is better suited to your environment; refer to the +[deployment guides](./deployment.mdx) for a full list of options. + +## How it works + +With Bound Keypair joining, instead of using a static token or relying solely on +renewable certificates, Machine ID bots generate a unique keypair. Teleport is +then configured to trust this public key. + +Later, when the bot attempts to join the cluster, Teleport issues it a challenge +that can only be completed using its private key. The bot returns the solved +challenge, attesting to its own identity, and is conditionally allowed to join +the cluster. + +As self attestation is inherently less secure than the external verification +that would be provided by a cloud provider like AWS or a dedicated TPM, bound +keypair joining enforces a number of additional checks to prevent abuse, +including: +- Join state verification to ensure the keypair cannot be usefully shared or + duplicated +- Certificate generation counter checks to ensure regular bot certificates + cannot be usefully shared or duplicated +- Configurable limits on how often - if at all - bots may be allowed to recover + using this keypair. + +Refer to the [admin guide][guide] for further details on how this join method +works. + +## Prerequisites + +{/* note: consider edition-prereqs-tabs.mdx include for v19; it is misleading due to the minor launch release */} + +- A running Teleport cluster version 18.1.0 or above. +- The `tsh` and `tctl` clients +- (!docs/pages/includes/tctl.mdx!) +- This guide assumes the bot node has mutable persistent storage for internal + bot data. While it is possible to use Bound Keypair joining can on immutable + hosts (like CI runs), doing so will reduce security guarantees; see the + [admin guide][guide] for further information. + +## Step 1/5. Install `tbot` + +**This step is completed on the Azure VM.** + +First, `tbot` needs to be installed on the node that you wish to use Machine ID +on. + +Download and install the appropriate Teleport package for your platform: + +(!docs/pages/includes/install-linux.mdx!) + +## Step 2/5. Create a Bot + +**This step is completed on your local machine.** + +(!docs/pages/includes/machine-id/create-a-bot.mdx!) + +## Step 3/5. Create a join token + +**This step is completed on your local machine.** + +Create `bot-token.yaml`: + +```yaml +kind: token +version: v2 +metadata: + # name will be specified in the `tbot` to use this token + name: example-bot +spec: + roles: [Bot] + # bot_name should match the name of the bot created earlier in this guide. + bot_name: example + join_method: bound_keypair +``` + +Replace: +- `example` with the name of the bot you created in the second step + +Use `tctl` to apply this file: + +```code +$ tctl create -f bot-token.yaml +``` + +## Step 4/5. Configure `tbot` + +**This step is completed on the bot node.** + +Create `/etc/tbot.yaml`: + +```yaml +version: v2 +proxy_server: example.teleport.sh:443 +onboarding: + join_method: bound_keypair + token: example-bot + bound_keypair: + registration_secret: +storage: + type: memory +# outputs will be filled in during the completion of an access guide. +outputs: [] +``` + +(!docs/pages/includes/machine-id/daemon-or-oneshot.mdx!) + +## Step 5/5. Configure outputs + +(!docs/pages/includes/machine-id/configure-outputs.mdx!) + +## Next steps + +- Read the [Bound Keypair Joining Reference and Admin Guide][guide] + for more details about the join method and the available configuration options. +- Follow the [access guides](../access-guides/access-guides.mdx) to finish configuring `tbot` for + your environment. +- Read the [configuration reference](../../../reference/machine-id/configuration.mdx) to explore + all the available configuration options. +- [More information about `TELEPORT_ANONYMOUS_TELEMETRY`.](../../../reference/machine-id/telemetry.mdx) + +[guide]: ../../../reference/machine-id/bound-keypair.mdx diff --git a/docs/pages/reference/join-methods.mdx b/docs/pages/reference/join-methods.mdx index e9e9df0d349d2..1f94e764a92d6 100644 --- a/docs/pages/reference/join-methods.mdx +++ b/docs/pages/reference/join-methods.mdx @@ -264,6 +264,19 @@ When a MachineID bot uses an ephemeral join token, the token is deleted. - [Deploying Machine ID on Linux](../machine-workload-identity/machine-id/deployment/linux.mdx) +### Bound Keypair: `bound_keypair` + +Bound Keypair tokens are an improved alternative to +[ephemeral tokens](#ephemeral-tokens) that aims to improve reliability for +on-prem deployments + +(!docs/pages/includes/provision-token/bound-keypair-spec.mdx!) + + +- [Deploying Machine ID on Bare-metal Nodes with Bound Keypair joining](../machine-workload-identity/machine-id/deployment/bound-keypair.mdx) +- [Bound Keypair Reference and Admin Guide](./machine-id/bound-keypair.mdx) + + ### AWS IAM role: `iam` The IAM join method is available to any Teleport process running anywhere with access to IAM credentials, diff --git a/docs/pages/reference/machine-id/bound-keypair.mdx b/docs/pages/reference/machine-id/bound-keypair.mdx new file mode 100644 index 0000000000000..52150e266c6e9 --- /dev/null +++ b/docs/pages/reference/machine-id/bound-keypair.mdx @@ -0,0 +1,557 @@ +--- +title: Bound Keypair Joining +description: "Bound Keypair Joining: Reference and admin guide" +--- + +Bound Keypair is a new join method designed to provide the best features of +[dynamic join methods][dynamic] join methods - like AWS, GCP, or Azure - but in +on-prem or otherwise unsupported environments where no external verification is +available. + +Specifically, this join method: +- Does not require dedicated TPM hardware or external identity attestation +- Does not require long-lived shared secrets +- Allows for limited automatic recovery if certificates expire +- Allows recovery restrictions to be relaxed or lifted to accommodate different + use cases and deployment scenarios +- Ensures failed bots can be recovered without client-side intervention in most + cases + + +Bound Keypair joining is available in v18.1.0 and is intended to replace `token` +joining as the default recommended join method in Teleport v19.0.0. + + +## Use Cases + +Bound Keypair joining can be used in any environment and is designed to function +as a drop-in replacement for the traditional [`token`][ephemeral] join method in +all situations where it is used today. This includes bare-metal and on-prem +hardware where TPMs are not available, or cloud providers not currently +supported by a delegated join method. + +As a drop-in replacement for `token` joining, Bound Keypair joining is also a +good replacement for local experimentation for testing, and can be easily + +Additionally, with [`insecure` recovery](#disabling-join-state-verification) and +in situations that can accommodate the security complications, Bound Keypair +joining can be used to join bots in otherwise unsupported CI/CD providers by +persisting the bot's keypair in the platform keystore. + +## Limitations + +While Bound Keypair joining does enable or simplify a number use cases, it does +have limitations that may make it unfit in some instances. + +In particular, the [secure recovery modes](#recovery) introduce some deployment +restrictions: +- Each bot deployment must be issued a unique token. For deployment at scale, + use of Teleport's + [Terraform provider](../terraform-provider/terraform-provider.mdx) is + recommended to create tokens in bulk for each deployment. +- Each bot deployment must be able to store client-side state (used for + [join state verification](#join-state-verification)). + +This limitation can be worked around using the +[`insecure` recovery mode](#disabling-join-state-verification), but doing so +does meaningfully reduce the join method's security protections and should be +used with care. + +## Concepts + +### Bound Keypair + +The term "bound keypair" refers to both the type of credential and way it's used +in Teleport: first, a bot generates a standard public/private keypair; next, +Teleport is configured to trust the public key, binding it to a token. + +Once this trust relationship has been established, bots can - in a sense - serve +as their own identity authority, similar to how cloud providers can attest to +the identity of a node or CI workflow run. Teleport issues the joining bot a +unique cryptographic challenge which the bot signs using its private key, and +the result is verified using the key registered with Teleport. + +As this self attestation is not by itself a sufficient security guarantee, Bound +Keypair joining makes use of additional server-side rules to control whether or +not a join attempt will be allowed, including: +- Limiting the number of [recovery attempts](#recovery) and optionally requiring + (server-side) human intervention for each recovery. +- Preventing reuse or theft of private keys using + [join state verification](#join-state-verification) + +### Onboarding + +Onboarding takes place at first join and is the stage at which a public key +becomes bound to a token. There are two methods bots may use to complete this +step: +- Preregistered keys: a private key is generated locally on the bot node, and + the public key is copied out-of-band to Teleport by a human user or a script. + This method ensures no secret values are ever copied between systems. +- Registration secrets: a one time use random secret is generated by Teleport + and provided to the bot node via either a human user or a script, similar to + the [ephemeral `token` join method][token]. + +If a registration secret is used, the bot uses the registration secret to prove +its identity for this first connection, generates a keypair on the fly, and +registers the public key with Teleport. + +Registration secrets are used by default: if a token is created and no initial +(preregistered) public key is provided, a registration secret will be randomly +generated by Teleport and written to the +`status.bound_keypair.registration_secret` field. + +In both cases - once registered, or when using preregistered keys - the joining +bot is then required to complete a joining challenge. A unique cryptographic +challenge is generated by Teleport, which the bot must sign and return. If the +signature matches the registered public key, onboarding has succeeded: the +public key is permanently bound to the token, a certificate bundle is issued, +and a join state document is provided for use on the next join attempt. + +### Recovery + +Recovery takes place when a bot uses its private key to request new +certificates. This can occur in two situations: +- On first join, when the bot has no certificates +- When the bot's certificates expire before they can be renewed, such as when a + bot is offline for longer than its configured `certificate_ttl`. + +The behavior of the recovery process depends on two values configured in the +token resource: the recovery mode (`spec.bound_keypair.recovery.mode`), and the +recovery limit (`spec.bound_keypair.recovery.limit`). The mode may have these +values: +- `standard` (default): only `limit` recovery attempts can be automatically + attempted. + + This mode is recommended for most situations. Bots can recover automatically + so long as the recovery count (`status.bound_keypair.recovery_count`) doesn't + exceed the configured limit, which allows cluster administrators to select + their tolerance for automatic bot recovery attempts. + + [Join state verification](#join-state-verification) is enabled, which helps + prevent keypair reuse, but does require clients to store additional on each + join. + + Note that as the initial join counts as a recovery attempt, `limit` must + always be at least `1` for the initial join to succeed. + +- `relaxed`: `limit` is ignored, but + [join state verification](#join-state-verification) remains enabled. + + This mode allows for unlimited recovery attempts and is useful for lower + security impact bots, or deployments where bots are regularly expected to go + offline for extended periods. + + Like `standard` mode, join state verification is enabled to help prevent + keypair reuse, but does require clients to store additional state on each + join. + +- `insecure`: `limit` is ignored, and join state verification is **disabled**. + + This mode is the most flexible, as bots can rejoin repeatedly and can operate + without mutable client-side state. However, this disables most additional + security checks, and + +### Join State Verification + +A **join state document** is an additional piece of information provided to +joining bots alongside their usual Teleport certificate bundle. It contains +signed information about the state of the joining process, including a sequence +number that uniquely identifies each join attempt. + +**Join state verification** both ensures that the bot can provide a valid and +signed join state document, and that the document contains the expected sequence +number. +* If successful, the join attempt can proceed and a new join state document will + be issued with an incremented sequence counter. +* If the sequence number presented by the client is outdated, the join + attempt is rejected, and a lock is created to ensure existing clients will be + denied further access to the Teleport cluster. + +This system relies on bots regularly reauthenticating to Teleport. If an +attacker somehow obtains a copy of a bot's keypair and attempts to use it to +retrieve Teleport certificates, they will initially succeed. However, the +original bot will eventually make another authentication attempt, using its +now-outdated join state document. This join attempt with an outdated document +will trigger a lockout and block both the original bot and any credentials in +use by the attacker. + +Note that join state verification is disabled when the token's +`spec.bound_keypair.recovery.mode` is set to `insecure`. + +## Token Resource Examples + +Bound Keypair tokens can be configured using Teleport's existing `token` +resource using the `bound_keypair` join method. For a complete reference, refer +to the [provision token reference](../join-methods.mdx#bound-keypair-bound_keypair). + +### Minimal Token + +A minimal bound keypair token does not require any special configuration beyond +specifying the `bound_keypair` join method value: +```yaml +kind: token +version: v2 +metadata: + name: example +spec: + roles: [Bot] + join_method: bound_keypair + bot_name: example +``` + +Without additional options specified, this will use the `standard` recovery mode +with a default recovery limit of 1 to allow for the initial join. A registration +secret will be automatically generated when the token is created on the Teleport +cluster. To create and use this token, adjust the name and bot name as needed, +and write it to a file, for example `token.yaml`, and run: + +```code +$ tctl create -f token.yaml +$ tctl get token/example-bkp --format=json | jq -r '.[0].status.bound_keypair.registration_secret' +``` + +This will print the registration secret which can be provided to the `tbot` +client using the `--registration-secret` flag: +```code +$ tbot configure identity \ + --proxy-server=example.teleport.sh:443 \ + --join-method=bound_keypair \ + --token=example \ + --registration-secret=$SECRET_VALUE \ + --destination=./destination > tbot.yaml +$ tbot start -c tbot.yaml +``` + +### Preregistered Key Example + +To use a preregistered key instead of a registration secret, a keypair needs to +be generated on the bot node. To do so, run the following commands: +```code +## If needed, create the bot's storage directory +$ mkdir /var/lib/teleport/bot +$ tbot keypair create --storage /var/lib/teleport/bot --proxy-server=example.teleport.sh:443 +2025-07-08T16:31:48.000-00:00 INFO [TBOT] keypair has been written to storage storage:directory: /var/lib/teleport/bot tbot/keypair.go:135 + +To register the keypair with Teleport, include this public key in the token's +`spec.bound_keypair.onboarding.initial_public_key`: + + ssh-ed25519 +``` + +Note that while a proxy server address must be provided, this is only used to +ping the cluster to determine its configured +[signature algorithms](../signature-algorithms.mdx). Once the keypair has been +generated, the public key will be printed to the console. + +Next, create a file named `token.yaml` with the following content: + +```yaml +kind: token +version: v2 +metadata: + name: example-token +spec: + roles: [Bot] + join_method: bound_keypair + bot_name: example + bound_keypair: + onboarding: + initial_public_key: "ssh-ed25519 " +``` + +Copy the public key (including the `ssh-ed25519 ...` prefix or similar) into the +`initial_public_key` field, and make certain the value is contained in quotes. + +Once finished, run the following to create the token and join the bot: +```code +## Create the token. +$ tctl create -f token.yaml +## Configure and start the bot. +$ tbot configure identity \ + --proxy-server=example.teleport.sh:443 \ + --join-method=bound_keypair \ + --token=example \ + --storage=/var/lib/teleport/bot \ + --destination=./destination > tbot.yaml +$ tbot start -c tbot.yaml +``` + +Note the following: +- In this example, the recovery limit defaults to 1, and will be reached as soon + as the bot joins. To enable automatic recovery, increase the recovery limit to + the desired value. +- The `tctl create` and `tbot` commands do not need to be run on the same node. + Typically, you might create tokens on your local machine and run `tbot` on a + remote server. + +## Admin Reference + +### Allowing additional recovery attempts + +When using the `standard` recovery mode, only a configured number of recovery +attempts can be made. If the limit is reached, no further recovery attempts can +be made until the limit is increased. + +To increase this limit and allow an expired bot to join again, edit the token +using `tctl edit`: +```code +$ tctl edit token/example-token +``` + +Find the `spec.bound_keypair.recovery.limit` field and increment the limit by +the desired amount. You are free to select any desired threshold. For example, +consider these use cases: +- If human intervention is desired for each join attempt you can increase this + value by 1. This single recovery attempt will be immediately consumed, so + future recoveries will again require human intervention, and may result in + downtime. + + While this approach makes downtime likely, it does ensure a human verifies the + state of the bot node on each recovery. +- If you want human intervention for each recovery, but want to avoid downtime, + you can increase this value by 2. The first attempt will be consumed + immediately, but the bot will have one recovery attempt for automatic future + use. + + A human user can periodically audit the recovery count and bot node to ensure + a recovery attempt is always available and the node is behaving as expected. +- Any larger value will increase the amount of time required between human + intervention. You can select your tolerance for automatic bot recoveries as + desired. + +Alternatively, if you wish to allow an unlimited number of automatic recovery +attempts, [refer to the entry below](#allowing-unlimited-recovery-attempts) on +the `relaxed` recovery mode. + +Note that the recovery limit is always relative to the recovery counter (in the +`status.bound_keypair.recovery_count` field in the token resource). It is valid +to decrease the limit or set it to zero, however doing so may prevent future +bot recovery attempts until the limit is increased again. + +Additionally, note that [join state verification](#join-state-verification) is +still required, and will prevent multiple concurrent uses of the same keypair +and token. In other words, increasing the recovery limit will not allow multiple +nodes to join. + +### Allowing unlimited recovery attempts + +To allow unlimited recovery attempts, the `spec.bound_keypair.recovery.mode` +field should be set to `relaxed`. To do this, use `tctl edit` to edit the token: +```code +$ tctl edit token/example-token +``` + +Find or create the `spec.bound_keypair.recovery.mode` field and set the value to +`relaxed`. Save the file and quit your editor to update the token. + +When the recovery mode is set to `relaxed`, the `limit` field is ignored and the +`status.bound_keypair.recovery_count` field may increase beyond the written +limit. If the mode is later changed back to `standard`, be aware that future +recovery attempts will fail unless the `limit` is increased to accommodate the +current value of `recovery_count`. + +Note that when `relaxed` mode is in use, +[join state verification](#join-state-verification) is still required and will +prevent multiple concurrent uses of the same keypair and token. If your use case +requires this, you can +[disable join state verification](#disabling-join-state-verification), but doing +so does impact the security of the token. + +### Requesting a keypair rotation + +To request a keypair rotation, set the `.spec.bound_keypair.rotate_after` field +to contain a timestamp. On the next authentication attempt after that timestamp +has elapsed, the bot will automatically rotate its keypair. + +To simplify this process, you can use the `tctl bound-keypair rotate` helper: +```code +$ tctl bound-keypair rotate token-name +``` + +This sets the timestamp to the current time. Note that by default bots only +reauthenticate every 20 minutes, so it may take some time for the request to be +acknowledged. You can monitor the rotation status by watching the token's +`.status.bound_keypair.last_rotated_at` field. + +If you want to force an early rotation and have access to the bot node, you can +restart the `tbot` process, or send it a signal with `pkill -usr1 tbot` to +request an early rotation. + +Note that the previous 10 keypairs are retained on the client for use in case of +a cluster rollback; refer to the +[cluster rollback](#recovery-after-a-cluster-rollback) section for additional +information. + +### Locking a `bound_keypair` bot or bot instance + +The simplest way to lock out a bot that joined using the `bound_keypair` join +method is to use a join token lock target: + +```code +$ tctl lock --join-token=token-name +``` + +As a bound keypair token is linked to a single bot, this will effectively lock +the bot. It will not be able to reauthenticate, recover, interact with the +Teleport API, or otherwise use its credentials until the lock is removed. + +Note that if a bot is locked for long enough - bots have a 1 hour certificate +TTL by default - its certificates will expire. If you intend to remove this lock +and reinstate the bot, you may also need to increase the recovery limit +(`.spec.bound_keypair.recovery.limit`) to accommodate the additional recovery +attempt. + +Other lock targets can also be used, but are not preferred: +- Bot instance (`tctl lock --bot-instance-id ...`): will lock only a single + instance of the bot. Note that if the recovery limit allows for it, the + [automatic recovery process](#recovery) will attempt to rejoin and, if + successful, will generate a new bot instance ID. +- Bot name (`tctl lock --user bot-`): will lock all bots using the same + bot / user. This may be overly broad and lock other instances running under + this bot user. + +### Recovering a locked `bound_keypair` bot instance + +Bots joined with the `bound_keypair` join method can become automatically locked +under various conditions, including: +- Failing to correctly complete [join state verification](#join-state-verification) +- Connecting with certificates that have an invalid [generation counter][ephemeral] +- Locked manually by a cluster admin + +To recover a bot that has become locked, first ensure the bot's internal storage +(`storage`) has not been compromised. These locking conditions are designed to +trigger if more than one client tries to join using a copy of the same +certificates and private key. This can occur due to a misconfiguration or due +to an attacker copying a bot's credentials, so ideally the latter should be +ruled out before unlocking a bot. + +Next, determine the name (UUID) of the lock or locks targeting the bot: +```code +$ tctl get lock +kind: lock +metadata: + name: 372af058-76d1-4e64-93da-3b04d7d03ac2 +spec: + target: + user: bot-example +version: v2 +--- +kind: lock +metadata: + name: 791d0b1d-01b4-4752-8a99-9b2908aebfae +spec: + target: + bot_instance_id: e7d494ae-a0ff-4d12-b935-de5e2025f667 +version: v2 +--- +kind: lock +metadata: + name: a69fdbb2-8e53-406a-b453-48b2cda6991d +spec: + target: + join_token: example-token-name +version: v2 +``` + +Note the different locks and lock targets shown above: bots can be targeted by +any of their Teleport user name (`bot-example`), the bot instance ID (a UUID), +or the join token name. Locks created automatically for bots using Bound Keypair +joining will typically use a `join_token` target, but a lock targeting any of +these values could be created manually. + +Note that locks may have a message field containing details about why the lock +was created. + +Once the lock name(s) have been determined, remove each using `tctl rm`: +```code +$ tctl rm lock/372af058-76d1-4e64-93da-3b04d7d03ac2 +``` + +Next, join state should be reset. Use `tctl edit` to set the token's recovery +mode to `insecure`, but make a note of the current value (`standard` or +`relaxed`): +```code +$ tctl edit token/example-token +``` + +Change the `.spec.bound_keypair.recovery.mode` field to `insecure`, save, and +quit the editor. + +The bot can now be allowed to rejoin. Given sufficient time it will retry on its +own, but if you have access to the node, `systemctl restart tbot` or similar can +be used to restart the bot process. + +The bot should now be able to join successfully. You can monitor progress by +watching for new audit events in Teleport's web UI, or by waiting for the +recovery counter to increase: +```code +$ tctl get token/example-token --format=json | jq '.[].status.bound_keypair.recovery_count' +``` + +Once the bot has joined successfully, reset the recovery mode to its previous +value using `tctl edit`: +```code +$ tctl edit token/example-token +``` + +If you do suspect the bot's credentials may have been compromised, you may also +want to [request a keypair rotation](#requesting-a-keypair-rotation). + +### Disabling Join State Verification + +It is occasionally useful to intentionally disable join state verification. For +example, this can enable use with: +- CI/CD providers without an explicit [dynamic join method][dynamic] +- Nodes with immutable storage that cannot store an updated join state document + after each join. + +Before continuing, be aware that disabling join state verification will prevent +Teleport from detecting if multiple clients are joining using the same bound +keypair token. In other words, if the private key is copied by an attacker, they +will be able to join indefinitely. Take care to protect the keypair, and make +certain to limit access from the bot identity using Teleport's +[RBAC system][rbac]. + +When ready, use `tctl edit` to modify the Bound Keypair token: +```code +$ tctl edit token/example-token +``` + +Find or add the `spec.bound_keypair.recovery.mode` field and set it to +`insecure`. Save and quit your editor to update the token. + +With the mode set to `insecure`, the `recovery.limit` is ignored, allowing +unlimited reuse of the token, and join state verification is disabled, allowing +concurrent or stateless reuse. + +### Recovery after a cluster rollback + +If your Teleport cluster is rolled back for any reason, joining bots may fail +[join state verification](#join-state-verification) as their local join state +document may not match the values currently (or previously) known to Teleport. + +The simplest workaround is to temporarily set all bound keypair tokens to +`insecure` recovery mode for the first join attempt following a cluster restore. +Once they've joined once, they will once again have a valid join state, so the +recovery mode can be restored to its previous value. + +To change the recovery mode, use `tctl edit` to modify the token resource: +```code +$ tctl edit token/example-token +``` + +Find the `spec.bound_keypair.recovery.mode` field, and set the value to +"insecure". Repeat this for each bound keypair token. Wait for all bound keypair +bots to reauthenticate, and repeat this process to restore the recovery mode to +its previous value. + +If [bot keypairs were rotated](#requesting-a-keypair-rotation) between the +snapshot and restore of the Teleport cluster, note that bots only keep a record +of the previous 10 keypairs. This means server-side recovery may impossible if +the keypair expected by the restored Teleport cluster has been rotated out of +the client-side history, or if the client-side history has been lost or deleted. + +[ephemeral]: ../architecture/machine-id-architecture.mdx#ephemeral +[dynamic]: ../architecture/machine-id-architecture.mdx#dynamic-join-tokens-eg-aws-iam +[rbac]: ../access-controls/roles.mdx +[token]: ../join-methods.mdx#ephemeral-tokens diff --git a/docs/pages/reference/machine-id/machine-id.mdx b/docs/pages/reference/machine-id/machine-id.mdx index 69ad4cf34eb20..6a11b039b4c83 100644 --- a/docs/pages/reference/machine-id/machine-id.mdx +++ b/docs/pages/reference/machine-id/machine-id.mdx @@ -11,3 +11,4 @@ description: Configuration and CLI reference for Teleport Machine ID. - [Telemetry](telemetry.mdx) - [V16 Upgrade Guide](v16-upgrade-guide.mdx) - [Bot Terraform Resource](../../reference/terraform-provider/resources/bot.mdx) +- [Bound Keypair Joining](bound-keypair.mdx) From 5324dad3cc73843c5996bf7e091e63da6778fe84 Mon Sep 17 00:00:00 2001 From: Tim Buckley Date: Tue, 8 Jul 2025 22:10:19 -0600 Subject: [PATCH 02/17] TODOs for deployment guide, add deployment guide to method list --- .../machine-id/deployment/bound-keypair.mdx | 6 +++++- .../machine-id/deployment/deployment.mdx | 17 +++++++++-------- 2 files changed, 14 insertions(+), 9 deletions(-) diff --git a/docs/pages/machine-workload-identity/machine-id/deployment/bound-keypair.mdx b/docs/pages/machine-workload-identity/machine-id/deployment/bound-keypair.mdx index c28eb6bbbc31a..e2994df13f31a 100644 --- a/docs/pages/machine-workload-identity/machine-id/deployment/bound-keypair.mdx +++ b/docs/pages/machine-workload-identity/machine-id/deployment/bound-keypair.mdx @@ -50,7 +50,7 @@ works. ## Step 1/5. Install `tbot` -**This step is completed on the Azure VM.** +**This step is completed on the bot node.** First, `tbot` needs to be installed on the node that you wish to use Machine ID on. @@ -69,6 +69,8 @@ Download and install the appropriate Teleport package for your platform: **This step is completed on your local machine.** +{/* TODO: option for preregistered key? Tabs maybe? */} + Create `bot-token.yaml`: ```yaml @@ -93,6 +95,8 @@ Use `tctl` to apply this file: $ tctl create -f bot-token.yaml ``` +{/* TODO: fetch the registration secret for use in the next step. */} + ## Step 4/5. Configure `tbot` **This step is completed on the bot node.** diff --git a/docs/pages/machine-workload-identity/machine-id/deployment/deployment.mdx b/docs/pages/machine-workload-identity/machine-id/deployment/deployment.mdx index 96ea2de48f299..007456b504327 100644 --- a/docs/pages/machine-workload-identity/machine-id/deployment/deployment.mdx +++ b/docs/pages/machine-workload-identity/machine-id/deployment/deployment.mdx @@ -52,14 +52,15 @@ and [Architecture](../../../reference/architecture/machine-id-architecture.mdx) Read the following guides for how to deploy Machine ID on your cloud platform or on-prem infrastructure. -| Platform | Installation method | Join method | -|-------------------------------------------|-------------------------------------------------|-----------------------------------------------------| -| [Linux](linux.mdx) | Package manager or TAR archive | Static join token | -| [Linux (TPM)](linux-tpm.mdx) | Package manager or TAR archive | Attestation from TPM 2.0 | -| [GCP](gcp.mdx) | Package manager, TAR archive, or Kubernetes pod | Identity document signed by GCP | -| [AWS](aws.mdx) | Package manager, TAR archive, or Kubernetes pod | Identity document signed by AWS | -| [Azure](azure.mdx) | Package manager or TAR archive | Identity document signed by Azure | -| [Kubernetes](kubernetes.mdx) | Kubernetes pod | Identity document signed by your Kubernetes cluster | +| Platform | Installation method | Join method | +|--------------------------------------------|-------------------------------------------------|-----------------------------------------------------| +| [Linux](linux.mdx) | Package manager or TAR archive | Static join token | +| [Linux (TPM)](linux-tpm.mdx) | Package manager or TAR archive | Attestation from TPM 2.0 | +| [Linux (Bound Keypair)](bound-keypair.mdx) | Package manager or TAR archive | Bound Keypair | +| [GCP](gcp.mdx) | Package manager, TAR archive, or Kubernetes pod | Identity document signed by GCP | +| [AWS](aws.mdx) | Package manager, TAR archive, or Kubernetes pod | Identity document signed by AWS | +| [Azure](azure.mdx) | Package manager or TAR archive | Identity document signed by Azure | +| [Kubernetes](kubernetes.mdx) | Kubernetes pod | Identity document signed by your Kubernetes cluster | ### CI/CD From 72667fd917603c84bd2dd7e688bc2c3e337eb90f Mon Sep 17 00:00:00 2001 From: Tim Buckley Date: Wed, 9 Jul 2025 20:17:39 -0600 Subject: [PATCH 03/17] Finish deployment guide and make small tweaks to other pages --- .../provision-token/bound-keypair-spec.mdx | 7 +- .../machine-id/deployment/bound-keypair.mdx | 68 +++++++++++++++---- docs/pages/reference/join-methods.mdx | 2 +- .../reference/machine-id/bound-keypair.mdx | 38 +++++++---- 4 files changed, 82 insertions(+), 33 deletions(-) diff --git a/docs/pages/includes/provision-token/bound-keypair-spec.mdx b/docs/pages/includes/provision-token/bound-keypair-spec.mdx index c39dc4d7df22f..2e4d38b383d54 100644 --- a/docs/pages/includes/provision-token/bound-keypair-spec.mdx +++ b/docs/pages/includes/provision-token/bound-keypair-spec.mdx @@ -14,12 +14,15 @@ spec: onboarding: # If set to a public key in SSH authorized_keys format, the # joining client must have the corresponding private key to join. This - # keypair may be created using `tbot keypair create`. + # keypair may be created using `tbot keypair create`. If set, + # `registration_secret` and `must_register_before` are ignored. initial_public_key: "" # If set to a secret string value, a client may use this secret to perform # the first join without pre-registering a public key in - # `initial_public_key`. + # `initial_public_key`. If unset and no `initial_public_key` is provided, + # a random value will be generated automatically into + # `.status.bound_keypair.registration_secret`. registration_secret: "" # If set to an RFC 3339 timestamp, attempts to register via diff --git a/docs/pages/machine-workload-identity/machine-id/deployment/bound-keypair.mdx b/docs/pages/machine-workload-identity/machine-id/deployment/bound-keypair.mdx index e2994df13f31a..da227126913f5 100644 --- a/docs/pages/machine-workload-identity/machine-id/deployment/bound-keypair.mdx +++ b/docs/pages/machine-workload-identity/machine-id/deployment/bound-keypair.mdx @@ -1,11 +1,11 @@ --- -title: Deploying Machine ID on Bare-metal with Bound Keypair -description: "How to install and configure Machine ID on a bare-metal server with Bound Keypair joining" +title: Deploying Machine ID with Bound Keypair Joining +description: "How to install and configure Machine ID with Bound Keypair joining" --- In this guide, you will install Machine ID's agent, `tbot`, on an arbitrary -node. This includes bare-metal machines, VMs, containers, or other hosts with -persistent storage. +node. This node could be a bare-metal machine, a VM, a container, or any other +host - the only requirement is that the node has persistent storage. Note that this is a generic join method. A platform-specific join method may be available that is better suited to your environment; refer to the @@ -15,7 +15,7 @@ available that is better suited to your environment; refer to the With Bound Keypair joining, instead of using a static token or relying solely on renewable certificates, Machine ID bots generate a unique keypair. Teleport is -then configured to trust this public key. +then configured to trust this public key for future joining attempts. Later, when the bot attempts to join the cluster, Teleport issues it a challenge that can only be completed using its private key. The bot returns the solved @@ -33,6 +33,11 @@ including: - Configurable limits on how often - if at all - bots may be allowed to recover using this keypair. +An important benefit to Bound Keypair joining is that all joining restrictions +can be reconfigured at any time, and bots that expire or go offline can be +recovered by making a server-side exemption without any client-side +intervention. + Refer to the [admin guide][guide] for further details on how this join method works. @@ -69,33 +74,59 @@ Download and install the appropriate Teleport package for your platform: **This step is completed on your local machine.** -{/* TODO: option for preregistered key? Tabs maybe? */} +In this guide, we'll demonstrate joining a bot using a registration secret: this +is a one-time use secret the bot can provide to Teleport to authenticate its +first join. Once authenticated, the bot automatically generates a keypair and +registers its public key with Teleport for use in all future join attempts. -Create `bot-token.yaml`: +Create `token-example.yaml`: ```yaml kind: token version: v2 metadata: - # name will be specified in the `tbot` to use this token - name: example-bot + # This name will be used in tbot's `onboarding.token` field. + name: example spec: roles: [Bot] # bot_name should match the name of the bot created earlier in this guide. bot_name: example join_method: bound_keypair + bound_keypair: + recovery: + mode: standard + limit: 1 ``` Replace: -- `example` with the name of the bot you created in the second step +- `example` in `spec.bot_name` with the name of the bot you created in the + second step + +For this example, we don't need to set any additional options for the bound +keypair token. We've allowed a single recovery attempt, which will be used to +allow the bot's initial join, and Teleport will generate a registration secret +automatically when the token is created as we have not preregistered a public +key to use. + + +To learn more about the various onboarding and recovery options, refer to the +[Reference and Admin Guide][guide]. + Use `tctl` to apply this file: ```code -$ tctl create -f bot-token.yaml +$ tctl create -f token-example.yaml ``` -{/* TODO: fetch the registration secret for use in the next step. */} +Next, retrieve the generated registration secret, which will be needed for the +next step: +```code +$ tctl get token/example --format=json | jq -r '.[0].status.bound_keypair.registration_secret' +``` + +This assumes `jq` is installed. If not, run `tctl get token/example` and inspect +the `.status.bound_keypair.registration_secret` field. ## Step 4/5. Configure `tbot` @@ -108,15 +139,22 @@ version: v2 proxy_server: example.teleport.sh:443 onboarding: join_method: bound_keypair - token: example-bot + token: example bound_keypair: - registration_secret: + registration_secret: SECRET storage: - type: memory + type: directory + path: /var/lib/teleport/bot # outputs will be filled in during the completion of an access guide. outputs: [] ``` +Replace the following: +- `example.teleport.sh:443` with the address of your Teleport Proxy. +- `example` with the name of the token created in the previous step, if you + changed it from `example`. +- `SECRET` with the registration secret retrieved in the previous step. + (!docs/pages/includes/machine-id/daemon-or-oneshot.mdx!) ## Step 5/5. Configure outputs diff --git a/docs/pages/reference/join-methods.mdx b/docs/pages/reference/join-methods.mdx index 1f94e764a92d6..a708894aae76c 100644 --- a/docs/pages/reference/join-methods.mdx +++ b/docs/pages/reference/join-methods.mdx @@ -273,7 +273,7 @@ on-prem deployments (!docs/pages/includes/provision-token/bound-keypair-spec.mdx!) -- [Deploying Machine ID on Bare-metal Nodes with Bound Keypair joining](../machine-workload-identity/machine-id/deployment/bound-keypair.mdx) +- [Deploying Machine ID with Bound Keypair joining](../machine-workload-identity/machine-id/deployment/bound-keypair.mdx) - [Bound Keypair Reference and Admin Guide](./machine-id/bound-keypair.mdx) diff --git a/docs/pages/reference/machine-id/bound-keypair.mdx b/docs/pages/reference/machine-id/bound-keypair.mdx index 52150e266c6e9..cb9b9bc3b5a88 100644 --- a/docs/pages/reference/machine-id/bound-keypair.mdx +++ b/docs/pages/reference/machine-id/bound-keypair.mdx @@ -77,7 +77,7 @@ not a join attempt will be allowed, including: - Limiting the number of [recovery attempts](#recovery) and optionally requiring (server-side) human intervention for each recovery. - Preventing reuse or theft of private keys using - [join state verification](#join-state-verification) + [join state verification](#join-state-verification). ### Onboarding @@ -97,12 +97,12 @@ registers the public key with Teleport. Registration secrets are used by default: if a token is created and no initial (preregistered) public key is provided, a registration secret will be randomly -generated by Teleport and written to the +generated by Teleport and written to the token's `status.bound_keypair.registration_secret` field. In both cases - once registered, or when using preregistered keys - the joining bot is then required to complete a joining challenge. A unique cryptographic -challenge is generated by Teleport, which the bot must sign and return. If the +challenge is generated by Teleport which the bot must sign and return. If the signature matches the registered public key, onboarding has succeeded: the public key is permanently bound to the token, a certificate bundle is issued, and a join state document is provided for use on the next join attempt. @@ -111,10 +111,16 @@ and a join state document is provided for use on the next join attempt. Recovery takes place when a bot uses its private key to request new certificates. This can occur in two situations: -- On first join, when the bot has no certificates +- On first join, when the bot has no certificates. - When the bot's certificates expire before they can be renewed, such as when a bot is offline for longer than its configured `certificate_ttl`. +So long as the bot has and maintains valid Teleport credentials - by default, +bot certificates are valid for an hour and renewed every 20 minutes, but these +values are configurable - no recovery will be needed after the initial join. +Recovery is only required when a bot is unable to refresh its certificates for +longer than its configured certificate TTL (e.g. 1 hour). + The behavior of the recovery process depends on two values configured in the token resource: the recovery mode (`spec.bound_keypair.recovery.mode`), and the recovery limit (`spec.bound_keypair.recovery.limit`). The mode may have these @@ -149,7 +155,8 @@ values: This mode is the most flexible, as bots can rejoin repeatedly and can operate without mutable client-side state. However, this disables most additional - security checks, and + security checks and should be used with care. + [See below](#disabling-join-state-verification) for more information. ### Join State Verification @@ -167,13 +174,13 @@ number. attempt is rejected, and a lock is created to ensure existing clients will be denied further access to the Teleport cluster. -This system relies on bots regularly reauthenticating to Teleport. If an -attacker somehow obtains a copy of a bot's keypair and attempts to use it to -retrieve Teleport certificates, they will initially succeed. However, the -original bot will eventually make another authentication attempt, using its -now-outdated join state document. This join attempt with an outdated document -will trigger a lockout and block both the original bot and any credentials in -use by the attacker. +This system takes advantage of Teleport's short lived certificates and frequent +bot reauthentication. If an attacker somehow obtains a copy of a bot's keypair +and attempts to use it to retrieve Teleport certificates, they will initially +succeed. However, the original bot will eventually make another authentication +attempt, using its now-outdated join state document. This join attempt with an +outdated document will trigger a lockout and block both the original bot and any +credentials in use by the attacker. Note that join state verification is disabled when the token's `spec.bound_keypair.recovery.mode` is set to `insecure`. @@ -228,7 +235,7 @@ To use a preregistered key instead of a registration secret, a keypair needs to be generated on the bot node. To do so, run the following commands: ```code ## If needed, create the bot's storage directory -$ mkdir /var/lib/teleport/bot +$ mkdir -p /var/lib/teleport/bot $ tbot keypair create --storage /var/lib/teleport/bot --proxy-server=example.teleport.sh:443 2025-07-08T16:31:48.000-00:00 INFO [TBOT] keypair has been written to storage storage:directory: /var/lib/teleport/bot tbot/keypair.go:135 @@ -495,13 +502,14 @@ $ tctl edit token/example-token ``` If you do suspect the bot's credentials may have been compromised, you may also -want to [request a keypair rotation](#requesting-a-keypair-rotation). +want to [request a keypair rotation](#requesting-a-keypair-rotation) in +addition to taking other steps to ensure the node is properly secured. ### Disabling Join State Verification It is occasionally useful to intentionally disable join state verification. For example, this can enable use with: -- CI/CD providers without an explicit [dynamic join method][dynamic] +- CI/CD providers without an explicit [dynamic join method][dynamic]. - Nodes with immutable storage that cannot store an updated join state document after each join. From cf2d999a85d70dce3ac76bfff074952173185cf3 Mon Sep 17 00:00:00 2001 From: Tim Buckley Date: Wed, 9 Jul 2025 20:39:28 -0600 Subject: [PATCH 04/17] Update tbot CLI and configuration reference --- docs/pages/reference/cli/tbot.mdx | 75 +++++++++++++++---- .../reference/machine-id/configuration.mdx | 7 +- 2 files changed, 66 insertions(+), 16 deletions(-) diff --git a/docs/pages/reference/cli/tbot.mdx b/docs/pages/reference/cli/tbot.mdx index 6b74a335e9d81..598694a971346 100644 --- a/docs/pages/reference/cli/tbot.mdx +++ b/docs/pages/reference/cli/tbot.mdx @@ -269,21 +269,22 @@ These flags are available to all `tbot start` commands. Note that `tbot start legacy` supports slightly different options, so refer to its specific section for details when using a YAML config file or legacy output. -| Flag | Description | -|----------------------|-------------| -| `-d/--debug` | Enable verbose logging to stderr. | -| `--[no-]fips` | Whether to run tbot in FIPS compliance mode. This requires the FIPS `tbot` binary. | -| `--log-format` | Controls the format of output logs. Can be `json` or `text`. Defaults to `text`. | -| `-a/--auth-server` | Address of the Teleport Auth Service. Prefer using `--proxy-server` where possible. | -| `--proxy-server` | Address of the Teleport Proxy Server. | -| `--token` | A bot join token or path to file with token value, if attempting to onboard a new bot; used on first connect. | -| `--ca-pin` | CA pin to validate the Teleport Auth Service; used on first connect. | -| `--certificate-ttl` | TTL of short-lived machine certificates. | -| `--renewal-interval` | Interval at which short-lived certificates are renewed; must be less than the certificate TTL. | -| `--join-method` | Method to use to join the cluster. One of: `azure`, `circleci`, `gcp`, `github`, `gitlab`, `iam`, `kubernetes`, `spacelift`, `token`, `tpm`, `terraform_cloud` | -| `--[no-]oneshot` | If set, quit after the first renewal. | -| `--diag-addr` | If set and the bot is in debug mode, a diagnostics service will listen on specified address. | -| `--storage` | A destination URI for tbot's internal storage, e.g. `file:///foo/bar`. See [Destination URIs](#destination-uris) for more info. | +| Flag | Description | +|-------------------------|-------------| +| `-d/--debug` | Enable verbose logging to stderr. | +| `--[no-]fips` | Whether to run tbot in FIPS compliance mode. This requires the FIPS `tbot` binary. | +| `--log-format` | Controls the format of output logs. Can be `json` or `text`. Defaults to `text`. | +| `-a/--auth-server` | Address of the Teleport Auth Service. Prefer using `--proxy-server` where possible. | +| `--proxy-server` | Address of the Teleport Proxy Server. | +| `--token` | A bot join token or path to file with token value, if attempting to onboard a new bot; used on first connect. | +| `--ca-pin` | CA pin to validate the Teleport Auth Service; used on first connect. | +| `--certificate-ttl` | TTL of short-lived machine certificates. | +| `--renewal-interval` | Interval at which short-lived certificates are renewed; must be less than the certificate TTL. | +| `--join-method` | Method to use to join the cluster. One of: `azure`, `circleci`, `gcp`, `github`, `gitlab`, `iam`, `kubernetes`, `spacelift`, `token`, `tpm`, `terraform_cloud` | +| `--[no-]oneshot` | If set, quit after the first renewal. | +| `--diag-addr` | If set and the bot is in debug mode, a diagnostics service will listen on specified address. | +| `--storage` | A destination URI for tbot's internal storage, e.g. `file:///foo/bar`. See [Destination URIs](#destination-uris) for more info. | +| `--registration-secret` | An optional joining secret to use on first join with the `bound_keypair` join method. | ## tbot start legacy @@ -710,6 +711,50 @@ $ tbot install systemd \ --write ``` +## tbot keypair create + +Generates a keypair for use with +[`bound_keypair` joining](../machine-id/bound-keypair.mdx) and stores it in the +specified bot internal storage directory. If a key already exists in the +storage directory, no new key will be generated and the existing public key will +be printed to the console; use the `--overwrite` flag to force the generation of +a new keypair. + +### Flags + +| Flag | Description | +|------|-------------| +| `--storage` | A destination URI to be used for bot internal storage. Required. | +| `--proxy-server` | A Teleport Proxy address. Required. | +| `--overwrite` | If set, always generate a new key. If unset, the existing public key will be printed if one already exists in the destination specified by `--storage` | + +### Examples + +First, ensure the desired internal storage directory exists: +```code +$ mkdir -p /var/lib/teleport/bot +``` + +Next, generate a keypair: +```code +$ tbot keypair create --proxy-server example.teleport.sh:443 --storage /var/lib/teleport/bot +2025-07-09T00:00:00.000-00:00 INFO [TBOT] keypair has been written to storage storage:directory: /var/lib/teleport/bot tbot/keypair.go:135 + +To register the keypair with Teleport, include this public key in the token's +`spec.bound_keypair.onboarding.initial_public_key`: + + ssh-ed25519 +``` + +This public key, including the algorithm identifier (`ssh-ed25519`, but may vary +depending on your cluster configuration) can then be copied into a Bound Keypair +join token to be used as a +[preregistered key](../machine-id/bound-keypair.mdx#preregistered-key-example). + +Note that the Teleport Proxy address is required to fetch the currently enabled +[signature suite](../signature-algorithms.mdx). No authentication takes place at +this time. + ## Destination URIs Many `tbot start` subcommands accept destination URIs via the `--storage` and diff --git a/docs/pages/reference/machine-id/configuration.mdx b/docs/pages/reference/machine-id/configuration.mdx index 20349a7e599a7..3fc743005c5b4 100644 --- a/docs/pages/reference/machine-id/configuration.mdx +++ b/docs/pages/reference/machine-id/configuration.mdx @@ -121,6 +121,11 @@ onboarding: # multiple Teleport clusters from a single GitLab CI job. token_env_var_name: "MY_GITLAB_ID_TOKEN" + # bound_keypair holds parameters specific to the "bound_keypair" join method + bound_keypair: + # registration_secret is an optional secret to use on first join in lieu of + # a preregistered keypair. + registration_secret: "secret" # storage specifies the destination that `tbot` should use to store its # internal state. This state is sensitive, and you should ensure that the @@ -216,7 +221,7 @@ output used in context. # always be `identity`. type: identity # ssh_config controls whether the identity output will attempt to generate an -# OpenSSH configuration file. This requires that `tbot` can connect to the +# OpenSSH configuration file. This requires that `tbot` can connect to the # Teleport Proxy Service. Must be "on" or "off". If unspecified, this defaults to # "on". ssh_config: on From fc898f522450f9268f77526330a821abb22669dc Mon Sep 17 00:00:00 2001 From: Tim Buckley Date: Wed, 9 Jul 2025 20:45:26 -0600 Subject: [PATCH 05/17] Fix docs lints --- docs/pages/reference/machine-id/bound-keypair.mdx | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/pages/reference/machine-id/bound-keypair.mdx b/docs/pages/reference/machine-id/bound-keypair.mdx index cb9b9bc3b5a88..76ae5be0e4597 100644 --- a/docs/pages/reference/machine-id/bound-keypair.mdx +++ b/docs/pages/reference/machine-id/bound-keypair.mdx @@ -315,6 +315,7 @@ consider these use cases: While this approach makes downtime likely, it does ensure a human verifies the state of the bot node on each recovery. + - If you want human intervention for each recovery, but want to avoid downtime, you can increase this value by 2. The first attempt will be consumed immediately, but the bot will have one recovery attempt for automatic future @@ -322,6 +323,7 @@ consider these use cases: A human user can periodically audit the recovery count and bot node to ensure a recovery attempt is always available and the node is behaving as expected. + - Any larger value will increase the amount of time required between human intervention. You can select your tolerance for automatic bot recoveries as desired. @@ -559,7 +561,7 @@ of the previous 10 keypairs. This means server-side recovery may impossible if the keypair expected by the restored Teleport cluster has been rotated out of the client-side history, or if the client-side history has been lost or deleted. -[ephemeral]: ../architecture/machine-id-architecture.mdx#ephemeral +[ephemeral]: ../architecture/machine-id-architecture.mdx#ephemeral-token [dynamic]: ../architecture/machine-id-architecture.mdx#dynamic-join-tokens-eg-aws-iam [rbac]: ../access-controls/roles.mdx [token]: ../join-methods.mdx#ephemeral-tokens From e7c42377e7e40bf5523429299db542cbe75230a7 Mon Sep 17 00:00:00 2001 From: Tim Buckley Date: Thu, 10 Jul 2025 18:46:24 -0600 Subject: [PATCH 06/17] Add notes about new lock targets for bots --- .../management/security/revoking-access.mdx | 7 ++--- docs/pages/identity-governance/locking.mdx | 27 ++++++++++++++++++- 2 files changed, 30 insertions(+), 4 deletions(-) diff --git a/docs/pages/admin-guides/management/security/revoking-access.mdx b/docs/pages/admin-guides/management/security/revoking-access.mdx index 42472b7802693..ade8d3e0d1b70 100644 --- a/docs/pages/admin-guides/management/security/revoking-access.mdx +++ b/docs/pages/admin-guides/management/security/revoking-access.mdx @@ -24,9 +24,10 @@ how to execute the procedure. Teleport locks allow you to permanently or temporarily revoke access to a number of different "targets". Supported lock targets include: specific users, roles, -servers, desktops, or MFA devices. After you create a lock, all existing -sessions where the lock applies are terminated and new sessions are rejected -while the lock remains in force. +servers, desktops, or MFA devices. For Machine ID bots, this additionally +includes the join token name (for delegated join methods) and the bot instance +UUID. After you create a lock, all existing sessions where the lock applies are +terminated and new sessions are rejected while the lock remains in force. For more information, read our [Session and Identity Locking Guide](../../../identity-governance/locking.mdx). diff --git a/docs/pages/identity-governance/locking.mdx b/docs/pages/identity-governance/locking.mdx index 434ab6b774449..e2786e71a0394 100644 --- a/docs/pages/identity-governance/locking.mdx +++ b/docs/pages/identity-governance/locking.mdx @@ -22,6 +22,8 @@ A lock can target the following objects or attributes: cluster) - a Windows desktop by the desktop's name - an [Access Request](access-requests/access-requests.mdx) by UUID +- a bot instance ID (for Machine ID bots) +- a join token name (for Machine ID bots using a [delegated join method](../reference/join-methods.mdx#delegated-join-methods)) ## Prerequisites @@ -83,6 +85,29 @@ with one of the following options: # Created a lock with name "dc7cee9d-fe5e-4534-a90d-db770f0234a1". ``` + + The most appropriate locking target for a Machine ID bot depends on its join + method. + + For [delegated join methods](../reference/join-methods.mdx#secret-vs-delegated), + it's best to target the specific join token the bot is using to join: + ```code + $ tctl lock --join-token=example-token-name + ``` + + The join token name cannot be targeted for bots joined using the `token` join + method, so it's best to use the + [bot instance ID](../reference/architecture/machine-id-architecture.mdx#bot-instances): + ```code + $ tctl lock --bot-instance-id aabbccdd-1234-5678-0000-3b04d7d03acc + ``` + + In all cases, you may also target the bot user, which will lock all instances + of a bot that share the same underlying user: + ```code + $ tctl lock --user bot-example + ``` +
@@ -209,7 +234,7 @@ auth_service: Restart or redeploy the Auth Service for the change to take effect. -If not, edit your cluster authentication preference resource: +If not, edit your cluster authentication preference resource: ```code $ tctl edit cap From c8ea87aa0d4eff3e1bda1303042e96a0ebf2a4bb Mon Sep 17 00:00:00 2001 From: Tim Buckley Date: Thu, 10 Jul 2025 18:47:23 -0600 Subject: [PATCH 07/17] Bound keypair guide feedback Various bits of review feedback, and attempt to wordsmith on the introduction paragraphs. --- .../machine-id/deployment/bound-keypair.mdx | 40 +++++++++++-------- 1 file changed, 24 insertions(+), 16 deletions(-) diff --git a/docs/pages/machine-workload-identity/machine-id/deployment/bound-keypair.mdx b/docs/pages/machine-workload-identity/machine-id/deployment/bound-keypair.mdx index da227126913f5..d3d00ab46e8e0 100644 --- a/docs/pages/machine-workload-identity/machine-id/deployment/bound-keypair.mdx +++ b/docs/pages/machine-workload-identity/machine-id/deployment/bound-keypair.mdx @@ -1,20 +1,28 @@ --- title: Deploying Machine ID with Bound Keypair Joining -description: "How to install and configure Machine ID with Bound Keypair joining" +description: "How to install and configure Machine ID with Bound Keypair Joining" --- In this guide, you will install Machine ID's agent, `tbot`, on an arbitrary -node. This node could be a bare-metal machine, a VM, a container, or any other -host - the only requirement is that the node has persistent storage. +host using Bound Keypair Joining. This host could be a bare-metal machine, a +VM, a container, or any other host - the only requirement is that the host has +persistent storage. -Note that this is a generic join method. A platform-specific join method may be -available that is better suited to your environment; refer to the -[deployment guides](./deployment.mdx) for a full list of options. +Bound Keypair Joining is an improved alternative to +[secret-based join methods][secret] and can function as a drop-in replacement. +It is more secure than static token joining, and is more flexible than ephemeral +token joining with renewable certificates: when its certificates expire, it can +perform an automated recovery to ensure the bot can rejoin even after an +extended outage. + +Note that platform-specific join methods may be available that are better suited +to your environment; refer to the [deployment guides](./deployment.mdx) for a +full list of options. ## How it works -With Bound Keypair joining, instead of using a static token or relying solely on -renewable certificates, Machine ID bots generate a unique keypair. Teleport is +With Bound Keypair Joining, Machine ID bots generate a unique keypair which is +persistently stored in their internal data directory. Teleport is then configured to trust this public key for future joining attempts. Later, when the bot attempts to join the cluster, Teleport issues it a challenge @@ -23,17 +31,17 @@ challenge, attesting to its own identity, and is conditionally allowed to join the cluster. As self attestation is inherently less secure than the external verification -that would be provided by a cloud provider like AWS or a dedicated TPM, bound -keypair joining enforces a number of additional checks to prevent abuse, +that would be provided by a cloud provider like AWS or a dedicated TPM, Bound +Keypair Joining enforces a number of additional checks to prevent abuse, including: - Join state verification to ensure the keypair cannot be usefully shared or duplicated - Certificate generation counter checks to ensure regular bot certificates cannot be usefully shared or duplicated - Configurable limits on how often - if at all - bots may be allowed to recover - using this keypair. + using this keypair -An important benefit to Bound Keypair joining is that all joining restrictions +An important benefit to Bound Keypair Joining is that all joining restrictions can be reconfigured at any time, and bots that expire or go offline can be recovered by making a server-side exemption without any client-side intervention. @@ -49,7 +57,7 @@ works. - The `tsh` and `tctl` clients - (!docs/pages/includes/tctl.mdx!) - This guide assumes the bot node has mutable persistent storage for internal - bot data. While it is possible to use Bound Keypair joining can on immutable + bot data. While it is possible to use Bound Keypair Joining can on immutable hosts (like CI runs), doing so will reduce security guarantees; see the [admin guide][guide] for further information. @@ -98,9 +106,8 @@ spec: limit: 1 ``` -Replace: -- `example` in `spec.bot_name` with the name of the bot you created in the - second step +Replace `example` in `spec.bot_name` with the name of the bot you created in the +second step. For this example, we don't need to set any additional options for the bound keypair token. We've allowed a single recovery attempt, which will be used to @@ -171,4 +178,5 @@ Replace the following: all the available configuration options. - [More information about `TELEPORT_ANONYMOUS_TELEMETRY`.](../../../reference/machine-id/telemetry.mdx) +[secret]: ../../../reference/join-methods.mdx#secret-vs-delegated [guide]: ../../../reference/machine-id/bound-keypair.mdx From 5e4929358802f797c9a222514d0ca203c6fc7026 Mon Sep 17 00:00:00 2001 From: Tim Buckley Date: Thu, 10 Jul 2025 18:50:41 -0600 Subject: [PATCH 08/17] Fix review feedback and missing references Adds some missing references and addresses review feedback. --- docs/pages/reference/cli/tbot.mdx | 4 ++-- docs/pages/reference/join-methods.mdx | 20 ++++++++++++++++---- 2 files changed, 18 insertions(+), 6 deletions(-) diff --git a/docs/pages/reference/cli/tbot.mdx b/docs/pages/reference/cli/tbot.mdx index 598694a971346..5207f121fd67f 100644 --- a/docs/pages/reference/cli/tbot.mdx +++ b/docs/pages/reference/cli/tbot.mdx @@ -275,7 +275,7 @@ specific section for details when using a YAML config file or legacy output. | `--[no-]fips` | Whether to run tbot in FIPS compliance mode. This requires the FIPS `tbot` binary. | | `--log-format` | Controls the format of output logs. Can be `json` or `text`. Defaults to `text`. | | `-a/--auth-server` | Address of the Teleport Auth Service. Prefer using `--proxy-server` where possible. | -| `--proxy-server` | Address of the Teleport Proxy Server. | +| `--proxy-server` | Address of the Teleport Proxy Service. | | `--token` | A bot join token or path to file with token value, if attempting to onboard a new bot; used on first connect. | | `--ca-pin` | CA pin to validate the Teleport Auth Service; used on first connect. | | `--certificate-ttl` | TTL of short-lived machine certificates. | @@ -725,7 +725,7 @@ a new keypair. | Flag | Description | |------|-------------| | `--storage` | A destination URI to be used for bot internal storage. Required. | -| `--proxy-server` | A Teleport Proxy address. Required. | +| `--proxy-server` | A Teleport Proxy Service address. Required. | | `--overwrite` | If set, always generate a new key. If unset, the existing public key will be printed if one already exists in the destination specified by `--storage` | ### Examples diff --git a/docs/pages/reference/join-methods.mdx b/docs/pages/reference/join-methods.mdx index a708894aae76c..9855e0c774b3b 100644 --- a/docs/pages/reference/join-methods.mdx +++ b/docs/pages/reference/join-methods.mdx @@ -117,6 +117,7 @@ Delegated join methods are: - [`azure`](#azure-managed-identity-azure) - [`azure_devops`](#azure-devops-azure_devops) - [`bitbucket`](#bitbucket-pipelines-bitbucket) +- [`bound_keypair`](#bound-keypair-bound_keypair) - [`circleci`](#circleci-circleci) - [`ec2`](#aws-ec2-identity-document-ec2) - [`gcp`](#gcp-service-account-gcp) @@ -143,6 +144,7 @@ Renewable join-methods are: - [ephemeral `token`](#ephemeral-tokens) - [static `token`](#static-tokens) - [`ec2`](#aws-ec2-identity-document-ec2) +- [`bound_keypair`](#bound-keypair-bound_keypair) Nodes with non-renewable certificates must join again in order to get a new certificate before expiry. The instance will have to prove again that it is legitimate. @@ -257,7 +259,12 @@ Or as Teleport resources: (!docs/pages/includes/provision-token/ephemeral-spec.mdx!) -When a MachineID bot uses an ephemeral join token, the token is deleted. +When a Machine ID bot uses an ephemeral join token, the token is deleted. + + +New Machine ID bot deployments should consider upgrading to the +[`bound_keypair` join method](#bound-keypair-bound_keypair). + - How to [Join Services with a Secure Token](../enroll-resources/agents/join-token.mdx). @@ -266,9 +273,14 @@ When a MachineID bot uses an ephemeral join token, the token is deleted. ### Bound Keypair: `bound_keypair` -Bound Keypair tokens are an improved alternative to -[ephemeral tokens](#ephemeral-tokens) that aims to improve reliability for -on-prem deployments +Bound Keypair tokens are an alternative to +[secret-based join methods](#secret-based-join-methods) that improve security +and flexibility. They are best used on platforms with persistent storage, but +can be configured for use in any environment. + +This join method is recommended for on-prem environments +[without TPMs](#trusted-platform-module-tpm) or cloud platforms +without a specialized [delegated join method](#delegated-join-methods). (!docs/pages/includes/provision-token/bound-keypair-spec.mdx!) From cf57e4c555b22bb595956e6909f05316d67ac16c Mon Sep 17 00:00:00 2001 From: Tim Buckley Date: Thu, 10 Jul 2025 18:56:27 -0600 Subject: [PATCH 09/17] Address review feedback, add missing detail Adds missing detail, especially around decoupled keypair from certs. --- .../reference/machine-id/bound-keypair.mdx | 55 ++++++++++++------- 1 file changed, 34 insertions(+), 21 deletions(-) diff --git a/docs/pages/reference/machine-id/bound-keypair.mdx b/docs/pages/reference/machine-id/bound-keypair.mdx index 76ae5be0e4597..67ea01170b33d 100644 --- a/docs/pages/reference/machine-id/bound-keypair.mdx +++ b/docs/pages/reference/machine-id/bound-keypair.mdx @@ -3,8 +3,8 @@ title: Bound Keypair Joining description: "Bound Keypair Joining: Reference and admin guide" --- -Bound Keypair is a new join method designed to provide the best features of -[dynamic join methods][dynamic] join methods - like AWS, GCP, or Azure - but in +Bound Keypair is a join method designed to provide the best features of +[delegated join methods][s-vs-d] join methods - like AWS, GCP, or Azure - but in on-prem or otherwise unsupported environments where no external verification is available. @@ -18,29 +18,32 @@ Specifically, this join method: cases -Bound Keypair joining is available in v18.1.0 and is intended to replace `token` +Bound Keypair Joining is available in v18.1.0 and is intended to replace `token` joining as the default recommended join method in Teleport v19.0.0. -## Use Cases +## Use cases -Bound Keypair joining can be used in any environment and is designed to function +Bound Keypair Joining can be used in any environment and is designed to function as a drop-in replacement for the traditional [`token`][ephemeral] join method in all situations where it is used today. This includes bare-metal and on-prem hardware where TPMs are not available, or cloud providers not currently -supported by a delegated join method. +supported by a [delegated join method][delegated]. -As a drop-in replacement for `token` joining, Bound Keypair joining is also a -good replacement for local experimentation for testing, and can be easily +Similar to `token` joining, Bound Keypair Joining is also a good replacement for +local experimentation for testing, with minimal configuration needed to onboard +a bot initially. When ready to deploy to production, it's trivial to adjust +onboarding and recovery settings to select your desired balance between +resiliency and security. Additionally, with [`insecure` recovery](#disabling-join-state-verification) and in situations that can accommodate the security complications, Bound Keypair -joining can be used to join bots in otherwise unsupported CI/CD providers by +Joining can be used to join bots in otherwise unsupported CI/CD providers by persisting the bot's keypair in the platform keystore. ## Limitations -While Bound Keypair joining does enable or simplify a number use cases, it does +While Bound Keypair Joining does enable or simplify a number use cases, it does have limitations that may make it unfit in some instances. In particular, the [secure recovery modes](#recovery) introduce some deployment @@ -59,12 +62,20 @@ used with care. ## Concepts -### Bound Keypair +### Bound keypair The term "bound keypair" refers to both the type of credential and way it's used in Teleport: first, a bot generates a standard public/private keypair; next, Teleport is configured to trust the public key, binding it to a token. +While bots using Bound Keypair Joining will receive regular, short-lived +Teleport certificates, and will refresh them regularly in a similar manner to +[renewable certificates][renewable], they also make use of an additional +public/private keypair that is not tied directly to a certificate. Critically, +this means the keypair does not have an expiration date, and that any trust +relationship established between Teleport and a Bound Keypair bot is controlled +by server-side configuration within Teleport and not due to the passage of time. + Once this trust relationship has been established, bots can - in a sense - serve as their own identity authority, similar to how cloud providers can attest to the identity of a node or CI workflow run. Teleport issues the joining bot a @@ -72,7 +83,7 @@ unique cryptographic challenge which the bot signs using its private key, and the result is verified using the key registered with Teleport. As this self attestation is not by itself a sufficient security guarantee, Bound -Keypair joining makes use of additional server-side rules to control whether or +Keypair Joining makes use of additional server-side rules to control whether or not a join attempt will be allowed, including: - Limiting the number of [recovery attempts](#recovery) and optionally requiring (server-side) human intervention for each recovery. @@ -158,7 +169,7 @@ values: security checks and should be used with care. [See below](#disabling-join-state-verification) for more information. -### Join State Verification +### Join state verification A **join state document** is an additional piece of information provided to joining bots alongside their usual Teleport certificate bundle. It contains @@ -185,13 +196,13 @@ credentials in use by the attacker. Note that join state verification is disabled when the token's `spec.bound_keypair.recovery.mode` is set to `insecure`. -## Token Resource Examples +## Token resource examples Bound Keypair tokens can be configured using Teleport's existing `token` resource using the `bound_keypair` join method. For a complete reference, refer to the [provision token reference](../join-methods.mdx#bound-keypair-bound_keypair). -### Minimal Token +### Minimal token A minimal bound keypair token does not require any special configuration beyond specifying the `bound_keypair` join method value: @@ -229,7 +240,7 @@ $ tbot configure identity \ $ tbot start -c tbot.yaml ``` -### Preregistered Key Example +### Preregistered key example To use a preregistered key instead of a registration secret, a keypair needs to be generated on the bot node. To do so, run the following commands: @@ -291,7 +302,7 @@ Note the following: Typically, you might create tokens on your local machine and run `tbot` on a remote server. -## Admin Reference +## Admin reference ### Allowing additional recovery attempts @@ -465,7 +476,7 @@ version: v2 Note the different locks and lock targets shown above: bots can be targeted by any of their Teleport user name (`bot-example`), the bot instance ID (a UUID), or the join token name. Locks created automatically for bots using Bound Keypair -joining will typically use a `join_token` target, but a lock targeting any of +Joining will typically use a `join_token` target, but a lock targeting any of these values could be created manually. Note that locks may have a message field containing details about why the lock @@ -507,11 +518,11 @@ If you do suspect the bot's credentials may have been compromised, you may also want to [request a keypair rotation](#requesting-a-keypair-rotation) in addition to taking other steps to ensure the node is properly secured. -### Disabling Join State Verification +### Disabling join state verification It is occasionally useful to intentionally disable join state verification. For example, this can enable use with: -- CI/CD providers without an explicit [dynamic join method][dynamic]. +- CI/CD providers without an explicit [delegated join method][delegated]. - Nodes with immutable storage that cannot store an updated join state document after each join. @@ -562,6 +573,8 @@ the keypair expected by the restored Teleport cluster has been rotated out of the client-side history, or if the client-side history has been lost or deleted. [ephemeral]: ../architecture/machine-id-architecture.mdx#ephemeral-token -[dynamic]: ../architecture/machine-id-architecture.mdx#dynamic-join-tokens-eg-aws-iam +[delegated]: ../join-methods.mdx#delegated-join-methods [rbac]: ../access-controls/roles.mdx [token]: ../join-methods.mdx#ephemeral-tokens +[renewable]: ../join-methods.mdx#renewable-vs-non-renewable +[s-vs-d]: ../join-methods.mdx#secret-vs-delegated From 6556cd0fa9e26c559e59b7ae25386b4329620c50 Mon Sep 17 00:00:00 2001 From: Tim Buckley Date: Thu, 10 Jul 2025 19:05:03 -0600 Subject: [PATCH 10/17] Further clarify what automatic recovery is --- .../machine-id/deployment/bound-keypair.mdx | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/pages/machine-workload-identity/machine-id/deployment/bound-keypair.mdx b/docs/pages/machine-workload-identity/machine-id/deployment/bound-keypair.mdx index d3d00ab46e8e0..d4e25af587d12 100644 --- a/docs/pages/machine-workload-identity/machine-id/deployment/bound-keypair.mdx +++ b/docs/pages/machine-workload-identity/machine-id/deployment/bound-keypair.mdx @@ -28,7 +28,9 @@ then configured to trust this public key for future joining attempts. Later, when the bot attempts to join the cluster, Teleport issues it a challenge that can only be completed using its private key. The bot returns the solved challenge, attesting to its own identity, and is conditionally allowed to join -the cluster. +the cluster. This process is repeated for every join attempt, but if the bot has +been offline long enough for its certificates to expire, it is additionally +forced to perform an automatic recovery to join again. As self attestation is inherently less secure than the external verification that would be provided by a cloud provider like AWS or a dedicated TPM, Bound @@ -38,8 +40,8 @@ including: duplicated - Certificate generation counter checks to ensure regular bot certificates cannot be usefully shared or duplicated -- Configurable limits on how often - if at all - bots may be allowed to recover - using this keypair +- Configurable limits on how often - if at all - bots may be allowed to + automatically recover using this keypair An important benefit to Bound Keypair Joining is that all joining restrictions can be reconfigured at any time, and bots that expire or go offline can be From e723101b1163e047e2b875b3f308b8a1d324aa47 Mon Sep 17 00:00:00 2001 From: Tim Buckley Date: Thu, 10 Jul 2025 19:08:07 -0600 Subject: [PATCH 11/17] Consistently refer to "bot hosts" rather than nodes --- .../machine-id/deployment/bound-keypair.mdx | 10 ++++---- .../reference/machine-id/bound-keypair.mdx | 24 +++++++++---------- 2 files changed, 17 insertions(+), 17 deletions(-) diff --git a/docs/pages/machine-workload-identity/machine-id/deployment/bound-keypair.mdx b/docs/pages/machine-workload-identity/machine-id/deployment/bound-keypair.mdx index d4e25af587d12..faa1ff0a4fdcb 100644 --- a/docs/pages/machine-workload-identity/machine-id/deployment/bound-keypair.mdx +++ b/docs/pages/machine-workload-identity/machine-id/deployment/bound-keypair.mdx @@ -56,18 +56,18 @@ works. {/* note: consider edition-prereqs-tabs.mdx include for v19; it is misleading due to the minor launch release */} - A running Teleport cluster version 18.1.0 or above. -- The `tsh` and `tctl` clients +- The `tsh` and `tctl` clients. - (!docs/pages/includes/tctl.mdx!) -- This guide assumes the bot node has mutable persistent storage for internal +- This guide assumes the bot host has mutable persistent storage for internal bot data. While it is possible to use Bound Keypair Joining can on immutable hosts (like CI runs), doing so will reduce security guarantees; see the [admin guide][guide] for further information. ## Step 1/5. Install `tbot` -**This step is completed on the bot node.** +**This step is completed on the bot host.** -First, `tbot` needs to be installed on the node that you wish to use Machine ID +First, `tbot` needs to be installed on the host that you wish to use Machine ID on. Download and install the appropriate Teleport package for your platform: @@ -139,7 +139,7 @@ the `.status.bound_keypair.registration_secret` field. ## Step 4/5. Configure `tbot` -**This step is completed on the bot node.** +**This step is completed on the bot host.** Create `/etc/tbot.yaml`: diff --git a/docs/pages/reference/machine-id/bound-keypair.mdx b/docs/pages/reference/machine-id/bound-keypair.mdx index 67ea01170b33d..d32c8a3c8b262 100644 --- a/docs/pages/reference/machine-id/bound-keypair.mdx +++ b/docs/pages/reference/machine-id/bound-keypair.mdx @@ -78,7 +78,7 @@ by server-side configuration within Teleport and not due to the passage of time. Once this trust relationship has been established, bots can - in a sense - serve as their own identity authority, similar to how cloud providers can attest to -the identity of a node or CI workflow run. Teleport issues the joining bot a +the identity of a host or CI workflow run. Teleport issues the joining bot a unique cryptographic challenge which the bot signs using its private key, and the result is verified using the key registered with Teleport. @@ -95,11 +95,11 @@ not a join attempt will be allowed, including: Onboarding takes place at first join and is the stage at which a public key becomes bound to a token. There are two methods bots may use to complete this step: -- Preregistered keys: a private key is generated locally on the bot node, and +- Preregistered keys: a private key is generated locally on the bot host, and the public key is copied out-of-band to Teleport by a human user or a script. This method ensures no secret values are ever copied between systems. - Registration secrets: a one time use random secret is generated by Teleport - and provided to the bot node via either a human user or a script, similar to + and provided to the bot host via either a human user or a script, similar to the [ephemeral `token` join method][token]. If a registration secret is used, the bot uses the registration secret to prove @@ -243,7 +243,7 @@ $ tbot start -c tbot.yaml ### Preregistered key example To use a preregistered key instead of a registration secret, a keypair needs to -be generated on the bot node. To do so, run the following commands: +be generated on the bot host. To do so, run the following commands: ```code ## If needed, create the bot's storage directory $ mkdir -p /var/lib/teleport/bot @@ -298,7 +298,7 @@ Note the following: - In this example, the recovery limit defaults to 1, and will be reached as soon as the bot joins. To enable automatic recovery, increase the recovery limit to the desired value. -- The `tctl create` and `tbot` commands do not need to be run on the same node. +- The `tctl create` and `tbot` commands do not need to be run on the same host. Typically, you might create tokens on your local machine and run `tbot` on a remote server. @@ -325,15 +325,15 @@ consider these use cases: downtime. While this approach makes downtime likely, it does ensure a human verifies the - state of the bot node on each recovery. + state of the bot host on each recovery. - If you want human intervention for each recovery, but want to avoid downtime, you can increase this value by 2. The first attempt will be consumed immediately, but the bot will have one recovery attempt for automatic future use. - A human user can periodically audit the recovery count and bot node to ensure - a recovery attempt is always available and the node is behaving as expected. + A human user can periodically audit the recovery count and bot host to ensure + a recovery attempt is always available and the host is behaving as expected. - Any larger value will increase the amount of time required between human intervention. You can select your tolerance for automatic bot recoveries as @@ -351,7 +351,7 @@ bot recovery attempts until the limit is increased again. Additionally, note that [join state verification](#join-state-verification) is still required, and will prevent multiple concurrent uses of the same keypair and token. In other words, increasing the recovery limit will not allow multiple -nodes to join. +clients to join. ### Allowing unlimited recovery attempts @@ -393,7 +393,7 @@ reauthenticate every 20 minutes, so it may take some time for the request to be acknowledged. You can monitor the rotation status by watching the token's `.status.bound_keypair.last_rotated_at` field. -If you want to force an early rotation and have access to the bot node, you can +If you want to force an early rotation and have access to the bot host, you can restart the `tbot` process, or send it a signal with `pkill -usr1 tbot` to request an early rotation. @@ -498,7 +498,7 @@ Change the `.spec.bound_keypair.recovery.mode` field to `insecure`, save, and quit the editor. The bot can now be allowed to rejoin. Given sufficient time it will retry on its -own, but if you have access to the node, `systemctl restart tbot` or similar can +own, but if you have access to the host, `systemctl restart tbot` or similar can be used to restart the bot process. The bot should now be able to join successfully. You can monitor progress by @@ -516,7 +516,7 @@ $ tctl edit token/example-token If you do suspect the bot's credentials may have been compromised, you may also want to [request a keypair rotation](#requesting-a-keypair-rotation) in -addition to taking other steps to ensure the node is properly secured. +addition to taking other steps to ensure the host is properly secured. ### Disabling join state verification From af9941be51d62a7fc941eb93130a2495e7e3e156 Mon Sep 17 00:00:00 2001 From: Tim Buckley Date: Thu, 10 Jul 2025 19:24:43 -0600 Subject: [PATCH 12/17] Address more review feedback - Retitle the reference guide - Link to preregistered keys in the guide --- .../machine-id/deployment/bound-keypair.mdx | 10 ++++++++-- docs/pages/reference/cli/tbot.mdx | 6 +++--- docs/pages/reference/machine-id/bound-keypair.mdx | 2 +- 3 files changed, 12 insertions(+), 6 deletions(-) diff --git a/docs/pages/machine-workload-identity/machine-id/deployment/bound-keypair.mdx b/docs/pages/machine-workload-identity/machine-id/deployment/bound-keypair.mdx index faa1ff0a4fdcb..c998b662ada4a 100644 --- a/docs/pages/machine-workload-identity/machine-id/deployment/bound-keypair.mdx +++ b/docs/pages/machine-workload-identity/machine-id/deployment/bound-keypair.mdx @@ -117,8 +117,14 @@ allow the bot's initial join, and Teleport will generate a registration secret automatically when the token is created as we have not preregistered a public key to use. - -To learn more about the various onboarding and recovery options, refer to the + +This example makes use of registration secrets to authenticate the initial join. +If desired, it is also possible to generate a key on the bot host first and +register it with Teleport out-of-band, avoiding the need to copy secrets between +hosts. + +To learn more about preregistering public keys and Bound Keypair Joining's other +onboarding and recovery options, refer to the [Reference and Admin Guide][guide]. diff --git a/docs/pages/reference/cli/tbot.mdx b/docs/pages/reference/cli/tbot.mdx index 5207f121fd67f..7448723185541 100644 --- a/docs/pages/reference/cli/tbot.mdx +++ b/docs/pages/reference/cli/tbot.mdx @@ -751,9 +751,9 @@ depending on your cluster configuration) can then be copied into a Bound Keypair join token to be used as a [preregistered key](../machine-id/bound-keypair.mdx#preregistered-key-example). -Note that the Teleport Proxy address is required to fetch the currently enabled -[signature suite](../signature-algorithms.mdx). No authentication takes place at -this time. +Note that the Teleport Proxy Service address is required to fetch the currently +enabled [signature suite](../signature-algorithms.mdx). No authentication takes +place at this time. ## Destination URIs diff --git a/docs/pages/reference/machine-id/bound-keypair.mdx b/docs/pages/reference/machine-id/bound-keypair.mdx index d32c8a3c8b262..6a295b1a53d71 100644 --- a/docs/pages/reference/machine-id/bound-keypair.mdx +++ b/docs/pages/reference/machine-id/bound-keypair.mdx @@ -1,5 +1,5 @@ --- -title: Bound Keypair Joining +title: Bound Keypair Joining Reference description: "Bound Keypair Joining: Reference and admin guide" --- From c87b4ad34157ca214abf6156edc80d799ac26588 Mon Sep 17 00:00:00 2001 From: Tim Buckley Date: Thu, 10 Jul 2025 19:40:05 -0600 Subject: [PATCH 13/17] Add guides section to reference page --- docs/pages/reference/machine-id/bound-keypair.mdx | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/pages/reference/machine-id/bound-keypair.mdx b/docs/pages/reference/machine-id/bound-keypair.mdx index 6a295b1a53d71..facedfd426690 100644 --- a/docs/pages/reference/machine-id/bound-keypair.mdx +++ b/docs/pages/reference/machine-id/bound-keypair.mdx @@ -60,6 +60,13 @@ This limitation can be worked around using the does meaningfully reduce the join method's security protections and should be used with care. +## Guides + +You can read step-by-step guides on using Bound Keypair Joining with Machine ID: + +- [Using Machine ID with Bound Keypair Joining](../../machine-workload-identity/machine-id/deployment/bound-keypair.mdx): + How to install and configure Machine ID with Bound Keypair Joining + ## Concepts ### Bound keypair From 8b80c8ae176b4b8c5bceeb408a085bcf1a809ff6 Mon Sep 17 00:00:00 2001 From: Tim Buckley Date: Tue, 15 Jul 2025 15:59:44 -0600 Subject: [PATCH 14/17] Remove admin guide from this PR This removes the admin guide from this PR and replaces references to it with placeholders with commented TODOs. The admin guide will be reintroduced in a separate PR stacked onto this one. --- .../machine-id/deployment/bound-keypair.mdx | 6 + docs/pages/reference/cli/tbot.mdx | 17 +- docs/pages/reference/join-methods.mdx | 5 + .../reference/machine-id/bound-keypair.mdx | 587 ------------------ .../pages/reference/machine-id/machine-id.mdx | 1 - 5 files changed, 21 insertions(+), 595 deletions(-) delete mode 100644 docs/pages/reference/machine-id/bound-keypair.mdx diff --git a/docs/pages/machine-workload-identity/machine-id/deployment/bound-keypair.mdx b/docs/pages/machine-workload-identity/machine-id/deployment/bound-keypair.mdx index c998b662ada4a..dfdc1e712fb31 100644 --- a/docs/pages/machine-workload-identity/machine-id/deployment/bound-keypair.mdx +++ b/docs/pages/machine-workload-identity/machine-id/deployment/bound-keypair.mdx @@ -187,4 +187,10 @@ Replace the following: - [More information about `TELEPORT_ANONYMOUS_TELEMETRY`.](../../../reference/machine-id/telemetry.mdx) [secret]: ../../../reference/join-methods.mdx#secret-vs-delegated +[guide]: ../../../reference/machine-id/machine-id.mdx + +{/* +TODO: guide link above is a placeholder, link to the real guide once merged in +follow-up PR. [guide]: ../../../reference/machine-id/bound-keypair.mdx +*/} diff --git a/docs/pages/reference/cli/tbot.mdx b/docs/pages/reference/cli/tbot.mdx index 7448723185541..362727ec41b0c 100644 --- a/docs/pages/reference/cli/tbot.mdx +++ b/docs/pages/reference/cli/tbot.mdx @@ -713,12 +713,11 @@ $ tbot install systemd \ ## tbot keypair create -Generates a keypair for use with -[`bound_keypair` joining](../machine-id/bound-keypair.mdx) and stores it in the -specified bot internal storage directory. If a key already exists in the -storage directory, no new key will be generated and the existing public key will -be printed to the console; use the `--overwrite` flag to force the generation of -a new keypair. +Generates a keypair for use with `bound_keypair` joining and stores it in the +specified bot internal storage directory. If a key already exists in the storage +directory, no new key will be generated and the existing public key will be +printed to the console; use the `--overwrite` flag to force the generation of a +new keypair. ### Flags @@ -748,8 +747,12 @@ To register the keypair with Teleport, include this public key in the token's This public key, including the algorithm identifier (`ssh-ed25519`, but may vary depending on your cluster configuration) can then be copied into a Bound Keypair -join token to be used as a +join token to be used as a preregistered key. + +{/* +TODO: Replace with a link into the admin guide once the follow up PR has merged. [preregistered key](../machine-id/bound-keypair.mdx#preregistered-key-example). +*/} Note that the Teleport Proxy Service address is required to fetch the currently enabled [signature suite](../signature-algorithms.mdx). No authentication takes diff --git a/docs/pages/reference/join-methods.mdx b/docs/pages/reference/join-methods.mdx index 9855e0c774b3b..80379ea9f986e 100644 --- a/docs/pages/reference/join-methods.mdx +++ b/docs/pages/reference/join-methods.mdx @@ -286,7 +286,12 @@ without a specialized [delegated join method](#delegated-join-methods). - [Deploying Machine ID with Bound Keypair joining](../machine-workload-identity/machine-id/deployment/bound-keypair.mdx) + +{/* +TODO: Uncomment after follow-up PR with admin guide has merged. - [Bound Keypair Reference and Admin Guide](./machine-id/bound-keypair.mdx) +*/} + ### AWS IAM role: `iam` diff --git a/docs/pages/reference/machine-id/bound-keypair.mdx b/docs/pages/reference/machine-id/bound-keypair.mdx deleted file mode 100644 index facedfd426690..0000000000000 --- a/docs/pages/reference/machine-id/bound-keypair.mdx +++ /dev/null @@ -1,587 +0,0 @@ ---- -title: Bound Keypair Joining Reference -description: "Bound Keypair Joining: Reference and admin guide" ---- - -Bound Keypair is a join method designed to provide the best features of -[delegated join methods][s-vs-d] join methods - like AWS, GCP, or Azure - but in -on-prem or otherwise unsupported environments where no external verification is -available. - -Specifically, this join method: -- Does not require dedicated TPM hardware or external identity attestation -- Does not require long-lived shared secrets -- Allows for limited automatic recovery if certificates expire -- Allows recovery restrictions to be relaxed or lifted to accommodate different - use cases and deployment scenarios -- Ensures failed bots can be recovered without client-side intervention in most - cases - - -Bound Keypair Joining is available in v18.1.0 and is intended to replace `token` -joining as the default recommended join method in Teleport v19.0.0. - - -## Use cases - -Bound Keypair Joining can be used in any environment and is designed to function -as a drop-in replacement for the traditional [`token`][ephemeral] join method in -all situations where it is used today. This includes bare-metal and on-prem -hardware where TPMs are not available, or cloud providers not currently -supported by a [delegated join method][delegated]. - -Similar to `token` joining, Bound Keypair Joining is also a good replacement for -local experimentation for testing, with minimal configuration needed to onboard -a bot initially. When ready to deploy to production, it's trivial to adjust -onboarding and recovery settings to select your desired balance between -resiliency and security. - -Additionally, with [`insecure` recovery](#disabling-join-state-verification) and -in situations that can accommodate the security complications, Bound Keypair -Joining can be used to join bots in otherwise unsupported CI/CD providers by -persisting the bot's keypair in the platform keystore. - -## Limitations - -While Bound Keypair Joining does enable or simplify a number use cases, it does -have limitations that may make it unfit in some instances. - -In particular, the [secure recovery modes](#recovery) introduce some deployment -restrictions: -- Each bot deployment must be issued a unique token. For deployment at scale, - use of Teleport's - [Terraform provider](../terraform-provider/terraform-provider.mdx) is - recommended to create tokens in bulk for each deployment. -- Each bot deployment must be able to store client-side state (used for - [join state verification](#join-state-verification)). - -This limitation can be worked around using the -[`insecure` recovery mode](#disabling-join-state-verification), but doing so -does meaningfully reduce the join method's security protections and should be -used with care. - -## Guides - -You can read step-by-step guides on using Bound Keypair Joining with Machine ID: - -- [Using Machine ID with Bound Keypair Joining](../../machine-workload-identity/machine-id/deployment/bound-keypair.mdx): - How to install and configure Machine ID with Bound Keypair Joining - -## Concepts - -### Bound keypair - -The term "bound keypair" refers to both the type of credential and way it's used -in Teleport: first, a bot generates a standard public/private keypair; next, -Teleport is configured to trust the public key, binding it to a token. - -While bots using Bound Keypair Joining will receive regular, short-lived -Teleport certificates, and will refresh them regularly in a similar manner to -[renewable certificates][renewable], they also make use of an additional -public/private keypair that is not tied directly to a certificate. Critically, -this means the keypair does not have an expiration date, and that any trust -relationship established between Teleport and a Bound Keypair bot is controlled -by server-side configuration within Teleport and not due to the passage of time. - -Once this trust relationship has been established, bots can - in a sense - serve -as their own identity authority, similar to how cloud providers can attest to -the identity of a host or CI workflow run. Teleport issues the joining bot a -unique cryptographic challenge which the bot signs using its private key, and -the result is verified using the key registered with Teleport. - -As this self attestation is not by itself a sufficient security guarantee, Bound -Keypair Joining makes use of additional server-side rules to control whether or -not a join attempt will be allowed, including: -- Limiting the number of [recovery attempts](#recovery) and optionally requiring - (server-side) human intervention for each recovery. -- Preventing reuse or theft of private keys using - [join state verification](#join-state-verification). - -### Onboarding - -Onboarding takes place at first join and is the stage at which a public key -becomes bound to a token. There are two methods bots may use to complete this -step: -- Preregistered keys: a private key is generated locally on the bot host, and - the public key is copied out-of-band to Teleport by a human user or a script. - This method ensures no secret values are ever copied between systems. -- Registration secrets: a one time use random secret is generated by Teleport - and provided to the bot host via either a human user or a script, similar to - the [ephemeral `token` join method][token]. - -If a registration secret is used, the bot uses the registration secret to prove -its identity for this first connection, generates a keypair on the fly, and -registers the public key with Teleport. - -Registration secrets are used by default: if a token is created and no initial -(preregistered) public key is provided, a registration secret will be randomly -generated by Teleport and written to the token's -`status.bound_keypair.registration_secret` field. - -In both cases - once registered, or when using preregistered keys - the joining -bot is then required to complete a joining challenge. A unique cryptographic -challenge is generated by Teleport which the bot must sign and return. If the -signature matches the registered public key, onboarding has succeeded: the -public key is permanently bound to the token, a certificate bundle is issued, -and a join state document is provided for use on the next join attempt. - -### Recovery - -Recovery takes place when a bot uses its private key to request new -certificates. This can occur in two situations: -- On first join, when the bot has no certificates. -- When the bot's certificates expire before they can be renewed, such as when a - bot is offline for longer than its configured `certificate_ttl`. - -So long as the bot has and maintains valid Teleport credentials - by default, -bot certificates are valid for an hour and renewed every 20 minutes, but these -values are configurable - no recovery will be needed after the initial join. -Recovery is only required when a bot is unable to refresh its certificates for -longer than its configured certificate TTL (e.g. 1 hour). - -The behavior of the recovery process depends on two values configured in the -token resource: the recovery mode (`spec.bound_keypair.recovery.mode`), and the -recovery limit (`spec.bound_keypair.recovery.limit`). The mode may have these -values: -- `standard` (default): only `limit` recovery attempts can be automatically - attempted. - - This mode is recommended for most situations. Bots can recover automatically - so long as the recovery count (`status.bound_keypair.recovery_count`) doesn't - exceed the configured limit, which allows cluster administrators to select - their tolerance for automatic bot recovery attempts. - - [Join state verification](#join-state-verification) is enabled, which helps - prevent keypair reuse, but does require clients to store additional on each - join. - - Note that as the initial join counts as a recovery attempt, `limit` must - always be at least `1` for the initial join to succeed. - -- `relaxed`: `limit` is ignored, but - [join state verification](#join-state-verification) remains enabled. - - This mode allows for unlimited recovery attempts and is useful for lower - security impact bots, or deployments where bots are regularly expected to go - offline for extended periods. - - Like `standard` mode, join state verification is enabled to help prevent - keypair reuse, but does require clients to store additional state on each - join. - -- `insecure`: `limit` is ignored, and join state verification is **disabled**. - - This mode is the most flexible, as bots can rejoin repeatedly and can operate - without mutable client-side state. However, this disables most additional - security checks and should be used with care. - [See below](#disabling-join-state-verification) for more information. - -### Join state verification - -A **join state document** is an additional piece of information provided to -joining bots alongside their usual Teleport certificate bundle. It contains -signed information about the state of the joining process, including a sequence -number that uniquely identifies each join attempt. - -**Join state verification** both ensures that the bot can provide a valid and -signed join state document, and that the document contains the expected sequence -number. -* If successful, the join attempt can proceed and a new join state document will - be issued with an incremented sequence counter. -* If the sequence number presented by the client is outdated, the join - attempt is rejected, and a lock is created to ensure existing clients will be - denied further access to the Teleport cluster. - -This system takes advantage of Teleport's short lived certificates and frequent -bot reauthentication. If an attacker somehow obtains a copy of a bot's keypair -and attempts to use it to retrieve Teleport certificates, they will initially -succeed. However, the original bot will eventually make another authentication -attempt, using its now-outdated join state document. This join attempt with an -outdated document will trigger a lockout and block both the original bot and any -credentials in use by the attacker. - -Note that join state verification is disabled when the token's -`spec.bound_keypair.recovery.mode` is set to `insecure`. - -## Token resource examples - -Bound Keypair tokens can be configured using Teleport's existing `token` -resource using the `bound_keypair` join method. For a complete reference, refer -to the [provision token reference](../join-methods.mdx#bound-keypair-bound_keypair). - -### Minimal token - -A minimal bound keypair token does not require any special configuration beyond -specifying the `bound_keypair` join method value: -```yaml -kind: token -version: v2 -metadata: - name: example -spec: - roles: [Bot] - join_method: bound_keypair - bot_name: example -``` - -Without additional options specified, this will use the `standard` recovery mode -with a default recovery limit of 1 to allow for the initial join. A registration -secret will be automatically generated when the token is created on the Teleport -cluster. To create and use this token, adjust the name and bot name as needed, -and write it to a file, for example `token.yaml`, and run: - -```code -$ tctl create -f token.yaml -$ tctl get token/example-bkp --format=json | jq -r '.[0].status.bound_keypair.registration_secret' -``` - -This will print the registration secret which can be provided to the `tbot` -client using the `--registration-secret` flag: -```code -$ tbot configure identity \ - --proxy-server=example.teleport.sh:443 \ - --join-method=bound_keypair \ - --token=example \ - --registration-secret=$SECRET_VALUE \ - --destination=./destination > tbot.yaml -$ tbot start -c tbot.yaml -``` - -### Preregistered key example - -To use a preregistered key instead of a registration secret, a keypair needs to -be generated on the bot host. To do so, run the following commands: -```code -## If needed, create the bot's storage directory -$ mkdir -p /var/lib/teleport/bot -$ tbot keypair create --storage /var/lib/teleport/bot --proxy-server=example.teleport.sh:443 -2025-07-08T16:31:48.000-00:00 INFO [TBOT] keypair has been written to storage storage:directory: /var/lib/teleport/bot tbot/keypair.go:135 - -To register the keypair with Teleport, include this public key in the token's -`spec.bound_keypair.onboarding.initial_public_key`: - - ssh-ed25519 -``` - -Note that while a proxy server address must be provided, this is only used to -ping the cluster to determine its configured -[signature algorithms](../signature-algorithms.mdx). Once the keypair has been -generated, the public key will be printed to the console. - -Next, create a file named `token.yaml` with the following content: - -```yaml -kind: token -version: v2 -metadata: - name: example-token -spec: - roles: [Bot] - join_method: bound_keypair - bot_name: example - bound_keypair: - onboarding: - initial_public_key: "ssh-ed25519 " -``` - -Copy the public key (including the `ssh-ed25519 ...` prefix or similar) into the -`initial_public_key` field, and make certain the value is contained in quotes. - -Once finished, run the following to create the token and join the bot: -```code -## Create the token. -$ tctl create -f token.yaml -## Configure and start the bot. -$ tbot configure identity \ - --proxy-server=example.teleport.sh:443 \ - --join-method=bound_keypair \ - --token=example \ - --storage=/var/lib/teleport/bot \ - --destination=./destination > tbot.yaml -$ tbot start -c tbot.yaml -``` - -Note the following: -- In this example, the recovery limit defaults to 1, and will be reached as soon - as the bot joins. To enable automatic recovery, increase the recovery limit to - the desired value. -- The `tctl create` and `tbot` commands do not need to be run on the same host. - Typically, you might create tokens on your local machine and run `tbot` on a - remote server. - -## Admin reference - -### Allowing additional recovery attempts - -When using the `standard` recovery mode, only a configured number of recovery -attempts can be made. If the limit is reached, no further recovery attempts can -be made until the limit is increased. - -To increase this limit and allow an expired bot to join again, edit the token -using `tctl edit`: -```code -$ tctl edit token/example-token -``` - -Find the `spec.bound_keypair.recovery.limit` field and increment the limit by -the desired amount. You are free to select any desired threshold. For example, -consider these use cases: -- If human intervention is desired for each join attempt you can increase this - value by 1. This single recovery attempt will be immediately consumed, so - future recoveries will again require human intervention, and may result in - downtime. - - While this approach makes downtime likely, it does ensure a human verifies the - state of the bot host on each recovery. - -- If you want human intervention for each recovery, but want to avoid downtime, - you can increase this value by 2. The first attempt will be consumed - immediately, but the bot will have one recovery attempt for automatic future - use. - - A human user can periodically audit the recovery count and bot host to ensure - a recovery attempt is always available and the host is behaving as expected. - -- Any larger value will increase the amount of time required between human - intervention. You can select your tolerance for automatic bot recoveries as - desired. - -Alternatively, if you wish to allow an unlimited number of automatic recovery -attempts, [refer to the entry below](#allowing-unlimited-recovery-attempts) on -the `relaxed` recovery mode. - -Note that the recovery limit is always relative to the recovery counter (in the -`status.bound_keypair.recovery_count` field in the token resource). It is valid -to decrease the limit or set it to zero, however doing so may prevent future -bot recovery attempts until the limit is increased again. - -Additionally, note that [join state verification](#join-state-verification) is -still required, and will prevent multiple concurrent uses of the same keypair -and token. In other words, increasing the recovery limit will not allow multiple -clients to join. - -### Allowing unlimited recovery attempts - -To allow unlimited recovery attempts, the `spec.bound_keypair.recovery.mode` -field should be set to `relaxed`. To do this, use `tctl edit` to edit the token: -```code -$ tctl edit token/example-token -``` - -Find or create the `spec.bound_keypair.recovery.mode` field and set the value to -`relaxed`. Save the file and quit your editor to update the token. - -When the recovery mode is set to `relaxed`, the `limit` field is ignored and the -`status.bound_keypair.recovery_count` field may increase beyond the written -limit. If the mode is later changed back to `standard`, be aware that future -recovery attempts will fail unless the `limit` is increased to accommodate the -current value of `recovery_count`. - -Note that when `relaxed` mode is in use, -[join state verification](#join-state-verification) is still required and will -prevent multiple concurrent uses of the same keypair and token. If your use case -requires this, you can -[disable join state verification](#disabling-join-state-verification), but doing -so does impact the security of the token. - -### Requesting a keypair rotation - -To request a keypair rotation, set the `.spec.bound_keypair.rotate_after` field -to contain a timestamp. On the next authentication attempt after that timestamp -has elapsed, the bot will automatically rotate its keypair. - -To simplify this process, you can use the `tctl bound-keypair rotate` helper: -```code -$ tctl bound-keypair rotate token-name -``` - -This sets the timestamp to the current time. Note that by default bots only -reauthenticate every 20 minutes, so it may take some time for the request to be -acknowledged. You can monitor the rotation status by watching the token's -`.status.bound_keypair.last_rotated_at` field. - -If you want to force an early rotation and have access to the bot host, you can -restart the `tbot` process, or send it a signal with `pkill -usr1 tbot` to -request an early rotation. - -Note that the previous 10 keypairs are retained on the client for use in case of -a cluster rollback; refer to the -[cluster rollback](#recovery-after-a-cluster-rollback) section for additional -information. - -### Locking a `bound_keypair` bot or bot instance - -The simplest way to lock out a bot that joined using the `bound_keypair` join -method is to use a join token lock target: - -```code -$ tctl lock --join-token=token-name -``` - -As a bound keypair token is linked to a single bot, this will effectively lock -the bot. It will not be able to reauthenticate, recover, interact with the -Teleport API, or otherwise use its credentials until the lock is removed. - -Note that if a bot is locked for long enough - bots have a 1 hour certificate -TTL by default - its certificates will expire. If you intend to remove this lock -and reinstate the bot, you may also need to increase the recovery limit -(`.spec.bound_keypair.recovery.limit`) to accommodate the additional recovery -attempt. - -Other lock targets can also be used, but are not preferred: -- Bot instance (`tctl lock --bot-instance-id ...`): will lock only a single - instance of the bot. Note that if the recovery limit allows for it, the - [automatic recovery process](#recovery) will attempt to rejoin and, if - successful, will generate a new bot instance ID. -- Bot name (`tctl lock --user bot-`): will lock all bots using the same - bot / user. This may be overly broad and lock other instances running under - this bot user. - -### Recovering a locked `bound_keypair` bot instance - -Bots joined with the `bound_keypair` join method can become automatically locked -under various conditions, including: -- Failing to correctly complete [join state verification](#join-state-verification) -- Connecting with certificates that have an invalid [generation counter][ephemeral] -- Locked manually by a cluster admin - -To recover a bot that has become locked, first ensure the bot's internal storage -(`storage`) has not been compromised. These locking conditions are designed to -trigger if more than one client tries to join using a copy of the same -certificates and private key. This can occur due to a misconfiguration or due -to an attacker copying a bot's credentials, so ideally the latter should be -ruled out before unlocking a bot. - -Next, determine the name (UUID) of the lock or locks targeting the bot: -```code -$ tctl get lock -kind: lock -metadata: - name: 372af058-76d1-4e64-93da-3b04d7d03ac2 -spec: - target: - user: bot-example -version: v2 ---- -kind: lock -metadata: - name: 791d0b1d-01b4-4752-8a99-9b2908aebfae -spec: - target: - bot_instance_id: e7d494ae-a0ff-4d12-b935-de5e2025f667 -version: v2 ---- -kind: lock -metadata: - name: a69fdbb2-8e53-406a-b453-48b2cda6991d -spec: - target: - join_token: example-token-name -version: v2 -``` - -Note the different locks and lock targets shown above: bots can be targeted by -any of their Teleport user name (`bot-example`), the bot instance ID (a UUID), -or the join token name. Locks created automatically for bots using Bound Keypair -Joining will typically use a `join_token` target, but a lock targeting any of -these values could be created manually. - -Note that locks may have a message field containing details about why the lock -was created. - -Once the lock name(s) have been determined, remove each using `tctl rm`: -```code -$ tctl rm lock/372af058-76d1-4e64-93da-3b04d7d03ac2 -``` - -Next, join state should be reset. Use `tctl edit` to set the token's recovery -mode to `insecure`, but make a note of the current value (`standard` or -`relaxed`): -```code -$ tctl edit token/example-token -``` - -Change the `.spec.bound_keypair.recovery.mode` field to `insecure`, save, and -quit the editor. - -The bot can now be allowed to rejoin. Given sufficient time it will retry on its -own, but if you have access to the host, `systemctl restart tbot` or similar can -be used to restart the bot process. - -The bot should now be able to join successfully. You can monitor progress by -watching for new audit events in Teleport's web UI, or by waiting for the -recovery counter to increase: -```code -$ tctl get token/example-token --format=json | jq '.[].status.bound_keypair.recovery_count' -``` - -Once the bot has joined successfully, reset the recovery mode to its previous -value using `tctl edit`: -```code -$ tctl edit token/example-token -``` - -If you do suspect the bot's credentials may have been compromised, you may also -want to [request a keypair rotation](#requesting-a-keypair-rotation) in -addition to taking other steps to ensure the host is properly secured. - -### Disabling join state verification - -It is occasionally useful to intentionally disable join state verification. For -example, this can enable use with: -- CI/CD providers without an explicit [delegated join method][delegated]. -- Nodes with immutable storage that cannot store an updated join state document - after each join. - -Before continuing, be aware that disabling join state verification will prevent -Teleport from detecting if multiple clients are joining using the same bound -keypair token. In other words, if the private key is copied by an attacker, they -will be able to join indefinitely. Take care to protect the keypair, and make -certain to limit access from the bot identity using Teleport's -[RBAC system][rbac]. - -When ready, use `tctl edit` to modify the Bound Keypair token: -```code -$ tctl edit token/example-token -``` - -Find or add the `spec.bound_keypair.recovery.mode` field and set it to -`insecure`. Save and quit your editor to update the token. - -With the mode set to `insecure`, the `recovery.limit` is ignored, allowing -unlimited reuse of the token, and join state verification is disabled, allowing -concurrent or stateless reuse. - -### Recovery after a cluster rollback - -If your Teleport cluster is rolled back for any reason, joining bots may fail -[join state verification](#join-state-verification) as their local join state -document may not match the values currently (or previously) known to Teleport. - -The simplest workaround is to temporarily set all bound keypair tokens to -`insecure` recovery mode for the first join attempt following a cluster restore. -Once they've joined once, they will once again have a valid join state, so the -recovery mode can be restored to its previous value. - -To change the recovery mode, use `tctl edit` to modify the token resource: -```code -$ tctl edit token/example-token -``` - -Find the `spec.bound_keypair.recovery.mode` field, and set the value to -"insecure". Repeat this for each bound keypair token. Wait for all bound keypair -bots to reauthenticate, and repeat this process to restore the recovery mode to -its previous value. - -If [bot keypairs were rotated](#requesting-a-keypair-rotation) between the -snapshot and restore of the Teleport cluster, note that bots only keep a record -of the previous 10 keypairs. This means server-side recovery may impossible if -the keypair expected by the restored Teleport cluster has been rotated out of -the client-side history, or if the client-side history has been lost or deleted. - -[ephemeral]: ../architecture/machine-id-architecture.mdx#ephemeral-token -[delegated]: ../join-methods.mdx#delegated-join-methods -[rbac]: ../access-controls/roles.mdx -[token]: ../join-methods.mdx#ephemeral-tokens -[renewable]: ../join-methods.mdx#renewable-vs-non-renewable -[s-vs-d]: ../join-methods.mdx#secret-vs-delegated diff --git a/docs/pages/reference/machine-id/machine-id.mdx b/docs/pages/reference/machine-id/machine-id.mdx index 6a11b039b4c83..69ad4cf34eb20 100644 --- a/docs/pages/reference/machine-id/machine-id.mdx +++ b/docs/pages/reference/machine-id/machine-id.mdx @@ -11,4 +11,3 @@ description: Configuration and CLI reference for Teleport Machine ID. - [Telemetry](telemetry.mdx) - [V16 Upgrade Guide](v16-upgrade-guide.mdx) - [Bot Terraform Resource](../../reference/terraform-provider/resources/bot.mdx) -- [Bound Keypair Joining](bound-keypair.mdx) From 1a086d235849fc6575c42a281a7565e633ee9306 Mon Sep 17 00:00:00 2001 From: Tim Buckley Date: Tue, 15 Jul 2025 16:03:54 -0600 Subject: [PATCH 15/17] Fix docs lint --- .../machine-id/deployment/bound-keypair.mdx | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/pages/machine-workload-identity/machine-id/deployment/bound-keypair.mdx b/docs/pages/machine-workload-identity/machine-id/deployment/bound-keypair.mdx index dfdc1e712fb31..f30229a09a11f 100644 --- a/docs/pages/machine-workload-identity/machine-id/deployment/bound-keypair.mdx +++ b/docs/pages/machine-workload-identity/machine-id/deployment/bound-keypair.mdx @@ -186,11 +186,11 @@ Replace the following: all the available configuration options. - [More information about `TELEPORT_ANONYMOUS_TELEMETRY`.](../../../reference/machine-id/telemetry.mdx) -[secret]: ../../../reference/join-methods.mdx#secret-vs-delegated -[guide]: ../../../reference/machine-id/machine-id.mdx - {/* TODO: guide link above is a placeholder, link to the real guide once merged in follow-up PR. [guide]: ../../../reference/machine-id/bound-keypair.mdx */} + +[secret]: ../../../reference/join-methods.mdx#secret-vs-delegated +[guide]: ../../../reference/machine-id/machine-id.mdx From a9acbbbff0c9c35adc8c8e2acf284768c30baced Mon Sep 17 00:00:00 2001 From: Tim Buckley Date: Wed, 16 Jul 2025 10:25:06 -0600 Subject: [PATCH 16/17] Fix Machine ID -> Machine & Workload Identity naming --- docs/pages/identity-governance/locking.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/pages/identity-governance/locking.mdx b/docs/pages/identity-governance/locking.mdx index e2786e71a0394..62de4fb178592 100644 --- a/docs/pages/identity-governance/locking.mdx +++ b/docs/pages/identity-governance/locking.mdx @@ -22,8 +22,8 @@ A lock can target the following objects or attributes: cluster) - a Windows desktop by the desktop's name - an [Access Request](access-requests/access-requests.mdx) by UUID -- a bot instance ID (for Machine ID bots) -- a join token name (for Machine ID bots using a [delegated join method](../reference/join-methods.mdx#delegated-join-methods)) +- a bot instance ID (for Machine & Workload Identity bots) +- a join token name (for Machine & Workload Identity bots using a [delegated join method](../reference/join-methods.mdx#delegated-join-methods)) ## Prerequisites From 7b32e94f2c6c75d1e370a881f7fde2f1e1a9bdff Mon Sep 17 00:00:00 2001 From: Tim Buckley Date: Wed, 16 Jul 2025 10:29:35 -0600 Subject: [PATCH 17/17] Update more Machine & Workload Identity references --- .../management/security/revoking-access.mdx | 9 +++++---- docs/pages/identity-governance/locking.mdx | 6 +++--- .../machine-id/deployment/bound-keypair.mdx | 14 +++++++------- docs/pages/reference/join-methods.mdx | 2 +- 4 files changed, 16 insertions(+), 15 deletions(-) diff --git a/docs/pages/admin-guides/management/security/revoking-access.mdx b/docs/pages/admin-guides/management/security/revoking-access.mdx index ade8d3e0d1b70..91de6f5dd9506 100644 --- a/docs/pages/admin-guides/management/security/revoking-access.mdx +++ b/docs/pages/admin-guides/management/security/revoking-access.mdx @@ -24,10 +24,11 @@ how to execute the procedure. Teleport locks allow you to permanently or temporarily revoke access to a number of different "targets". Supported lock targets include: specific users, roles, -servers, desktops, or MFA devices. For Machine ID bots, this additionally -includes the join token name (for delegated join methods) and the bot instance -UUID. After you create a lock, all existing sessions where the lock applies are -terminated and new sessions are rejected while the lock remains in force. +servers, desktops, or MFA devices. For Machine & Workload Identity bots, this +additionally includes the join token name (for delegated join methods) and the +bot instance UUID. After you create a lock, all existing sessions where the loc +applies are terminated and new sessions are rejected while the lock remains in +force. For more information, read our [Session and Identity Locking Guide](../../../identity-governance/locking.mdx). diff --git a/docs/pages/identity-governance/locking.mdx b/docs/pages/identity-governance/locking.mdx index 62de4fb178592..f2eed1318ab63 100644 --- a/docs/pages/identity-governance/locking.mdx +++ b/docs/pages/identity-governance/locking.mdx @@ -85,9 +85,9 @@ with one of the following options: # Created a lock with name "dc7cee9d-fe5e-4534-a90d-db770f0234a1". ``` - - The most appropriate locking target for a Machine ID bot depends on its join - method. + + The most appropriate locking target for a Machine & Workload Identity bot + depends on its join method. For [delegated join methods](../reference/join-methods.mdx#secret-vs-delegated), it's best to target the specific join token the bot is using to join: diff --git a/docs/pages/machine-workload-identity/machine-id/deployment/bound-keypair.mdx b/docs/pages/machine-workload-identity/machine-id/deployment/bound-keypair.mdx index f30229a09a11f..fa012d0cf61e3 100644 --- a/docs/pages/machine-workload-identity/machine-id/deployment/bound-keypair.mdx +++ b/docs/pages/machine-workload-identity/machine-id/deployment/bound-keypair.mdx @@ -3,10 +3,10 @@ title: Deploying Machine ID with Bound Keypair Joining description: "How to install and configure Machine ID with Bound Keypair Joining" --- -In this guide, you will install Machine ID's agent, `tbot`, on an arbitrary -host using Bound Keypair Joining. This host could be a bare-metal machine, a -VM, a container, or any other host - the only requirement is that the host has -persistent storage. +In this guide, you will install Machine & Workload Identity's agent, `tbot`, on +an arbitrary host using Bound Keypair Joining. This host could be a bare-metal +machine, a VM, a container, or any other host - the only requirement is that the +host has persistent storage. Bound Keypair Joining is an improved alternative to [secret-based join methods][secret] and can function as a drop-in replacement. @@ -21,9 +21,9 @@ full list of options. ## How it works -With Bound Keypair Joining, Machine ID bots generate a unique keypair which is -persistently stored in their internal data directory. Teleport is -then configured to trust this public key for future joining attempts. +With Bound Keypair Joining, Machine & Workload Identity bots generate a unique +keypair which is persistently stored in their internal data directory. Teleport +is then configured to trust this public key for future joining attempts. Later, when the bot attempts to join the cluster, Teleport issues it a challenge that can only be completed using its private key. The bot returns the solved diff --git a/docs/pages/reference/join-methods.mdx b/docs/pages/reference/join-methods.mdx index 80379ea9f986e..f66dd9b0ea0b1 100644 --- a/docs/pages/reference/join-methods.mdx +++ b/docs/pages/reference/join-methods.mdx @@ -262,7 +262,7 @@ Or as Teleport resources: When a Machine ID bot uses an ephemeral join token, the token is deleted. -New Machine ID bot deployments should consider upgrading to the +New Machine & Workload Identity bot deployments should consider upgrading to the [`bound_keypair` join method](#bound-keypair-bound_keypair).