Add renewable certificate generation checks

[timothyb89]

This adds a new validation check for renewable certificates that maintains a renewal counter as both a certificate extension and a user label. This counter is used to ensure only a single certificate lineage can exist: for example, if a renewable certificate is stolen, only one copy of the certificate can be renewed as the generation counter will not match.

When renewing a certificate, first the generation counter presented by the user (via their TLS identity) is compared to a value stored with the associated user (in a new teleport.dev/bot-generation label field). If they aren't equal, the renewal attempt fails. Otherwise, the generation counter is incremented by 1, stored to the database using a CompareAndSwap() to ensure atomicity, and set on the generated certificate for use in future renewals.

Summary of changes:

1. Adds a new Generation extension to TLS and SSH certificates, with associated fields in Identity, IdentityContext, UserCertParams, certRequest, etc.
2. Adds a new CompareAndSwapUser method for performing compare-and-swap operation on User objects.
3. Adds a new validateGenerationLabel() method, called by generateUserCerts(), to check and update the generation field + label. If the counters do not match, a lock is created on the bot user and an audit warning is emitted.

For more information about the intent behind this, refer to the "Preventing Certificate Propagation" section in the certificate bot RFD.

Reliability considerations

• CompareAndSwap() doesn't seem to be hugely battle-tested and until now has only been used during CA rotations. A couple of bugs were found during the implementation (see also Enable map key sorting in utils.FastMarshal #10070)
• Adds a modest amount of complexity to the already-complex generateUserCerts() process, particularly:
  • Adds a cache-bypassing GetUser() read and a CompareAndSwap() write to an API call that was otherwise (mostly) read-only (I think the only writes previously were to the audit log).
• In general this change does severely increase the failure potential for certificate renewal attempts. Most of the biggest failure points have been smoothed out but there are some that probably can't be (e.g. now the bot simply breaks if it fails to save a set of freshly-renewed certs)

Security considerations

• The CompareAndSwapUser() call in validateGenerationLabel() bypasses RBAC controls to update the User. It's a very narrow update so I don't think this should pose a problem by itself.
• We're relying on CompareAndSwapUser() to act as a security control and prevent race conditions if e.g. an attacker steals a renewable certificate.
• This mechanism is mainly designed to limit abuse potential in case certificates are stolen, rather than eliminate it. There is still a fairly wide window of time where an attacker could steal and use a certificate before being discovered.

Outstanding TODOs

• Certificate generation mismatch should produce an audit log event and lock the associated bot user (per the bot RFD)
• Thorough unit testing

[timothyb89]

Just hit (I think) gogo/protobuf#693 due to user traits now being set, so now we just fetch the user from the database a bunch of times to copy it. Which is insane, but I'm not sure there's a better way. We can't marshal/unmarshal it because importing the code to do that creates an import cycle 🙃

[zmb3]

Food for thought:

I wonder if we need a new label prefix. Thus far, I believe teleport.dev has been used for things that are set by teleport but potentially useful for users. For example:

• teleport.dev/origin indicates whether a resource was defined in a config file or discovered dynamically
• teleport.dev/os Contains the OS of a Windows desktop discovered via LDAP

I'm not aware of any instances where teleport.dev is necessary for teleport itself to function correctly. This would be the first. If a user changes or removes this label, things break, right?

Maybe a teleport.internal/ prefix for things that are required by Teleport and should never be touched by users makes sense. We could always hide these in the UI and CLIs unless some sort of --all-labels flag was present.

[timothyb89]

Maybe a teleport.internal/ prefix for things that are required by Teleport and should never be touched by users makes sense. We could always hide these in the UI and CLIs unless some sort of --all-labels flag was present.

I like that, it clarifies the intent nicely.

I've also debated whether or not some of these fields could just be added to the User struct directly. Realistically though I don't think I see that happening before v9.

[timothyb89]

Now that locking is working, I figure this is ready enough to at least merge into the main PR (#10099), pending review of course.

[timothyb89]

I'm going to force squash+merge this one into the parent PR (#10099) for the sake of reviewers