[v13] Support proxy version server

[bernardjkim]

Backport #35150, #35342, #35996, #35998 to branch/v13

changelog: Support running a version server in the proxy for automatic agent upgrades.

[bernardjkim]

Testing

• Verify default version server

❯ curl https://bernard-dev.cloud.gravitational.io/webapi/automaticupgrades/channel/stable/cloud/version
v13.4.14%

❯ curl https://updates.releases.teleport.dev/v1/stable/cloud/version
v13.4.14

• Verify forward_url and static_url

# teleport.yaml
proxy_service:                                                              
  enabled: yes                                                              
  automatic_upgrades_channels:                                              
    stable/cloud:                                                           
      forward_url: https://updates.releases.teleport.dev/v1/stable/cloud/v13
    stable/cloud/v13:                                                       
      forward_url: https://updates.releases.teleport.dev/v1/stable/cloud/v13
    stable/cloud/v14:                                                       
      forward_url: https://updates.releases.teleport.dev/v1/stable/cloud/v14
    static/cloud/v13:                                                       
      static_version: v13.4.14                                                
    static/cloud/v14:                                                       
      static_version: v14.3.0                                              
    static/cloud/v15:                                                       
      static_version: v15.0.0       

❯ curl https://bernard-dev.cloud.gravitational.io/webapi/automaticupgrades/channel/stable/cloud/version
v13.4.14%
❯ curl https://bernard-dev.cloud.gravitational.io/webapi/automaticupgrades/channel/stable/cloud/v13/version
v13.4.14%
❯ curl https://bernard-dev.cloud.gravitational.io/webapi/automaticupgrades/channel/stable/cloud/v14/version
13.4.14%
❯ curl https://bernard-dev.cloud.gravitational.io/webapi/automaticupgrades/channel/static/cloud/v13/version
v13.4.14%
❯ curl https://bernard-dev.cloud.gravitational.io/webapi/automaticupgrades/channel/static/cloud/v14/version
13.4.14%
❯ curl https://bernard-dev.cloud.gravitational.io/webapi/automaticupgrades/channel/static/cloud/v15/version
13.4.14%

• Verify install scripts

# kube agent install script
helm install teleport-agent teleport/teleport-kube-agent -f prod-cluster-values.yaml --version 13.4.14 --create-namespace --namespace teleport

# systemd agent install script
❯ curl -fsSL https://bernard-dev.cloud.gravitational.io/scripts/.../install-node.sh | grep TELEPORT_VERSION -m 1
TELEPORT_VERSION='13.4.14'

• Verify kube agent upgrade v13.4.14

# kubectl -n teleport logs teleport-agent-updater...
2024-01-05T02:06:17Z    INFO    starting the updater    {"version": "13.4.14", "url": "https://bernard-dev.cloud.gravitational.io/webapi/automaticupgrades/channel/stable/cloud"}
...
2024-01-05T01:39:27Z    DEBUG   New version candidate   {"controller": "statefulset", "controllerGroup": "apps", "controllerKind": "StatefulSet", "StatefulSet": {"name":"teleport-agent","namespace":"teleport"}, "namespace": "teleport", "name": "teleport-agent", "reconcileID": "20bc75f0-1d27-4aab-afef-a64e24f09cc7", "namespacedname": {"name":"teleport-agent","namespace":"teleport"}, "kind": "StatefulSet", "nextVersion": "v13.4.13"}
2024-01-05T01:39:27Z    DEBUG   Version change is valid, building img candidate {"controller": "statefulset", "controllerGroup": "apps", "controllerKind": "StatefulSet", "StatefulSet": {"name":"teleport-agent","namespace":"teleport"}, "namespace": "teleport", "name": "teleport-agent", "reconcileID": "20bc75f0-1d27-4aab-afef-a64e24f09cc7", "namespacedname": {"name":"teleport-agent","namespace":"teleport"}, "kind": "StatefulSet"}
2024-01-05T01:39:27Z    DEBUG   Verifying candidate img {"controller": "statefulset", "controllerGroup": "apps", "controllerKind": "StatefulSet", "StatefulSet": {"name":"teleport-agent","namespace":"teleport"}, "namespace": "teleport", "name": "teleport-agent", "reconcileID": "20bc75f0-1d27-4aab-afef-a64e24f09cc7", "namespacedname": {"name":"teleport-agent","namespace":"teleport"}, "kind": "StatefulSet", "img": "public.ecr.aws/gravitational/teleport-ent-distroless:13.4.13"}
2024-01-05T01:39:27Z    DEBUG   Image approved by the validator {"controller": "statefulset", "controllerGroup": "apps", "controllerKind": "StatefulSet", "StatefulSet": {"name":"teleport-agent","namespace":"teleport"}, "namespace": "teleport", "name": "teleport-agent", "reconcileID": "20bc75f0-1d27-4aab-afef-a64e24f09cc7", "namespacedname": {"name":"teleport-agent","namespace":"teleport"}, "kind": "StatefulSet", "image": "public.ecr.aws/gravitational/teleport-ent-distroless:13.4.13", "validator": "cosign signature validator-a55977c6d752759f68c4883ac10ad8a85f5cfd0f", "resolvedImages": "public.ecr.aws/gravitational/teleport-ent-distroless:13.4.13@sha256:349463559827f10262b8b4e9b61702336d4c5031bac4f929c3265633b9bdcc1d"}
2024-01-05T01:39:27Z    DEBUG   The following image was verified        {"controller": "statefulset", "controllerGroup": "apps", "controllerKind": "StatefulSet", "StatefulSet": {"name":"teleport-agent","namespace":"teleport"}, "namespace": "teleport", "name": "teleport-agent", "reconcileID": "20bc75f0-1d27-4aab-afef-a64e24f09cc7", "namespacedname": {"name":"teleport-agent","namespace":"teleport"}, "kind": "StatefulSet", "verifiedImage": "public.ecr.aws/gravitational/teleport-ent-distroless:13.4.13@sha256:349463559827f10262b8b4e9b61702336d4c5031bac4f929c3265633b9bdcc1d"}
2024-01-05T01:39:27Z    INFO    Updating podSpec with image     {"controller": "statefulset", "controllerGroup": "apps", "controllerKind": "StatefulSet", "StatefulSet": {"name":"teleport-agent","namespace":"teleport"}, "namespace": "teleport", "name": "teleport-agent", "reconcileID": "20bc75f0-1d27-4aab-afef-a64e24f09cc7", "namespacedname": {"name":"teleport-agent","namespace":"teleport"}, "kind": "StatefulSet", "image": "public.ecr.aws/gravitational/teleport-ent-distroless:13.4.13@sha256:349463559827f10262b8b4e9b61702336d4c5031bac4f929c3265633b9bdcc1d"}
2024-01-05T01:39:27Z    DEBUG   statefulset managed pods        {"controller": "statefulset", "controllerGroup": "apps", "controllerKind": "StatefulSet", "StatefulSet": {"name":"teleport-agent","namespace":"teleport"}, "namespace": "teleport", "name": "teleport-agent", "reconcileID": "20bc75f0-1d27-4aab-afef-a64e24f09cc7", "namespacedname": {"name":"teleport-agent","namespace":"teleport"}, "kind": "StatefulSet", "managedPodsList": ["teleport-agent-0", "teleport-agent-1"]}
2024-01-05T01:39:27Z    DEBUG    no statefulset unhealthy pods from old revisions       {"controller": "statefulset", "controllerGroup": "apps", "controllerKind": "StatefulSet", "StatefulSet": {"name":"teleport-agent","namespace":"teleport"}, "namespace": "teleport", "name": "teleport-agent", "reconcileID": "20bc75f0-1d27-4aab-afef-a64e24f09cc7", "namespacedname": {"name":"teleport-agent","namespace":"teleport"}, "kind": "StatefulSet"}

[public-teleport-github-review-bot]

@bernardjkim - this PR will require admin approval to merge due to its size. Consider breaking it up into a series smaller changes.

[bernardjkim]

@hugoShaka I'm keeping integrations/kube-agent-updater as a separate module for this v13 backport. Merging into the teleport module results in a number of dependency conflicts that I'm having trouble resolving.

If it because an issue later, I'll take another look at removing the integrations/kube-agent-updater module.

[hugoShaka]

I'm keeping integrations/kube-agent-updater as a separate module for this v13 backport. Merging into the teleport module results in a number of dependency conflicts that I'm having trouble resolving.

Thank you for the heads up 👍

I don't think some code duplication in v13 is a no-go, but we'll have to be very careful when backporting future updater changes to v13 as the built-in version server and the kube-agent-updater will use two different implementations.