Add the ability to override KSAs. (#1020)

This is a new opt-in feature that will allow developers to supply custom
KSAs for their applciations. This will allow consistent KSAs to be used
for blue/green applications, and for integrations like GCP's workload
identity to work with the KSAs in Kf -- supporting Kf users who want to
move away from Service Brokers and use rotating creds.

config/config-defaults.yaml

@@ -162,6 +162,9 @@ data:
     # up the start command for Apps which requires fetching the container
     # configuration for every App.
     appDisableStartCommandLookup: "false"
+    # AppEnableServiceAccountOverride allows apps to override the service account via annotation:
+    # apps.kf.dev/service-account-name.
+    appEnableServiceAccountOverride: "false"
     # Maximum time in seconds for a deployment to make progress before it is considered to be failed
     progressDeadlineSeconds: "600"
     # The grace period is the duration in seconds after the processes running in the pod are sent a termination signal and
@@ -251,6 +254,7 @@ data:
   appCPUPerGBOfRAM: "100m"
   appCPUMin: "100m"
   appDisableStartCommandLookup: "false"
+  appEnableServiceAccountOverride: "false"
   progressDeadlineSeconds: "600"
   terminationGracePeriodSeconds: "30"
   routeTrackVirtualService: "false"

docs/content/en/docs/v2.11/_index.md

@@ -3,3 +3,6 @@ title: "Documentation"
 linkTitle: "Kf Documentation"
 ---
 
+Kf offers developers the Cloud Foundry experience while empowering operators to
+adopt declarative Kubernetes practices. It makes migrating Cloud Foundry workloads to Kubernetes
+straightforward, and most importantly, avoids major changes to developer workflows.

docs/content/en/docs/v2.11/developer/build-and-deploy/manifest.md

@@ -84,7 +84,7 @@ The hint is used by the [service mesh](https://cloud.google.com/service-mesh/doc
 {{< warning >}} Kf doesn't currently support TCP port-based routing. You must use a
 [Kubernetes LoadBalancer](https://kubernetes.io/docs/tutorials/stateless-application/expose-external-ip-address/) if you want to expose a TCP port to the Internet. Ports are available on the cluster internal App address `<app-name>.<space>`.{{< /warning >}}
 
-## Metadata fields
+## Metadata fields {#metadata-fields}
 
 The following fields are valid for `application.metadata` objects:
 

docs/content/en/docs/v2.11/developer/configure-service-accounts.md

@@ -0,0 +1,99 @@
+---
+title: "Configure and Use Service Accounts"
+weight: 51
+description: >
+    Configure and Use Kubernetes Service Accounts from your Apps.
+---
+
+By default, all applications in Kf are assigned a unique Kubernetes Service Account (KSA) named `sa-<APP_NAME>`.
+Kf uses this KSA as the "user" it runs application instances and tasks under.
+
+Each App KSA receives a copy of the container registry credentials used by the Space's build KSA so Kf apps can pull
+container images that were created during `kf push`.
+
+## Using the service account
+
+Kuberenetes Pods (the building blocks of Apps and Tasks) automatically receive a JWT for the KSA
+mounted in the container:
+
+```sh
+$ ls /var/run/secrets/kubernetes.io/serviceaccount/
+ca.crt
+namespace
+token
+```
+
+* `ca.crt` The Kubernetes control plane's certificate.
+* `namespace` The Kubernetes namespace of the workload.
+* `token` A Base64 encoded JWT for the Kf App's Service Account.
+
+Below is an example of what the JWT looks like, note that:
+
+* It expires and needs to be periodically refreshed from disk.
+* It's audience is only valid within the Kubernetes cluster.
+
+```json
+{
+    "aud": [
+        "<KUBERNETES_CLUSTER_URI>"
+    ],
+    "exp": 3600,
+    "iat": 0,
+    "iss": "<KUBERNETES_CLUSTER_URI>",
+    "kubernetes.io": {
+        "namespace": "<SPACE_NAME>",
+        "pod": {
+            "name": "<APP_NAME>-<RANDOM_SUFFIX>",
+            "uid": "<APP_GUID>"
+        },
+        "serviceaccount": {
+            "name": "sa-<APP_NAME>",
+            "uid": "<SERVICE_ACCOUNT_GUID>"
+        },
+        "warnafter": 3500
+    },
+    "nbf": 0,
+    "sub": "system:serviceaccount:<SPACE_NAME>:sa-<APP_NAME>"
+}
+```
+
+You can use this credential to connect to the Kubernetes control plane listed in the issuer (`iss`) field.
+
+## Customizing the service account
+
+{{< note >}}
+[This feature must be enabled]({{<relref "customizing-features#ksa-overrides">}}) by your platform operator before being used.
+{{< /note >}}
+
+You want to use a different service account than the default one Kf provides, for example to:
+
+* Allow blue/green apps to have the same identity.
+* Use Kf with a federated identity system.
+* Provide custom image pull credentials for a specific app.
+
+You can enable this by adding the `apps.kf.dev/service-account-name`
+[annotation to your app manifest]({{<relref "manifest#metadata-fields">}}).
+The value should be the name of the KSA you want the application and tasks to use.
+
+Example:
+
+```yaml
+applications:
+- name: my-app
+  metadata:
+    annotations:
+      "apps.kf.dev/service-account-name": "override-sa-name"
+```
+
+Limitations:
+
+* Only KSAs within the same Kubernetes namespace--corresponding to a Kf Space--are allowed.
+* The KSA must exist and be readable by Kf, otherwise the app will not deploy.
+* The KSA or the cluster must have permission to pull the application's container images.
+
+## Additional resources
+
+* If you use GKE, learn how to
+  [inject apps with a Google Service Account credential](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity).
+* Learn how to use [federated identity](https://cloud.google.com/iam/docs/workload-identity-federation) in GCP to allow authenticating
+  KSAs to GCP infrastructure.

docs/content/en/docs/v2.11/operator/customizing/customizing-features.md

@@ -252,6 +252,27 @@ kubectl patch \
     -p="[{'op':'add','path':'/spec/kf/config/appDisableStartCommandLookup','value':true}]"
 ```
 
+## Enable/Disable Kubernetes service Account (KSA) overrides{#ksa-overrides}
+
+Allows enabling/disbaling the ability to override the Kubernetes Service Account for Apps via annotation.
+
+{{< warning >}}
+Enabling this allows developers to run code as any KSA in the App's Namespace.
+This may change the security posture of your cluster.
+{{< /warning >}}
+
+Values for `appEnableServiceAccountOverride`:
+
+* `false` Don't allow overriding service account. (Default)
+* `true` Allow developers to override KSAs for their Apps.
+
+```sh
+kubectl patch \
+    kfsystem kfsystem \
+    --type='json' \
+    -p="[{'op':'add','path':'/spec/kf/config/appEnableServiceAccountOverride','value':true}]"
+```
+
 ## Set default Kf Task timeout
 
 Kf uses Tekton TaskRuns as its mechanism to run Kf Tasks. 

operator/pkg/apis/kfsystem/kf/defaults.go

@@ -24,31 +24,32 @@ import (
 
 const (
 	// DefaultsConfigName is the name of the defaults configmap.
-	DefaultsConfigName               = "config-defaults"
-	spaceContainerRegistryKey        = "spaceContainerRegistry"
-	spaceClusterDomainsKey           = "spaceClusterDomains"
-	spaceBuildpacksV2Key             = "spaceBuildpacksV2"
-	spaceStacksV2Key                 = "spaceStacksV2"
-	spaceStacksV3Key                 = "spaceStacksV3"
-	spaceDefaultToV3StackKey         = "spaceDefaultToV3Stack"
-	routeServiceProxyImageKey        = "routeServiceProxyImage"
-	featureFlagsKey                  = "featureFlags"
-	buildDisableIstioSidecarKey      = "buildDisableIstioSidecar"
-	buildPodResourcesKey             = "buildPodResources"
-	buildRetentionCountKey           = "buildRetentionCount"
-	taskRetentionCountKey            = "taskRetentionCount"
-	buildTimeoutKey                  = "buildTimeout"
-	buildNodeSelectorsKey            = "buildNodeSelectors"
-	appCPUPerGBOfRAMKey              = "appCPUPerGBOfRAM"
-	appCPUMinKey                     = "appCPUMin"
-	appDisableStartCommandLookupKey  = "appDisableStartCommandLookup"
-	progressDeadlineSecondsKey       = "progressDeadlineSeconds"
-	terminationGracePeriodSecondsKey = "terminationGracePeriodSeconds"
-	routeTrackVirtualServiceKey      = "routeTrackVirtualService"
-	routeDisableRetriesKey           = "routeDisableRetries"
-	routeHostIgnoringPortKey         = "routeHostIgnoringPort"
-	taskDefaultTimeoutMinutesKey     = "taskDefaultTimeoutMinutes"
-	taskDisableVolumeMountsKey       = "taskDisableVolumeMounts"
+	DefaultsConfigName                 = "config-defaults"
+	spaceContainerRegistryKey          = "spaceContainerRegistry"
+	spaceClusterDomainsKey             = "spaceClusterDomains"
+	spaceBuildpacksV2Key               = "spaceBuildpacksV2"
+	spaceStacksV2Key                   = "spaceStacksV2"
+	spaceStacksV3Key                   = "spaceStacksV3"
+	spaceDefaultToV3StackKey           = "spaceDefaultToV3Stack"
+	routeServiceProxyImageKey          = "routeServiceProxyImage"
+	featureFlagsKey                    = "featureFlags"
+	buildDisableIstioSidecarKey        = "buildDisableIstioSidecar"
+	buildPodResourcesKey               = "buildPodResources"
+	buildRetentionCountKey             = "buildRetentionCount"
+	taskRetentionCountKey              = "taskRetentionCount"
+	buildTimeoutKey                    = "buildTimeout"
+	buildNodeSelectorsKey              = "buildNodeSelectors"
+	appCPUPerGBOfRAMKey                = "appCPUPerGBOfRAM"
+	appCPUMinKey                       = "appCPUMin"
+	appDisableStartCommandLookupKey    = "appDisableStartCommandLookup"
+	appEnableServiceAccountOverrideKey = "appEnableServiceAccountOverride"
+	progressDeadlineSecondsKey         = "progressDeadlineSeconds"
+	terminationGracePeriodSecondsKey   = "terminationGracePeriodSeconds"
+	routeTrackVirtualServiceKey        = "routeTrackVirtualService"
+	routeDisableRetriesKey             = "routeDisableRetries"
+	routeHostIgnoringPortKey           = "routeHostIgnoringPort"
+	taskDefaultTimeoutMinutesKey       = "taskDefaultTimeoutMinutes"
+	taskDisableVolumeMountsKey         = "taskDisableVolumeMounts"
 
 	// Images used for build purposes
 
@@ -144,6 +145,10 @@ type DefaultsConfig struct {
 	// configuration for every App.
 	AppDisableStartCommandLookup bool `json:"appDisableStartCommandLookup,omitempty"`
 
+	// AppEnableServiceAccountOverride allows apps to override the service account via annotation:
+	// apps.kf.dev/service-account-name.
+	AppEnableServiceAccountOverride bool `json:"appEnableServiceAccountOverride,omitempty"`
+
 	// ProgressDeadlineSeconds contains the maximum time in seconds for a deployment to make progress before it
 	// is considered to be failed.
 	ProgressDeadlineSeconds *int32 `json:"progressDeadlineSeconds,omitempty"`
@@ -270,22 +275,23 @@ func (defaultsConfig *DefaultsConfig) getStringValues() map[string]*string {
 // getInterfaceValues returns a map of the key/value pairs on a DefaultsConfig that are JSON/YAML encoded values.
 func (defaultsConfig *DefaultsConfig) getInterfaceValues(leaveEmpty bool) map[string]interface{} {
 	m := map[string]interface{}{
-		spaceDefaultToV3StackKey:         &defaultsConfig.SpaceDefaultToV3Stack,
-		buildDisableIstioSidecarKey:      &defaultsConfig.BuildDisableIstioSidecar,
-		buildPodResourcesKey:             &defaultsConfig.BuildPodResources,
-		buildRetentionCountKey:           &defaultsConfig.BuildRetentionCount,
-		taskRetentionCountKey:            &defaultsConfig.TaskRetentionCount,
-		buildNodeSelectorsKey:            &defaultsConfig.BuildNodeSelectors,
-		appCPUPerGBOfRAMKey:              &defaultsConfig.AppCPUPerGBOfRAM,
-		appCPUMinKey:                     &defaultsConfig.AppCPUMin,
-		appDisableStartCommandLookupKey:  &defaultsConfig.AppDisableStartCommandLookup,
-		progressDeadlineSecondsKey:       &defaultsConfig.ProgressDeadlineSeconds,
-		terminationGracePeriodSecondsKey: &defaultsConfig.TerminationGracePeriodSeconds,
-		routeTrackVirtualServiceKey:      &defaultsConfig.RouteTrackVirtualService,
-		routeDisableRetriesKey:           &defaultsConfig.RouteDisableRetries,
-		routeHostIgnoringPortKey:         &defaultsConfig.RouteHostIgnoringPort,
-		taskDefaultTimeoutMinutesKey:     &defaultsConfig.TaskDefaultTimeoutMinutes,
-		taskDisableVolumeMountsKey:       &defaultsConfig.TaskDisableVolumeMounts,
+		spaceDefaultToV3StackKey:           &defaultsConfig.SpaceDefaultToV3Stack,
+		buildDisableIstioSidecarKey:        &defaultsConfig.BuildDisableIstioSidecar,
+		buildPodResourcesKey:               &defaultsConfig.BuildPodResources,
+		buildRetentionCountKey:             &defaultsConfig.BuildRetentionCount,
+		taskRetentionCountKey:              &defaultsConfig.TaskRetentionCount,
+		buildNodeSelectorsKey:              &defaultsConfig.BuildNodeSelectors,
+		appCPUPerGBOfRAMKey:                &defaultsConfig.AppCPUPerGBOfRAM,
+		appCPUMinKey:                       &defaultsConfig.AppCPUMin,
+		appDisableStartCommandLookupKey:    &defaultsConfig.AppDisableStartCommandLookup,
+		appEnableServiceAccountOverrideKey: &defaultsConfig.AppEnableServiceAccountOverride,
+		progressDeadlineSecondsKey:         &defaultsConfig.ProgressDeadlineSeconds,
+		terminationGracePeriodSecondsKey:   &defaultsConfig.TerminationGracePeriodSeconds,
+		routeTrackVirtualServiceKey:        &defaultsConfig.RouteTrackVirtualService,
+		routeDisableRetriesKey:             &defaultsConfig.RouteDisableRetries,
+		routeHostIgnoringPortKey:           &defaultsConfig.RouteHostIgnoringPort,
+		taskDefaultTimeoutMinutesKey:       &defaultsConfig.TaskDefaultTimeoutMinutes,
+		taskDisableVolumeMountsKey:         &defaultsConfig.TaskDisableVolumeMounts,
 	}
 
 	if !leaveEmpty {

pkg/apis/kf/config/defaults.go

@@ -24,31 +24,32 @@ import (
 
 const (
 	// DefaultsConfigName is the name of the defaults configmap.
-	DefaultsConfigName               = "config-defaults"
-	spaceContainerRegistryKey        = "spaceContainerRegistry"
-	spaceClusterDomainsKey           = "spaceClusterDomains"
-	spaceBuildpacksV2Key             = "spaceBuildpacksV2"
-	spaceStacksV2Key                 = "spaceStacksV2"
-	spaceStacksV3Key                 = "spaceStacksV3"
-	spaceDefaultToV3StackKey         = "spaceDefaultToV3Stack"
-	routeServiceProxyImageKey        = "routeServiceProxyImage"
-	featureFlagsKey                  = "featureFlags"
-	buildDisableIstioSidecarKey      = "buildDisableIstioSidecar"
-	buildPodResourcesKey             = "buildPodResources"
-	buildRetentionCountKey           = "buildRetentionCount"
-	taskRetentionCountKey            = "taskRetentionCount"
-	buildTimeoutKey                  = "buildTimeout"
-	buildNodeSelectorsKey            = "buildNodeSelectors"
-	appCPUPerGBOfRAMKey              = "appCPUPerGBOfRAM"
-	appCPUMinKey                     = "appCPUMin"
-	appDisableStartCommandLookupKey  = "appDisableStartCommandLookup"
-	progressDeadlineSecondsKey       = "progressDeadlineSeconds"
-	terminationGracePeriodSecondsKey = "terminationGracePeriodSeconds"
-	routeTrackVirtualServiceKey      = "routeTrackVirtualService"
-	routeDisableRetriesKey           = "routeDisableRetries"
-	routeHostIgnoringPortKey         = "routeHostIgnoringPort"
-	taskDefaultTimeoutMinutesKey     = "taskDefaultTimeoutMinutes"
-	taskDisableVolumeMountsKey       = "taskDisableVolumeMounts"
+	DefaultsConfigName                 = "config-defaults"
+	spaceContainerRegistryKey          = "spaceContainerRegistry"
+	spaceClusterDomainsKey             = "spaceClusterDomains"
+	spaceBuildpacksV2Key               = "spaceBuildpacksV2"
+	spaceStacksV2Key                   = "spaceStacksV2"
+	spaceStacksV3Key                   = "spaceStacksV3"
+	spaceDefaultToV3StackKey           = "spaceDefaultToV3Stack"
+	routeServiceProxyImageKey          = "routeServiceProxyImage"
+	featureFlagsKey                    = "featureFlags"
+	buildDisableIstioSidecarKey        = "buildDisableIstioSidecar"
+	buildPodResourcesKey               = "buildPodResources"
+	buildRetentionCountKey             = "buildRetentionCount"
+	taskRetentionCountKey              = "taskRetentionCount"
+	buildTimeoutKey                    = "buildTimeout"
+	buildNodeSelectorsKey              = "buildNodeSelectors"
+	appCPUPerGBOfRAMKey                = "appCPUPerGBOfRAM"
+	appCPUMinKey                       = "appCPUMin"
+	appDisableStartCommandLookupKey    = "appDisableStartCommandLookup"
+	appEnableServiceAccountOverrideKey = "appEnableServiceAccountOverride"
+	progressDeadlineSecondsKey         = "progressDeadlineSeconds"
+	terminationGracePeriodSecondsKey   = "terminationGracePeriodSeconds"
+	routeTrackVirtualServiceKey        = "routeTrackVirtualService"
+	routeDisableRetriesKey             = "routeDisableRetries"
+	routeHostIgnoringPortKey           = "routeHostIgnoringPort"
+	taskDefaultTimeoutMinutesKey       = "taskDefaultTimeoutMinutes"
+	taskDisableVolumeMountsKey         = "taskDisableVolumeMounts"
 
 	// Images used for build purposes
 
@@ -144,6 +145,10 @@ type DefaultsConfig struct {
 	// configuration for every App.
 	AppDisableStartCommandLookup bool `json:"appDisableStartCommandLookup,omitempty"`
 
+	// AppEnableServiceAccountOverride allows apps to override the service account via annotation:
+	// apps.kf.dev/service-account-name.
+	AppEnableServiceAccountOverride bool `json:"appEnableServiceAccountOverride,omitempty"`
+
 	// ProgressDeadlineSeconds contains the maximum time in seconds for a deployment to make progress before it
 	// is considered to be failed.
 	ProgressDeadlineSeconds *int32 `json:"progressDeadlineSeconds,omitempty"`
@@ -270,22 +275,23 @@ func (defaultsConfig *DefaultsConfig) getStringValues() map[string]*string {
 // getInterfaceValues returns a map of the key/value pairs on a DefaultsConfig that are JSON/YAML encoded values.
 func (defaultsConfig *DefaultsConfig) getInterfaceValues(leaveEmpty bool) map[string]interface{} {
 	m := map[string]interface{}{
-		spaceDefaultToV3StackKey:         &defaultsConfig.SpaceDefaultToV3Stack,
-		buildDisableIstioSidecarKey:      &defaultsConfig.BuildDisableIstioSidecar,
-		buildPodResourcesKey:             &defaultsConfig.BuildPodResources,
-		buildRetentionCountKey:           &defaultsConfig.BuildRetentionCount,
-		taskRetentionCountKey:            &defaultsConfig.TaskRetentionCount,
-		buildNodeSelectorsKey:            &defaultsConfig.BuildNodeSelectors,
-		appCPUPerGBOfRAMKey:              &defaultsConfig.AppCPUPerGBOfRAM,
-		appCPUMinKey:                     &defaultsConfig.AppCPUMin,
-		appDisableStartCommandLookupKey:  &defaultsConfig.AppDisableStartCommandLookup,
-		progressDeadlineSecondsKey:       &defaultsConfig.ProgressDeadlineSeconds,
-		terminationGracePeriodSecondsKey: &defaultsConfig.TerminationGracePeriodSeconds,
-		routeTrackVirtualServiceKey:      &defaultsConfig.RouteTrackVirtualService,
-		routeDisableRetriesKey:           &defaultsConfig.RouteDisableRetries,
-		routeHostIgnoringPortKey:         &defaultsConfig.RouteHostIgnoringPort,
-		taskDefaultTimeoutMinutesKey:     &defaultsConfig.TaskDefaultTimeoutMinutes,
-		taskDisableVolumeMountsKey:       &defaultsConfig.TaskDisableVolumeMounts,
+		spaceDefaultToV3StackKey:           &defaultsConfig.SpaceDefaultToV3Stack,
+		buildDisableIstioSidecarKey:        &defaultsConfig.BuildDisableIstioSidecar,
+		buildPodResourcesKey:               &defaultsConfig.BuildPodResources,
+		buildRetentionCountKey:             &defaultsConfig.BuildRetentionCount,
+		taskRetentionCountKey:              &defaultsConfig.TaskRetentionCount,
+		buildNodeSelectorsKey:              &defaultsConfig.BuildNodeSelectors,
+		appCPUPerGBOfRAMKey:                &defaultsConfig.AppCPUPerGBOfRAM,
+		appCPUMinKey:                       &defaultsConfig.AppCPUMin,
+		appDisableStartCommandLookupKey:    &defaultsConfig.AppDisableStartCommandLookup,
+		appEnableServiceAccountOverrideKey: &defaultsConfig.AppEnableServiceAccountOverride,
+		progressDeadlineSecondsKey:         &defaultsConfig.ProgressDeadlineSeconds,
+		terminationGracePeriodSecondsKey:   &defaultsConfig.TerminationGracePeriodSeconds,
+		routeTrackVirtualServiceKey:        &defaultsConfig.RouteTrackVirtualService,
+		routeDisableRetriesKey:             &defaultsConfig.RouteDisableRetries,
+		routeHostIgnoringPortKey:           &defaultsConfig.RouteHostIgnoringPort,
+		taskDefaultTimeoutMinutesKey:       &defaultsConfig.TaskDefaultTimeoutMinutes,
+		taskDisableVolumeMountsKey:         &defaultsConfig.TaskDisableVolumeMounts,
 	}
 
 	if !leaveEmpty {

pkg/apis/kf/config/defaults_test.go

@@ -49,6 +49,7 @@ func TestPatchConfigMap(t *testing.T) {
 		appCPUMinKey,
 		appCPUPerGBOfRAMKey,
 		appDisableStartCommandLookupKey,
+		appEnableServiceAccountOverrideKey,
 		progressDeadlineSecondsKey,
 		terminationGracePeriodSecondsKey,
 		routeTrackVirtualServiceKey,

pkg/apis/kf/config/store_test.go

@@ -55,6 +55,7 @@ func TestStoreLoadWithContext(t *testing.T) {
 		appCPUMinKey,
 		appCPUPerGBOfRAMKey,
 		appDisableStartCommandLookupKey,
+		appEnableServiceAccountOverrideKey,
 		progressDeadlineSecondsKey,
 		terminationGracePeriodSecondsKey,
 		routeTrackVirtualServiceKey,