feat(authz): Authorino for Service Mesh (opendatahub-io#784)

* feat(authz): Authorino for Service Mesh This first iteration is to cover authentication needs for KServe * Add templates to install Authorino * Add templates to configure Service Mesh to use Authorino to delegate Authorization * Add KServe-specific templates add ability to secure KServe Inference Services * Add relevant fields to DSCInitialization resource * Code for proper cleanup, in case of uninstalling Most (if not all) of this code comes from pull request opendatahub-io#605. Attribution to original authors: @bartoszmajsak, @aslakknutsen, @cam-garrison, et. al. Related opendatahub-io/kserve#128 Signed-off-by: Edgar Hernández <[email protected]> * Fix linter issues Signed-off-by: Edgar Hernández <[email protected]> * Resolve feedback: Bartosz Signed-off-by: Edgar Hernández <[email protected]> * fix: Remove port from the authorization policy Also, add `/metrics` to the ignored paths for auth. Signed-off-by: Edgar Hernández <[email protected]> * Fix feedback: Bartosz Signed-off-by: Edgar Hernández <[email protected]> * More feedback: Bartosz Co-authored-by: Bartosz Majsak <[email protected]> * Fix feedback: Reto - Adjust AuthorizationPolicy Signed-off-by: Edgar Hernández <[email protected]> * Fix more feedback: Bartosz - Remove Authorino namespace field from DSCI. - Move around some code in kserve.go to servicemesh_setup.go Signed-off-by: Edgar Hernández <[email protected]> * chore: adds sec. prefix to authorino label selector * fix: adds base dir to manifest sources * chore: uses security instead of sec as a prefix in authorino label * fix: /healthz is called by _something_, skipp * fix: adopt ODH-ADR-0006 for clean up label * fix: uses correct CRD name for authconfigs Co-authored-by: Cameron Garrison <[email protected]> * Remove left-over file Signed-off-by: Edgar Hernández <[email protected]> * Feedback: remove auth-refs ConfigMap Signed-off-by: Edgar Hernández <[email protected]> * Add missing role.yaml changes Signed-off-by: Edgar Hernández <[email protected]> * Go back to installing Authorino on its own namespace Signed-off-by: Edgar Hernández <[email protected]> * Feedback: Add clean-up for KServe/OSSM-auth Signed-off-by: Edgar Hernández <[email protected]> * Feedback: Simplify namings Signed-off-by: Edgar Hernández <[email protected]> * fix: add auth-refs cm * Feedback: adjust labels and a log message Signed-off-by: Edgar Hernández <[email protected]> * Bugfix: Extension provider terminating with error when SMCP is gone Signed-off-by: Edgar Hernández <[email protected]> * Fix: add missing RBAC for ConfigMaps func Signed-off-by: Edgar Hernández <[email protected]> * Fix: Run `make bundle` and commit resulting changes Signed-off-by: Edgar Hernández <[email protected]> * Feedback: Wen - Better feature namings Signed-off-by: Edgar Hernández <[email protected]> * Feedback: Bartosz * Use feature logger * Don't trim -applications suffix on ResolveAuthNamespace Signed-off-by: Edgar Hernández <[email protected]> * Feedback: Wen - revert image placeholder was replaced Signed-off-by: Edgar Hernández <[email protected]> --------- Signed-off-by: Edgar Hernández <[email protected]> Co-authored-by: Bartosz Majsak <[email protected]> Co-authored-by: Aslak Knutsen <[email protected]> Co-authored-by: Cameron Garrison <[email protected]>
red-hat-data-services · Feb 19, 2024 · e32a7c2 · e32a7c2
1 parent e7e3982
commit e32a7c2
Show file tree

Hide file tree

Showing 29 changed files with 655 additions and 48 deletions.
diff --git a/apis/dscinitialization/v1/zz_generated.deepcopy.go b/apis/dscinitialization/v1/zz_generated.deepcopy.go
diff --git a/bundle/manifests/dscinitialization.opendatahub.io_dscinitializations.yaml b/bundle/manifests/dscinitialization.opendatahub.io_dscinitializations.yaml
@@ -88,6 +88,28 @@ spec:
                   user experience; e.g. it provides unified authentication giving
                   a Single Sign On experience.
                 properties:
+                  auth:
+                    description: Auth holds configuration of authentication and authorization
+                      services used by Service Mesh in Opendatahub.
+                    properties:
+                      audiences:
+                        default:
+                        - https://kubernetes.default.svc
+                        description: Audiences is a list of the identifiers that the
+                          resource server presented with the token identifies as.
+                          Audience-aware token authenticators will verify that the
+                          token was intended for at least one of the audiences in
+                          this list. If no audiences are provided, the audience will
+                          default to the audience of the Kubernetes apiserver (kubernetes.default.svc).
+                        items:
+                          type: string
+                        type: array
+                      namespace:
+                        description: Namespace where it is deployed. If not provided,
+                          the default is to use '-auth-provider' suffix on the ApplicationsNamespace
+                          of the DSCI.
+                        type: string
+                    type: object
                   controlPlane:
                     description: ControlPlane holds configuration of Service Mesh
                       used by Opendatahub.

diff --git a/bundle/manifests/opendatahub-operator.clusterserviceversion.yaml b/bundle/manifests/opendatahub-operator.clusterserviceversion.yaml
@@ -328,6 +328,12 @@ spec:
           - tokenreviews
           verbs:
           - create
+        - apiGroups:
+          - authorino.kuadrant.io
+          resources:
+          - authconfigs
+          verbs:
+          - '*'
         - apiGroups:
           - authorization.k8s.io
           resources:
@@ -548,6 +554,7 @@ spec:
           - get
           - list
           - patch
+          - update
           - watch
         - apiGroups:
           - ""
@@ -1180,6 +1187,12 @@ spec:
           - deletecollection
           - get
           - patch
+        - apiGroups:
+          - networking.istio.io
+          resources:
+          - envoyfilters
+          verbs:
+          - '*'
         - apiGroups:
           - networking.istio.io
           resources:
@@ -1260,6 +1273,12 @@ spec:
           - patch
           - update
           - watch
+        - apiGroups:
+          - operator.authorino.kuadrant.io
+          resources:
+          - authorinos
+          verbs:
+          - '*'
         - apiGroups:
           - operator.knative.dev
           resources:
@@ -1401,6 +1420,12 @@ spec:
           - patch
           - update
           - watch
+        - apiGroups:
+          - security.istio.io
+          resources:
+          - authorizationpolicies
+          verbs:
+          - '*'
         - apiGroups:
           - security.openshift.io
           resources:

diff --git a/components/kserve/kserve.go b/components/kserve/kserve.go
@@ -163,11 +163,16 @@ func (k *Kserve) ReconcileComponent(ctx context.Context, cli client.Client, resC
 			return err
 		}
 	}
-	return nil
+
+	return k.configureServiceMesh(dscispec)
 }
 
 func (k *Kserve) Cleanup(_ client.Client, instance *dsciv1.DSCInitializationSpec) error {
-	return k.removeServerlessFeatures(instance)
+	if removeServerlessErr := k.removeServerlessFeatures(instance); removeServerlessErr != nil {
+		return removeServerlessErr
+	}
+
+	return k.removeServiceMeshConfigurations(instance)
 }
 
 func (k *Kserve) configureServerless(instance *dsciv1.DSCInitializationSpec) error {

diff --git a/components/kserve/servicemesh_setup.go b/components/kserve/servicemesh_setup.go
@@ -0,0 +1,46 @@
+package kserve
+
+import (
+	"path"
+
+	operatorv1 "github.com/openshift/api/operator/v1"
+
+	dsciv1 "github.com/opendatahub-io/opendatahub-operator/v2/apis/dscinitialization/v1"
+	"github.com/opendatahub-io/opendatahub-operator/v2/pkg/feature"
+	"github.com/opendatahub-io/opendatahub-operator/v2/pkg/feature/servicemesh"
+)
+
+func (k *Kserve) configureServiceMesh(dscispec *dsciv1.DSCInitializationSpec) error {
+	if dscispec.ServiceMesh.ManagementState == operatorv1.Managed && k.GetManagementState() == operatorv1.Managed {
+		serviceMeshInitializer := feature.ComponentFeaturesHandler(k, dscispec, k.defineServiceMeshFeatures())
+		return serviceMeshInitializer.Apply()
+	}
+	if dscispec.ServiceMesh.ManagementState == operatorv1.Unmanaged && k.GetManagementState() == operatorv1.Managed {
+		return nil
+	}
+
+	return k.removeServiceMeshConfigurations(dscispec)
+}
+
+func (k *Kserve) removeServiceMeshConfigurations(dscispec *dsciv1.DSCInitializationSpec) error {
+	serviceMeshInitializer := feature.ComponentFeaturesHandler(k, dscispec, k.defineServiceMeshFeatures())
+	return serviceMeshInitializer.Delete()
+}
+
+func (k *Kserve) defineServiceMeshFeatures() feature.FeaturesProvider {
+	return func(handler *feature.FeaturesHandler) error {
+		kserveExtAuthzErr := feature.CreateFeature("kserve-external-authz").
+			For(handler).
+			Manifests(
+				path.Join(feature.KServeDir),
+			).
+			WithData(servicemesh.ClusterDetails).
+			Load()
+
+		if kserveExtAuthzErr != nil {
+			return kserveExtAuthzErr
+		}
+
+		return nil
+	}
+}
diff --git a/config/crd/bases/dscinitialization.opendatahub.io_dscinitializations.yaml b/config/crd/bases/dscinitialization.opendatahub.io_dscinitializations.yaml
@@ -89,6 +89,28 @@ spec:
                   user experience; e.g. it provides unified authentication giving
                   a Single Sign On experience.
                 properties:
+                  auth:
+                    description: Auth holds configuration of authentication and authorization
+                      services used by Service Mesh in Opendatahub.
+                    properties:
+                      audiences:
+                        default:
+                        - https://kubernetes.default.svc
+                        description: Audiences is a list of the identifiers that the
+                          resource server presented with the token identifies as.
+                          Audience-aware token authenticators will verify that the
+                          token was intended for at least one of the audiences in
+                          this list. If no audiences are provided, the audience will
+                          default to the audience of the Kubernetes apiserver (kubernetes.default.svc).
+                        items:
+                          type: string
+                        type: array
+                      namespace:
+                        description: Namespace where it is deployed. If not provided,
+                          the default is to use '-auth-provider' suffix on the ApplicationsNamespace
+                          of the DSCI.
+                        type: string
+                    type: object
                   controlPlane:
                     description: ControlPlane holds configuration of Service Mesh
                       used by Opendatahub.

diff --git a/config/rbac/role.yaml b/config/rbac/role.yaml
@@ -132,6 +132,12 @@ rules:
   - tokenreviews
   verbs:
   - create
+- apiGroups:
+  - authorino.kuadrant.io
+  resources:
+  - authconfigs
+  verbs:
+  - '*'
 - apiGroups:
   - authorization.k8s.io
   resources:
@@ -352,6 +358,7 @@ rules:
   - get
   - list
   - patch
+  - update
   - watch
 - apiGroups:
   - ""
@@ -984,6 +991,12 @@ rules:
   - deletecollection
   - get
   - patch
+- apiGroups:
+  - networking.istio.io
+  resources:
+  - envoyfilters
+  verbs:
+  - '*'
 - apiGroups:
   - networking.istio.io
   resources:
@@ -1064,6 +1077,12 @@ rules:
   - patch
   - update
   - watch
+- apiGroups:
+  - operator.authorino.kuadrant.io
+  resources:
+  - authorinos
+  verbs:
+  - '*'
 - apiGroups:
   - operator.knative.dev
   resources:
@@ -1205,6 +1224,12 @@ rules:
   - patch
   - update
   - watch
+- apiGroups:
+  - security.istio.io
+  resources:
+  - authorizationpolicies
+  verbs:
+  - '*'
 - apiGroups:
   - security.openshift.io
   resources:

diff --git a/controllers/datasciencecluster/kubebuilder_rbac.go b/controllers/datasciencecluster/kubebuilder_rbac.go
@@ -4,14 +4,25 @@ package datasciencecluster
 //+kubebuilder:rbac:groups="datasciencecluster.opendatahub.io",resources=datascienceclusters/finalizers,verbs=update;patch
 //+kubebuilder:rbac:groups="datasciencecluster.opendatahub.io",resources=datascienceclusters,verbs=get;list;watch;create;update;patch;delete
 
-/* Service Mesh prerequisite */
-// +kubebuilder:rbac:groups="maistra.io",resources=servicemeshcontrolplanes,verbs=create;get;list;patch;update;use;watch
-
 /* Serverless prerequisite */
 // +kubebuilder:rbac:groups="networking.istio.io",resources=gateways,verbs=*
 // +kubebuilder:rbac:groups="operator.knative.dev",resources=knativeservings,verbs=*
 // +kubebuilder:rbac:groups="config.openshift.io",resources=ingresses,verbs=get
 
+/* Service Mesh Integration */
+// +kubebuilder:rbac:groups="maistra.io",resources=servicemeshcontrolplanes,verbs=create;get;list;patch;update;use;watch
+// +kubebuilder:rbac:groups="maistra.io",resources=servicemeshmemberrolls,verbs=create;get;list;patch;update;use;watch
+// +kubebuilder:rbac:groups="maistra.io",resources=servicemeshmembers,verbs=create;get;list;patch;update;use;watch
+// +kubebuilder:rbac:groups="maistra.io",resources=servicemeshmembers/finalizers,verbs=create;get;list;patch;update;use;watch
+// +kubebuilder:rbac:groups="networking.istio.io",resources=virtualservices/status,verbs=update;patch;delete
+// +kubebuilder:rbac:groups="networking.istio.io",resources=virtualservices/finalizers,verbs=get;list;watch;create;update;patch;delete
+// +kubebuilder:rbac:groups="networking.istio.io",resources=virtualservices,verbs=*
+// +kubebuilder:rbac:groups="networking.istio.io",resources=gateways,verbs=*
+// +kubebuilder:rbac:groups="networking.istio.io",resources=envoyfilters,verbs=*
+// +kubebuilder:rbac:groups="security.istio.io",resources=authorizationpolicies,verbs=*
+// +kubebuilder:rbac:groups="authorino.kuadrant.io",resources=authconfigs,verbs=*
+// +kubebuilder:rbac:groups="operator.authorino.kuadrant.io",resources=authorinos,verbs=*
+
 /* This is for DSP */
 //+kubebuilder:rbac:groups="datasciencepipelinesapplications.opendatahub.io",resources=datasciencepipelinesapplications/status,verbs=update;patch;get
 //+kubebuilder:rbac:groups="datasciencepipelinesapplications.opendatahub.io",resources=datasciencepipelinesapplications/finalizers,verbs=update;patch
@@ -93,10 +104,6 @@ package datasciencecluster
 // +kubebuilder:rbac:groups="networking.k8s.io",resources=networkpolicies,verbs=get;create;list;watch;delete;update;patch
 // +kubebuilder:rbac:groups="networking.k8s.io",resources=ingresses,verbs=create;delete;list;update;watch;patch;get
 
-// +kubebuilder:rbac:groups="networking.istio.io",resources=virtualservices/status,verbs=update;patch;delete
-// +kubebuilder:rbac:groups="networking.istio.io",resources=virtualservices/finalizers,verbs=get;list;watch;create;update;patch;delete
-// +kubebuilder:rbac:groups="networking.istio.io",resources=virtualservices,verbs=*
-
 // +kubebuilder:rbac:groups="monitoring.coreos.com",resources=servicemonitors,verbs=get;create;delete;update;watch;list;patch;deletecollection
 // +kubebuilder:rbac:groups="monitoring.coreos.com",resources=podmonitors,verbs=get;create;delete;update;watch;list;patch
 // +kubebuilder:rbac:groups="monitoring.coreos.com",resources=prometheusrules,verbs=get;create;patch;delete;deletecollection
@@ -189,7 +196,7 @@ package datasciencecluster
 // +kubebuilder:rbac:groups="core",resources=endpoints,verbs=watch;list;get;create;update;delete
 
 // +kubebuilder:rbac:groups="core",resources=configmaps/status,verbs=get;update;patch;delete
-// +kubebuilder:rbac:groups="core",resources=configmaps,verbs=get;create;watch;patch;delete;list
+// +kubebuilder:rbac:groups="core",resources=configmaps,verbs=get;create;update;watch;patch;delete;list
 
 // +kubebuilder:rbac:groups="core",resources=clusterversions,verbs=watch;list
 // +kubebuilder:rbac:groups="config.openshift.io",resources=clusterversions,verbs=watch;list
@@ -248,11 +255,6 @@ package datasciencecluster
 
 // +kubebuilder:rbac:groups="*",resources=customresourcedefinitions,verbs=get;list;watch
 
-// +kubebuilder:rbac:groups="maistra.io",resources=servicemeshcontrolplanes,verbs=create;get;list;patch;update;use;watch
-// +kubebuilder:rbac:groups="maistra.io",resources=servicemeshmemberrolls,verbs=create;get;list;patch;update;use;watch
-// +kubebuilder:rbac:groups="maistra.io",resources=servicemeshmembers,verbs=create;get;list;patch;update;use;watch
-// +kubebuilder:rbac:groups="maistra.io",resources=servicemeshmembers/finalizers,verbs=create;get;list;patch;update;use;watch
-
 /* Only for RHODS */
 // +kubebuilder:rbac:groups="user.openshift.io",resources=groups,verbs=get;create;list;watch;patch;delete
 // +kubebuilder:rbac:groups="console.openshift.io",resources=consolelinks,verbs=create;get;patch;delete
diff --git a/controllers/dscinitialization/servicemesh_setup.go b/controllers/dscinitialization/servicemesh_setup.go
@@ -86,6 +86,47 @@ func configureServiceMeshFeatures() feature.FeaturesProvider {
 			}
 		}
 
+		cfgMapErr := feature.CreateFeature("mesh-shared-configmap").
+			For(handler).
+			WithResources(servicemesh.ConfigMaps).
+			Load()
+		if cfgMapErr != nil {
+			return cfgMapErr
+		}
+
+		extAuthzErr := feature.CreateFeature("mesh-control-plane-external-authz").
+			For(handler).
+			Manifests(
+				path.Join(feature.AuthDir, "auth-smm.tmpl"),
+				path.Join(feature.AuthDir, "base"),
+				path.Join(feature.AuthDir, "mesh-authz-ext-provider.patch.tmpl"),
+			).
+			WithData(servicemesh.ClusterDetails).
+			PreConditions(
+				feature.EnsureCRDIsInstalled("authconfigs.authorino.kuadrant.io"),
+				servicemesh.EnsureServiceMeshInstalled,
+				servicemesh.EnsureAuthNamespaceExists,
+			).
+			PostConditions(
+				feature.WaitForPodsToBeReady(serviceMeshSpec.ControlPlane.Namespace),
+				func(f *feature.Feature) error {
+					return feature.WaitForPodsToBeReady(f.Spec.Auth.Namespace)(f)
+				},
+				func(f *feature.Feature) error {
+					// We do not have the control over deployment resource creation.
+					// It is created by Authorino operator using Authorino CR
+					//
+					// To make it part of Service Mesh we have to patch it with injection
+					// enabled instead, otherwise it will not have proxy pod injected.
+					return f.ApplyManifest(path.Join(feature.AuthDir, "deployment.injection.patch.tmpl"))
+				},
+			).
+			OnDelete(servicemesh.RemoveExtensionProvider).
+			Load()
+		if extAuthzErr != nil {
+			return extAuthzErr
+		}
+
 		return nil
 	}
 }
diff --git a/infrastructure/v1/servicemesh_types.go b/infrastructure/v1/servicemesh_types.go
@@ -9,6 +9,9 @@ type ServiceMeshSpec struct {
 	ManagementState operatorv1.ManagementState `json:"managementState,omitempty"`
 	// ControlPlane holds configuration of Service Mesh used by Opendatahub.
 	ControlPlane ControlPlaneSpec `json:"controlPlane,omitempty"`
+	// Auth holds configuration of authentication and authorization services
+	// used by Service Mesh in Opendatahub.
+	Auth AuthSpec `json:"auth,omitempty"`
 }
 
 type ControlPlaneSpec struct {
@@ -38,3 +41,16 @@ type IngressGatewaySpec struct {
 	// the for Ingress Gateway.
 	Certificate CertificateSpec `json:"certificate,omitempty"`
 }
+
+type AuthSpec struct {
+	// Namespace where it is deployed. If not provided, the default is to
+	// use '-auth-provider' suffix on the ApplicationsNamespace of the DSCI.
+	Namespace string `json:"namespace,omitempty"`
+	// Audiences is a list of the identifiers that the resource server presented
+	// with the token identifies as. Audience-aware token authenticators will verify
+	// that the token was intended for at least one of the audiences in this list.
+	// If no audiences are provided, the audience will default to the audience of the
+	// Kubernetes apiserver (kubernetes.default.svc).
+	// +kubebuilder:default={"https://kubernetes.default.svc"}
+	Audiences *[]string `json:"audiences,omitempty"`
+}