Merged in MP14-access-local (pull request #10)

MP14 access local

* add instruction to setup ingresses for accessing service (local setup)

* add access management stuff for local setup

* add extra info on how to create kubeflow users

resources/local-setup/access_management/create_user/README.md

@@ -0,0 +1,15 @@
+# Helper script to create new users
+
+This is a helper script for adding new users to the cluster and giving them access.
+
+It was created following the step indicated in the
+[6_Access_management.md](/tutorials/local_deployment/06_Access_Management.md) tutorial.
+
+## Usage
+
+```bash
+# e.g. ./create_user.sh john users
+./create_user.sh <user_name> <group_name>
+```
+
+It will create a kubeconfig file named `<username>-kubeconfig`. This file can be sent to the user, so that he can use it to access the cluster.

resources/local-setup/access_management/create_user/create_user.sh

@@ -0,0 +1,57 @@
+#!/bin/bash
+
+# usage: $./create_user.sh <username> <group_name>
+# e.g. $./create_user.sh john users
+
+set -eoau pipefail
+
+USER_NAME=$1
+GROUP_NAME=$2
+
+echo "Username: $USER_NAME"
+echo "Group name: $GROUP_NAME"
+
+OUTDIR="$USER_NAME"
+mkdir "$OUTDIR"
+
+# create CSR config for the user
+cat csr.cnf.template | envsubst > "$OUTDIR"/"$USER_NAME"-csr.cnf
+
+# create user key
+openssl genrsa -out "$OUTDIR"/"$USER_NAME".key 4096
+
+# create CSR
+openssl req -config "$OUTDIR"/"$USER_NAME"-csr.cnf -new -key "$OUTDIR"/"$USER_NAME".key -nodes -out "$OUTDIR"/"$USER_NAME".csr
+
+# encode the .csr file in base64
+BASE64_CSR=$(cat "$OUTDIR"/"$USER_NAME".csr | base64 | tr -d '\n')
+
+# create the CertificateSigninRequest
+cat csr.yaml.template | envsubst > "$OUTDIR"/csr.yaml
+
+kubectl apply -f "$OUTDIR"/csr.yaml
+
+# approve the certificate
+sleep 2
+kubectl certificate approve "$USER_NAME"-csr
+
+### generate kubeconfig ###
+
+# Cluster Name (from the current context)
+export CLUSTER_NAME=$(kubectl config view --minify -o jsonpath={.current-context})
+# Client certificate
+export CLIENT_CERTIFICATE_DATA=$(kubectl get csr "$USER_NAME"-csr -o jsonpath='{.status.certificate}')
+# Cluster Certificate Authority
+export CLUSTER_CA=$(kubectl config view --raw -o json | jq -r '.clusters[] | select(.name == "'$(kubectl config current-context)'") | .cluster."certificate-authority-data"')
+# API Server endpoint
+export CLUSTER_ENDPOINT=$(kubectl config view --raw -o json | jq -r '.clusters[] | select(.name == "'$(kubectl config current-context)'") | .cluster."server"')
+
+echo "user: $USER_NAME, cluster name: $CLUSTER_NAME, client certificate (length): ${#CLIENT_CERTIFICATE_DATA}, Cluster Certificate Authority (length): ${#CLUSTER_CA}, API Server endpoint: $CLUSTER_ENDPOINT"
+
+cat kubeconfig.template | envsubst > "$OUTDIR"/"$USER_NAME"-kubeconfig
+
+# add user private key
+kubectl config set-credentials $USER_NAME --client-key="$PWD"/"$OUTDIR"/"$USER_NAME".key --embed-certs=true --kubeconfig="$PWD"/"$OUTDIR"/"$USER_NAME"-kubeconfig
+
+# confirm user access
+kubectl version --kubeconfig="$PWD"/"$OUTDIR"/"$USER_NAME"-kubeconfig

resources/local-setup/access_management/create_user/csr.cnf.template

@@ -0,0 +1,15 @@
+[ req ]
+default_bits = 2048
+prompt = no
+default_md = sha256
+distinguished_name = dn
+
+[ dn ]
+CN = ${USER_NAME}
+O = ${GROUP_NAME}
+
+[ v3_ext ]
+authorityKeyIdentifier=keyid,issuer:always
+basicConstraints=CA:FALSE
+keyUsage=keyEncipherment,dataEncipherment
+extendedKeyUsage=clientAuth

resources/local-setup/access_management/create_user/csr.yaml.template

@@ -0,0 +1,14 @@
+apiVersion: certificates.k8s.io/v1
+kind: CertificateSigningRequest
+metadata:
+  name: ${USER_NAME}-csr
+spec:
+  signerName: kubernetes.io/kube-apiserver-client
+
+  groups:
+  - system:authenticated
+  request: ${BASE64_CSR}
+  usages:
+  - client auth
+  - digital signature
+  - key encipherment

resources/local-setup/access_management/create_user/kubeconfig.template

@@ -0,0 +1,17 @@
+apiVersion: v1
+kind: Config
+clusters:
+- cluster:
+    certificate-authority-data: ${CLUSTER_CA}
+    server: ${CLUSTER_ENDPOINT}
+  name: ${CLUSTER_NAME}
+users:
+- name: ${USER_NAME}
+  user:
+    client-certificate-data: ${CLIENT_CERTIFICATE_DATA}
+contexts:
+- context:
+    cluster: ${CLUSTER_NAME}
+    user: ${USER_NAME}
+  name: ${USER_NAME}-${CLUSTER_NAME}
+current-context: ${USER_NAME}-${CLUSTER_NAME}

tutorials/gcp_deployment/03_Deploy_Kubeflow.md

@@ -154,3 +154,7 @@ from kfp.gcp import use_gcp_secret
 pull_data_step = ... # Define Kubeflow Pipeline step
 pull_data_step.apply(use_gcp_secret(secret_name="user-gcp-sa"))
 ```
+
+## Creating User Profiles
+
+To see how to create user profiles, please refer to official documentation: [link](https://www.kubeflow.org/docs/components/multi-tenancy/getting-started/).

tutorials/local_deployment/05_Setup_Ingress.md

@@ -0,0 +1,181 @@
+# Setup ingress
+
+In order to access Kubeflow, MLflow, etc. through the network without having to
+use `port-forward`, we need to set up NGINX ingress controller and create ingresses for
+the different services.
+
+In this tutorial we will see how to get the ingresses working. We will use NGINX to
+route to these services.
+
+## 1. Deploy Ingress controller (NGINX)
+
+NGINX is a free, open-source, high-performance HTTP server and reverse proxy.
+
+```bash
+# /deployment
+mkdir nginx
+wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml -O nginx/nginx-kind-deployment.yaml
+kubectl apply -f nginx/nginx-kind-deployment.yaml
+```
+
+The manifests contain kind specific patches to forward the hostPorts to the ingress
+controller, set taint tolerations and schedule it to the custom labelled node.
+
+## 2. Create ingresses
+
+We will create ingresses for the following services:
+
+- Kubeflow Dashboard
+- MLflow server
+- MinIO (MLflow)
+- Prometheus
+- Grafana
+
+
+```bash
+# mlflow-ingress.yaml
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: mlflow-ingress
+  namespace: mlflow
+  annotations:
+    kubernetes.io/ingress.class: nginx
+    nginx.ingress.kubernetes.il/add-base-url: "true"
+spec:
+  rules:
+  - host: mlflow-server.local
+    http:
+      paths:
+        - backend:
+            service:
+              name: mlflow
+              port:
+                number: 5000
+          path: /
+          pathType: Prefix
+```
+
+```bash
+# kubeflow-ingress.yaml
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: kubeflow-ingress
+  namespace: istio-system
+  annotations:
+    kubernetes.io/ingress.class: nginx
+    nginx.ingress.kubernetes.il/add-base-url: "true"
+spec:
+  rules:
+  - host: kubeflow.local
+    http:
+      paths:
+        - backend:
+            service:
+              name: istio-ingressgateway
+              port:
+                number: 80
+          path: /
+          pathType: Prefix
+```
+
+```bash
+# minio-ingress.yaml
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: mlflow-minio-ingress
+  annotations:
+    kubernetes.io/ingress.class: nginx
+    nginx.ingress.kubernetes.il/add-base-url: "true"
+    nginx.ingress.kubernetes.io/ssl-redirect: "false"
+spec:
+  rules:
+  - host: mlflow-minio.local
+    http:
+      paths:
+        - backend:
+            service:
+              name: mlflow-minio-service
+              port:
+                number: 9001
+          path: /
+          pathType: Prefix
+```
+
+```bash
+# prometheus-ingress.yaml
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: prometheus-ingress
+  namespace: monitoring
+  annotations:
+    kubernetes.io/ingress.class: nginx
+    nginx.ingress.kubernetes.il/add-base-url: "true"
+spec:
+    rules:
+    - host: prometheus.local
+        http:
+        paths:
+            - backend:
+                service:
+                name: prometheus-service
+                port:
+                    number: 8080
+            path: /
+            pathType: Prefix
+```
+
+```bash
+# grafana-ingress.yaml
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: grafana-ingress
+  namespace: monitoring
+  annotations:
+    kubernetes.io/ingress.class: nginx
+    nginx.ingress.kubernetes.il/add-base-url: "true"
+spec:
+    rules:
+    - host: grafana.local
+        http:
+        paths:
+            - backend:
+                service:
+                name: grafana
+                port:
+                    number: 3000
+            path: /
+            pathType: Prefix
+```
+
+
+## 3. Update hosts
+
+Map the cluster IP to the host names in your `/etc/hosts` file.
+
+Open your `/etc/hosts` file. E.g.
+```bash
+sudo nano /etc/hosts
+```
+
+Append the following line with the IP of the cluster and save the changes.
+
+```
+0.0.0.0 mlflow-server.local mlflow-minio.local kubeflow.local prometheus.local grafana.local
+```
+
+> To access the ingresses from another computer in the local network, replace `0.0.0.0`
+> with the real IP of the computer running the cluster. 
+
+Now the ingress is all setup. You should be able to access these services by simple
+navigating on your browser to their addresses:
+
+- KFP UI: [http://ml-pipeline-ui.local](http://ml-pipeline-ui.local)
+- MLflow UI: [http://mlflow-server.local](http://mlflow-server.local)
+- MinIO (MLflow) UI: [http://mlflow-minio.local](http://mlflow-minio.local)
+- Prometheus UI: [http://prometheus.local](http://prometheus.local)
+- Grafana UI: [http://grafana.local](http://grafana.local)