unify the local k8s cluster setup tool with kind (#116)

* unify local k8s cluster setup with kind; MacOS can use prepare_env script

* fix

* fix

* update

* remove darwin code

* update README and docs

* update

* update

Author: acezen

README-zh.md

@@ -19,17 +19,19 @@ GraphScope 整合了达摩院的多项学术研究成果，其中的核心技术
 
 ## 快速开始
 
-GraphScope 设计在 [Kubernetes (k8s)](https://kubernetes.io/) 管理的群集上运行。为了快速上手，我们可以利用 [minikube](https://minikube.sigs.k8s.io/) 来创建一个本地的 Kubernetes 集群，并根据如下步骤利用我们预先构建的 GraphScope 镜像。
+GraphScope 设计在 [Kubernetes (k8s)](https://kubernetes.io/) 管理的群集上运行。为了快速上手，我们可以按照本文档的以下步骤部署一个本地 Kubernetes 集群，并加载预编译好的镜像。。
 
-### 环境准备 
+### 环境准备
 
-为了在本地环境下使用 minikube 跑通我们的示例，以下软件需要您预先安装：
+本地运行 GraphScope 需要预先安装以下依赖:
 
 - Docker
-- minikube
 - Python >= 3.6 (以及 pip)
+- Local Kubernetes cluster set-up tool (例如 [Kind](https://kind.sigs.k8s.io))
 
-对于 Linux 环境，我们也提供了一个脚本来安装以上软件来准备环境。
+对于 Windows 和 MacOS 的用户，可通过官方文档来安装上述依赖, 并在Docker中开启Kubernetes功能。
+对于 Ubuntu/CentOS Linux 发行版用户，我们提供了脚本来准备运行时环境。
+您也可以在 Windows 上安装 `WSL2 <https://docs.microsoft.com/zh-cn/windows/wsl/install-win10>`_ 以使用脚本。
 
 ```bash
 # run the environment preparing script.
@@ -97,6 +99,12 @@ k8s_volumes = {
 sess = graphscope.session(k8s_volumes=k8s_volumes)
 ```
 
+对于 macOS，创建会话需要使用 LoadBalancer 服务类型（默认是 NodePort）。
+
+```python
+sess = graphscope.session(k8s_volumes=k8s_volumes, k8s_service_type="LoadBalancer")
+```
+
 会话的建立过程中，首选会在背后尝试拉起一个 `coordinator` 作为后端引擎的入口。
 该 `coordinator` 负责管理该次会话的所有资源（k8s pods），以及交互式查询、图分析、图学习引擎的生命周期。
 在 `coordinator` 后续拉起的其他每个 pod 中，都有一个 vineyard 实例作为内存管理层，分布式的管理图数据。
@@ -111,9 +119,9 @@ GraphScope 以属性图（property graph）建模图数据。属性图中，点
 </div>
 
 该图具有四种顶点，分别标记为“论文”、“作者”、“机构”和“研究领域”。有四种连接它们的边，
-每种边都有一个标签，并且边的两端顶点的标签也是确定的。 
+每种边都有一个标签，并且边的两端顶点的标签也是确定的。
 例如，“引用”这种标签的边连接两个“论文”顶点。另一个例子是标记为“撰写”的边，
-它要求该起始点的标记为“作者”，终止点的标记为“论文”。 
+它要求该起始点的标记为“作者”，终止点的标记为“论文”。
 所有的顶点和边都可以具有属性。 例如，“论文”顶点具有诸如发布年份、主题标签等属性。
 
 要将此图加载到 GraphScope，可以将以下代码与

README.md

@@ -26,9 +26,9 @@ To run GraphScope on your local computer, the following dependencies or tools ar
 
 - Docker
 - Python >= 3.6 (with pip)
-- Local Kubernetes cluster set-up tool (e.g. [Minikube](https://minikube.sigs.k8s.io) or [Kind](https://kind.sigs.k8s.io))
+- Local Kubernetes cluster set-up tool (e.g. [Kind](https://kind.sigs.k8s.io))
 
-On macOS, you can follow the official guides to install them.
+On macOS, you can follow the official guides to install them and enable Kubernetes in Docker.
 For Ubuntu/CentOS Linux distributions, we provide a script to install the above
 dependencies and prepare the environment.
 On Windows, you may want to install [Ubuntu](https://ubuntu.com/blog/ubuntu-on-wsl-2-is-generally-available) on [WSL2](https://docs.microsoft.com/zh-cn/windows/wsl/install-win10) to use the script.
@@ -78,26 +78,29 @@ To use GraphScope, we need to establish a session in a python interpreter.
 import os
 import graphscope
 
-# Setting an env for mounting test data from local disk,
-# hence we can access data inside the pods.
-
 # assume we mount `~/test_data` to `/testingdata` in pods.
 k8s_volumes = {
     "data": {
         "type": "hostPath",
         "field": {
-          "path": os.path.expanduser("~/test_data/"),
-          "type": "Directory"
+            "path": os.path.expanduser("~/test_data/"),
+            "type": "Directory"
         },
         "mounts": {
-          "mountPath": "/testingdata"
+            "mountPath": "/testingdata"
         }
     }
 }
 
 sess = graphscope.session(k8s_volumes=k8s_volumes)
 ```
 
+For macOS, the session needs to establish with the LoadBalancer service type (which is NodePort by default).
+
+```python
+sess = graphscope.session(k8s_volumes=k8s_volumes, k8s_service_type="LoadBalancer")
+```
+
 A session tries to launch a `coordinator`, which is the entry for the back-end engines.
 The coordinator manages a cluster of resources (k8s pods),
 and the interactive/analytical/learning engines ran on them.

docs/getting_started.rst

@@ -53,8 +53,30 @@ To use GraphScope, we need to establish a :ref:`Session` in a python interpreter
 
 .. code:: python
 
+    import os
     import graphscope
-    sess = graphscope.session()
+
+    # assume we mount `~/test_data` to `/testingdata` in pods.
+    k8s_volumes = {
+        "data": {
+            "type": "hostPath",
+            "field": {
+                "path": os.path.expanduser("~/test_data/"),
+                "type": "Directory"
+            },
+            "mounts": {
+                "mountPath": "/testingdata"
+            }
+        }
+    }
+
+    sess = graphscope.session(k8s_volumes=k8s_volumes)
+
+For macOS, the session needs to establish with the LoadBalancer service type (which is NodePort by default).
+
+.. code:: python
+
+    sess = graphscope.session(k8s_volumes=k8s_volumes, k8s_service_type="LoadBalancer")
 
 A :ref:`Session` tries to launch a coordinator,
 which is the entry for the back-end engines.
@@ -125,6 +147,14 @@ To load this graph to GraphScope, one may use the code below.
         }
     )
 
+Alternatively, we provide a function to load this graph for convenience.
+
+.. code:: python
+
+    from graphscope.dataset.ogbn_mag import load_ogbn_mag
+
+    g = load_ogbn_mag(sess, "/testingdata/ogbn_mag_small/")
+
 Here, the ``g`` is loaded in parallel via vineyard and stored
 in vineyard instances in the cluster managed by the session.
 See more details in :ref:`Loading Graphs`

docs/installation.rst

@@ -11,9 +11,9 @@ To run GraphScope on your local computer, the following dependencies or tools ar
 
 - Docker
 - Python 3.8 (with pip)
-- Local Kubernetes cluster set-up tool (e.g. [Minikube](https://minikube.sigs.k8s.io) or [Kind](https://kind.sigs.k8s.io))
+- Local Kubernetes cluster set-up tool (e.g. [Kind](https://kind.sigs.k8s.io))
 
-On Windows and macOS, you can follow the official guides to install them.
+On Windows and macOS, you can follow the official guides to install them and enable Kubernetes in Docker.
 For Ubuntu/CentOS Linux distributions, we provide a script to install the above
 dependencies and prepare the environment.
 Alternatively, you may want to install `WSL2 <https://docs.microsoft.com/zh-cn/windows/wsl/install-win10>`_ on Windows to use the script.

docs/zh/getting_started.rst

@@ -44,8 +44,29 @@
 
 .. code:: python
 
+    import os
     import graphscope
-    sess = graphscope.session()
+
+    # assume we mount `~/test_data` to `/testingdata` in pods.
+    k8s_volumes = {
+        "data": {
+            "type": "hostPath",
+            "field": {
+                "path": os.path.expanduser("~/test_data/"),
+                "type": "Directory"
+            },
+            "mounts": {
+                "mountPath": "/testingdata"
+            }
+        }
+    }
+
+    sess = graphscope.session(k8s_volumes=k8s_volumes)
+
+对于 macOS，创建会话需要使用 LoadBalancer 服务类型（默认是 NodePort）。
+
+.. code:: python
+    sess = graphscope.session(k8s_volumes=k8s_volumes, k8s_service_type="LoadBalancer")
 
 
 会话（:ref:`Session`）的建立过程中，首选会在背后尝试拉起一个 `coordinator` 作为后端引擎的入口。
@@ -65,9 +86,9 @@ GraphScope 以属性图（property graph）建模图数据。属性图中，点
     :alt: a sample property graph.
 
 该图具有四种顶点，分别标记为“论文”、“作者”、“机构”和“研究领域”。有四种连接它们的边，
-每种边都有一个标签，并且边的两端顶点的标签也是确定的。 
+每种边都有一个标签，并且边的两端顶点的标签也是确定的。
 例如，“引用”这种标签的边连接两个“论文”顶点。另一个例子是标记为“撰写”的边，
-它要求该起始点的标记为“作者”，终止点的标记为“论文”。 
+它要求该起始点的标记为“作者”，终止点的标记为“论文”。
 所有的顶点和边都可以具有属性。 例如，“论文”顶点具有诸如发布年份、主题标签等属性。
 
 

docs/zh/installation.rst

@@ -9,9 +9,9 @@ GraphScope 被设计为运行在 Kubernetes 管理的群集上。
 
 - Docker
 - Python 3.8 (with pip)
-- Local Kubernetes cluster set-up tool (e.g. [Minikube](https://minikube.sigs.k8s.io) or [Kind](https://kind.sigs.k8s.io))
+- Local Kubernetes cluster set-up tool (e.g. [Kind](https://kind.sigs.k8s.io))
 
-对于 Windows 和 MacOS 的用户，可通过官方文档来安装上述依赖。
+对于 Windows 和 MacOS 的用户，可通过官方文档来安装上述依赖, 并在Docker中开启Kubernetes功能。
 对于Ubuntu/CentOS Linux 发行版用户，我们提供了脚本来准备运行时环境。
 您也可以在 Windows 上安装 `WSL2 <https://docs.microsoft.com/zh-cn/windows/wsl/install-win10>`_ 以使用脚本。
 

scripts/prepare_env.sh

@@ -42,9 +42,9 @@ function get_os_version() {
   elif [ -f /etc/centos-release ]; then
     # Older Red Hat, CentOS, etc.
     platform=CentOS
-    os_version=$(cat /etc/centos-release | sed 's/.* \([0-9]\).*/\1/'))
+    os_version=$(cat /etc/centos-release | sed 's/.* \([0-9]\).*/\1/')
   else
-    # Fall back to uname, e.g. "Linux <version>", also works for BSD, etc.
+    # Fall back to uname, e.g. "Linux <version>", also works for BSD, Darwin, etc.
     platform=$(uname -s)
     os_version=$(uname -r)
   fi
@@ -57,7 +57,7 @@ function check_os_compatibility() {
   fi
 
   if [[ "${platform}" != *"Ubuntu"* && "${platform}" != *"CentOS"* ]]; then
-    echo "This script is only available on Ubuntu/CentOS."
+    echo "This script is only available on Ubuntu/CentOS"
     exit 1
   fi
 
@@ -116,15 +116,9 @@ function install_dependencies() {
   chmod +x kubectl && sudo mv kubectl /usr/local/bin/ && sudo ln /usr/local/bin/kubectl /usr/bin/kubectl || true
   echo "$(date '+%Y-%m-%d %H:%M:%S') kubectl ${K8S_VERSION} installed."
 
-  if [[ "${is_in_wsl}" = false ]]; then
-      curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 && \
-      chmod +x minikube && sudo mv minikube /usr/local/bin/ && sudo ln /usr/local/bin/minikube /usr/bin/minikube || true
-      echo "$(date '+%Y-%m-%d %H:%M:%S') minikube ${K8S_VERSION} installed."
-  else
-      curl -Lo kind https://kind.sigs.k8s.io/dl/v0.9.0/kind-linux-amd64 && \
-      chmod +x kind && sudo mv kind /usr/local/bin/ && sudo ln /usr/local/bin/kind /usr/bin/kind || true
-      echo "$(date '+%Y-%m-%d %H:%M:%S') kind v0.9.0 installed."
-  fi
+  curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.10.0/kind-linux-amd64
+  chmod +x kind && sudo mv kind /usr/local/bin/ && sudo ln /usr/local/bin/kind /usr/bin/kind || true
+  echo "$(date '+%Y-%m-%d %H:%M:%S') kind v0.10.0 installed."
 }
 
 function start_docker() {
@@ -142,24 +136,11 @@ function start_docker() {
 
 function launch_k8s_cluster() {
   echo "$(date '+%Y-%m-%d %H:%M:%S') launching k8s cluster"
-  if [[ "${is_in_wsl}" = false ]]; then
-    export CHANGE_MINIKUBE_NONE_USER=true
-    sudo sysctl fs.protected_regular=0 || true
-    sudo minikube start --vm-driver=none --kubernetes-version="${K8S_VERSION}"
-
-    sudo cp -r /root/.kube /root/.minikube "${HOME}" || true
-    sudo chown -R "$(id -u)":"$(id -g)" "${HOME}"/.minikube || true
-    sudo chown -R "$(id -u)":"$(id -g)" "${HOME}"/.kube || true
-    sed -i 's@/root@'"${HOME}"'@g' "${HOME}"/.kube/config || true
-    minikube update-context
-  else
-    curl -Lo config-with-mounts.yaml https://kind.sigs.k8s.io/examples/config-with-mounts.yaml
-    # mount $HOME dir to cluster container, which is kind-control-plane
-    sed -i 's@/path/to/my/files/@'"${HOME}"'@g; s@/files@'"${HOME}"'@g' ./config-with-mounts.yaml  || true
-    sudo kind create cluster --config config-with-mounts.yaml
-    sudo cp -r /root/.kube ${HOME} || true
-    sudo chown -R "$(id -u)":"$(id -g)" "${HOME}"/.kube || true
-  fi
+  curl -Lo config-with-mounts.yaml https://kind.sigs.k8s.io/examples/config-with-mounts.yaml
+  # mount $HOME dir to cluster container, which is kind-control-plane
+  sed -i 's@/path/to/my/files/@'"${HOME}"'@g; s@/files@'"${HOME}"'@g' ./config-with-mounts.yaml  || true
+  sudo kind create cluster --config config-with-mounts.yaml
+  sudo chown -R "$(id -u)":"$(id -g)" "${HOME}"/.kube || true
   echo "$(date '+%Y-%m-%d %H:%M:%S') cluster is lauched successfully."
 }
 
@@ -171,14 +152,13 @@ function pull_images() {
   sudo docker pull quay.io/coreos/etcd:v3.4.13 || true
   echo "$(date '+%Y-%m-%d %H:%M:%S') images pulled successfully."
 
-  if [[ "${is_in_wsl}" = true ]]; then
-    echo "$(date '+%Y-%m-%d %H:%M:%S') loading images into kind cluster."
-    sudo kind load registry.cn-hongkong.aliyuncs.com/graphscope/graphscope:${version} || true
-    sudo kind load registry.cn-hongkong.aliyuncs.com/graphscope/maxgraph_standalone_manager:${version} || true
-    sudo kind load zookeeper:3.4.14 || true
-    sudo kind load quay.io/coreos/etcd:v3.4.13 || true
-    echo "$(date '+%Y-%m-%d %H:%M:%S') images loaded."
-  fi
+  echo "$(date '+%Y-%m-%d %H:%M:%S') loading images into kind cluster."
+  sudo kind load docker-image registry.cn-hongkong.aliyuncs.com/graphscope/graphscope:${version} || true
+  sudo kind load docker-image registry.cn-hongkong.aliyuncs.com/graphscope/maxgraph_standalone_manager:${version} || true
+  sudo kind load docker-image zookeeper:3.4.14 || true
+  sudo kind load docker-image quay.io/coreos/etcd:v3.4.13 || true
+  echo "$(date '+%Y-%m-%d %H:%M:%S') images loaded."
+
 }
 
 if [ -f "${HOME}/.kube/config" ];