refactor: Upgrade gsctl and modify files associated with gsctl

.cirrus.yml

@@ -48,8 +48,9 @@ macosx_arm64_wheel_task:
 
     # Not sure when the first installation would fail, saying that the lock of python3.10 is taken
     # However the second trial would success.
-    ./gs install-deps dev || true
-    ./gs install-deps dev
+    python3 -m pip install click 
+    python3 gsctl.py install-deps dev || true
+    python3 gsctl.py install-deps dev
 
     brew install openjdk@11
     sudo ln -sfn /opt/homebrew/opt/openjdk@11/libexec/openjdk.jdk /Library/Java/JavaVirtualMachines/openjdk-11.jdk

.github/workflows/build-graphscope-wheels-macos.yml

@@ -35,8 +35,9 @@ jobs:
 
         # Not sure when the first installation would fail, saying that the lock of python3.10 is taken
         # However the second trial would success.
-        ./gs install-deps dev || true
-        ./gs install-deps dev
+        python3 -m pip install click 
+        python3 gsctl.py install-deps dev || true
+        python3 gsctl.py install-deps dev
         echo "export PATH=/usr/local/opt/coreutils/libexec/gnubin:\$PATH" >> ~/.graphscope_env
 
     - name: Install JDK 11
@@ -150,8 +151,8 @@ jobs:
         sudo mkdir /opt/graphscope
         sudo chown -R $(id -u):$(id -g) /opt/graphscope
 
-        ./gs install-deps dev || true
-        ./gs install-deps dev
+        python3 gsctl.py install-deps dev || true
+        python3 gsctl.py install-deps dev
 
     - name: Install latest libgrape-lite
       if: false

.github/workflows/nightly.yml

@@ -43,7 +43,8 @@ jobs:
         fi
         sudo mkdir /opt/graphscope
         sudo chown -R $(id -u):$(id -g) /opt/graphscope
-        ./gs install-deps dev --v6d-version v0.16.4
+        python3 -m pip install click 
+        python3 gsctl.py install-deps dev --v6d-version v0.16.4
 
     - name: Setup tmate session
       if: false

README-zh.md

@@ -294,10 +294,10 @@ sess.close()
 为了构建 graphscope Python 包以及引擎，你需要安装一些依赖和构建工具。
 
 ```bash
-./gs install-deps dev
+python3 gsctl.py install-deps dev
 
 # 如果在中国，加入参数 `--cn` 来加速下载
-./gs install-deps dev --cn
+python3 gsctl.py install-deps dev --cn
 ```
 
 现在你可以使用 `make` 来构建 GraphScope

README.md

@@ -333,10 +333,10 @@ Please note that we have not hardened this release for production use and it lac
 To build graphscope Python package and the engine binaries, you need to install some dependencies and build tools.
 
 ```bash
-./gs install-deps dev
+python3 gsctl.py install-deps dev
 
 # With argument --cn to speed up the download if you are in China.
-./gs install-deps dev --cn
+python3 gsctl.py install-deps dev --cn
 ```
 
 Then you can build GraphScope with pre-configured `make` commands.

docs/analytical_engine/dev_and_test.md

@@ -85,7 +85,7 @@ With `gs` command-line utility, you can build analytical engine of GraphScope wi
 # Clone a repo if needed
 # git clone https://github.com/alibaba/graphscope
 # cd graphscope
-./gs make analytical
+python3 gsctl.py make analytical
 ```
 
 The code of analytical engine is a cmake project, with a `CMakeLists.txt` in the its root directory (`/analytical_engine`). After the building with `gs`, you may found the built artifacts in `analytical_engine/build/grape_engine`.
@@ -95,7 +95,7 @@ Together with the `grape_engine` are shared libraries, or there may have a bunch
 You could install it to a location by
 
 ```bash
-./gs make analytical-install --install-prefix /usr/local
+python3 gsctl.py make analytical-install --install-prefix /usr/local
 ```
 
 ````{note}
@@ -116,7 +116,7 @@ export GRAPHSCOPE_HOME=`pwd`
 See more about `GRAPHSCOPE_HOME` in [run tests](../development/how_to_test.md#run-tests)
 
 ```bash
-./gs test analytical
+python3 gsctl.py test analytical
 ```
 
 It would download the test dataset to the `/tmp/gstest` (if not exists) and run multiple algorithms against various graphs, and compare the result with the ground truth.

docs/deployment/install_on_local.md

@@ -68,7 +68,7 @@ to help you install all the dependencies.
 git clone https://github.com/alibaba/graphscope
 cd graphscope
 
-./gs install-deps dev
+python3 gsctl.py install-deps dev
 # use --help to get more usage.
 ```
 

docs/development/dev_guide.md

@@ -37,10 +37,10 @@ To use the dev containers for GraphScope development, you can follow these steps
 To install all dependencies on your local, use the GraphScope command-line utility [gs](https://github.com/alibaba/GraphScope/blob/main/gs) with the subcommand `install-deps` like this
 
 ```bash
-./gs install-deps dev
+python3 gsctl.py install-deps dev
 
 # for more usage, try
-# ./gs install-deps -h
+# python3 gsctl.py install-deps -h
 ```
 
 You could download the `gs` directly or clone the [GraphScope](https://github.com/alibaba/GraphScope) to local, the `gs` is located in the root directory of GraphScope.
@@ -52,7 +52,7 @@ You could download the `gs` directly or clone the [GraphScope](https://github.co
 With `gs` command-line utility, you can build all targets for GraphScope with a single command.
 
 ```bash
-./gs make all
+python3 gsctl.py make all
 ```
 
 This would build all targets sequentially, here we
@@ -65,7 +65,7 @@ You may found the built artifacts in several places according to each components
 And you could install them to one place by
 
 ```bash
-./gs make install [--prefix=/opt/graphscope]
+python3 gsctl.py make install [--prefix=/opt/graphscope]
 ```
 
 By default it would install all artifacts to `/opt/graphscope`, and you could specify another location by assigning the value of `--prefix`.
@@ -94,7 +94,7 @@ The package would be installed in [editable mode](https://pip.pypa.io/en/stable/
 ````
 
 ```shell
-./gs make coordinator
+python3 gsctl.py make coordinator
 ```
 
 ### Build Python Client
@@ -108,5 +108,5 @@ This package would also be installed in [editable mode](https://pip.pypa.io/en/s
 ````
 
 ```shell
-./gs make client
+python3 gsctl.py make client
 ```

docs/development/how_to_test.md

@@ -27,14 +27,14 @@ Please refer to [Dev Environment](../development/dev_guide.md#dev-environment) t
 With `gs` command-line utility, you can build all targets for GraphScope with a single command.
 
 ```bash
-./gs make install
+python3 gsctl.py make install
 ```
 
 ### Run tests
 
 Run a bunch of test cases that involves 3 engines
 ```bash
-./gs test e2e --local
+python3 gsctl.py test e2e --local
 ```
 
 ## Test GraphScope on Kubernetes cluster
@@ -44,7 +44,7 @@ Make sure you have [docker](https://www.docker.com) installed.
 ### Build images
 
 ```bash
-./gs make-image all
+python3 gsctl.py make-image all
 ```
 
 Executing this command will generate the corresponding images for each component.
@@ -63,10 +63,10 @@ Images of GraphScope
 Or you could choose to generate image for a specific component. For example
 
 ```bash
-./gs make-image analytical
+python3 gsctl.py make-image analytical
 ```
 
-Try `./gs make-image -h` for more available options.
+Try `python3 gsctl.py make-image -h` for more available options.
 
 ### Prepare a Kubernetes cluster
 
@@ -79,5 +79,5 @@ The tests would launch a couple of pods on your cluster, so you need to set appr
 Take our previous built image for example, which has an empty registry (or you could say it's `docker.io` by default), and tagged `latest`
 
 ```bash
-./gs test e2e --k8s --registry="docker.io" --tag="latest"
+python3 gsctl.py test e2e --k8s --registry="docker.io" --tag="latest"
 ```

docs/interactive_engine/dev_and_test.md

@@ -23,20 +23,20 @@ cd graphscope
 
 Now you are ready to build the GIE engine (on vineyard store) with the following command:
 ```bash
-./gs make interactive --storage-type=vineyard
+python3 gsctl.py make interactive --storage-type=vineyard
 ```
 You can find the built artifacts in `interactive_engine/assembly/target/graphscope`.
 
 You could install it to a location by
 
 ```bash
-./gs make interactive-install --storage-type=vineyard --install-prefix /opt/graphscope
+python3 gsctl.py make interactive-install --storage-type=vineyard --install-prefix /opt/graphscope
 ```
 
 ## Test GIE with Vineyard Store on Local
 You could test the GIE engine on vineyard store with the following command:
 ```bash
-./gs test interactive --local --storage-type=vineyard
+python3 gsctl.py test interactive --local --storage-type=vineyard
 ```
 
 This will run end2end tests, from compiling a gremlin queries to obtaining and verifying the results from the computed engine. The test includes:

docs/learning_engine/dev_and_test.md

@@ -14,10 +14,10 @@ To install all dependencies on your local, use the GraphScope command-line utili
 `install-deps` like this
 
 ```bash
-./gs install-deps dev
+python3 gsctl.py install-deps dev
 
 # for more usage, try
-# ./gs install-deps -h
+# python3 gsctl.py install-deps -h
 ```
 
 ### Dev on docker container
@@ -40,13 +40,13 @@ More options about `docker` command can be found [here](https://docs.docker.com/
 You can build all targets for GraphScope Learning Engine with a single command.
 
 ```bash
-./gs make learning
+python3 gsctl.py make learning
 ```
 
 You could install it to a location by:
 
 ```bash
-./gs make learning-install --install-prefix /opt/graphscope
+python3 gsctl.py make learning-install --install-prefix /opt/graphscope
 ```
 
 ## How to Test

docs/zh/deployment.rst

@@ -106,12 +106,12 @@ Coordinator 作为 GraphScope 后端服务的入口，通过 grpc 接收来自 P
 -----------------
 
 我们提供了一个可在本地安装GraphScope相关依赖的脚本，该脚本可以运行在 Ubuntu 20.04+ 或 MacOS 11+ (Intel) / MacOS 12+ (Apple silicon) 平台上, 主要的用法如下：
-你可以通过 `./gs -h` 获取更详细的帮助信息。
+你可以通过 `python3 gsctl.py -h` 获取更详细的帮助信息。
 
 * 安装 GraphScope 开发相关依赖
 .. code:: shell
 
-    ./gs install-deps dev
+    python3 gsctl.py install-deps dev
 
 * 本地部署 GraphScope
 .. code:: shell

docs/zh/developer_guide.rst

@@ -106,7 +106,7 @@ macOS
 .. code:: bash
 
     cd GraphScope/k8s/internal
-    ./gs install-deps dev
+    python3 gsctl.py install-deps dev
     source ~/.graphscope_env
 
 - 构建 GraphScope Server Wheels

python/graphscope/gsctl/gsctl.py

@@ -19,14 +19,14 @@
 import io
 import os
 import subprocess
-import tempfile
 
 import click
 
 scripts_dir = os.path.join(os.path.dirname(os.path.realpath(__file__)), "scripts")
 install_deps_script = os.path.join(scripts_dir, "install_deps_command.sh")
 make_script = os.path.join(scripts_dir, "make_command.sh")
 make_image_script = os.path.join(scripts_dir, "make_image_command.sh")
+test_script = os.path.join(scripts_dir, "test_command.sh")
 
 
 def run_shell_cmd(cmd, workingdir):
@@ -267,14 +267,81 @@ def make_image(repo, component, registry, tag):
 
 
 @click.command()
+@click.argument(
+    "type",
+    type=click.Choice(
+        [
+            "analytical",
+            "analytical-java",
+            "interactive",
+            "learning",
+            "local-e2e",
+            "k8s-e2e",
+            "groot",
+        ],
+        case_sensitive=False,
+    ),
+    required=False,
+)
+@click.option(
+    "--testdata",
+    type=click.Path(),
+    default="/tmp/gstest",
+    show_default=True,
+    help="""assign a custom test data location. This could be cloned from
+    https://github.com/graphscope/gstest""",
+)
+@click.option(
+    "--local",
+    is_flag=True,
+    default=False,
+    help="Run local tests",
+)
+@click.option(
+    "--storage-type",
+    default="default",
+    show_default=True,
+    help="test gie with specified storage type",
+)
+@click.option(
+    "--k8s",
+    is_flag=True,
+    default=False,
+    help="Run local tests",
+)
+@click.option(
+    "--nx",
+    is_flag=True,
+    default=False,
+    help="Run nx tests",
+)
 @click.pass_obj
-def test(repo):
+def test(repo, type, testdata, local, storage_type, k8s, nx):
     """Trigger tests on built artifacts.
 
     \f
     TODO: fulfill this."""
     click.secho(f"repo.home = {repo.home}", fg="green")
     click.echo("test")
+    if type is None:
+        type = ""
+    cmd = [
+        "bash",
+        test_script,
+        "-t",
+        type,
+        "-d",
+        testdata,
+        "-l",
+        str(local),
+        "-s",
+        storage_type,
+        "-k",
+        str(k8s),
+        "-n",
+        str(nx),
+    ]
+    run_shell_cmd(cmd, repo.home)
 
 
 cli.add_command(install_deps)