Skip to content

Commit

Permalink
Modify files associated with gsctl
Browse files Browse the repository at this point in the history
  • Loading branch information
FLYLX committed Aug 31, 2023
1 parent 3a9b666 commit 6a2837c
Show file tree
Hide file tree
Showing 15 changed files with 43 additions and 42 deletions.
5 changes: 3 additions & 2 deletions .cirrus.yml
Original file line number Diff line number Diff line change
Expand Up @@ -48,8 +48,9 @@ macosx_arm64_wheel_task:
# Not sure when the first installation would fail, saying that the lock of python3.10 is taken
# However the second trial would success.
./gs install-deps dev || true
./gs install-deps dev
python3 -m pip install click
python3 gsctl.py install-deps dev || true
python3 gsctl.py install-deps dev
brew install openjdk@11
sudo ln -sfn /opt/homebrew/opt/openjdk@11/libexec/openjdk.jdk /Library/Java/JavaVirtualMachines/openjdk-11.jdk
Expand Down
9 changes: 5 additions & 4 deletions .github/workflows/build-graphscope-wheels-macos.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,9 @@ jobs:
# Not sure when the first installation would fail, saying that the lock of python3.10 is taken
# However the second trial would success.
./gs install-deps dev || true
./gs install-deps dev
python3 -m pip install click
python3 gsctl.py install-deps dev || true
python3 gsctl.py install-deps dev
echo "export PATH=/usr/local/opt/coreutils/libexec/gnubin:\$PATH" >> ~/.graphscope_env
- name: Install JDK 11
Expand Down Expand Up @@ -150,8 +151,8 @@ jobs:
sudo mkdir /opt/graphscope
sudo chown -R $(id -u):$(id -g) /opt/graphscope
./gs install-deps dev || true
./gs install-deps dev
python3 gsctl.py install-deps dev || true
python3 gsctl.py install-deps dev
- name: Install latest libgrape-lite
if: false
Expand Down
3 changes: 2 additions & 1 deletion .github/workflows/nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,8 @@ jobs:
fi
sudo mkdir /opt/graphscope
sudo chown -R $(id -u):$(id -g) /opt/graphscope
./gs install-deps dev --v6d-version v0.16.4
python3 -m pip install click
python3 gsctl.py install-deps dev --v6d-version v0.16.4
- name: Setup tmate session
if: false
Expand Down
4 changes: 2 additions & 2 deletions README-zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -294,10 +294,10 @@ sess.close()
为了构建 graphscope Python 包以及引擎,你需要安装一些依赖和构建工具。

```bash
./gs install-deps dev
python3 gsctl.py install-deps dev

# 如果在中国,加入参数 `--cn` 来加速下载
./gs install-deps dev --cn
python3 gsctl.py install-deps dev --cn
```

现在你可以使用 `make` 来构建 GraphScope
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -333,10 +333,10 @@ Please note that we have not hardened this release for production use and it lac
To build graphscope Python package and the engine binaries, you need to install some dependencies and build tools.

```bash
./gs install-deps dev
python3 gsctl.py install-deps dev

# With argument --cn to speed up the download if you are in China.
./gs install-deps dev --cn
python3 gsctl.py install-deps dev --cn
```

Then you can build GraphScope with pre-configured `make` commands.
Expand Down
6 changes: 3 additions & 3 deletions docs/analytical_engine/dev_and_test.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ With `gs` command-line utility, you can build analytical engine of GraphScope wi
# Clone a repo if needed
# git clone https://github.com/alibaba/graphscope
# cd graphscope
./gs make analytical
python3 gsctl.py make analytical
```

The code of analytical engine is a cmake project, with a `CMakeLists.txt` in the its root directory (`/analytical_engine`). After the building with `gs`, you may found the built artifacts in `analytical_engine/build/grape_engine`.
Expand All @@ -95,7 +95,7 @@ Together with the `grape_engine` are shared libraries, or there may have a bunch
You could install it to a location by

```bash
./gs make analytical-install --install-prefix /usr/local
python3 gsctl.py make analytical-install --install-prefix /usr/local
```

````{note}
Expand All @@ -116,7 +116,7 @@ export GRAPHSCOPE_HOME=`pwd`
See more about `GRAPHSCOPE_HOME` in [run tests](../development/how_to_test.md#run-tests)

```bash
./gs test analytical
python3 gsctl.py test analytical
```

It would download the test dataset to the `/tmp/gstest` (if not exists) and run multiple algorithms against various graphs, and compare the result with the ground truth.
2 changes: 1 addition & 1 deletion docs/deployment/install_on_local.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ to help you install all the dependencies.
git clone https://github.com/alibaba/graphscope
cd graphscope

./gs install-deps dev
python3 gsctl.py install-deps dev
# use --help to get more usage.
```

Expand Down
12 changes: 6 additions & 6 deletions docs/development/dev_guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,10 +37,10 @@ To use the dev containers for GraphScope development, you can follow these steps
To install all dependencies on your local, use the GraphScope command-line utility [gs](https://github.com/alibaba/GraphScope/blob/main/gs) with the subcommand `install-deps` like this

```bash
./gs install-deps dev
python3 gsctl.py install-deps dev

# for more usage, try
# ./gs install-deps -h
# python3 gsctl.py install-deps -h
```

You could download the `gs` directly or clone the [GraphScope](https://github.com/alibaba/GraphScope) to local, the `gs` is located in the root directory of GraphScope.
Expand All @@ -52,7 +52,7 @@ You could download the `gs` directly or clone the [GraphScope](https://github.co
With `gs` command-line utility, you can build all targets for GraphScope with a single command.

```bash
./gs make all
python3 gsctl.py make all
```

This would build all targets sequentially, here we
Expand All @@ -65,7 +65,7 @@ You may found the built artifacts in several places according to each components
And you could install them to one place by

```bash
./gs make install [--prefix=/opt/graphscope]
python3 gsctl.py make install [--prefix=/opt/graphscope]
```

By default it would install all artifacts to `/opt/graphscope`, and you could specify another location by assigning the value of `--prefix`.
Expand Down Expand Up @@ -94,7 +94,7 @@ The package would be installed in [editable mode](https://pip.pypa.io/en/stable/
````

```shell
./gs make coordinator
python3 gsctl.py make coordinator
```

### Build Python Client
Expand All @@ -108,5 +108,5 @@ This package would also be installed in [editable mode](https://pip.pypa.io/en/s
````

```shell
./gs make client
python3 gsctl.py make client
```
12 changes: 6 additions & 6 deletions docs/development/how_to_test.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,14 +27,14 @@ Please refer to [Dev Environment](../development/dev_guide.md#dev-environment) t
With `gs` command-line utility, you can build all targets for GraphScope with a single command.

```bash
./gs make install
python3 gsctl.py make install
```

### Run tests

Run a bunch of test cases that involves 3 engines
```bash
./gs test e2e --local
python3 gsctl.py test e2e --local
```

## Test GraphScope on Kubernetes cluster
Expand All @@ -44,7 +44,7 @@ Make sure you have [docker](https://www.docker.com) installed.
### Build images

```bash
./gs make-image all
python3 gsctl.py make-image all
```

Executing this command will generate the corresponding images for each component.
Expand All @@ -63,10 +63,10 @@ Images of GraphScope
Or you could choose to generate image for a specific component. For example

```bash
./gs make-image analytical
python3 gsctl.py make-image analytical
```

Try `./gs make-image -h` for more available options.
Try `python3 gsctl.py make-image -h` for more available options.

### Prepare a Kubernetes cluster

Expand All @@ -79,5 +79,5 @@ The tests would launch a couple of pods on your cluster, so you need to set appr
Take our previous built image for example, which has an empty registry (or you could say it's `docker.io` by default), and tagged `latest`

```bash
./gs test e2e --k8s --registry="docker.io" --tag="latest"
python3 gsctl.py test e2e --k8s --registry="docker.io" --tag="latest"
```
6 changes: 3 additions & 3 deletions docs/interactive_engine/dev_and_test.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,20 +23,20 @@ cd graphscope

Now you are ready to build the GIE engine (on vineyard store) with the following command:
```bash
./gs make interactive --storage-type=vineyard
python3 gsctl.py make interactive --storage-type=vineyard
```
You can find the built artifacts in `interactive_engine/assembly/target/graphscope`.

You could install it to a location by

```bash
./gs make interactive-install --storage-type=vineyard --install-prefix /opt/graphscope
python3 gsctl.py make interactive-install --storage-type=vineyard --install-prefix /opt/graphscope
```

## Test GIE with Vineyard Store on Local
You could test the GIE engine on vineyard store with the following command:
```bash
./gs test interactive --local --storage-type=vineyard
python3 gsctl.py test interactive --local --storage-type=vineyard
```

This will run end2end tests, from compiling a gremlin queries to obtaining and verifying the results from the computed engine. The test includes:
Expand Down
8 changes: 4 additions & 4 deletions docs/learning_engine/dev_and_test.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,10 @@ To install all dependencies on your local, use the GraphScope command-line utili
`install-deps` like this

```bash
./gs install-deps dev
python3 gsctl.py install-deps dev

# for more usage, try
# ./gs install-deps -h
# python3 gsctl.py install-deps -h
```

### Dev on docker container
Expand All @@ -40,13 +40,13 @@ More options about `docker` command can be found [here](https://docs.docker.com/
You can build all targets for GraphScope Learning Engine with a single command.

```bash
./gs make learning
python3 gsctl.py make learning
```

You could install it to a location by:

```bash
./gs make learning-install --install-prefix /opt/graphscope
python3 gsctl.py make learning-install --install-prefix /opt/graphscope
```

## How to Test
Expand Down
4 changes: 2 additions & 2 deletions docs/zh/deployment.rst
Original file line number Diff line number Diff line change
Expand Up @@ -106,12 +106,12 @@ Coordinator 作为 GraphScope 后端服务的入口,通过 grpc 接收来自 P
-----------------

我们提供了一个可在本地安装GraphScope相关依赖的脚本,该脚本可以运行在 Ubuntu 20.04+ 或 MacOS 11+ (Intel) / MacOS 12+ (Apple silicon) 平台上, 主要的用法如下:
你可以通过 `./gs -h` 获取更详细的帮助信息。
你可以通过 `python3 gsctl.py -h` 获取更详细的帮助信息。

* 安装 GraphScope 开发相关依赖
.. code:: shell
./gs install-deps dev
python3 gsctl.py install-deps dev
* 本地部署 GraphScope
.. code:: shell
Expand Down
2 changes: 1 addition & 1 deletion docs/zh/developer_guide.rst
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ macOS
.. code:: bash
cd GraphScope/k8s/internal
./gs install-deps dev
python3 gsctl.py install-deps dev
source ~/.graphscope_env
- 构建 GraphScope Server Wheels
Expand Down
2 changes: 0 additions & 2 deletions python/graphscope/gsctl/gsctl.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,6 @@
import io
import os
import subprocess
import tempfile

import click

Expand Down Expand Up @@ -326,7 +325,6 @@ def test(repo, type, testdata, local, storage_type, k8s, nx):
click.echo("test")
if type is None:
type = ""

cmd = [
"bash",
test_script,
Expand Down
6 changes: 3 additions & 3 deletions python/graphscope/gsctl/scripts/install_deps_command.sh
Original file line number Diff line number Diff line change
Expand Up @@ -338,7 +338,7 @@ write_env_config() {
echo "export CC=${homebrew_prefix}/opt/llvm/bin/clang"
echo "export CXX=${homebrew_prefix}/opt/llvm/bin/clang++"
echo "export CARGO_TARGET_X86_64_APPLE_DARWIN_LINKER=${CC}"
if [ "${JAVA_HOME}" == "False" ]; then
if [ -z "${JAVA_HOME}" ]; then
echo "export JAVA_HOME=\$(/usr/libexec/java_home -v11)"
fi
echo "export OPENSSL_ROOT_DIR=${homebrew_prefix}/opt/openssl"
Expand All @@ -350,7 +350,7 @@ write_env_config() {

elif [[ "${OS_PLATFORM}" == *"Ubuntu"* ]]; then
{
if [ "${JAVA_HOME}" == "False" ]; then
if [ -z "${JAVA_HOME}" ]; then
echo "export JAVA_HOME=/usr/lib/jvm/default-java"
fi
} >>"${OUTPUT_ENV_FILE}"
Expand All @@ -362,7 +362,7 @@ write_env_config() {
echo "source /opt/rh/llvm-toolset-7.0/enable || true"
echo "export LIBCLANG_PATH=/opt/rh/llvm-toolset-7.0/root/usr/lib64/"
fi
if [ "${JAVA_HOME}" == "False" ]; then
if [ -z "${JAVA_HOME}" ]; then
echo "export JAVA_HOME=/usr/lib/jvm/jre-openjdk"
fi
echo "export OPENSSL_ROOT_DIR=${install_prefix}"
Expand Down

0 comments on commit 6a2837c

Please sign in to comment.