coordinator启动失败

[dakeleblack]

创建session的时候报了下面超时的错误，请问各位，这是什么问题？

s = graphscope.session()
2021-05-24 15:46:33,486 [INFO][session:457]: Initializing graphscope session with parameters: {'addr': None, 'cluster_type': 'k8s', 'num_workers': 2, 'preemptive': True, 'k8s_namespace': None, 'k8s_service_type': 'NodePort', 'k8s_gs_image': 'registry.cn-hongkong.aliyuncs.com/graphscope/graphscope:0.4.1', 'k8s_etcd_image': 'quay.io/coreos/etcd:v3.4.13', 'k8s_image_pull_policy': 'IfNotPresent', 'k8s_image_pull_secrets': [], 'k8s_gie_graph_manager_image': 'registry.cn-hongkong.aliyuncs.com/graphscope/maxgraph_standalone_manager:0.4.1', 'k8s_zookeeper_image': 'zookeeper:3.4.14', 'k8s_coordinator_cpu': 1.0, 'k8s_coordinator_mem': '4Gi', 'k8s_etcd_num_pods': 3, 'k8s_etcd_cpu': 0.5, 'k8s_etcd_mem': '128Mi', 'k8s_zookeeper_cpu': 0.5, 'k8s_zookeeper_mem': '256Mi', 'k8s_gie_graph_manager_cpu': 1.0, 'k8s_gie_graph_manager_mem': '4Gi', 'k8s_vineyard_daemonset': 'none', 'k8s_vineyard_cpu': 0.5, 'k8s_vineyard_mem': '512Mi', 'vineyard_shared_mem': '4Gi', 'k8s_engine_cpu': 0.5, 'k8s_engine_mem': '4Gi', 'k8s_mars_worker_cpu': 0.5, 'k8s_mars_worker_mem': '4Gi', 'k8s_mars_scheduler_cpu': 0.5, 'k8s_mars_scheduler_mem': '2Gi', 'with_mars': False, 'k8s_volumes': {}, 'k8s_waiting_for_delete': False, 'timeout_seconds': 600, 'dangling_timeout_seconds': 600, 'k8s_client_config': {}}
2021-05-24 15:46:33,618 [INFO][cluster:451]: Launching coordinator...
2021-05-24 15:46:36,679 [INFO][utils:167]: coordinator-oqucbi-6d85f4549f-zl9l9: 0/1 nodes are available: 1 Insufficient memory.
Traceback (most recent call last):
File "/Users/taoge666/opt/anaconda3/envs/graphscope/lib/python3.8/site-packages/graphscope/deploy/kubernetes/cluster.py", line 706, in start
self._waiting_for_services_ready()
File "/Users/taoge666/opt/anaconda3/envs/graphscope/lib/python3.8/site-packages/graphscope/deploy/kubernetes/cluster.py", line 633, in _waiting_for_services_ready
if wait_for_deployment_complete(
File "/Users/taoge666/opt/anaconda3/envs/graphscope/lib/python3.8/site-packages/graphscope/deploy/kubernetes/utils.py", line 122, in wait_for_deployment_complete
raise TimeoutError("Waiting timeout for deployment {}".format(name))
TimeoutError: Waiting timeout for deployment coordinator-oqucbi

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "", line 1, in
File "/Users/taoge666/opt/anaconda3/envs/graphscope/lib/python3.8/site-packages/graphscope/client/utils.py", line 156, in wrapper
return_value = func(*args, **kwargs)
File "/Users/taoge666/opt/anaconda3/envs/graphscope/lib/python3.8/site-packages/graphscope/client/session.py", line 491, in init
self._connect()
File "/Users/taoge666/opt/anaconda3/envs/graphscope/lib/python3.8/site-packages/graphscope/client/session.py", line 828, in _connect
self._launcher.start()
File "/Users/taoge666/opt/anaconda3/envs/graphscope/lib/python3.8/site-packages/graphscope/deploy/kubernetes/cluster.py", line 713, in start
raise K8sError(
graphscope.framework.errors.K8sError: 'Error when launching Coordinator on kubernetes cluster'

[lidongze0629]

@bitao666

From the log above, 0/1 nodes are available: 1 Insufficient memory shows the reason that your node memory is not enough.

You can usekubectl describe node [NODE_NAME] to view the resource information of your k8s node. And for GraphScope, you can adjust the values by session params.

graphscope.session(k8s_coordinator_mem='1Gi')

[DanielaCasilli]

Hi @lidongze0629
I tried to create a cluster using kind and have run into similar error messages. I am running it on a Macbook Pro with 8GB RAM

My node details for memory capacity are as follows for the kind-control-plane

Capacity:
cpu: 4
ephemeral-storage: 61255492Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 8153416Ki
pods: 110
Allocatable:
cpu: 4
ephemeral-storage: 61255492Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 8153416Ki
pods: 110

and I've tried to deploy a graphscope session with the following configs:

config ={'num_workers': 1,
'log_level': 'info',
'show_log': True,
'k8s_namespace': 'gs-viriue',
'k8s_gs_image': 'registry.cn-hongkong.aliyuncs.com/graphscope/graphscope:0.2.1',
'k8s_etcd_image': 'quay.io/coreos/etcd:v3.4.13',
'k8s_image_pull_policy': 'IfNotPresent',
'k8s_image_pull_secrets': [],
'k8s_gie_graph_manager_image': 'registry.cn-hongkong.aliyuncs.com/graphscope/maxgraph_standalone_manager:0.2.1',
'k8s_zookeeper_image': 'zookeeper:3.4.14',
'k8s_coordinator_cpu': 0.25,
'k8s_coordinator_mem': '256Mi',
'k8s_vineyard_cpu': 0.25,
'k8s_vineyard_mem': '256Mi',
'k8s_vineyard_shared_mem': '256Mi',
'k8s_engine_cpu': 0.5,
'k8s_engine_mem': '256Mi',
'k8s_waiting_for_delete': False,
'timeout_seconds': 600}

I originally had higher memory but have reduced it quite low but am still seeing:
graphscope.framework.errors.K8sError: 'Error when launching Coordinator on kubernetes cluster'

Any advice would be appreciated.

[lidongze0629]

@bitao666 May I know more detail to describe this problem? Such as the full log.