[BUG] There is an error when loading a large graph with about 60 million edges.

[wym126]

Describe the bug
There is an error when loading a large graph whit about 60 million edges.

Environment (please complete the following information):

• GraphScope version: [v0.15]
• OS: [Linux]
• Version python3.7

Additional context

Check failed: Not enough memory: Failed to allocate memory of size 828797872, total available memory size are 4294967296, and 3617130488 are already in use in "this->Build(client)", in function virtual std::shared_ptr<vineyard::Object> vineyard::FixedSizeBinaryArrayBaseBuilder::_Seal(vineyard::Client&), file /tmp/v6d/modules/basic/ds/arrow.vineyard.h, line 1125
*** Aborted at 1658818708 (unix time) try "date -d @1658818708" if you are using GNU date ***
PC: @                0x0 (unknown)
*** SIGSEGV (@0x0) received by PID 55826 (TID 0x7f477a7da640) from PID 0; stack trace: ***
    @     0x7f4782f441f0 (unknown)
    @     0x7f4779e0f9be vineyard::ArrowFragment<>::initPointers()
    @     0x7f4779e110b4 vineyard::ArrowFragment<>::PostConstruct()
    @     0x7f4779e09265 vineyard::ArrowFragmentBaseBuilder<>::_Seal()
    @     0x7f4779e0b40a vineyard::ArrowFragmentBaseBuilder<>::_Seal()
    @     0x7f4782fda6da vineyard::ObjectBuilder::Seal()
    @     0x7f4779da739d vineyard::ArrowFragment<>::AddNewVertexEdgeLabels()
    @     0x7f4779da9dfa vineyard::ArrowFragment<>::AddVerticesAndEdges()
    @     0x7f4779db11a1 vineyard::BasicEVFragmentLoader<>::AddVerticesAndEdgesToFragment()
    @     0x7f4779ddeb96 gs::ArrowFragmentLoader<>::addVerticesAndEdges()
    @     0x7f4779ddffa1 gs::ArrowFragmentLoader<>::AddLabelsToGraphAsFragmentGroup()
    @     0x7f4779ca1257 (unknown)
    @     0x7f4779ca1821 AddLabelsToGraph
    @           0x4969a7 gs::GrapeInstance::addLabelsToGraph()
    @           0x4a5bf2 gs::GrapeInstance::OnReceive()
    @           0x5242cb gs::Dispatcher::processCmd()
    @           0x5256ba gs::Dispatcher::subscriberLoop()
    @     0x7f4783012290 execute_native_thread_routine
    @     0x7f4782f39450 start_thread
    @     0x7f4781b9fd53 clone
    @                0x0 (unknown)
[error] Check failed: Not enough memory: Failed to allocate memory of size 826884720, total available memory size are 4294967296, and 3653055104 are already in use in "this->Build(client)", in function virtual std::shared_ptr<vineyard::Object> vineyard::FixedSizeBinaryArrayBaseBuilder::_Seal(vineyard::Client&), file /tmp/v6d/modules/basic/ds/arrow.vineyard.h, line 1125
*** Aborted at 1658818709 (unix time) try "date -d @1658818709" if you are using GNU date ***
PC: @                0x0 (unknown)
*** SIGSEGV (@0x0) received by PID 55824 (TID 0x7f37bcb8f640) from PID 0; stack trace: ***
    @     0x7f37c1ae81f0 (unknown)
    @     0x7f37ace2f9be vineyard::ArrowFragment<>::initPointers()
    @     0x7f37ace310b4 vineyard::ArrowFragment<>::PostConstruct()
    @     0x7f37ace29265 vineyard::ArrowFragmentBaseBuilder<>::_Seal()
    @     0x7f37ace2b40a vineyard::ArrowFragmentBaseBuilder<>::_Seal()
    @     0x7f37c1b7e6da vineyard::ObjectBuilder::Seal()
    @     0x7f37acdc739d vineyard::ArrowFragment<>::AddNewVertexEdgeLabels()
    @     0x7f37acdc9dfa vineyard::ArrowFragment<>::AddVerticesAndEdges()
    @     0x7f37acdd11a1 vineyard::BasicEVFragmentLoader<>::AddVerticesAndEdgesToFragment()
    @     0x7f37acdfeb96 gs::ArrowFragmentLoader<>::addVerticesAndEdges()
    @     0x7f37acdfffa1 gs::ArrowFragmentLoader<>::AddLabelsToGraphAsFragmentGroup()
    @     0x7f37accc1257 (unknown)
    @     0x7f37accc1821 AddLabelsToGraph
    @           0x4969a7 gs::GrapeInstance::addLabelsToGraph()
    @           0x4a5bf2 gs::GrapeInstance::OnReceive()
    @           0x5242cb gs::Dispatcher::processCmd()
    @           0x526227 gs::Dispatcher::publisherLoop()
    @     0x7f37c1bb6290 execute_native_thread_routine
    @     0x7f37c1add450 start_thread
    @     0x7f37c0743d53 clone

    @                0x0 (unknown)
--------------------------------------------------------------------------

Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------

[welcome[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! And a maintainer will get back to you shortly!
Please feel free to contact us on DingTalk, WeChat account(graphscope) or Slack. We are happy to answer your questions responsively.

[sighingnow]

Please specify a larger vineyard_shared_mem (in bytes). The default value is 4G.

[wym126]

Thanks, but how do I specify vineyard_shared_mem in jupyter, API or other methods?

[siyuan0322]

In session constructors.

GraphScope/python/graphscope/tests/kubernetes/test_demo_script.py

Line 71 in 13aca2e

vineyard_shared_mem="4Gi",