implement louvain algorithm #183

acezen · 2021-03-10T00:30:32Z

What do these changes do?

implement a distributed louvain algorithm

Related issue number

Fixes #182

acezen · 2021-03-10T02:15:32Z

community quality test

compare the modularity of result partition with https://github.com/taynaud/python-louvain

dataset: p2p (62586 nodes, 147892 edges)

graphscope louvain: 0.571986 find community num: 103
python-louvain: 0.578780 find community num: 110

com-youtube dataset (1134890 nodes, 2987624 edges), community num: 8,385

graphscope louvain: 0.717838, find community_num: 4980
python-louvain: 0.7204, find community num: 5340

analytical_engine/CMakeLists.txt

analytical_engine/core/app/pregel/pregel_compute_context.h

acezen · 2021-03-18T02:23:29Z

performance test

dataset: com-friendster ( 65,608,366 nodes, 1,806,067,135 edges)
tolerance=0, min_progress=1

parallel GraphScope louvain (8 worker)

time: 7 min (remove the atomic operation, but bring in a three-level vector buffer to process messages to prevent thread conflict.)

grape louvain( 8 worker, 128 frag)

time: 5 min

siyuan0322

Since it's will be open sourced, please revise the code style, such as function naming (Camel Style), class encapsulation, variable naming, comments, fixme, non-necessary header and so on.

Maybe you could add a python test case for louvain, mainly to demonstrate it's usage, and the format of result.
Reference: https://google.github.io/styleguide/cppguide.html

analytical_engine/apps/pregel/louvain/auxiliary.h

analytical_engine/apps/pregel/louvain/louvain_app_base.h

analytical_engine/core/app/pregel/pregel_vertex.h

acezen · 2021-03-19T05:37:42Z

Since it's will be open sourced, please revise the code style, such as function naming (Camel Style), class encapsulation, variable naming, comments, fixme, non-necessary header and so on.

Maybe you could add a python test case for louvain, mainly to demonstrate it's usage, and the format of result.
Reference: https://google.github.io/styleguide/cppguide.html

Hi, @siyuan0322, thanks for reviewing, could you quote some bad code examples as a hint, that would be much help.

codecov-io · 2021-03-19T08:11:47Z

Codecov Report

Merging #183 (5a19fcc) into main (7a8f3f6) will decrease coverage by 0.55%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main     #183      +/-   ##
==========================================
- Coverage   78.91%   78.35%   -0.56%     
==========================================
  Files          52       53       +1     
  Lines        4767     4986     +219     
==========================================
+ Hits         3762     3907     +145     
- Misses       1005     1079      +74

Impacted Files	Coverage Δ
python/graphscope/analytical/app/__init__.py	`100.00% <100.00%> (ø)`
python/graphscope/analytical/app/louvain.py	`100.00% <100.00%> (ø)`
...n/graphscope/deploy/kubernetes/resource_builder.py	`57.65% <0.00%> (-6.06%)`	⬇️
python/graphscope/framework/graph_utils.py	`84.05% <0.00%> (-4.79%)`	⬇️
python/graphscope/framework/errors.py	`92.64% <0.00%> (-1.48%)`	⬇️
python/graphscope/client/session.py	`72.30% <0.00%> (-1.41%)`	⬇️
python/graphscope/interactive/query.py	`89.28% <0.00%> (-0.72%)`	⬇️
python/graphscope/config.py	`100.00% <0.00%> (ø)`
python/graphscope/client/rpc.py	`83.70% <0.00%> (ø)`
python/graphscope/dataset/ldbc.py	`100.00% <0.00%> (ø)`
... and 10 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7a8f3f6...5a19fcc. Read the comment docs.

add louvain context and vertex add louvain test update update fix fix fix update update update compiled update runnable runable update update add louvain to client update adapt to ArrowProjectedFragment sync community after comm ready vd_t be oid_t use SendToFragment runnable of p2p n = 4 fix use parallel worker fix sendp2p and aggregate fix pallell workers update update cleanup parallel processing messages revise parallel process messages clean up code remove halted aggregator edge weight as double format update update fix update revise code style remove unnecessary virtual declare update

analytical_engine/apps/pregel/louvain/louvain.h

lidongze0629

LGTM.

analytical_engine/apps/pregel/louvain/louvain_vertex.h

sighingnow · 2021-03-25T02:15:05Z

analytical_engine/core/app/pregel/pregel_compute_context.h

@@ -89,8 +98,7 @@ class PregelComputeContext {
        message_manager_->SyncStateOnOuterVertex<fragment_t, MD_T>(*fragment_,
                                                                   v, value);
      } else {
-        messages_in_[v].emplace_back(value);
-        has_messages_ = true;
+        messages_out_[v].emplace_back(value);


@lidongze0629 Any comment on this part?

analytical_engine/core/app/pregel/pregel_compute_context.h

sighingnow

LGTM. I'm also ccing @lidongze0629 for a quick review on the pregel app base related changes.

…libaba#183

…sistent with #183 (#3472) ## What do these changes do? This change try to fix the louvain algorithm that the result is not consistent with #183 ## Related issue number Close #3457

acezen changed the title ~~implement louvain algorithm~~ [WIP] implement louvain algorithm Mar 10, 2021

sighingnow reviewed Mar 11, 2021

View reviewed changes

analytical_engine/CMakeLists.txt Outdated Show resolved Hide resolved

analytical_engine/core/app/pregel/pregel_compute_context.h Outdated Show resolved Hide resolved

acezen changed the title ~~[WIP] implement louvain algorithm~~ implement louvain algorithm Mar 18, 2021

acezen marked this pull request as ready for review March 18, 2021 12:21

acezen requested review from lidongze0629, yecol, siyuan0322 and luoxiaojian March 18, 2021 13:05

siyuan0322 reviewed Mar 19, 2021

View reviewed changes

analytical_engine/apps/pregel/louvain/auxiliary.h Outdated Show resolved Hide resolved

analytical_engine/apps/pregel/louvain/louvain_app_base.h Outdated Show resolved Hide resolved

analytical_engine/core/app/pregel/pregel_vertex.h Outdated Show resolved Hide resolved

acezen force-pushed the zwb/louvain branch 2 times, most recently from 4684fc5 to 31aaf25 Compare March 22, 2021 01:49

acezen added 2 commits March 22, 2021 09:50

format

316cd58

acezen force-pushed the zwb/louvain branch from 31aaf25 to 316cd58 Compare March 22, 2021 01:51

siyuan0322 reviewed Mar 22, 2021

View reviewed changes

analytical_engine/apps/pregel/louvain/louvain.h Outdated Show resolved Hide resolved

lidongze0629 reviewed Mar 22, 2021

View reviewed changes

analytical_engine/apps/pregel/louvain/louvain_vertex.h Outdated Show resolved Hide resolved

analytical_engine/apps/pregel/louvain/louvain_vertex.h Outdated Show resolved Hide resolved

analytical_engine/apps/pregel/louvain/louvain_vertex.h Outdated Show resolved Hide resolved

acezen added 10 commits March 22, 2021 14:41

just for perf test

e63b578

add comments

cd8e6c7

revise

d13ad71

update

6e62769

update

64dcfaa

update

20f2889

update

06bd01e

update

c1d07c4

revise comments

105f42b

add directed check before run app

0c08d46

acezen added 5 commits March 24, 2021 16:06

fix

4ee2f82

remove unnessary header

6ff2314

fix

df5d108

fix

f75e836

update

20d151e

sighingnow reviewed Mar 25, 2021

View reviewed changes

analytical_engine/core/app/pregel/pregel_compute_context.h Outdated Show resolved Hide resolved

sighingnow approved these changes Mar 25, 2021

View reviewed changes

sighingnow requested a review from lidongze0629 March 25, 2021 02:16

delete not use code

5a19fcc

lidongze0629 approved these changes Mar 25, 2021

View reviewed changes

acezen mentioned this pull request Mar 25, 2021

Release plan for v0.3.0 #128

Closed

18 tasks

sighingnow merged commit 4f5db65 into alibaba:main Mar 25, 2021

acezen deleted the zwb/louvain branch September 3, 2021 10:52

JackyYangPassion mentioned this pull request Jan 3, 2024

[BUG] Run louvain algo on k8s error #3457

Closed

acezen added a commit to acezen/GraphScope that referenced this pull request Jan 8, 2024

fix: fix the louvain algorithm that the result is not consistent with a…

22c4050

…libaba#183

acezen mentioned this pull request Jan 8, 2024

fix(analytical): fix the louvain algorithm that the result is not consistent with #183 #3472

Merged

acezen added a commit to acezen/GraphScope that referenced this pull request Jan 15, 2024

fix: fix the louvain algorithm that the result is not consistent with a…

64dda49

…libaba#183

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

implement louvain algorithm #183

implement louvain algorithm #183

acezen commented Mar 10, 2021

acezen commented Mar 10, 2021 •

edited

Loading

acezen commented Mar 18, 2021 •

edited

Loading

siyuan0322 left a comment

acezen commented Mar 19, 2021 •

edited

Loading

codecov-io commented Mar 19, 2021 •

edited

Loading

lidongze0629 left a comment

sighingnow Mar 25, 2021

lidongze0629 Mar 25, 2021

sighingnow left a comment

implement louvain algorithm #183

implement louvain algorithm #183

Conversation

acezen commented Mar 10, 2021

What do these changes do?

Related issue number

acezen commented Mar 10, 2021 • edited Loading

community quality test

dataset: p2p (62586 nodes, 147892 edges)

com-youtube dataset (1134890 nodes, 2987624 edges), community num: 8,385

acezen commented Mar 18, 2021 • edited Loading

performance test

parallel GraphScope louvain (8 worker)

grape louvain( 8 worker, 128 frag)

siyuan0322 left a comment

Choose a reason for hiding this comment

acezen commented Mar 19, 2021 • edited Loading

codecov-io commented Mar 19, 2021 • edited Loading

Codecov Report

lidongze0629 left a comment

Choose a reason for hiding this comment

sighingnow Mar 25, 2021

Choose a reason for hiding this comment

lidongze0629 Mar 25, 2021

Choose a reason for hiding this comment

sighingnow left a comment

Choose a reason for hiding this comment

acezen commented Mar 10, 2021 •

edited

Loading

acezen commented Mar 18, 2021 •

edited

Loading

acezen commented Mar 19, 2021 •

edited

Loading

codecov-io commented Mar 19, 2021 •

edited

Loading