Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve the memory usage of fragment (CSR) using delta encoding and varint #1373

Closed
sighingnow opened this issue May 12, 2023 · 0 comments · Fixed by #1372 or #1516
Closed

Improve the memory usage of fragment (CSR) using delta encoding and varint #1373

sighingnow opened this issue May 12, 2023 · 0 comments · Fixed by #1372 or #1516
Assignees
@vegetableysm
Labels
component:graph enhancement New feature or request

Comments

@sighingnow
Copy link
Member

Additional context

See also: alibaba/GraphScope#2628
@sighingnow sighingnow added enhancement New feature or request component:graph labels May 12, 2023
@sighingnow sighingnow moved this from Todo to In Progress in Vineyard Releases May 12, 2023
@sighingnow sighingnow mentioned this issue May 26, 2023
Merged
sighingnow added a commit that referenced this issue May 26, 2023
@vegetableysm @sighingnow
Implements varint + delta encoding on CSR to optimize the fragment me…
0c9c16e 
…mory usage (#1372)

Related issue number
--------------------

- Fixes #1373
- Part of alibaba/GraphScope#2628

---------

Signed-off-by: vegetableysm <[email protected]>
Signed-off-by: Tao He <[email protected]>
Co-authored-by: Tao He <[email protected]>
@github-project-automation github-project-automation bot moved this from In Progress to Done in Vineyard Releases May 26, 2023
@sighingnow sighingnow mentioned this issue May 29, 2023
Merged
sighingnow added a commit that referenced this issue May 30, 2023
@sighingnow
Batch encoding/decoding (with SIMD) of varint CSR (#1403)
2c0957b 
What do these changes do?
-------------------------

- add back the offset array for correct degree
- use v8dec/nec for varint encoding
- implements batch decoding in CompactNbr

Part of #1373

Signed-off-by: Tao He <[email protected]>
@sighingnow sighingnow mentioned this issue Aug 11, 2023
Merged
sighingnow added a commit that referenced this issue Aug 12, 2023
@sighingnow
Improve the varint encoding for edges by avoiding copy and extra memo…
6d7dfc4 
…ry (#1516)

This pull request enhances the previous varint encoding implementation by:

- use serialized, batch encoding (not slow than previous implementation, thanks to batching)
- reuse the ie list when encoding, avoid extra memory allocation and copy
- use the shrink API to adjust the size of the encoded blob writer.

After this pull request, compacting edges with varint no longer requires extra memory,
and the encoding execution time has been optimized (almost half) as well.

Fixes #1373

Signed-off-by: Tao He <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vegetableysm vegetableysm

Labels
component:graph enhancement New feature or request
Projects
Vineyard Releases
Archived in project
Milestone
No milestone
2 participants
@sighingnow @vegetableysm