Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BugFix] upgrade jemalloc from 5.2.1 to 5.3.0 #26326

Merged
merged 1 commit into from
Jun 30, 2023

Conversation

trueeyu
Copy link
Contributor

@trueeyu trueeyu commented Jun 30, 2023

Fixes #issue

Thread 912 (Thread 0x7f50197ff700 (LWP 3944)):
#0  0x00007f501ad7854d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f501ad73e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2  0x00007f501ad73d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x0000000006059a27 in pthread_mutex_lock_impl (mutex=<optimized out>) at src/bthread/mutex.cpp:564
#4  pthread_mutex_lock (__mutex=0x7f5019a032a8) at src/bthread/mutex.cpp:818
#5  pthread_mutex_lock (__mutex=0x7f5019a032a8) at src/bthread/mutex.cpp:817
#6  0x000000000697819f in malloc_mutex_lock_final (mutex=0x7f5019a03268) at include/jemalloc/internal/mutex.h:155
#7  je_malloc_mutex_lock_slow (mutex=mutex@entry=0x7f5019a03268) at src/mutex.c:85
#8  0x000000000696f8c8 in malloc_mutex_lock (mutex=0x7f5019a03268, tsdn=0x7f50197f9040) at include/jemalloc/internal/mutex.h:221
#9  je_extents_evict (tsdn=tsdn@entry=0x7f50197f9040, arena=arena@entry=0x7f5019a00980, r_extent_hooks=r_extent_hooks@entry=0x7f50197f84d8, extents=extents@entry=0x7f5019a03268, npages_min=npages_min@entry=18533) at src/extent.c:572
#10 0x0000000006936004 in arena_stash_decayed (decay_extents=<synthetic pointer>, npages_decay_max=492, npages_limit=<optimized out>, extents=0x7f5019a03268, r_extent_hooks=0x7f50197f84d8, arena=0x7f5019a00980, tsdn=0x7f50197f9040) at src/arena.c:833
#11 arena_decay_to_limit (tsdn=tsdn@entry=0x7f50197f9040, arena=arena@entry=0x7f5019a00980, decay=decay@entry=0x7f5019a06ca0, extents=extents@entry=0x7f5019a03268, all=all@entry=false, npages_limit=<optimized out>, npages_decay_max=<optimized out>, is_background_thread=<optimized out>) at src/arena.c:934
#12 0x00000000069363fb in arena_decay_try_purge (is_background_thread=<optimized out>, npages_limit=<optimized out>, current_npages=<optimized out>, extents=<optimized out>, decay=<optimized out>, arena=<optimized out>, tsdn=<optimized out>) at src/arena.c:722
#13 arena_decay_try_purge (is_background_thread=<optimized out>, npages_limit=<optimized out>, current_npages=<optimized out>, extents=<optimized out>, decay=<optimized out>, arena=<optimized out>, tsdn=<optimized out>) at src/arena.c:611
#14 arena_maybe_decay (is_background_thread=<optimized out>, extents=<optimized out>, decay=<optimized out>, arena=<optimized out>, tsdn=<optimized out>) at src/arena.c:762
#15 arena_maybe_decay (tsdn=<optimized out>, arena=0x7f5019a00980, decay=0x7f5019a06ca0, extents=0x7f5019a03268, is_background_thread=<optimized out>) at src/arena.c:714
#16 0x00000000069389ab in arena_decay_impl (all=false, is_background_thread=true, extents=0x7f5019a03268, decay=0x7f5019a06ca0, arena=0x7f5019a00980, tsdn=0x7f50197f9040) at src/arena.c:964
#17 arena_decay_dirty (all=false, is_background_thread=true, arena=0x7f5019a00980, tsdn=0x7f50197f9040) at src/arena.c:985
#18 je_arena_decay (tsdn=0x7f50197f9040, arena=0x7f5019a00980, is_background_thread=true, all=<optimized out>) at src/arena.c:998
#19 0x00000000069403e9 in background_work_sleep_once (ind=0, info=<optimized out>, tsdn=<optimized out>) at src/background_thread.c:295
#20 background_thread0_work (tsd=<optimized out>) at src/background_thread.c:452
#21 background_work (ind=<optimized out>, tsd=<optimized out>) at src/background_thread.c:490
#22 background_thread_entry () at src/background_thread.c:522
#23 0x00007f501ad71ea5 in start_thread () from /lib64/libpthread.so.0
#24 0x00007f501a38cb0d in clone () from /lib64/libc.so.6

upgrade jemalloc from 5.2.1 to 5.3.0 to fix the deadlock:

jemalloc/jemalloc#2019

Redis also encountered this problem

redis/redis#12115

What type of PR is this:

  • BugFix
  • Feature
  • Enhancement
  • Refactor
  • UT
  • Doc
  • Tool

Checklist:

  • I have added test cases for my bug fix or my new feature
  • This pr will affect users' behaviors
  • This pr needs user documentation (for new or modified features or behaviors)
    • I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

  • I have checked the version labels which the pr will be auto-backported to the target branch
    • 3.1
    • 3.0
    • 2.5
    • 2.4

@trueeyu trueeyu merged commit 33b1342 into StarRocks:main Jun 30, 2023
33 checks passed
@wanpengfei-git
Copy link
Collaborator

@Mergifyio backport branch-3.1

@github-actions github-actions bot removed the 3.1 label Jun 30, 2023
@wanpengfei-git
Copy link
Collaborator

@Mergifyio backport branch-3.0

@github-actions github-actions bot removed the 3.0 label Jun 30, 2023
@mergify
Copy link
Contributor

mergify bot commented Jun 30, 2023

backport branch-3.1

✅ Backports have been created

@mergify
Copy link
Contributor

mergify bot commented Jun 30, 2023

backport branch-3.0

✅ Backports have been created

mergify bot pushed a commit that referenced this pull request Jun 30, 2023
Fixes #issue

```
Thread 912 (Thread 0x7f50197ff700 (LWP 3944)):
#0  0x00007f501ad7854d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f501ad73e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2  0x00007f501ad73d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x0000000006059a27 in pthread_mutex_lock_impl (mutex=<optimized out>) at src/bthread/mutex.cpp:564
#4  pthread_mutex_lock (__mutex=0x7f5019a032a8) at src/bthread/mutex.cpp:818
#5  pthread_mutex_lock (__mutex=0x7f5019a032a8) at src/bthread/mutex.cpp:817
#6  0x000000000697819f in malloc_mutex_lock_final (mutex=0x7f5019a03268) at include/jemalloc/internal/mutex.h:155
#7  je_malloc_mutex_lock_slow (mutex=mutex@entry=0x7f5019a03268) at src/mutex.c:85
#8  0x000000000696f8c8 in malloc_mutex_lock (mutex=0x7f5019a03268, tsdn=0x7f50197f9040) at include/jemalloc/internal/mutex.h:221
#9  je_extents_evict (tsdn=tsdn@entry=0x7f50197f9040, arena=arena@entry=0x7f5019a00980, r_extent_hooks=r_extent_hooks@entry=0x7f50197f84d8, extents=extents@entry=0x7f5019a03268, npages_min=npages_min@entry=18533) at src/extent.c:572
#10 0x0000000006936004 in arena_stash_decayed (decay_extents=<synthetic pointer>, npages_decay_max=492, npages_limit=<optimized out>, extents=0x7f5019a03268, r_extent_hooks=0x7f50197f84d8, arena=0x7f5019a00980, tsdn=0x7f50197f9040) at src/arena.c:833
#11 arena_decay_to_limit (tsdn=tsdn@entry=0x7f50197f9040, arena=arena@entry=0x7f5019a00980, decay=decay@entry=0x7f5019a06ca0, extents=extents@entry=0x7f5019a03268, all=all@entry=false, npages_limit=<optimized out>, npages_decay_max=<optimized out>, is_background_thread=<optimized out>) at src/arena.c:934
#12 0x00000000069363fb in arena_decay_try_purge (is_background_thread=<optimized out>, npages_limit=<optimized out>, current_npages=<optimized out>, extents=<optimized out>, decay=<optimized out>, arena=<optimized out>, tsdn=<optimized out>) at src/arena.c:722
#13 arena_decay_try_purge (is_background_thread=<optimized out>, npages_limit=<optimized out>, current_npages=<optimized out>, extents=<optimized out>, decay=<optimized out>, arena=<optimized out>, tsdn=<optimized out>) at src/arena.c:611
#14 arena_maybe_decay (is_background_thread=<optimized out>, extents=<optimized out>, decay=<optimized out>, arena=<optimized out>, tsdn=<optimized out>) at src/arena.c:762
#15 arena_maybe_decay (tsdn=<optimized out>, arena=0x7f5019a00980, decay=0x7f5019a06ca0, extents=0x7f5019a03268, is_background_thread=<optimized out>) at src/arena.c:714
#16 0x00000000069389ab in arena_decay_impl (all=false, is_background_thread=true, extents=0x7f5019a03268, decay=0x7f5019a06ca0, arena=0x7f5019a00980, tsdn=0x7f50197f9040) at src/arena.c:964
#17 arena_decay_dirty (all=false, is_background_thread=true, arena=0x7f5019a00980, tsdn=0x7f50197f9040) at src/arena.c:985
#18 je_arena_decay (tsdn=0x7f50197f9040, arena=0x7f5019a00980, is_background_thread=true, all=<optimized out>) at src/arena.c:998
#19 0x00000000069403e9 in background_work_sleep_once (ind=0, info=<optimized out>, tsdn=<optimized out>) at src/background_thread.c:295
#20 background_thread0_work (tsd=<optimized out>) at src/background_thread.c:452
#21 background_work (ind=<optimized out>, tsd=<optimized out>) at src/background_thread.c:490
#22 background_thread_entry () at src/background_thread.c:522
#23 0x00007f501ad71ea5 in start_thread () from /lib64/libpthread.so.0
#24 0x00007f501a38cb0d in clone () from /lib64/libc.so.6
```

upgrade jemalloc from 5.2.1 to 5.3.0 to fix the deadlock:

jemalloc/jemalloc#2019

Redis also encountered this problem

redis/redis#12115

Signed-off-by: trueeyu <[email protected]>
(cherry picked from commit 33b1342)
mergify bot pushed a commit that referenced this pull request Jun 30, 2023
Fixes #issue

```
Thread 912 (Thread 0x7f50197ff700 (LWP 3944)):
#0  0x00007f501ad7854d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f501ad73e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2  0x00007f501ad73d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x0000000006059a27 in pthread_mutex_lock_impl (mutex=<optimized out>) at src/bthread/mutex.cpp:564
#4  pthread_mutex_lock (__mutex=0x7f5019a032a8) at src/bthread/mutex.cpp:818
#5  pthread_mutex_lock (__mutex=0x7f5019a032a8) at src/bthread/mutex.cpp:817
#6  0x000000000697819f in malloc_mutex_lock_final (mutex=0x7f5019a03268) at include/jemalloc/internal/mutex.h:155
#7  je_malloc_mutex_lock_slow (mutex=mutex@entry=0x7f5019a03268) at src/mutex.c:85
#8  0x000000000696f8c8 in malloc_mutex_lock (mutex=0x7f5019a03268, tsdn=0x7f50197f9040) at include/jemalloc/internal/mutex.h:221
#9  je_extents_evict (tsdn=tsdn@entry=0x7f50197f9040, arena=arena@entry=0x7f5019a00980, r_extent_hooks=r_extent_hooks@entry=0x7f50197f84d8, extents=extents@entry=0x7f5019a03268, npages_min=npages_min@entry=18533) at src/extent.c:572
#10 0x0000000006936004 in arena_stash_decayed (decay_extents=<synthetic pointer>, npages_decay_max=492, npages_limit=<optimized out>, extents=0x7f5019a03268, r_extent_hooks=0x7f50197f84d8, arena=0x7f5019a00980, tsdn=0x7f50197f9040) at src/arena.c:833
#11 arena_decay_to_limit (tsdn=tsdn@entry=0x7f50197f9040, arena=arena@entry=0x7f5019a00980, decay=decay@entry=0x7f5019a06ca0, extents=extents@entry=0x7f5019a03268, all=all@entry=false, npages_limit=<optimized out>, npages_decay_max=<optimized out>, is_background_thread=<optimized out>) at src/arena.c:934
#12 0x00000000069363fb in arena_decay_try_purge (is_background_thread=<optimized out>, npages_limit=<optimized out>, current_npages=<optimized out>, extents=<optimized out>, decay=<optimized out>, arena=<optimized out>, tsdn=<optimized out>) at src/arena.c:722
#13 arena_decay_try_purge (is_background_thread=<optimized out>, npages_limit=<optimized out>, current_npages=<optimized out>, extents=<optimized out>, decay=<optimized out>, arena=<optimized out>, tsdn=<optimized out>) at src/arena.c:611
#14 arena_maybe_decay (is_background_thread=<optimized out>, extents=<optimized out>, decay=<optimized out>, arena=<optimized out>, tsdn=<optimized out>) at src/arena.c:762
#15 arena_maybe_decay (tsdn=<optimized out>, arena=0x7f5019a00980, decay=0x7f5019a06ca0, extents=0x7f5019a03268, is_background_thread=<optimized out>) at src/arena.c:714
#16 0x00000000069389ab in arena_decay_impl (all=false, is_background_thread=true, extents=0x7f5019a03268, decay=0x7f5019a06ca0, arena=0x7f5019a00980, tsdn=0x7f50197f9040) at src/arena.c:964
#17 arena_decay_dirty (all=false, is_background_thread=true, arena=0x7f5019a00980, tsdn=0x7f50197f9040) at src/arena.c:985
#18 je_arena_decay (tsdn=0x7f50197f9040, arena=0x7f5019a00980, is_background_thread=true, all=<optimized out>) at src/arena.c:998
#19 0x00000000069403e9 in background_work_sleep_once (ind=0, info=<optimized out>, tsdn=<optimized out>) at src/background_thread.c:295
#20 background_thread0_work (tsd=<optimized out>) at src/background_thread.c:452
#21 background_work (ind=<optimized out>, tsd=<optimized out>) at src/background_thread.c:490
#22 background_thread_entry () at src/background_thread.c:522
#23 0x00007f501ad71ea5 in start_thread () from /lib64/libpthread.so.0
#24 0x00007f501a38cb0d in clone () from /lib64/libc.so.6
```

upgrade jemalloc from 5.2.1 to 5.3.0 to fix the deadlock:

jemalloc/jemalloc#2019

Redis also encountered this problem

redis/redis#12115

Signed-off-by: trueeyu <[email protected]>
(cherry picked from commit 33b1342)
wanpengfei-git pushed a commit that referenced this pull request Jun 30, 2023
Fixes #issue

```
Thread 912 (Thread 0x7f50197ff700 (LWP 3944)):
#0  0x00007f501ad7854d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f501ad73e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2  0x00007f501ad73d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x0000000006059a27 in pthread_mutex_lock_impl (mutex=<optimized out>) at src/bthread/mutex.cpp:564
#4  pthread_mutex_lock (__mutex=0x7f5019a032a8) at src/bthread/mutex.cpp:818
#5  pthread_mutex_lock (__mutex=0x7f5019a032a8) at src/bthread/mutex.cpp:817
#6  0x000000000697819f in malloc_mutex_lock_final (mutex=0x7f5019a03268) at include/jemalloc/internal/mutex.h:155
#7  je_malloc_mutex_lock_slow (mutex=mutex@entry=0x7f5019a03268) at src/mutex.c:85
#8  0x000000000696f8c8 in malloc_mutex_lock (mutex=0x7f5019a03268, tsdn=0x7f50197f9040) at include/jemalloc/internal/mutex.h:221
#9  je_extents_evict (tsdn=tsdn@entry=0x7f50197f9040, arena=arena@entry=0x7f5019a00980, r_extent_hooks=r_extent_hooks@entry=0x7f50197f84d8, extents=extents@entry=0x7f5019a03268, npages_min=npages_min@entry=18533) at src/extent.c:572
#10 0x0000000006936004 in arena_stash_decayed (decay_extents=<synthetic pointer>, npages_decay_max=492, npages_limit=<optimized out>, extents=0x7f5019a03268, r_extent_hooks=0x7f50197f84d8, arena=0x7f5019a00980, tsdn=0x7f50197f9040) at src/arena.c:833
#11 arena_decay_to_limit (tsdn=tsdn@entry=0x7f50197f9040, arena=arena@entry=0x7f5019a00980, decay=decay@entry=0x7f5019a06ca0, extents=extents@entry=0x7f5019a03268, all=all@entry=false, npages_limit=<optimized out>, npages_decay_max=<optimized out>, is_background_thread=<optimized out>) at src/arena.c:934
#12 0x00000000069363fb in arena_decay_try_purge (is_background_thread=<optimized out>, npages_limit=<optimized out>, current_npages=<optimized out>, extents=<optimized out>, decay=<optimized out>, arena=<optimized out>, tsdn=<optimized out>) at src/arena.c:722
#13 arena_decay_try_purge (is_background_thread=<optimized out>, npages_limit=<optimized out>, current_npages=<optimized out>, extents=<optimized out>, decay=<optimized out>, arena=<optimized out>, tsdn=<optimized out>) at src/arena.c:611
#14 arena_maybe_decay (is_background_thread=<optimized out>, extents=<optimized out>, decay=<optimized out>, arena=<optimized out>, tsdn=<optimized out>) at src/arena.c:762
#15 arena_maybe_decay (tsdn=<optimized out>, arena=0x7f5019a00980, decay=0x7f5019a06ca0, extents=0x7f5019a03268, is_background_thread=<optimized out>) at src/arena.c:714
#16 0x00000000069389ab in arena_decay_impl (all=false, is_background_thread=true, extents=0x7f5019a03268, decay=0x7f5019a06ca0, arena=0x7f5019a00980, tsdn=0x7f50197f9040) at src/arena.c:964
#17 arena_decay_dirty (all=false, is_background_thread=true, arena=0x7f5019a00980, tsdn=0x7f50197f9040) at src/arena.c:985
#18 je_arena_decay (tsdn=0x7f50197f9040, arena=0x7f5019a00980, is_background_thread=true, all=<optimized out>) at src/arena.c:998
#19 0x00000000069403e9 in background_work_sleep_once (ind=0, info=<optimized out>, tsdn=<optimized out>) at src/background_thread.c:295
#20 background_thread0_work (tsd=<optimized out>) at src/background_thread.c:452
#21 background_work (ind=<optimized out>, tsd=<optimized out>) at src/background_thread.c:490
#22 background_thread_entry () at src/background_thread.c:522
#23 0x00007f501ad71ea5 in start_thread () from /lib64/libpthread.so.0
#24 0x00007f501a38cb0d in clone () from /lib64/libc.so.6
```

upgrade jemalloc from 5.2.1 to 5.3.0 to fix the deadlock:

jemalloc/jemalloc#2019

Redis also encountered this problem

redis/redis#12115

Signed-off-by: trueeyu <[email protected]>
(cherry picked from commit 33b1342)
wanpengfei-git pushed a commit that referenced this pull request Jun 30, 2023
Fixes #issue

```
Thread 912 (Thread 0x7f50197ff700 (LWP 3944)):
#0  0x00007f501ad7854d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f501ad73e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2  0x00007f501ad73d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x0000000006059a27 in pthread_mutex_lock_impl (mutex=<optimized out>) at src/bthread/mutex.cpp:564
#4  pthread_mutex_lock (__mutex=0x7f5019a032a8) at src/bthread/mutex.cpp:818
#5  pthread_mutex_lock (__mutex=0x7f5019a032a8) at src/bthread/mutex.cpp:817
#6  0x000000000697819f in malloc_mutex_lock_final (mutex=0x7f5019a03268) at include/jemalloc/internal/mutex.h:155
#7  je_malloc_mutex_lock_slow (mutex=mutex@entry=0x7f5019a03268) at src/mutex.c:85
#8  0x000000000696f8c8 in malloc_mutex_lock (mutex=0x7f5019a03268, tsdn=0x7f50197f9040) at include/jemalloc/internal/mutex.h:221
#9  je_extents_evict (tsdn=tsdn@entry=0x7f50197f9040, arena=arena@entry=0x7f5019a00980, r_extent_hooks=r_extent_hooks@entry=0x7f50197f84d8, extents=extents@entry=0x7f5019a03268, npages_min=npages_min@entry=18533) at src/extent.c:572
#10 0x0000000006936004 in arena_stash_decayed (decay_extents=<synthetic pointer>, npages_decay_max=492, npages_limit=<optimized out>, extents=0x7f5019a03268, r_extent_hooks=0x7f50197f84d8, arena=0x7f5019a00980, tsdn=0x7f50197f9040) at src/arena.c:833
#11 arena_decay_to_limit (tsdn=tsdn@entry=0x7f50197f9040, arena=arena@entry=0x7f5019a00980, decay=decay@entry=0x7f5019a06ca0, extents=extents@entry=0x7f5019a03268, all=all@entry=false, npages_limit=<optimized out>, npages_decay_max=<optimized out>, is_background_thread=<optimized out>) at src/arena.c:934
#12 0x00000000069363fb in arena_decay_try_purge (is_background_thread=<optimized out>, npages_limit=<optimized out>, current_npages=<optimized out>, extents=<optimized out>, decay=<optimized out>, arena=<optimized out>, tsdn=<optimized out>) at src/arena.c:722
#13 arena_decay_try_purge (is_background_thread=<optimized out>, npages_limit=<optimized out>, current_npages=<optimized out>, extents=<optimized out>, decay=<optimized out>, arena=<optimized out>, tsdn=<optimized out>) at src/arena.c:611
#14 arena_maybe_decay (is_background_thread=<optimized out>, extents=<optimized out>, decay=<optimized out>, arena=<optimized out>, tsdn=<optimized out>) at src/arena.c:762
#15 arena_maybe_decay (tsdn=<optimized out>, arena=0x7f5019a00980, decay=0x7f5019a06ca0, extents=0x7f5019a03268, is_background_thread=<optimized out>) at src/arena.c:714
#16 0x00000000069389ab in arena_decay_impl (all=false, is_background_thread=true, extents=0x7f5019a03268, decay=0x7f5019a06ca0, arena=0x7f5019a00980, tsdn=0x7f50197f9040) at src/arena.c:964
#17 arena_decay_dirty (all=false, is_background_thread=true, arena=0x7f5019a00980, tsdn=0x7f50197f9040) at src/arena.c:985
#18 je_arena_decay (tsdn=0x7f50197f9040, arena=0x7f5019a00980, is_background_thread=true, all=<optimized out>) at src/arena.c:998
#19 0x00000000069403e9 in background_work_sleep_once (ind=0, info=<optimized out>, tsdn=<optimized out>) at src/background_thread.c:295
#20 background_thread0_work (tsd=<optimized out>) at src/background_thread.c:452
#21 background_work (ind=<optimized out>, tsd=<optimized out>) at src/background_thread.c:490
#22 background_thread_entry () at src/background_thread.c:522
#23 0x00007f501ad71ea5 in start_thread () from /lib64/libpthread.so.0
#24 0x00007f501a38cb0d in clone () from /lib64/libc.so.6
```

upgrade jemalloc from 5.2.1 to 5.3.0 to fix the deadlock:

jemalloc/jemalloc#2019

Redis also encountered this problem

redis/redis#12115

Signed-off-by: trueeyu <[email protected]>
(cherry picked from commit 33b1342)
JackeyLee007 pushed a commit to JackeyLee007/starrocks that referenced this pull request Jul 1, 2023
Fixes #issue

```
Thread 912 (Thread 0x7f50197ff700 (LWP 3944)):
#0  0x00007f501ad7854d in __lll_lock_wait () from /lib64/libpthread.so.0
StarRocks#1  0x00007f501ad73e9b in _L_lock_883 () from /lib64/libpthread.so.0
StarRocks#2  0x00007f501ad73d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
StarRocks#3  0x0000000006059a27 in pthread_mutex_lock_impl (mutex=<optimized out>) at src/bthread/mutex.cpp:564
StarRocks#4  pthread_mutex_lock (__mutex=0x7f5019a032a8) at src/bthread/mutex.cpp:818
StarRocks#5  pthread_mutex_lock (__mutex=0x7f5019a032a8) at src/bthread/mutex.cpp:817
StarRocks#6  0x000000000697819f in malloc_mutex_lock_final (mutex=0x7f5019a03268) at include/jemalloc/internal/mutex.h:155
StarRocks#7  je_malloc_mutex_lock_slow (mutex=mutex@entry=0x7f5019a03268) at src/mutex.c:85
StarRocks#8  0x000000000696f8c8 in malloc_mutex_lock (mutex=0x7f5019a03268, tsdn=0x7f50197f9040) at include/jemalloc/internal/mutex.h:221
StarRocks#9  je_extents_evict (tsdn=tsdn@entry=0x7f50197f9040, arena=arena@entry=0x7f5019a00980, r_extent_hooks=r_extent_hooks@entry=0x7f50197f84d8, extents=extents@entry=0x7f5019a03268, npages_min=npages_min@entry=18533) at src/extent.c:572
StarRocks#10 0x0000000006936004 in arena_stash_decayed (decay_extents=<synthetic pointer>, npages_decay_max=492, npages_limit=<optimized out>, extents=0x7f5019a03268, r_extent_hooks=0x7f50197f84d8, arena=0x7f5019a00980, tsdn=0x7f50197f9040) at src/arena.c:833
StarRocks#11 arena_decay_to_limit (tsdn=tsdn@entry=0x7f50197f9040, arena=arena@entry=0x7f5019a00980, decay=decay@entry=0x7f5019a06ca0, extents=extents@entry=0x7f5019a03268, all=all@entry=false, npages_limit=<optimized out>, npages_decay_max=<optimized out>, is_background_thread=<optimized out>) at src/arena.c:934
StarRocks#12 0x00000000069363fb in arena_decay_try_purge (is_background_thread=<optimized out>, npages_limit=<optimized out>, current_npages=<optimized out>, extents=<optimized out>, decay=<optimized out>, arena=<optimized out>, tsdn=<optimized out>) at src/arena.c:722
StarRocks#13 arena_decay_try_purge (is_background_thread=<optimized out>, npages_limit=<optimized out>, current_npages=<optimized out>, extents=<optimized out>, decay=<optimized out>, arena=<optimized out>, tsdn=<optimized out>) at src/arena.c:611
StarRocks#14 arena_maybe_decay (is_background_thread=<optimized out>, extents=<optimized out>, decay=<optimized out>, arena=<optimized out>, tsdn=<optimized out>) at src/arena.c:762
StarRocks#15 arena_maybe_decay (tsdn=<optimized out>, arena=0x7f5019a00980, decay=0x7f5019a06ca0, extents=0x7f5019a03268, is_background_thread=<optimized out>) at src/arena.c:714
StarRocks#16 0x00000000069389ab in arena_decay_impl (all=false, is_background_thread=true, extents=0x7f5019a03268, decay=0x7f5019a06ca0, arena=0x7f5019a00980, tsdn=0x7f50197f9040) at src/arena.c:964
StarRocks#17 arena_decay_dirty (all=false, is_background_thread=true, arena=0x7f5019a00980, tsdn=0x7f50197f9040) at src/arena.c:985
StarRocks#18 je_arena_decay (tsdn=0x7f50197f9040, arena=0x7f5019a00980, is_background_thread=true, all=<optimized out>) at src/arena.c:998
StarRocks#19 0x00000000069403e9 in background_work_sleep_once (ind=0, info=<optimized out>, tsdn=<optimized out>) at src/background_thread.c:295
StarRocks#20 background_thread0_work (tsd=<optimized out>) at src/background_thread.c:452
StarRocks#21 background_work (ind=<optimized out>, tsd=<optimized out>) at src/background_thread.c:490
StarRocks#22 background_thread_entry () at src/background_thread.c:522
StarRocks#23 0x00007f501ad71ea5 in start_thread () from /lib64/libpthread.so.0
StarRocks#24 0x00007f501a38cb0d in clone () from /lib64/libc.so.6
```

upgrade jemalloc from 5.2.1 to 5.3.0 to fix the deadlock:

jemalloc/jemalloc#2019

Redis also encountered this problem

redis/redis#12115

Signed-off-by: trueeyu <[email protected]>
JackeyLee007 pushed a commit to JackeyLee007/starrocks that referenced this pull request Jul 1, 2023
Fixes #issue

```
Thread 912 (Thread 0x7f50197ff700 (LWP 3944)):
#0  0x00007f501ad7854d in __lll_lock_wait () from /lib64/libpthread.so.0
StarRocks#1  0x00007f501ad73e9b in _L_lock_883 () from /lib64/libpthread.so.0
StarRocks#2  0x00007f501ad73d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
StarRocks#3  0x0000000006059a27 in pthread_mutex_lock_impl (mutex=<optimized out>) at src/bthread/mutex.cpp:564
StarRocks#4  pthread_mutex_lock (__mutex=0x7f5019a032a8) at src/bthread/mutex.cpp:818
StarRocks#5  pthread_mutex_lock (__mutex=0x7f5019a032a8) at src/bthread/mutex.cpp:817
StarRocks#6  0x000000000697819f in malloc_mutex_lock_final (mutex=0x7f5019a03268) at include/jemalloc/internal/mutex.h:155
StarRocks#7  je_malloc_mutex_lock_slow (mutex=mutex@entry=0x7f5019a03268) at src/mutex.c:85
StarRocks#8  0x000000000696f8c8 in malloc_mutex_lock (mutex=0x7f5019a03268, tsdn=0x7f50197f9040) at include/jemalloc/internal/mutex.h:221
StarRocks#9  je_extents_evict (tsdn=tsdn@entry=0x7f50197f9040, arena=arena@entry=0x7f5019a00980, r_extent_hooks=r_extent_hooks@entry=0x7f50197f84d8, extents=extents@entry=0x7f5019a03268, npages_min=npages_min@entry=18533) at src/extent.c:572
StarRocks#10 0x0000000006936004 in arena_stash_decayed (decay_extents=<synthetic pointer>, npages_decay_max=492, npages_limit=<optimized out>, extents=0x7f5019a03268, r_extent_hooks=0x7f50197f84d8, arena=0x7f5019a00980, tsdn=0x7f50197f9040) at src/arena.c:833
StarRocks#11 arena_decay_to_limit (tsdn=tsdn@entry=0x7f50197f9040, arena=arena@entry=0x7f5019a00980, decay=decay@entry=0x7f5019a06ca0, extents=extents@entry=0x7f5019a03268, all=all@entry=false, npages_limit=<optimized out>, npages_decay_max=<optimized out>, is_background_thread=<optimized out>) at src/arena.c:934
StarRocks#12 0x00000000069363fb in arena_decay_try_purge (is_background_thread=<optimized out>, npages_limit=<optimized out>, current_npages=<optimized out>, extents=<optimized out>, decay=<optimized out>, arena=<optimized out>, tsdn=<optimized out>) at src/arena.c:722
StarRocks#13 arena_decay_try_purge (is_background_thread=<optimized out>, npages_limit=<optimized out>, current_npages=<optimized out>, extents=<optimized out>, decay=<optimized out>, arena=<optimized out>, tsdn=<optimized out>) at src/arena.c:611
StarRocks#14 arena_maybe_decay (is_background_thread=<optimized out>, extents=<optimized out>, decay=<optimized out>, arena=<optimized out>, tsdn=<optimized out>) at src/arena.c:762
StarRocks#15 arena_maybe_decay (tsdn=<optimized out>, arena=0x7f5019a00980, decay=0x7f5019a06ca0, extents=0x7f5019a03268, is_background_thread=<optimized out>) at src/arena.c:714
StarRocks#16 0x00000000069389ab in arena_decay_impl (all=false, is_background_thread=true, extents=0x7f5019a03268, decay=0x7f5019a06ca0, arena=0x7f5019a00980, tsdn=0x7f50197f9040) at src/arena.c:964
StarRocks#17 arena_decay_dirty (all=false, is_background_thread=true, arena=0x7f5019a00980, tsdn=0x7f50197f9040) at src/arena.c:985
StarRocks#18 je_arena_decay (tsdn=0x7f50197f9040, arena=0x7f5019a00980, is_background_thread=true, all=<optimized out>) at src/arena.c:998
StarRocks#19 0x00000000069403e9 in background_work_sleep_once (ind=0, info=<optimized out>, tsdn=<optimized out>) at src/background_thread.c:295
StarRocks#20 background_thread0_work (tsd=<optimized out>) at src/background_thread.c:452
StarRocks#21 background_work (ind=<optimized out>, tsd=<optimized out>) at src/background_thread.c:490
StarRocks#22 background_thread_entry () at src/background_thread.c:522
StarRocks#23 0x00007f501ad71ea5 in start_thread () from /lib64/libpthread.so.0
StarRocks#24 0x00007f501a38cb0d in clone () from /lib64/libc.so.6
```

upgrade jemalloc from 5.2.1 to 5.3.0 to fix the deadlock:

jemalloc/jemalloc#2019

Redis also encountered this problem

redis/redis#12115

Signed-off-by: trueeyu <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants