Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The jemalloc's background thread is disabled #9812

Closed
CalvinNeo opened this issue Jan 22, 2025 · 0 comments · Fixed by #9813
Closed

The jemalloc's background thread is disabled #9812

CalvinNeo opened this issue Jan 22, 2025 · 0 comments · Fixed by #9813
Assignees
Labels
affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. affects-8.1 This bug affects the 8.1.x(LTS) versions. affects-8.5 This bug affects the 8.5.x(LTS) versions. component/storage severity/major type/bug The issue is confirmed as a bug.

Comments

@CalvinNeo
Copy link
Member

CalvinNeo commented Jan 22, 2025

Bug Report

Please answer these questions before submitting your issue. Thanks!

The current implementation is

    auto * malloc_conf = getenv("MALLOC_CONF");

    if (!malloc_conf && !old_b)
    {
        // enable background thread
    }

In previous version, there is not MALLOC_CONF set, so the background thread is enabled by default.

However, after pingcap/tiup@bac3c2e, TiFlash is bootstrapped with a given MALLOC_CONF env string which does not specify background_thread at all. So the background thread will not be enabled.

If the background thread is not enabled, dirty pages could not be purged in the expected time, if using jemalloc 5.2.1.

There are mainly several ways to dirty pages in jemalloc:

  • Manual purge called by mallctl
  • The value of dirty_decay_ms is changed
  • A counter named decay_ticker will decrease from 1000, on alloc/dealloc events. Once it goes to 0, it would check if the current epoch should be advanced according to deadline. If so, a purge will trigger.
  • The background thread is enabled, and it would trigger purge in a given interval.

The problem is, if there is no frequent alloc/dealloc events after one dealloc, the decay_ticker will not be decreased. thus the epoch is not checked, and the memory is not released.

So, we'd better enable background_thread again.

1. Minimal reproduce step (Required)

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiFlash version? (Required)

@CalvinNeo CalvinNeo added the type/bug The issue is confirmed as a bug. label Jan 22, 2025
@CalvinNeo CalvinNeo added affects-8.5 This bug affects the 8.5.x(LTS) versions. affects-8.1 This bug affects the 8.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-6.1 This bug affects the 6.1.x(LTS) versions. and removed may-affects-5.4 may-affects-6.1 may-affects-6.5 may-affects-7.1 may-affects-7.5 may-affects-8.1 may-affects-8.5 labels Jan 22, 2025
@CalvinNeo CalvinNeo self-assigned this Jan 22, 2025
@ti-chi-bot ti-chi-bot bot closed this as completed in 9832885 Jan 22, 2025
ti-chi-bot bot pushed a commit that referenced this issue Jan 23, 2025
ref #9722, close #9812

Re-enable jemalloc's background thread by default

Signed-off-by: ti-chi-bot <[email protected]>
Signed-off-by: JaySon-Huang <[email protected]>

Co-authored-by: Calvin Neo <[email protected]>
Co-authored-by: JaySon-Huang <[email protected]>
Co-authored-by: JaySon <[email protected]>
ti-chi-bot bot pushed a commit that referenced this issue Feb 11, 2025
ref #9722, close #9812

Re-enable jemalloc's background thread by default

Signed-off-by: ti-chi-bot <[email protected]>
Signed-off-by: JaySon-Huang <[email protected]>

Co-authored-by: Calvin Neo <[email protected]>
Co-authored-by: JaySon-Huang <[email protected]>
Co-authored-by: JaySon <[email protected]>
ti-chi-bot bot pushed a commit that referenced this issue Feb 17, 2025
ref #9722, close #9812

Re-enable jemalloc's background thread by default

Signed-off-by: ti-chi-bot <[email protected]>
Signed-off-by: JaySon-Huang <[email protected]>

Co-authored-by: Calvin Neo <[email protected]>
Co-authored-by: JaySon-Huang <[email protected]>
Co-authored-by: JaySon <[email protected]>
ti-chi-bot bot pushed a commit that referenced this issue Feb 27, 2025
ref #9722, close #9812

Re-enable jemalloc's background thread by default

Signed-off-by: ti-chi-bot <[email protected]>

Co-authored-by: Calvin Neo <[email protected]>
Co-authored-by: JaySon <[email protected]>
ti-chi-bot bot pushed a commit that referenced this issue Feb 27, 2025
ref #9722, close #9812

Re-enable jemalloc's background thread by default

Signed-off-by: ti-chi-bot <[email protected]>
Signed-off-by: JaySon-Huang <[email protected]>

Co-authored-by: Calvin Neo <[email protected]>
Co-authored-by: JaySon-Huang <[email protected]>
Co-authored-by: JaySon <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. affects-8.1 This bug affects the 8.1.x(LTS) versions. affects-8.5 This bug affects the 8.5.x(LTS) versions. component/storage severity/major type/bug The issue is confirmed as a bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant