Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(query): record state_offsets to reduce the memory leak of agg #17285

Merged
merged 8 commits into from
Jan 16, 2025

Conversation

sundy-li
Copy link
Member

@sundy-li sundy-li commented Jan 14, 2025

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

  1. Record state_offsets to reduce the memory leak of agg
  2. Always use jemalloc in hashtable instead of MMAP allocator
  3. Use StackHashSet

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

This change is Reviewable

@github-actions github-actions bot added the pr-refactor this PR changes the code base without new features or bugfix label Jan 14, 2025
Copy link

what-the-diff bot commented Jan 14, 2025

PR Summary

  • Memory Allocator Switch
    This update changes our memory management system in various components of our software. By switching from the MmapAllocator to the JEAllocator, we improve how we allocate and manage the memory usage in the system, which can lead to increased performance and efficiency.

  • Struct Modification
    We have updated several key components or 'structs' to use the new memory allocator as their default. This means these components should also benefit from the performance improvements.

  • Payload Refactoring
    We have made significant changes in a component named payload. This involved renaming state_rows to state_offsets, improving how we manage and check states, and introduced better logging for troubleshooting. These adjustments increase efficiency and provide better system state tracking.

@sundy-li sundy-li requested a review from zhang2014 January 14, 2025 14:37
@sundy-li sundy-li marked this pull request as draft January 15, 2025 12:34
@sundy-li sundy-li marked this pull request as ready for review January 16, 2025 06:12
@sundy-li sundy-li enabled auto-merge January 16, 2025 07:18
@sundy-li sundy-li added this pull request to the merge queue Jan 16, 2025
Merged via the queue into databendlabs:main with commit c89e4a3 Jan 16, 2025
74 checks passed
@sundy-li sundy-li deleted the agg-leak2 branch January 16, 2025 08:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pr-refactor this PR changes the code base without new features or bugfix
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants