-
Notifications
You must be signed in to change notification settings - Fork 5.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Optimize Memory Usage for MergeGlobalStatus's TopN #45727
Labels
affects-6.5
This bug affects the 6.5.x(LTS) versions.
affects-7.1
This bug affects the 7.1.x(LTS) versions.
report/customer
Customers have encountered this bug.
sig/planner
SIG: Planner
type/enhancement
The issue or PR belongs to an enhancement.
Comments
12 tasks
This was referenced Aug 10, 2023
ti-chi-bot
added
affects-7.1
This bug affects the 7.1.x(LTS) versions.
affects-6.5
This bug affects the 6.5.x(LTS) versions.
labels
Aug 11, 2023
12 tasks
This was referenced Aug 14, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
affects-6.5
This bug affects the 6.5.x(LTS) versions.
affects-7.1
This bug affects the 7.1.x(LTS) versions.
report/customer
Customers have encountered this bug.
sig/planner
SIG: Planner
type/enhancement
The issue or PR belongs to an enhancement.
Enhancement
During the MergeGlobalStatus process, the current method of waiting for all TopN values to be collected before merging them can lead to high memory consumption. This is especially problematic when dealing with large datasets or when the number of unique values is high.
The proposed enhancement involves modifying the mergeglobalstatus process to collect and process the TopN values concurrently. Rather than waiting for all TopN values to be collected, the process should start merging and processing the available TopN values as soon as they are collected. This approach would help optimize memory usage by reducing the need to hold all TopN values in memory before processing.
The text was updated successfully, but these errors were encountered: