Skip to content

Optimize cleaner run time #6815

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Jun 16, 2025
Merged

Optimize cleaner run time #6815

merged 3 commits into from
Jun 16, 2025

Conversation

@alexqyle
Copy link
Contributor

What this PR does:

Two optimization being made for cleaner:

  • Use non-block way to push cleaner jobs to active user channel and delete user channel. So that long run time deletion job would not block cleaning job in the next cycle.
  • Introduced concurrent way to do DeletePrefix. So it is faster to delete directory with large number of files. Tested with 1k files in one directory. Sequential deletion took 17s and Concurrent deletion took 1.3s

Which issue(s) this PR fixes:

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

SungJin1212
SungJin1212 reviewed Jun 13, 2025
View reviewed changes
danielblando
danielblando reviewed Jun 13, 2025
View reviewed changes
@danielblando
Copy link
Contributor

We can add an ENHANCEMENT in the changelog. Good to know what is being deployed in a new rev

yeya24
yeya24 reviewed Jun 14, 2025
View reviewed changes
yeya24
yeya24 approved these changes Jun 14, 2025
View reviewed changes
Copy link
Contributor

@yeya24 yeya24 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. We can update changelog as @danielblando mentioned

@alexqyle
added metric to record failure pushing to job channel. updated CHANGE…
9705f8b 
…LOG. put more logs with timing values

Signed-off-by: Alex Le <[email protected]>
@pull-request-size pull-request-size bot added size/L and removed size/M labels Jun 16, 2025
@alexqyle
Copy link
Contributor Author

We can add an ENHANCEMENT in the changelog. Good to know what is being deployed in a new rev

Updated CHANGELOG

@danielblando danielblando merged commit ffecf3b into cortexproject:master Jun 16, 2025
17 checks passed
@alexqyle alexqyle deleted the cleaner-optimization branch June 16, 2025 18:48
bogdan-st pushed a commit to bogdan-st/cortex that referenced this pull request Jun 30, 2025
@alexqyle @bogdan-st
Optimize cleaner run time (cortexproject#6815)
5cb75dc 
* optimize cleaner run time

Signed-off-by: Alex Le <[email protected]>

* make old bucket.DeletePrefix concurrent

Signed-off-by: Alex Le <[email protected]>

* added metric to record failure pushing to job channel. updated CHANGELOG. put more logs with timing values

Signed-off-by: Alex Le <[email protected]>

---------

Signed-off-by: Alex Le <[email protected]>
@CharlieTLe CharlieTLe mentioned this pull request Jul 26, 2025
Closed
@CharlieTLe CharlieTLe mentioned this pull request Aug 10, 2025
Closed
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@danielblando danielblando danielblando approved these changes

@SungJin1212 SungJin1212 SungJin1212 approved these changes

@yeya24 yeya24 yeya24 approved these changes

Assignees

No one assigned

Labels

component/compactor size/L type/performance

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@alexqyle @danielblando @SungJin1212 @yeya24