-
Notifications
You must be signed in to change notification settings - Fork 6.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
FIFO Compaction with TTL #2480
FIFO Compaction with TTL #2480
Commits on Jun 23, 2017
-
- Added a new TableProperty, creation_time, to keep track of when the SST file is created. - Creation_time: - On Flush: Set to the time of flush. - On Compaction: Set to the max creation_time of all the files involved in the compaction. - Added a new TTL option to FIFO compaction options.
Configuration menu - View commit details
-
Copy full SHA for 353da0e - Browse repository at this point
Copy the full SHA 353da0eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 2d87ed4 - Browse repository at this point
Copy the full SHA 2d87ed4View commit details
Commits on Jun 25, 2017
-
Cleaned up FIFO compaction with TTL logic, and added tests.
- Added unit tests. - Split PickCompaction into PickTTLCompaction and PickSizeCompaction. - Fixed a bug in BuildTable where time wasn't being passed to NewTableBuilder.
Configuration menu - View commit details
-
Copy full SHA for 5d8f184 - Browse repository at this point
Copy the full SHA 5d8f184View commit details -
Added creation_time to the tables built during repair and recovery.
Also handled the return status of GetCurrentTime, so that a garbage value is not used when return statu is not ok. time is set to 0 instead.
Configuration menu - View commit details
-
Copy full SHA for 3a90501 - Browse repository at this point
Copy the full SHA 3a90501View commit details -
Add ttl param to the end in CompactionOptionsFIFO constructor.
with a default value of 0.
Configuration menu - View commit details
-
Copy full SHA for 20514a9 - Browse repository at this point
Copy the full SHA 20514a9View commit details
Commits on Jun 26, 2017
-
Configuration menu - View commit details
-
Copy full SHA for ee95191 - Browse repository at this point
Copy the full SHA ee95191View commit details -
Return MaxInputFileCreationTime irrespective of compaction.
Also moved GetTotalFilesSize into anonymouse namespace from FIFOCompactionPicker class.
Configuration menu - View commit details
-
Copy full SHA for e6fc920 - Browse repository at this point
Copy the full SHA e6fc920View commit details -
Move log message to the right place.
Since there is a potential chance that we could discard the files picked in the main loop in PickTTLCompaction, move the log message to the right place, to be right before creating the Compaction object.
Configuration menu - View commit details
-
Copy full SHA for 7c096ef - Browse repository at this point
Copy the full SHA 7c096efView commit details -
Support FIFO-Compaction-with-TTL only with unlimited open files.
Support FIFO-compaction-with-TTL only when max_open_files=-1, as the creation_time embedded in every file's table properties need to be consulted to figure out the files that need to be deleted. table_reader embedded in a FileDescriptor could potentially get deleted if max_open_files is not set to -1. We could initialize the table_reader again to get the table properties again, but it would involve a performance penalty due to reading new blocks from disk. We could potentially support it in the future, but may be not in the first version.
Configuration menu - View commit details
-
Copy full SHA for a87140a - Browse repository at this point
Copy the full SHA a87140aView commit details
Commits on Jun 27, 2017
-
Add tests to check the compatiblity of FIFO-with-TTL with other options.
1. It is only supported with max_open_files = -1. 2. It is only supported with Block based table format.
Configuration menu - View commit details
-
Copy full SHA for 05bf810 - Browse repository at this point
Copy the full SHA 05bf810View commit details -
Pick only continguous files for deletion based on TTL.
Addressing review comments: 1. Pick only continguous files for deletion in FIFO-with-TTL. 2. Add a check to make sure that GetTableProperties() does not return a null pointer, before accessing fields fruther.
Configuration menu - View commit details
-
Copy full SHA for 12a5eda - Browse repository at this point
Copy the full SHA 12a5edaView commit details