Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FIFO Compaction with TTL #2480

Closed
wants to merge 11 commits into from
Closed

Commits on Jun 23, 2017

  1. FIFO Compaction with TTL.

    - Added a new TableProperty, creation_time, to keep track of when the
      SST file is created.
    - Creation_time:
      - On Flush: Set to the time of flush.
      - On Compaction: Set to the max creation_time of all the files
        involved in the compaction.
    - Added a new TTL option to FIFO compaction options.
    sagar0 committed Jun 23, 2017
    Configuration menu
    Copy the full SHA
    353da0e View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    2d87ed4 View commit details
    Browse the repository at this point in the history

Commits on Jun 25, 2017

  1. Cleaned up FIFO compaction with TTL logic, and added tests.

    - Added unit tests.
    - Split PickCompaction into PickTTLCompaction and PickSizeCompaction.
    - Fixed a bug in BuildTable where time wasn't being passed to
      NewTableBuilder.
    sagar0 committed Jun 25, 2017
    Configuration menu
    Copy the full SHA
    5d8f184 View commit details
    Browse the repository at this point in the history
  2. Added creation_time to the tables built during repair and recovery.

    Also handled the return status of GetCurrentTime, so that a garbage
    value is not used when return statu is not ok. time is set to 0 instead.
    sagar0 committed Jun 25, 2017
    Configuration menu
    Copy the full SHA
    3a90501 View commit details
    Browse the repository at this point in the history
  3. Add ttl param to the end in CompactionOptionsFIFO constructor.

    with a default value of 0.
    sagar0 committed Jun 25, 2017
    Configuration menu
    Copy the full SHA
    20514a9 View commit details
    Browse the repository at this point in the history

Commits on Jun 26, 2017

  1. Configuration menu
    Copy the full SHA
    ee95191 View commit details
    Browse the repository at this point in the history
  2. Return MaxInputFileCreationTime irrespective of compaction.

    Also moved GetTotalFilesSize into anonymouse namespace from
    FIFOCompactionPicker class.
    sagar0 committed Jun 26, 2017
    Configuration menu
    Copy the full SHA
    e6fc920 View commit details
    Browse the repository at this point in the history
  3. Move log message to the right place.

    Since there is a potential chance that we could discard the files picked
    in the main loop in PickTTLCompaction, move the log message to the right
    place, to be right before creating the Compaction object.
    sagar0 committed Jun 26, 2017
    Configuration menu
    Copy the full SHA
    7c096ef View commit details
    Browse the repository at this point in the history
  4. Support FIFO-Compaction-with-TTL only with unlimited open files.

    Support FIFO-compaction-with-TTL only when max_open_files=-1, as the
    creation_time embedded in every file's table properties need to be
    consulted to figure out the files that need to be deleted. table_reader
    embedded in a FileDescriptor could potentially get deleted if
    max_open_files is not set to -1.
    
    We could initialize the table_reader again to get the table properties
    again, but it would involve a performance penalty due to reading new
    blocks from disk. We could potentially support it in the future, but may
    be not in the first version.
    sagar0 committed Jun 26, 2017
    Configuration menu
    Copy the full SHA
    a87140a View commit details
    Browse the repository at this point in the history

Commits on Jun 27, 2017

  1. Add tests to check the compatiblity of FIFO-with-TTL with other options.

    1. It is only supported with max_open_files = -1.
    2. It is only supported with Block based table format.
    sagar0 committed Jun 27, 2017
    Configuration menu
    Copy the full SHA
    05bf810 View commit details
    Browse the repository at this point in the history
  2. Pick only continguous files for deletion based on TTL.

    Addressing review comments:
    1. Pick only continguous files for deletion in FIFO-with-TTL.
    2. Add a check to make sure that GetTableProperties() does not return a
    null pointer, before accessing fields fruther.
    sagar0 committed Jun 27, 2017
    Configuration menu
    Copy the full SHA
    12a5eda View commit details
    Browse the repository at this point in the history