Skip to content

Commit

Permalink
storage: Evict high priority files (#257)
Browse files Browse the repository at this point in the history
Signed-off-by: Wish <[email protected]>
  • Loading branch information
breezewish authored and JaySon-Huang committed Aug 6, 2024
1 parent 0f1bdc3 commit 9811bf1
Show file tree
Hide file tree
Showing 2 changed files with 8 additions and 4 deletions.
8 changes: 6 additions & 2 deletions dbms/src/Storages/S3/FileCache.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -406,15 +406,19 @@ bool FileCache::reserveSpaceImpl(FileType reserve_for, UInt64 size, EvictMode ev
// Distinguish cache priority according to file type. The larger the file type, the lower the priority.
// First, try to evict files which not be used recently with the same type. => Try to evict old files.
// Second, try to evict files with lower priority. => Try to evict lower priority files.
// Finally, evict files with higher priority, if space is still not sufficient. Higher priority files
// are usually smaller. If we don't evict them, it is very possible that cache is full of these higher
// priority small files and we can't effectively cache any lower-priority large files.
std::vector<FileType> FileCache::getEvictFileTypes(FileType evict_for)
{
std::vector<FileType> evict_types;
evict_types.push_back(evict_for); // First, try evict with the same file type.
constexpr auto all_file_types = magic_enum::enum_values<FileType>(); // all_file_types are sorted by enum value.
// Second, try evict from the lower proirity file type.
for (auto itr = std::rbegin(all_file_types); itr != std::rend(all_file_types) && *itr > evict_for; ++itr)
for (auto itr = std::rbegin(all_file_types); itr != std::rend(all_file_types); ++itr)
{
evict_types.push_back(*itr);
if (*itr != evict_for)
evict_types.push_back(*itr);
}
return evict_types;
}
Expand Down
4 changes: 2 additions & 2 deletions dbms/src/Storages/S3/tests/gtest_filecache.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -946,7 +946,7 @@ try
auto cache_not_contains = [&](const String & file) {
const auto all = file_cache.getAll();
for (const auto & file_seg : all)
if (file_seg->getLocalFileName().contains(file))
if (file_seg->getLocalFileName().find(file) != std::string::npos)
return false;
return true;
};
Expand Down Expand Up @@ -1003,7 +1003,7 @@ try
}
catch (Exception & e)
{
ASSERT_TRUE(e.message().contains("Cannot reserve 999 space for object"));
ASSERT_TRUE(e.message().find("Cannot reserve 999 space for object") != std::string::npos);
}
ASSERT_EQ(file_cache.getAll().size(), 0);
}
Expand Down

0 comments on commit 9811bf1

Please sign in to comment.