Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(mercury): use SHA1 as key in LRU cache #743

Merged
merged 3 commits into from
Dec 13, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 10 additions & 4 deletions mercury/src/internal/pack/cache.rs
Original file line number Diff line number Diff line change
Expand Up @@ -26,14 +26,20 @@ pub trait _Cache {
fn clear(&self);
}

impl lru_mem::HeapSize for SHA1 {
fn heap_size(&self) -> usize {
0
}
}

pub struct Caches {
map_offset: DashMap<usize, SHA1>, // offset to hash
hash_set: DashSet<SHA1>, // item in the cache
// dropping large lru cache will take a long time on Windows without multi-thread IO
// because "multi-thread IO" clone Arc<CacheObject>, so it won't be dropped in the main thread,
// and `CacheObjects` will be killed by OS after Process ends abnormally
// Solution: use `mimalloc`
lru_cache: Mutex<LruCache<String, ArcWrapper<CacheObject>>>, // *lru_cache require the key to implement lru::MemSize trait, so didn't use SHA1 as the key*
lru_cache: Mutex<LruCache<SHA1, ArcWrapper<CacheObject>>>,
mem_size: Option<usize>,
tmp_path: PathBuf,
pool: Arc<ThreadPool>,
Expand All @@ -44,7 +50,7 @@ impl Caches {
/// only get object from memory, not from tmp file
fn try_get(&self, hash: SHA1) -> Option<Arc<CacheObject>> {
let mut map = self.lru_cache.lock().unwrap();
map.get(&hash.to_string()).map(|x| x.data.clone())
map.get(&hash).map(|x| x.data.clone())
}

/// !IMPORTANT: because of the process of pack, the file must be written / be writing before, so it won't be dead lock
Expand Down Expand Up @@ -72,7 +78,7 @@ impl Caches {
Some(self.pool.clone()),
);
x.set_store_path(Caches::generate_temp_path(&self.tmp_path, hash));
let _ = map.insert(hash.to_string(), x); // handle the error
let _ = map.insert(hash, x); // handle the error
Ok(obj)
}

Expand Down Expand Up @@ -154,7 +160,7 @@ impl _Cache for Caches {
Some(self.pool.clone()),
);
a_obj.set_store_path(Caches::generate_temp_path(&self.tmp_path, hash));
let _ = map.insert(hash.to_string(), a_obj);
let _ = map.insert(hash, a_obj);
}
//order maters as for reading in 'get_by_offset()'
self.hash_set.insert(hash);
Expand Down
Loading