Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory-mapped Utility for Ankerl Hashmap #222

Merged
merged 8 commits into from
Jul 20, 2023

Conversation

weiliw-amz
Copy link
Contributor

@weiliw-amz weiliw-amz commented May 18, 2023

Issue #, if available:
N/A

Description of changes:
Added memory-mapped utility for ankerl hashmap.

Ankerl hashmap is extremely fast at retrieval, comparing to usual B-tree based key-value stores, but without the utility of memory-map to support out-of-core lookups.

This PR adds such functionality to make ankerl hashmap a good substitution for the usual key-value stores, if user needs a memory-mappable read-only single-machine key-value store, and cares more about latency.

For 2.5Billion key-value pairs, to randomly retrieve 500 entries, the time cost:
LMDB: 5.2 ms
mmap ankerl (C/C++): 0.05ms
mmap ankerl (Python binding): 0.8ms

Usage (Python-binding): Please refer to the added unittests.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@weiliw-amz weiliw-amz self-assigned this May 18, 2023
@weiliw-amz weiliw-amz requested a review from rofuyu May 18, 2023 05:10
@weiliw-amz weiliw-amz force-pushed the ankerl-mmap branch 4 times, most recently from e92f5e2 to 221cec0 Compare June 16, 2023 04:18
@weiliw-amz weiliw-amz marked this pull request as ready for review June 16, 2023 04:25
@weiliw-amz weiliw-amz requested a review from hallogameboy June 16, 2023 04:25
@OctoberChang OctoberChang self-requested a review June 16, 2023 18:45
@weiliw-amz weiliw-amz requested a review from jiong-zhang June 17, 2023 05:05
@weiliw-amz weiliw-amz force-pushed the ankerl-mmap branch 3 times, most recently from beb9e1c to faa4e23 Compare June 19, 2023 03:52
@weiliw-amz weiliw-amz merged commit 2283da8 into amzn:mainline Jul 20, 2023
@weiliw-amz weiliw-amz deleted the ankerl-mmap branch July 20, 2023 20:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants