Barebones python/S3 implementation #10
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dirkgr
requested changes
Mar 1, 2024
| for _ in 0..number_of_u32 { | ||
| bits.push(AtomicU32::new(0)); | ||
| } | ||
| let mut bits = { |
| @@ -0,0 +1,169 @@ | |||
| """ Quick'n'dirty mapping of bff for python before I can make it pure-rust | |||
There was a problem hiding this comment.
This is fine, if you want to put it into a scripts/ directory, but I feel like this is not really necessary. You could replace the entire thing with a few lines of bash:
aws cp "s3://bucket/input_dir/*.jsonl.gz" ./input_files
bff --bloom-filter-file filter.bff --bloom-filter-size <something> --expected-ngram-count <something> --output-directory ./output_files ./input_files/*.json.gz
rm -r input_files
aws cp ./output_files s3://bucket/output_dir
added 8 commits
March 5, 2024 15:15
… to describe some new features
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added
bff_v0.pywhich is a simple python script to:Also modified the
main.rsto use the faster init for bloom filters