Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Bulk Loading/Writing for Chunks #535

Open
wants to merge 70 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
d5f3797
feat: add linear chunk format with zstd lib dependency
Jan 24, 2025
65f1904
fixing a typo
Jan 24, 2025
9517837
Merge remote-tracking branch 'upstream/master' into linear_chunk_format
Jan 24, 2025
a02003f
refactor: rename file handling methods for clarity and more lint fixes
Jan 24, 2025
9f85aa6
refactor: improve clarity on validation methods
Jan 25, 2025
28e1222
fix: invert version check in LinearFile header validation
Jan 25, 2025
c9f6a7e
fix: remove leading dot from file path formatting in LinearChunkFormat
Jan 25, 2025
a1cbe33
refactor: improve error handling and method naming in LinearFile oper…
Jan 25, 2025
f2d51fd
fix: ensure file is flushed after writing in LinearFile operations
Jan 25, 2025
2f8c589
fix: lints and warnings
Jan 25, 2025
63a73a9
fix: more lints
Jan 25, 2025
8f8ef47
refactor: streamline LinearFile structure and improve header management
Jan 25, 2025
21c46e4
refactor: change chunks_headers to use Box for improved memory manage…
Jan 25, 2025
9308469
refactor: update path parameter types to PathBuf for consistency and …
Jan 25, 2025
48a2883
refactor: update path parameter types to Path avoiding uknown changes…
Jan 25, 2025
7563a81
fix: avoid using the same path for different tests
Jan 26, 2025
34604ea
fix: fix some bytes sizes for the header based on the spec
Jan 26, 2025
c14ee64
refactor: enable file truncation when opening for writing and simplif…
Jan 26, 2025
f5405b2
fix: removing IoSlices for windows issues.
Jan 26, 2025
0ca59aa
fix: clippy lints
Jan 26, 2025
fc543cf
refactor: improve incompatible files cheking, some documentation and …
Jan 27, 2025
3612b71
feat: add file_format to ChunkConfig and implement format handling in…
Jan 27, 2025
9340bbc
Merge branch 'master' into linear_chunk_format
Mili-ssm Jan 27, 2025
3404700
feat: add error logging for invalid file headers and data size checks…
Jan 28, 2025
1953773
feat: implement file locking mechanism for concurrent chunk read/writ…
Jan 29, 2025
c89bbc7
fix: bug at desserializing the chunks headers.
Jan 29, 2025
5c8f799
feat: (Bulk API) refactor chunk reading and writing to handle multipl…
Jan 30, 2025
293886e
refactor: streamline chunk reading and writing methods, improve chunk…
Jan 31, 2025
7c89480
feat: enhance chunk reading and writing to support bulk and improve c…
Feb 1, 2025
f469867
feat: add file locking during chunk writing to ensure safe concurrent…
Feb 5, 2025
d768b3d
Merge branch 'chunk_bulk_api' into chunk_bulk_api_v2
Mili-ssm Feb 6, 2025
5782014
fix: resolve some issues with the merge
Feb 6, 2025
0b48790
fix: update chunk header size calculation in linear chunk parsing
Feb 6, 2025
4c57d85
refactor: optimize chunk processing and improve concurrency handling
Feb 6, 2025
6f0bf27
feat: implement chunk I/O management with file locking for safe concu…
Feb 9, 2025
44cd9eb
Merge remote-tracking branch 'upstream/master' into chunk_bulk_api_v2
Feb 9, 2025
ab110ae
refactor: simplify public load method and fix clippy issues
Feb 9, 2025
3616e7e
feature: add file cache for files with ongoing IO ops
Feb 10, 2025
239e2c1
refactor: update LoadedData enum variants for consistency and clarity…
Feb 10, 2025
d29e90f
refactor: restore compression enum and replace ChunkSerializingError …
Feb 10, 2025
40d70ca
refactor: enhance chunk cache cleaning logic and improve logging duri…
Feb 10, 2025
3c1d91a
refactor: optimize chunk data handling with parallel processing and i…
Feb 13, 2025
32496b0
fix: lints and logging
Feb 13, 2025
58ce0c6
fix: correct typo and import format
Feb 13, 2025
709efb3
refactor: anvil code organization and llinear chunk reading code.
Feb 13, 2025
d1acce2
Merge branch 'master' into chunk_bulk_api_v2
Mili-ssm Feb 17, 2025
5b46f40
fix logging issue and refactor some locks bindings
Feb 19, 2025
66c31f0
refactor: reduce chunk cloning, ensure drop bindings/guards
Feb 20, 2025
2ced481
Merge remote-tracking branch 'upstream/master' into chunk_bulk_api_v2
Feb 20, 2025
f3bfb48
refactor: reduce some threading overhead when not needed
Feb 20, 2025
9a5c4d7
fix: clippy lints
Feb 20, 2025
8cd5352
refactor: simplify chunk data processing in Anvil and Linear serializers
Feb 20, 2025
f77625c
Merge remote-tracking branch 'upstream/master' into chunk_bulk_api_v2
Feb 20, 2025
7855381
fix: clippy lints rust 1.84.1 to 1.85.0
Feb 20, 2025
e46956b
add file drop guard and rework some logic
kralverde Feb 21, 2025
373b4bb
refactor: improve AnvilChunkFile and AnvilChunkData serialization log…
Feb 21, 2025
4e34235
fix: typos
Feb 21, 2025
cab6493
Merge pull request #2 from kralverde/bulk_chunk
Mili-ssm Feb 21, 2025
fb271f9
feature: initial async implementation of Bulk API
Feb 22, 2025
40e4c34
fix: chunk writing with async read
Mili-ssm Feb 22, 2025
2fa6e6d
refactor: improve async handling and use asyn IO for reading files
Feb 22, 2025
002aac2
fix: sorry i delet a mod xd
Feb 22, 2025
08e0704
refactor: enhance chunk management with async locks and OneCells
Feb 22, 2025
c927364
use async methods where avaliable and stream chunks instead of collec…
kralverde Feb 23, 2025
b4563a3
Merge pull request #4 from kralverde/chunk_batching_3
Mili-ssm Feb 23, 2025
d1fdb9c
refactor: streamline chunk management by removing unnecessary mutex a…
Feb 23, 2025
9e4e24f
Split data
suprohub Feb 23, 2025
2d4e7f8
Move format related stuff from chunk
suprohub Feb 23, 2025
0e792c0
Merge pull request #5 from suprohub/bulkapi
Mili-ssm Feb 23, 2025
e362d04
Merge branch 'Pumpkin-MC:master' into chunk_bulk_api_v2
Mili-ssm Feb 24, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,8 @@ members = [
"pumpkin-protocol/",
"pumpkin-registry/",
"pumpkin-world",
"pumpkin/", "pumpkin-data",
"pumpkin/",
"pumpkin-data",
]

[workspace.package]
Expand Down Expand Up @@ -40,6 +41,7 @@ tokio = { version = "1.43", features = [
"sync",
"io-std",
"signal",
"fs",
] }

thiserror = "2.0"
Expand All @@ -48,7 +50,6 @@ bytes = "1.10"

# Concurrency/Parallelism and Synchronization
rayon = "1.10"
parking_lot = { version = "0.12", features = ["send_guard"] }
crossbeam = "0.8"

uuid = { version = "1.14", features = ["serde", "v3", "v4"] }
Expand Down
5 changes: 5 additions & 0 deletions pumpkin-world/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,9 @@ pumpkin-config = { path = "../pumpkin-config" }
pumpkin-macros = { path = "../pumpkin-macros" }
pumpkin-data = { path = "../pumpkin-data" }


async-trait.workspace = true

bytes.workspace = true


Expand All @@ -27,6 +30,8 @@ dashmap = "6.1"

num-traits = "0.2"

parking_lot = "0.12.3"

# Compression
flate2 = "1.0"
lz4 = "1.28"
Expand Down
Loading
Loading