Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

obstore-based Store implementation #1661

Open
wants to merge 80 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
80 commits
Select commit Hold shift + click to select a range
14be826
Initial object-store implementation
kylebarron Feb 8, 2024
a492bf0
Merge branch 'v3' into kyle/object-store
d-v-b Feb 12, 2024
50b6c47
Merge branch 'v3' into kyle/object-store
jhamman Feb 27, 2024
afa79af
Update src/zarr/v3/store/object_store.py
kylebarron Feb 27, 2024
c466f9f
Merge branch 'main' into kyle/object-store
kylebarron Oct 21, 2024
c3e7296
Merge branch 'main' into kyle/object-store
kylebarron Oct 22, 2024
f5c884b
update
kylebarron Oct 22, 2024
af2a39b
Handle list streams
kylebarron Nov 1, 2024
d7cfbee
Update get
kylebarron Nov 1, 2024
cb40015
wip refactor get_partial_values
kylebarron Nov 1, 2024
619df43
Merge branch 'main' into kyle/object-store
kylebarron Nov 1, 2024
b976450
Fixes to _get_partial_values
kylebarron Nov 7, 2024
cca70d7
Merge branch 'main' into kyle/object-store
kylebarron Nov 7, 2024
f2c827d
Fix constructing prototype from get
kylebarron Nov 7, 2024
5c8903f
lint
kylebarron Nov 7, 2024
50e1dec
Merge branch 'main' into kyle/object-store
kylebarron Nov 18, 2024
8bb252e
Add docstring
kylebarron Nov 18, 2024
559eafd
Make names private
kylebarron Nov 18, 2024
5486e69
Implement eq
kylebarron Nov 18, 2024
9a05c01
Add obstore as a test dep
maxrjones Nov 18, 2024
56b7a0b
Run store tests on ObjectStore
maxrjones Nov 18, 2024
d5d0d4d
Merge pull request #1 from maxrjones/object-store-tests
kylebarron Nov 20, 2024
b38ada1
import or skip
kylebarron Nov 21, 2024
ab00b46
Bump obstore beta version
kylebarron Nov 22, 2024
9c65e4d
bump pre-commit
kylebarron Nov 22, 2024
77d7c12
Add read_only param for __init__
kylebarron Dec 5, 2024
4418426
Bump obstore
maxrjones Dec 13, 2024
7a71174
Add fixtures for object store tests
maxrjones Dec 13, 2024
e73bcc9
Cast return from __eq__ as bool
maxrjones Dec 13, 2024
c2cd6b8
Avoid recursion error on repr
maxrjones Dec 13, 2024
a95ec59
Check store type at runtime
maxrjones Dec 13, 2024
fb8b16d
Merge pull request #2 from maxrjones/update-tests
kylebarron Dec 13, 2024
ca261b1
Check if store is writable for setting or deleting objects
maxrjones Dec 13, 2024
0eb416a
Add test for object store repr
maxrjones Dec 13, 2024
247432f
Add attribute tests
maxrjones Dec 13, 2024
4b31b3c
Add get and set methods to test class
maxrjones Dec 14, 2024
d49d1ff
Raise an exeption for previously set key
maxrjones Dec 14, 2024
c2ebc8f
Update src/zarr/testing/store.py
maxrjones Dec 14, 2024
eb76698
Update _transform_list_dir to not remove all items
maxrjones Dec 15, 2024
f310260
Return bytes from GetResult
maxrjones Dec 15, 2024
86951b8
Don't raise an exception on set_if_not_exists
maxrjones Dec 15, 2024
f989884
Remove test that stores empty file
maxrjones Dec 15, 2024
40e1b25
Handle None as start or end of byte range request
maxrjones Dec 15, 2024
3aa3578
Merge pull request #3 from maxrjones/check_writable
kylebarron Dec 16, 2024
6da7976
Merge branch 'main' into object-store-update
maxrjones Jan 10, 2025
26fa37e
Use new ByteRequest syntax
maxrjones Jan 10, 2025
315e22e
Raise not implemented error on pickling
maxrjones Jan 10, 2025
264eac6
Merge pull request #4 from maxrjones/object-store-update
kylebarron Jan 13, 2025
fc93029
Bump obstore
maxrjones Jan 17, 2025
1b9f9f2
Catch allowed exceptions
maxrjones Jan 17, 2025
72c9b30
Merge pull request #5 from maxrjones/object-store
kylebarron Jan 17, 2025
0f8820d
Merge branch 'main' into kyle/object-store
kylebarron Jan 17, 2025
f1a0534
Merge branch 'main' into kyle/object-store
maxrjones Feb 3, 2025
b18de38
WIP
maxrjones Feb 4, 2025
6976738
Fix list dir
kylebarron Feb 4, 2025
65f2db4
Merge branch 'main' into kyle/object-store
kylebarron Feb 4, 2025
ebf7be1
Merge branch 'kyle/object-store' into kyle/fix-transform-list-dir
kylebarron Feb 4, 2025
72bf9f4
Merge pull request #6 from kylebarron/kyle/fix-transform-list-dir
kylebarron Feb 4, 2025
f287780
Fix failing tests
maxrjones Feb 5, 2025
08b7771
Make module structure consistent with other stores
maxrjones Feb 5, 2025
8038b94
Remove override of pytest parameterization
maxrjones Feb 5, 2025
b65d439
Use store prefix
kylebarron Feb 5, 2025
1fa0125
Remove xfail for pickle
kylebarron Feb 5, 2025
b0b9d56
Restore pickle xfail
kylebarron Feb 5, 2025
aa5bdac
Mark xfail
maxrjones Feb 5, 2025
01de03d
Mark xfails
maxrjones Feb 5, 2025
fdd7f8e
Serialization should now be supported by obstore
maxrjones Feb 5, 2025
73fa459
Update equality
maxrjones Feb 5, 2025
6af1e85
Bump obstore
maxrjones Feb 5, 2025
f4bb423
Fix serialization
maxrjones Feb 5, 2025
1f7df52
Improve equality checking
maxrjones Feb 6, 2025
fc728a4
Merge pull request #8 from maxrjones/obstore-xfails
kylebarron Feb 6, 2025
12ccb49
Update src/zarr/storage/_object.py
kylebarron Feb 6, 2025
25676a3
Update src/zarr/storage/_object.py
kylebarron Feb 6, 2025
4d15e43
update __eq__
kylebarron Feb 6, 2025
9a25057
Fix serialization
maxrjones Feb 6, 2025
9e90a02
Merge pull request #9 from maxrjones/fix-serialization
kylebarron Feb 6, 2025
6f5f960
Expand test coverage
maxrjones Feb 7, 2025
d3ce999
Merge pull request #10 from maxrjones/improve-test-coverage
kylebarron Feb 7, 2025
0fab774
Merge branch 'main' into kyle/object-store
jhamman Feb 7, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@ repos:
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.9.4
hooks:
- id: ruff
args: ["--fix", "--show-fixes"]
- id: ruff-format
- id: ruff
args: ["--fix", "--show-fixes"]
- id: ruff-format
- repo: https://github.com/codespell-project/codespell
rev: v2.4.1
hooks:
Expand All @@ -34,6 +34,7 @@ repos:
- numpy==2.1 # until https://github.com/numpy/numpy/issues/28034 is resolved
- typing_extensions
- universal-pathlib
- obstore==0.3.0-beta.8
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: update to latest obstore release when we merge

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: obstore is currently on version 0.4.0-beta.4. I'm ready to publish 0.4.0 but wanted to ensure it works with zarr-python before publishing it.

# Tests
- pytest
- repo: https://github.com/scientific-python/cookie
Expand Down
2 changes: 2 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ keywords = ["Python", "compressed", "ndimensional-arrays", "zarr"]
# User extras
remote = [
"fsspec>=2023.10.0",
"obstore==0.4.0b4",
]
gpu = [
"cupy-cuda12x",
Expand Down Expand Up @@ -211,6 +212,7 @@ dependencies = [
'universal_pathlib @ git+https://github.com/fsspec/universal_pathlib',
'typing_extensions @ git+https://github.com/python/typing_extensions',
'donfig @ git+https://github.com/pytroll/donfig',
"obstore @ git+https://github.com/developmentseed/obstore@main#subdirectory=obstore",
# test deps
'hypothesis',
'pytest',
Expand Down
2 changes: 2 additions & 0 deletions src/zarr/storage/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
from zarr.storage._local import LocalStore
from zarr.storage._logging import LoggingStore
from zarr.storage._memory import GpuMemoryStore, MemoryStore
from zarr.storage._object import ObjectStore
from zarr.storage._wrapper import WrapperStore
from zarr.storage._zip import ZipStore

Expand All @@ -17,6 +18,7 @@
"LocalStore",
"LoggingStore",
"MemoryStore",
"ObjectStore",
"StoreLike",
"StorePath",
"WrapperStore",
Expand Down
Loading