Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support headerless format in getFrameParameters and decompress #217

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

swehner
Copy link

@swehner swehner commented Mar 16, 2024

Adding support for magic-less format in a few functions:

  • Add optional paramater format to get_frame_paramaters and switch to using getFrameHeader_advanced internally
  • Use getFrameHeader_advanced in decompress instead of getFrameContentSize

Testing done:

$ export ZSTD_SLOW_TESTS=1  

$ for i in cext rust cffi; do echo "=== PYTHON_ZSTANDARD_IMPORT_POLICY=$i python setup.py test"; PYTHON_ZSTANDARD_IMPORT_POLICY=$i python setup.py test; done 2>&1 | grep "\(==\|OK\|Ran\|failed\|ERROR\)"
=== PYTHON_ZSTANDARD_IMPORT_POLICY=cext python setup.py test
Ran 292 tests in 13.942s
OK
=== PYTHON_ZSTANDARD_IMPORT_POLICY=rust python setup.py test
Ran 292 tests in 15.512s
OK
=== PYTHON_ZSTANDARD_IMPORT_POLICY=cffi python setup.py test
Ran 292 tests in 20.358s
OK (skipped=26)

@@ -767,7 +767,7 @@ def test_data_equivalence(self, original, threads, use_dict):
dctx = zstd.ZstdDecompressor(**kwargs)

for i, frame in enumerate(result):
self.assertEqual(dctx.decompress(frame), original[i])
self.assertEqual(dctx.decompress(frame.tobytes()), original[i])
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the rust backend the test was erroring out becase frame wasn't a buffer but a zstandard.backend_rust.BufferSegment - not sure what the best way to fix it is...
This does work, so I added the conversion

@swehner swehner changed the title Add format to get frame params Support headerless format in getFrameParameters and decompress Mar 16, 2024
Comment on lines +4 to +5
[target.aarch64-apple-darwin]
rustflags = ["-C", "link-arg=-undefined", "-C", "link-arg=dynamic_lookup"]
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For compiling on M1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant