Allow reading utf-8 encoded json files #1312

nhz2 · 2022-12-27T20:05:19Z

Currently, zarr-python errors when reading a json file with non-ascii characters encoded with utf-8, however, Zarr.jl writes json files that include non-ascii characters using utf-8 encoding.
This PR will enable zarr attributes written in Zarr.jl to be read by zarr-python.

TODO:

Add unit tests and/or doctests in docstrings
Add docstrings and API docs for any new/modified user-facing classes and functions
New/modified features documented in docs/tutorial.rst
Changes documented in docs/release.rst
GitHub Actions have all passed
Test coverage is 100% (Codecov passes)

nhz2 · 2023-01-10T19:54:04Z

@MSanKeys963 @jakirkham This PR is ready for review. Also, because I am a first-time contributor, I am not able to trigger Github Actions myself.

sanketverma1704 · 2023-01-11T11:50:03Z

Thanks for sending the PR @nhz2.
@jakirkham, @joshmoore and @rabernat, please have a look.

joshmoore · 2023-01-16T13:49:21Z

Thanks for this, @nhz2! I've triggered the actions and will go through the code ASAP.

codecov · 2023-01-16T14:02:37Z

Codecov Report

Merging #1312 (2024d7f) into main (4dc6f1f) will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##              main     #1312   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           36        36           
  Lines        14744     14748    +4     
=========================================
+ Hits         14744     14748    +4

Impacted Files	Coverage Δ
zarr/meta.py	`100.00% <100.00%> (ø)`
zarr/tests/test_attrs.py	`100.00% <100.00%> (ø)`
zarr/util.py	`100.00% <100.00%> (ø)`

joshmoore · 2023-01-16T14:18:47Z

Relaunched the Windows built after:

    requests.exceptions.ChunkedEncodingError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))

joshmoore · 2023-01-16T14:51:24Z

Green 🎉 in case anyone else wants to now take a closer look. (release.rst will need its conflict handled)

For my part, reading this makes complete sense, but I do wonder what the impact on downstream code will be and therefore what types of warnings/modifications might be needed.

nhz2 · 2023-01-17T00:29:09Z

This PR should have no effect on the JSON files zarr-python writes, they will still be ASCII only with non-ASCII characters escaped. It only affects what kind of JSON files zarr-python can read.

joshmoore · 2023-01-17T09:13:42Z

Thanks, @nhz2. Guess I was more concerned whether any calling code (and/or subclasses) would need modification.

nhz2 · 2023-01-24T21:36:13Z

In terms of the effect on other code, the main change is
from

def json_loads(s: str) -> Dict[str, Any]:
    """Read JSON in a consistent way."""
    return json.loads(ensure_text(s, 'ascii'))

to

def json_loads(s: bytes) -> Dict[str, Any]:
    """Read JSON in a consistent way."""
    return json.loads(ensure_text(s, 'utf-8'))

Functions called by `json_loads`

The ensure_text is defined in numcodecs: https://github.com/zarr-developers/numcodecs/blob/c9f47a80981dbefe372c4ae7ff3938ba000f2de9/numcodecs/compat.py#L178

If a bytes is input to ensure_text, it will decode it and output a str, or fail if the bytes cannot be decoded.

json.loads takes a str and will return a python dict, list, str, int, float, True, False, or None or raise a JSONDecodeError if the input is not a valid JSON document according to https://docs.python.org/3/library/json.html

Should json_loads error if the object returned by json.loads isn't a dict? Maybe that can be added in a future PR?

Functions that call `json_loads`

consolidate_metadata:

zarr-python/zarr/convenience.py

Line 1230 in 0bf0b3b

key: json_loads(store[key])

parse_metadata:

zarr-python/zarr/meta.py

Line 108 in 0bf0b3b

meta = json_loads(s)

It is also called a few times in n5.py

ConsolidatedMetadataStore.__init__:

zarr-python/zarr/storage.py

Line 2884 in 0bf0b3b

meta = json_loads(self.store[metadata_key])

ConsolidatedMetadataStoreV3.__init__:

zarr-python/zarr/_storage/v3.py

Line 543 in 0bf0b3b

meta = json_loads(self.store[metadata_key])

All of these cases directly, or indirectly through parse_metadata, call json_loads on an object obtained from a store. Since store values are bytes, not strings, I changed the type hints to match the current usage.

joshmoore · 2023-02-02T15:11:10Z

Thanks for the analysis, @nhz2! Sounds like it should not impact external consumers.

Any objections to rolling this into 2.14, anyone?

zarr/meta.py

zarr/util.py

jakirkham · 2023-02-08T09:25:19Z

Thanks for the PR Nathan! 🙏

This is a good improvement to include.

Noting some changes in the typing to still allow str in functions (since it should still technically be valid). Though agree bytes should be included in a few places as well for clarity

Ideally we would capture any bytes-like type here (since they are all valid), but that is somewhat complicated to do in practice. Maybe there will be guidance on this in the future ( python/typing#593 )

Co-authored-by: jakirkham <[email protected]>

joshmoore · 2023-02-10T14:31:39Z

I think this has failed on the zarr-feedstock: conda-forge/zarr-feedstock#72

jakirkham · 2023-02-10T20:43:12Z

zarr/tests/test_attrs.py

+    def test_utf8_encoding(self, zarr_version):
+
+        # fixture data
+        fixture = group(store=DirectoryStore('fixture'))
+        assert fixture['utf8attrs'].attrs.asdict() == dict(foo='た')


This test is failing there

jakirkham

The issue is neither of these files were included in the Python sdist

jakirkham · 2023-02-10T20:44:59Z

fixture/utf8attrs/.zattrs

@@ -0,0 +1 @@
+{"foo": "た"}


This file is missing from the sdist on PyPI

Thanks, @jakirkham. Argh, I thought I had caught these issues but likely only if there's code to generate it them as well. Are you already working on a fix or shall I?

Feel free to go ahead. I might not get to this for a while

Forget if we already discussed this before, but maybe we can make some fixes to CI to ensure we catch these issues in PRs. Wrote up some thoughts on this in issue ( #1347 )

jakirkham · 2023-02-10T20:45:14Z

fixture/utf8attrs/.zgroup

+{
+    "zarr_format": 2
+}


Same with this one

This is a temporary fix for the larger issue of out-of-tree testing described in zarr-developers#1347, but this should allow a release of 2.14.1 which passes on conda.

This is a temporary fix for the larger issue of out-of-tree testing described in #1347, but this should allow a release of 2.14.1 which passes on conda.

see: zarr-developers#1312 see: zarr-developers#1347 see: zarr-developers#1348

see: #1312 see: #1347 see: #1348

github-actions bot added needs release notes Automatically applied to PRs which haven't added release notes and removed needs release notes Automatically applied to PRs which haven't added release notes labels Dec 27, 2022

nhz2 marked this pull request as ready for review December 27, 2022 20:12

sanketverma1704 requested a review from jakirkham December 27, 2022 21:38

nhz2 added 2 commits January 13, 2023 16:12

read utf-8 in json

33057b9

update release

5290626

Merge branch 'main' into fix-utf8-attrs

a5dcc40

nhz2 added 2 commits January 24, 2023 16:41

Merge branch 'main' into fix-utf8-attrs

acdeb6e

Merge branch 'main' into fix-utf8-attrs

010f5cb

Merge branch 'main' into fix-utf8-attrs

adc55e4

jakirkham reviewed Feb 8, 2023

View reviewed changes

zarr/meta.py Outdated Show resolved Hide resolved

jakirkham reviewed Feb 8, 2023

View reviewed changes

zarr/util.py Outdated Show resolved Hide resolved

joshmoore mentioned this pull request Feb 8, 2023

Prepare 2.14 changelog #1337

Merged

nhz2 and others added 2 commits February 8, 2023 17:25

Update zarr/util.py

207627e

Co-authored-by: jakirkham <[email protected]>

allow str

2024d7f

jakirkham approved these changes Feb 8, 2023

View reviewed changes

joshmoore merged commit 280d969 into zarr-developers:main Feb 9, 2023

jakirkham reviewed Feb 10, 2023

View reviewed changes

jakirkham mentioned this pull request Feb 10, 2023

Run Zarr CI tests out-of-tree #1347

Closed

joshmoore added a commit that referenced this pull request Feb 12, 2023

Generate fixture for #1312 if it is missing (#1348)

76fce14

This is a temporary fix for the larger issue of out-of-tree testing described in #1347, but this should allow a release of 2.14.1 which passes on conda.

jakirkham mentioned this pull request Feb 24, 2023

zarr v2.14.2 conda-forge/zarr-feedstock#74

Merged

3 tasks

joshmoore added a commit to joshmoore/zarr-python that referenced this pull request Mar 10, 2023

Don't use relative fixture path

6ddb9bd

see: zarr-developers#1312 see: zarr-developers#1347 see: zarr-developers#1348

joshmoore mentioned this pull request Mar 10, 2023

Don't use relative fixture path #1364

Merged

6 tasks

joshmoore added a commit that referenced this pull request Mar 10, 2023

Don't use relative fixture path (#1364)

6e63fe9

see: #1312 see: #1347 see: #1348

joshmoore mentioned this pull request Mar 16, 2023

One more fix for missing directories #1367

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow reading utf-8 encoded json files #1312

Allow reading utf-8 encoded json files #1312

nhz2 commented Dec 27, 2022 •

edited

Loading

nhz2 commented Jan 10, 2023

sanketverma1704 commented Jan 11, 2023

joshmoore commented Jan 16, 2023

codecov bot commented Jan 16, 2023 •

edited

Loading

joshmoore commented Jan 16, 2023

joshmoore commented Jan 16, 2023 •

edited

Loading

nhz2 commented Jan 17, 2023

joshmoore commented Jan 17, 2023

nhz2 commented Jan 24, 2023 •

edited

Loading

joshmoore commented Feb 2, 2023

jakirkham commented Feb 8, 2023

joshmoore commented Feb 10, 2023

jakirkham Feb 10, 2023

jakirkham left a comment

jakirkham Feb 10, 2023

joshmoore Feb 10, 2023

jakirkham Feb 10, 2023

jakirkham Feb 10, 2023

jakirkham Feb 10, 2023

Allow reading utf-8 encoded json files #1312

Allow reading utf-8 encoded json files #1312

Conversation

nhz2 commented Dec 27, 2022 • edited Loading

nhz2 commented Jan 10, 2023

sanketverma1704 commented Jan 11, 2023

joshmoore commented Jan 16, 2023

codecov bot commented Jan 16, 2023 • edited Loading

Codecov Report

joshmoore commented Jan 16, 2023

joshmoore commented Jan 16, 2023 • edited Loading

nhz2 commented Jan 17, 2023

joshmoore commented Jan 17, 2023

nhz2 commented Jan 24, 2023 • edited Loading

Functions called by json_loads

Functions that call json_loads

joshmoore commented Feb 2, 2023

jakirkham commented Feb 8, 2023

joshmoore commented Feb 10, 2023

jakirkham Feb 10, 2023

Choose a reason for hiding this comment

jakirkham left a comment

Choose a reason for hiding this comment

jakirkham Feb 10, 2023

Choose a reason for hiding this comment

joshmoore Feb 10, 2023

Choose a reason for hiding this comment

jakirkham Feb 10, 2023

Choose a reason for hiding this comment

jakirkham Feb 10, 2023

Choose a reason for hiding this comment

jakirkham Feb 10, 2023

Choose a reason for hiding this comment

nhz2 commented Dec 27, 2022 •

edited

Loading

codecov bot commented Jan 16, 2023 •

edited

Loading

joshmoore commented Jan 16, 2023 •

edited

Loading

nhz2 commented Jan 24, 2023 •

edited

Loading

Functions called by `json_loads`

Functions that call `json_loads`