feat: generate frictionless data package metadata

[dangotbanned]

Closes #629

• Adds build_datapackage.py script
• Generates initial metadata in both .yaml and .json formats

[dangotbanned]

@domoritz @dsmedia

I'm not sure how you're managing python dependencies/environment(s) for the existing scripts.
Therefore, I've left some links showing how you can run this one using inline dependencies.

The feature is quite new, but effectively it makes the script portable and doesn't require a pyproject.toml
e.g. (https://github.com/vega/altair/blob/922eac35da764d3faae928b870a2c70d0cf1435b/pyproject.toml)

[dsmedia]

Thank you, @dangotbanned. This looks very promising. The script nicely incorporates the intrinsic metadata of the dataset (i.e. what can be inferred from the data file itself) into the datapackage file. While we're at it, should we try to incorporate into the datapackage yaml/json files some of the extrinsic metadata like dataset source, dataset license, dataset column description, and dataset description? It would seem a shame to leave that information stranded in an unstructured SOURCES.md. That said, I'm also content with moving forward on this as is, and leaving the SOURCES.md work for another PR.

Fortunately, the Frictionless Data specs have room for this:

• The Frictionless Data Resource standard allows for a description, source, and license for the specific datasets, which would be nice to include. Without a source or license, the resource appears to inherit these from the (higher-level) Data Package, which could be misleading.

• Also, the Table Schema definition appears to allow for column names to include descriptions, which can be crucial to making the datasets usable. (See, for example, the column descriptions in SOURCES.md for countries.json.)

Since the datapackage files are generated by script, it would seem that the extrinsic metadata would need to be available to the script in some machine-readable format. If it required a one-time restructuring of SOURCES.md, we could probably do that pretty quickly.

[dangotbanned]

#631 (comment)

@dsmedia all sounds good to me, I'm glad you picked up on what else we could add.

I'm happy with either that being part of this PR or following it another one - if keeping things atomic is a preference here.

Right now I'm working on an adapter for flights-200k.arrow - since that format wasn't natively supported.
Was almost finished, but I could rework the script more generally to support supplementing with external metadata - which it seems we'll need for these requirements

[domoritz]

Is there a big benefit to including the yaml in addition to json? json is much more common (and the only of the two natively supported in python/js) and the readability difference is small that I would say let's only have json.

Otherwise looks great. Thanks for the script!

[dangotbanned]

Is there a big benefit to including the yaml in addition to json? json is much more common (and the only of the two natively supported in python/js) and the readability difference is small that I would say let's only have json.

@domoritz having yaml doesn't benefit me personally, just thought I'd provide the options @dsmedia mentioned in #629 (comment):

Just thinking out loud, but instead of directly maintaining the sources.md file, we could keep the dataset metadata in a json or yaml file, and generate the sources.md file from this machine-readable format.

I'm happy with just json

Otherwise looks great. Thanks for the script!

🎉

[dangotbanned]

Right now I'm working on an adapter for flights-200k.arrow - since that format wasn't natively supported. Was almost finished, ...

#631 (comment)

Was less trivial than I was expecting - no longer planning to include .arrow support in this PR

[domoritz]

I think it would be good to think through how sources.md and the data package relate. I'm not sure how feasible it is to pare the current markdown file to include the source in the package. Or we could convert the sources file to yaml and then include that in the json package. But this is beyond the scope of this pull request.

Regarding arrow. We do include it in the index (but don't include parquet) so that's inconsistent. Should we add support for the parquet file in the index and also include arrow in the data package even though we don't have column information?

[dangotbanned]

Regarding arrow. We do include it in the index (but don't include parquet) so that's inconsistent. Should we add support for the parquet file in the index and also include arrow in the data package even though we don't have column information?

@domoritz the column info I solved fairly quickly - but the framework is producing a very opaque traceback that I cannot follow at all:

Traceback

Traceback (most recent call last):
  File "C:\Users\danie\Documents\GitHub\vega-datasets\scripts\build_datapackage.py", line 364, in main
    pkg.to_json(p)
  File "C:\Users\danie\AppData\Local\uv\cache\archive-v0\Gx6PRXGoUfWB67iQwxXch\Lib\site-packages\frictionless\metadata\metadata.py", line 229, in to_json
    text = helpers.to_json(self.to_descriptor(), encoder_class=encoder_class)
                           ^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\danie\AppData\Local\uv\cache\archive-v0\Gx6PRXGoUfWB67iQwxXch\Lib\site-packages\frictionless\metadata\metadata.py", line 195, in to_descriptor
  File "C:\Users\danie\AppData\Local\uv\cache\archive-v0\Gx6PRXGoUfWB67iQwxXch\Lib\site-packages\frictionless\metadata\metadata.py", line 195, in to_descriptor
    descriptor = self.metadata_export()
                 ^^^^^^^^^^^^^^^^^^^^^^
    descriptor = self.metadata_export()
                 ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\danie\AppData\Local\uv\cache\archive-v0\Gx6PRXGoUfWB67iQwxXch\Lib\site-packages\frictionless\package\package.py", line 701, in metadata_export
  File "C:\Users\danie\AppData\Local\uv\cache\archive-v0\Gx6PRXGoUfWB67iQwxXch\Lib\site-packages\frictionless\package\package.py", line 701, in metadata_export
    descriptor = super().metadata_export()
                 ^^^^^^^^^^^^^^^^^^^^^^^^^
    descriptor = super().metadata_export()
                 ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\danie\AppData\Local\uv\cache\archive-v0\Gx6PRXGoUfWB67iQwxXch\Lib\site-packages\frictionless\metadata\metadata.py", line 440, in metadata_export
    value = [item.to_descriptor_source() for item in value]  # type: ignore
  File "C:\Users\danie\AppData\Local\uv\cache\archive-v0\Gx6PRXGoUfWB67iQwxXch\Lib\site-packages\frictionless\metadata\metadata.py", line 440, in metadata_export
    value = [item.to_descriptor_source() for item in value]  # type: ignore
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\danie\AppData\Local\uv\cache\archive-v0\Gx6PRXGoUfWB67iQwxXch\Lib\site-packages\frictionless\metadata\metadata.py", line 206, in to_descriptor_source
    descriptor = self.to_descriptor()
    descriptor = self.to_descriptor()
                 ^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\danie\AppData\Local\uv\cache\archive-v0\Gx6PRXGoUfWB67iQwxXch\Lib\site-packages\frictionless\metadata\metadata.py", line 195, in to_descriptor
  File "C:\Users\danie\AppData\Local\uv\cache\archive-v0\Gx6PRXGoUfWB67iQwxXch\Lib\site-packages\frictionless\metadata\metadata.py", line 195, in to_descriptor
    descriptor = self.metadata_export()
    descriptor = self.metadata_export()
                 ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\danie\AppData\Local\uv\cache\archive-v0\Gx6PRXGoUfWB67iQwxXch\Lib\site-packages\frictionless\resource\resource.py", line 865, in metadata_export
    descriptor = super().metadata_export()
                 ^^^^^^^^^^^^^^^^^^^^^^^^^
    value = value.to_descriptor_source()  # type: ignore
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'dict' object has no attribute 'to_descriptor_source'

[domoritz]

Oh I see. Did you file/find an issue we can watch to update this script when it's fixed?

[dangotbanned]

Oh I see. Did you file/find an issue we can watch to update this script when it's fixed?

Okay I think I've found the issue:

frictionlessdata/frictionless-py#1466 (comment)

https://framework.frictionlessdata.io/docs/fields/datetime.html

So I expected this would be valid:

from frictionless import Resource

Resource(
    ...,
    schema={
        "fields": [
            {"name": "field_one", "type": "string"},
            {"name": "field_two", "type": "number"},
            {"name": "field_three", "type": "datetime"},
        ]
    },
)

Turns out it needs to be written as:

from frictionless import Resource, Schema, fields

Resource(
    ...,
    schema=Schema(
        fields=[
            fields.StringField(name="field_one"),
            fields.NumberField(name="field_two"),
            fields.DatetimeField(name="field_three"),
        ]
    ),
)

[domoritz]

Good find. So we can remove SUFFIX_UNSUPPORTED?

[dangotbanned]

Good find. So we can remove SUFFIX_UNSUPPORTED?

Yeah I'll be pushing with a few changes like that soon

Note

Ready to merge from my end as of (6727f85) @domoritz

[domoritz]

Thank you