feat: replace SOURCES.md with datapackage.md

[dsmedia]

Resolves #634

Tasks

• Add SOURCES.toml to provide supplemental (extrinisic) metadata on datasets, from SOURCES.md, in a form usable by build_datapackage.py
• Include resource descriptions, sources, and licenses to supplement script output
• Preserve existing markdown content for future documentation
• Remove duplicated content between descriptions and sources
• Incorporate resource-level column descriptions into table schema, where available
• Migrate license link, where available, into [[resources.licenses]]
• Determine if root-level $schema property should be specified in the TOML file with the value "https://datapackage.org/profiles/2.0/datapackage.json" per Frictionless Data guidelines
• Rename datapackage-tabular.md -> datapackage.md (comment)

[dsmedia]

I'm currently revising now to remove duplicate sources from descriptions.

[dsmedia]

Completed in commit 42b5b25:

• Remove duplicated content between descriptions and sources
• Incorporate resource-level column descriptions into table schema
• Migrate license link, where available, into [[resources.licenses]]

Root-level schema property is optional in the Frictionelss Spec and isn't essential at this stage, so I'm marking this as ready for review.

[dangotbanned]

@dsmedia I wanted to get your opinion on formatting the (longer) descriptions like this?

• feat: replace SOURCES.md with datapackage.md #643 (comment)
• feat: replace SOURCES.md with datapackage.md #643 (comment)

Note

I'm not sure why GitHub doesn't understand multi-line strings, but they are valid toml.

I found it pretty difficult to read some of these descriptions with how long the lines were.
We have multiple ways to format these, depending on if we actually want to keep the whitespace.

With these suggestions added, the longest line is still 1280 characters - so I'd want to make changes like this more broadly.
Ideally aiming for 80-100 characters per line. See pep-0008/#maximum-line-length if this number is new to you

[dangotbanned]

Overall, really great work getting everything translated over from markdown @dsmedia!

[domoritz]

Looks good to me (pending other comments).

Let's delete SOURCES.md as part of this pull request and update the references in the pull request template and readme.

[domoritz]

I'm not a fan of aligned =.

Suggested change

	path = "7zip.png"
	path = "7zip.png"

is simpler and easier to edit

[dangotbanned]

@domoritz having align_entries=false is fine with me.

If you've got any other preferences, maybe we could add those in a taplo.toml?

[domoritz]

If there is an automatic tool (ruff?) I am okay either way btw. I just don't want to manually align stuff.

[dangotbanned]

If there is an automatic tool (ruff?) I am okay either way btw. I just don't want to manually align stuff.

@domoritz

• https://taplo.tamasfe.dev/configuration/file.html
• https://taplo.tamasfe.dev/cli/usage/formatting.html
• https://taplo.tamasfe.dev/configuration/formatter-options.html

[dangotbanned]

@domoritz
I should've explained this better in the description for (1ea2812)

That commit uses automatic formatting via taplo.
The two non-default configuration options applied were:

align_entries=true

Align entries vertically. Entries that have table headers, comments, or blank lines between them are not aligned.

allowed_blank_lines=1

The maximum amount of consecutive blank lines allowed.

---

taplo can be used via its CLI and as a VSCode extension (Even Better TOML)

I haven't used it in CI before, but it is available on PyPI - so we could run it with uvx?
https://github.com/tamasfe/taplo/blob/a8bc571ee28775e7d5ad84c2ea87cf5b61ab42f5/.github/workflows/releases.yaml#L503-L504

NPM is another option for install

[domoritz]

If we have a formatter, it should run on the ci as well.

[dangotbanned]

@domoritz I've opened #646 for this

[dsmedia]

Let's delete SOURCES.md as part of this pull request and update the references in the pull request template and readme.

@dangotbanned If I understand the process correctly, SOURCES.toml is designed to contain only the information not captured (or captured incorrectly) by the datapackage script. In this sense, I'm not sure we can simply replace the references to SOURCES.md with SOURCES.toml, since the toml file won't contain the complete source information. Ideally, I think, the script could, after generating datapakcage.json, convert that json into a human-readable SOURCES.md with the synthesized output of the script-generated metadata and the hard-coded metadata from SOURCES.toml.

Here is a related repo

This is a command-line tool to extract the resources within a Frictionless Data Package into a variety of formats such as Markdown, HTML, CSV, etc.

[dangotbanned]

I think you can see the schema from the script instead of toml but I dint have strong feelings.

@domoritz are you talking about this point?

• Determine if root-level $schema property should be specified in the TOML file with the value "datapackage.org/profiles/2.0/datapackage.json" per Frictionless Data guidelines

I don't think we need this at all, since we're using the framework directly there are options under Validating Data if we want to do this.

[domoritz]

Yep. Great if we don't need it.

[dangotbanned]

Happy to merge after feedback on (#643 (comment))

Thanks for working on this @dsmedia 😄

[dsmedia]

One slight concern on the licensing section of the json/md.

{
  "name": "vega-datasets",
  "description": "Common repository for example datasets used by Vega related projects.",
  "homepage": "https://github.com/vega/vega-datasets.git",
  "licenses": [
    {
      "name": "BSD-3-Clause",
      "path": "https://opensource.org/license/bsd-3-clause",
      "title": "The 3-Clause BSD License"
    }

The licensing section could benefit from clarification regarding the distinction between package code and dataset licenses. Here is how it's handled in the European data portal:

Most of the resources published display a specific reference to the licence under which the owner has chosen to release them.

For the resources without licence information, users must consult the licence conditions in the original portal where the resources were initially published.

We could strengthen this further with explicit guidance:

"BSD-3-Clause license applies to package code and infrastructure. Users are solely responsible for ensuring their use of datasets complies with the license terms of the original sources where the datasets were published. Dataset license information that may be included here is provided as a reference starting point only and without any warranty of accuracy or completeness."

This addition would clearly communicate the dual nature of licensing (code vs datasets) to users. The clarification could be added either in the package license title or in the package description. While it might also fit in the repo README.md, having this in datapackage files ensures the licensing clarity travels with the dataset metadata itself.

[dangotbanned]

#643 (comment)

datapackage Docs

Package

https://datapackage.org/standard/data-package/#licenses

The license(s) under which the package is provided.

Caution

This property is not legally binding and does not guarantee the package is licensed under the terms defined in this property.

licenses MUST be an array. Each item in the array is a License. Each MUST be an object. > The object MUST contain a name property and/or a path property, and it MAY contain a title property:

name: A string containing an Open Definition license ID
path: A URL or Path, that is a fully qualified HTTP address, or a relative POSIX path.
title: A string containing human-readable title.

Resource

https://datapackage.org/standard/data-resource/#licenses

List of licenses as for Data Package. If not specified the resource inherits from the data package.

---

@dsmedia my read on this would be I think you'd need to raise this over at https://github.com/frictionlessdata/datapackage/issues
The distinction package/resource licenses I think is already covered.

However, inheriting the license from the package seems like the wrong move for our situation.

Also, there isn't any room in the spec for adding context (e.g. a description) per-license.
Your proposal would require at least one more field - which seems reasonable to me personally

[dsmedia]

How about within the datapackage description itself?

description
A description of the package. The description MUST be markdown formatted — this also allows for simple plain text as plain text is itself valid markdown. The first paragraph (up to the first double line break) SHOULD be usable as summary information for the package.

This is the top-level description property in the spec root object

{
  "title": "Data Package",
  "description": "Data Package", 
  "type": "object",

}

[dangotbanned]

#643 (comment)

Sure @dsmedia that seems like a good place you could add a disclaimer

Feel free to add 🙌

[dangotbanned]

Hope the title change in (#643 (comment)) makes sense to others.

I think this is more descriptive of the larger change this PR will have

[dangotbanned]

Thanks for all the work you put into this @dsmedia, merging now