Is a (Semantic) Data Product a Catalog or a Dataset (or both or something else)?

[jgeluk]

In the Data Mesh world, it seems to me that a Data Product is a piece of code that is deployed somewhere serving data.

Which is traditionally called many other things such as "a server", "a database server", "a data service" and so forth.

In an Enterprise Knowledge Graph (EKG) context though, we need something that represents the logical version (or even virtual version) of a Data Product. Something that humans and AIs can reason about, can buy, consume etc.

Should we see such a Data Product as:

a) a "Catalog (or collection) of Datasets"?
b) a "Catalog of (other) Data Products"?
c) an extended (marketable, deployable) version of a Dataset?

[andrea-gioia]

The concept of data product can be associated with existing concepts in DCAT. In particular, the following are the possible mappings:

1. Data Product => Resource: A data product can be seen as a specific type of catalogable resource, just like a dataset and a data service.
2. Data Product => Dataset: A data product can be seen as a specific type of dataset. However, this association is limiting because, in the data mesh approach, a data product is a richer entity than just the exposed data. Additionally, a data product may expose multiple datasets.
3. Data Product => Catalog: A data product can be seen as a catalog that exposes the resources it is composed of.

IMHO the 3rd is the preferable option as shown in this image. Initially, a data product can be seen as a catalog of its datasets and data services, and in the future, even its internal components (infrastructure and applications).

[andrea-gioia]

Why not see a data product as a collection of data exposed by a single producer instead? Even a software product is ultimately composed of multiple services, each with its own service agreements. Consumers may only consume some of the services within the software product, but that doesn't make it any less of a product. What's different for a data product?

[jgeluk]

I agree! If we want to make this as easy to understand as possible, for everyone, then we should stay as close to the general connotation that everyone has about what a product is in economic terms. Something provided by one supplier and consumed by one consumer, via an agreement in the form of a contract, in exchange for money.

One data product may however be served in many different ways:

1. By one data-service
2. By multiple data-services
3. By multiple different protocols (JDBC, REST, GraphQL, Kafka, whatever, even FTP) where each of these protocols could be one "output port" on a given host, collection of hosts, or even many services (access points) across the world...
4. It could be that any of those data-services or access points also serves any other number of data products.

[jgeluk]

I'd even say that when anyone accesses a data product through whatever port or access point, that in the given protocol, whatever that may be, it has to be clear, via a header or token or so, what the corresponding agreement/contract/context is in which the data product is being consumed/accessed.

[jgeluk]

For any given data-product we have tons of "detail" that people should be able to specify, which may be referred to in data contracts (and should be stable for the duration of the contract, blockchain comes to mind, and/or versioning of metadata).

Such as:

• Entitlement & Usage policies (bloomberg restricting access to a market data feed to just a number of use cases in a number of departments for instance)
• IP policy
• Usage policies
• Pricing policies
• Jurisdiction specific regulatory and legal policies
• Many others

Basically all of that screams for "graph", or actually "semantic graph", quite complex metadata that all combined would allow for highly automated robust enterprise-grade data services that can serve any data product as long as it has all this information onboard.

[MikeHypercube]

Isn't the data-set playing the role of a product here?

Specifically it is the 'thing of value' (the economic resource) that is the subject of one side of the contract (the reciprocal commitment from the other party being a commitment to pay for that product under whatever terms e.g. one-off, subscription etc.)