Support for “new” RDF serialisation

[gouttegd]

Now that the normalized form of the RDF serialisation is about to get merged, we should think about how it is going to be supported by SSSOM-Py.

The normalized form is purposefully very close to the form that SSSOM-Py has been producing and reading since the beginning, but it does deviates from it on some aspects – it deviates enough for the current SSSOM/RDF parser in SSSOM-Py to reject a file compliant with the normalized form.

Both the current SSSOM/RDF parser and the current SSSOM/RDF writer in SSSOM-Py are heavily reliant on the LinkML runtime: the parser is mostly a thin wrapper around LinkML’s RDFLibLoader and the writer is mostly a thin wrapper around LinkML’s rdflib_dumper.

Unfortunately, that approach is only valid for a RDF serialisation that is exactly conform to LinkML’s RDF translation rules, which the new normalized form of SSSOM/RDF is not.

Possible new approaches to support the form:

(A) Forget about LinkML’s RDF-related methods and write entirely custom code to convert to and from RDF, using SSSOM’s own translation rules.

In my (not-so-)humble opinion the best solution, simultaneously the cleanest and the most flexible, but possibly the one that will require the most work (though it’s unclear to me that the other approaches below are actually any better on that front).

(B) Keep using the LinkML runtime, with appropriate pre- and post-processing steps. That is:

• when converting from RDF, add a pre-processing step to transform the graph into the form expected by the LinkML runtime, then feed the transformed graph to the runtime;
• when converting to RDF, use the LinkML runtime to produce a first graph, then add a post-processing step to transform that graph into the new normalized form.

Possibly quicker to code, but that’s a lot of completely avoidable complexity.

(C) Going through JSON-LD, as suggested in #592 (at least for the conversion to RDF).

The problem here is the JSON-LD serialisation of SSSOM is still not formally defined (which incidentally is also the case of the non-LD JSON serialisation). Now I think it would make much sense for SSSOM/JSON-LD to be defined in such a way that is exactly equivalent to SSSOM/RDF, but we have not decided that yet.

[matentzn]

Now I think it would make much sense for SSSOM/JSON-LD to be defined in such a way that is exactly equivalent to SSSOM/RDF, but we have not decided that yet.

I would be much in favour of that!

Possible new approaches to support the form:

I personally would Option (B), but NOT in the interest of SSSOM - more in the interest of the entire vision behind LinkML. Ideally the LinkML team would analyse the necessary pre and post-processing step and see if these make sense as defaults in the LinkML converters - now maybe thats silly but I am also not entirely certain anymore which exact pre and post processing steps will we need to add!

[cthoyt]

If we go the route of asking the LinkML team to make improvements to LinkML, are they just going to put a hack in for our use case, or is this a more wide use case to do this? At what point are we simply past what can be modeled in LinkML?

I have always been luke warm about relying on LinkML, and it might be a good time to rip off the bandage now rather than after spending a lot of precious developer hours on coordination

[gouttegd]

Now I think it would make much sense for SSSOM/JSON-LD to be defined in such a way that is exactly equivalent to SSSOM/RDF, but we have not decided that yet.

I would be much in favour of that!

So would I (well, kind of: I would not necessarily be “much” in favour, mostly because I in fact don’t care in the slightest for JSON-LD once we have RDF; I would be in favour of saying “JSON-LD SSSOM is the same thing as RDF SSSOM, but in JSON” because at least we wouldn’t have to spend months discussing about what JSON-LD SSSOM should look like, and we could spend that time discussing about the non-LD version of JSON instead), but that doesn’t mean I would like the (C) solution; having to wait for JSON-LD as a prerequisite for supporting the normalized form of SSSOM RDF is not appealing to me.

I personally would Option (B), but NOT in the interest of SSSOM - more in the interest of the entire vision behind LinkML.

[…]

If we go the route of asking the LinkML team to make improvements to LinkML,

We are not really talking about “improvements” here. We’re talking about the need to deal with, either

• SSSOM-specific idiosyncrasies, like the NonRelativeURI type;

• SSSOM-specific backwards compatibility issues.

For example, we want slots typed as NonRelativeURI (e.g., license, mapping_set_id, mapping_provider, etc.) to be

• rendered as RDF named resources (e.g. ?mapping_set dcterms:license <http://creativecommons.org/licenses/by/4.0/>) when writing to RDF;
• accepted both from named resources (as written) and from xsd:anyURI literals (?mapping_set dcterms:license "http://creativecommons.org/licenses/by/4.0/"^^xsd:anyURI) when reading from RDF, because that’s what SSSOM-Py has always produced.

The LinkML RDF loader accepts both when reading, but the RDF dumper only writes the literal form.

(Possibly, this could easily be solved by declaring the NonRelativeURI type, within the SSSOM schema, with a type_uri of rdfs:Resource; I need to check the behaviour of LinkML’s runtime in that case – in particular, would the loader still accept literals once the type has been defined as a named resource?)

But the biggest problem is about the rendering of the mapping_set_id and record_id slots.

We want mapping_set_id to be rendered as the named resource that represents the mapping set, as in:

<https://example.org/my-mapping-set-id/> a sssom:MappingSet ;
  dcterms:title "My mapping set" ;
  ...

But the LinkML RDF dumper will instead use a blank node to represent the mapping set, and write the mapping_set_id as any other slot:

[] a sssom:MappingSet ;
  sssom:mapping_set_id "https://example.org/my-mapping-set-id" ;
  dcterms:title "My mapping set";
  ...

We could fix that by declaring mapping_set_id as the “primary key” for the mapping set class, which would make the dumper produce the output we want. But that would also make the loader no longer accept the old form!

Same problem for the individual records and the record_id slot, except that for this slot we don’t even have the option of declaring the slot the “primary key” for the mapping class, because that slot is optional and LinkML (quite reasonably) does not support the idea of an “optional primary key”.

So I am not sure there is anything for the LinkML folks to “improve” here. Those are SSSOM-specific issues.

[gouttegd]

At what point are we simply past what can be modeled in LinkML?

I’ve said that before and I can say it again: one of the biggest problems with LinkML (or, at least, with the LinkML runtime) is the complete inability to separate the data model from the serialisation formats. For LinkML, it’s all the same thing; the data model entirely defines the different serialisations.

Now, I can see how people can consider that a good thing because it dispenses them from having to (1) decide what their serialisations should look like and (2) implement parsers and writers for those serialisations (at least if you’re working in Python).

But it also means that you have no flexibility at all, and that any change in the data model also becomes a change in the serialisations (with nothing to “dampen” the impact of a data model change).

That’s the reason for the above issue with the record_id and mapping_set_id slots. For LinkML, “that slot acts as the named resource that represents an instance of this class” (which is a RDF serialisation detail) is the same thing as “that slot is defined in the model as the primary key for the class” (which is a data modeling decision).

This is also why SSSOM-Py has no support for backwards compatibility (which is a shame, in my opinion), i.e. why you can’t use SSSOM-Py to read a SSSOM file that was produced to be compliant with version 0.9 of the spec (before the match_type slot was renamed to mapping_justification): the LinkML parsers are only aware of the current version of the schema, so they only know of the mapping_justification slot and have no idea of what to do is they encounter a match_type slot.