[Feature] Optional fields

[yannham]

Optional fields

Closes #373.

Overview

This PR adds the keyword optional (subject to bikeshed) to the metadata syntax to describe a field that may be filled, but is not required.

The general guideline for the semantics of optional fields is that an optional field without definition should be transparent to all record operations, and serialization. Once it get merged with a definition, or with the same field without definition but that is not optional, it acts like a normal field.

In particular, an empty optional field won't appear when calling to record.fields and will return false when calling to record.has_field. The rationale is that we don't want record.has_field "foo" r to return true on an empty optional field and then have r.foo to fail. We may want functions like fields_all or fields_optional that include empty optional fields later, but I don't see an obvious usage for now (dynamic contract inspection/building, maybe?), so let's wait for someone to actually ask for them.

Deciding "is an empty optional"

The way we handle metadata in Nickel is by nature very dynamic: any value can have metadata, and in principle we can't fetch the full metadata of an expression, or decide properties like "is it defined" without performing a form of evaluation, which can involve arbitrary computations. We can have nested metadata, etc.

This means that deciding "is this field an empty optional" is potentially arbitrary long, and causes the evaluation of the field, which sounds bad. For example, record.fields requires to decide "is empty optional" on each field, to filter the list. It feels more natural to store this information at the field level, as a simple flag, and update upon merge, such that we have a direct access in constant time to this property for any given field.

Solution implemented in this PR

However, this would make optional fields quite special, without a good reason. The same dilemma applies to many other metadata. This is why I think we should have a general reflection for metadata (should they only be allowed on record fields and stored there, basically), instead of special casing optional. In the meantime, this PR uses an heuristics to try to detect quickly an empty optional, which I hope will be more than enough for most concrete cases (indeed, while it may be arbitrarily complex to detect in theory, the natural use of optional fields should stay rather simple). The lack of optional fields have been limiting in several experimentation already, when trying to write contracts for concrete use-cases, so in my humble opinion, it's better than nothing to have them with some limitations now, and potentially make them better later, rather than waiting for a restructuring of metadata.

[thufschmitt]

It feels a bit weird to have optional be a metadata field and not a property of the structure of the record itself. What's the reason for that? You mention that “this would make optional fields quite special, without a good reason”, but it intuitively seems to me that they have to be a first-class construct for records 🤔

[yannham]

@thufschmitt to be clear, I think having optional be a property of the record itself is eventually the right solution. But to me, it seems that this is not specific to optional: this is the same dilemma for e.g. doc, the priority (default, etc.), possibly the future hidden, serialize_with, custom merge function, etc. The rationale is then not to introduce an ad-hoc parameter for optionals right now, but to do like we've done until now temporarily, and to think about basically moving all metadata but annotations to be attached directly to record fields, at once.

I'm just not 100% convinced the somehow hacky solution of this PR is fine, even temporarily. But still a bit convinced.

[thufschmitt]

Fair point. It's less obvious (to me at least) for stuff like doc, but it makes sense. Let's 👍 this and wait for #819 to have a better representation