Need to specify how to null a value.

[birdsarah]

If you have a field: award.supplier.address.postcode = 'ABC'

and you want to set it to empty award.supplier.address.postcode = ''

but at the moment, merging will ignore the change, assuming its just an empty field.

[birdsarah]

This is now possible.

Should be documented though.

Currently, in ocds version of jsonmerge, null values will be ignored, and will not change a field. But can set to empty string to empty a value. This wouldn't work for emptying a number. Setting to 0 is not the same thing.

Not entirely sure what to do, because if we don't ignore null values then our versionedRelease gets massive, but maybe that doesn't matter as its only intended to be read by a machine. Alternatively, we could ignore null if the versioned release field is empty, but not if its not.

See: http://nbviewer.ipython.org/github/open-contracting/sample-data/blob/master/buyandsell/processing/Check%20nulling%20a%20value.ipynb

The more I think about this, the more I think this is not good.

Would really appreciate thoughts from @avian2, @practicalparticipation, @jpmckinney.

[jpmckinney]

Do any current publishers have versioned releases? All the data I've seen is one-shot. There are alternatives to the current versioning system that may work better, so if no publisher currently does versioning, then maybe it'd be possible to include later.

Under the current system, if an earlier version has a non-null value, and a later version has a null value, the non-null value should not pass through.

[avian2]

If I understand correctly, in incremental updates you need to differentiate between 1) value that is not changed from the current state and 2) a value that should be removed.

From the perspective of jsonmerge, I think the correct way is in case of 1) that the value is not present at all in the update document (e.g. no award.supplier.address.postcode element). In case of 2) the element is set to null.

[jpmckinney]

@avian2 Yes, that's my understanding, too.

[LindseyAM]

+1 to @jpmckinney 's question, @birdsarah would love to hear what you have seen on versioning from the supply side.

[practicalparticipation]

@avian2's suggestion also matches with the JSON Merge Patch approach.

I've drafted the following documentation on this:

Null values, and removing fields

Publishers providing regular releases of data should pay careful attention to the inclusion of fields and the use of null values as this affects how data is merged when compiling a record with a version history of the contracting process.

If a field is included in one release, and then ommitted entirely from a subsequent release, the compiled record will continue to contain the original field.

ToDo: Worked example

If a field is included in one release, and then set to null in a subsequent release, that field will be removed from the compiled record (though past values of it should remain in the versioned section of the record).

ToDo: Worked example

If a field is set to null in one release, and is also set to null in a subsequent release, it will be ommitted from both the record and the versioned section of the record. I.e. fields that are always set as null will be ignored in the record.

ToDo: Worked example

If a field is set to null in one release, and then has a value in a subsequent release, it will first appear in the record, and the version section of the record when it has a value.

ToDo: Worked example

[birdsarah]

@jpmckinney just getting my head back in the game, so i may be answering the wrong question. "Do any current publishers have versioned releases?" A number of datasets update their data and mark it as revised / amended.

[birdsarah]

@practicalparticipation - that is perfect - I'm not sure that's what our version of jsonmerge does at the moment, but I think it's the right behavior.

See open-contracting-archive/jsonmerge#1

[birdsarah]

@practicalparticipation looking at this again, would the following be acceptable.

If a field is included in one release, and then set to null in a subsequent release, that field will be recorded as null in the compiled record (though past values of it should remain in the versioned section of the record).

If a field is set to null in one release, and is also set to null in a subsequent release, it will be recorded as null as per the original release, and the record will not show any change based on the subsequent release I.e. fields that are set to null are recorded as such and then only changed if the value is set.

If a field is set to null in one release, and then has a value in a subsequent release, it will first appear in the record, and the version section of the record as null and then with the subsequent value.

The short way of saying this, is if you want a tidy compiled & versioned record, then omit fields rather than set to null.

[birdsarah]

(I'm still haven't checked behavior) am just covering my bases.

[practicalparticipation]

Documented at http://ocds.open-contracting.org/standard/r/master/en/schema/reference/#emptying-fields-and-values

[birdsarah]

I still need to confirm this - will do so on 17th.

[timgdavies]

Closed for now. Will revisit in testing tools in future.