[Enhancement] Change the kafka-versions file into JSON/YAML format

[tomncooper]

Is your feature request related to a problem? Please describe.

At the moment the kafka-versions file is in a custom format that makes adding information difficult as you then need to check and possibly update every method that reads it.

Describe the solution you'd like

Changing this file to a standard format such as JSON or YAML would mean standard parsing libraries could be used to extract the information in each relevant method and any additional information added later to the kafka versions would be ignored by those methods that do not use it.

Describe alternatives you've considered
Leaving it how it is and avoiding adding any additional information to the file.

Additional context

I am working on the process for upgrading ZK when the new Kafka version requires a newer version. In order to do this robustly I need to add the ZK version information to the kafka-versions file. This would mean changing all the sections of Strimzi that touch this file so I thought we could bite the bullet and do this work to change the file format now to hopefully save time in the future.

[tombentley]

I guess one consideration for the schema would be the ease of using jq to extract the various parts needed. But how about this as a starting point:

{
  "2.3.0": {
    "sha1sum": ...,
    "protocol": "2.3",
    "format": "2.3",
    "default": true,
    "3rd-party-libs": "2.3.x"
  },
  ...
}

The other choice would be an array of those inner objects (with an extra version key), rather than the outer object.

In the future we could consider adding a url key if, for example, there was a url for downloading kafka nightlies.

[tomncooper]

I was thinking of an array, but this is likely to be parsed into a map (with the version as key) at some point anyway so we might as well store it as one.

Do you think JSON and jq rather than YAML and yq are the better way to go? I have no preference either way.

[tombentley]

I don't have a strong opinion. The build already depends on yq, so in that sense YAML might be better as it's one less build-time requirement. YAML also supports comments, which might occasionally be useful.

[tomncooper]

So the YAML version might be something like:

--- 
"2.2.1": 
  3rd-party-libs: 2.2.x
  default: false
  format: 2.2
  protocol: 2.2
  sha1sum: asdajslkdjaskjdlksjdl

"2.3.0": 
  3rd-party-libs: 2.3.x
  default: true
  format: 2.3
  protocol: 2.3
  sha1sum: asdajslkdjaskjdlksjdl

The only issue I can think of is do we already pull in a Java YAML library? Does fabric8 include something to do this? I am thinking specifically of io.strimzi.operator.cluster.model.KafkaVersion.

[tombentley]

Yes fabric8 depends on jackson's YAML support, which we already use directly in a few places.

[tombentley]

We could filter the sha1sum and 3rd-party-libs from the file on the CO classpath, since those are not runtime concerns.

[ppatierno]

tbh I am not a fan of using a potential value of a field as a key (I am referring to using "2.3.0" instead of having an array of objects and then the field "version": "2.3.0"). I think it makes less readable the YAML or JSON. I think it's something Kubernetes team doesn't use as well.

[tombentley]

I think the reasons behind why kube doesn't do this don't apply here.

[ppatierno]

True but that representation remains to be less readable to me.

[tomncooper]

I have switched to using an array of maps as that is easier to parse in the KafkaVersion file, but I need to see how this effects usingyq in the command line scripts.