Add command to generate jsonschema for limayaml

[afbjorklund]

Closes #2305

https://pypi.org/project/check-jsonschema/

Actually found two bugs, with the current code:

Schema validation errors were encountered.
  examples/default.yaml::$.vmType: None is not of type 'string'
  examples/default.yaml::$.os: None is not of type 'string'
  examples/default.yaml::$.arch: None is not of type 'string'
  examples/default.yaml::$.cpuType.armv7l: None is not of type 'string'
  examples/default.yaml::$.cpuType.aarch64: None is not of type 'string'
  examples/default.yaml::$.cpuType.x86_64: None is not of type 'string'
  examples/default.yaml::$.cpus: None is not of type 'integer'
  examples/default.yaml::$.memory: None is not of type 'string'
  examples/default.yaml::$.disk: None is not of type 'string'

Schema validation errors were encountered.
  examples/docker.yaml::$.probes[0]: Additional properties are not allowed ('hint', 'script' were unexpected)
  examples/docker.yaml::$.probes[0]: 'Mode' is a required property
  examples/docker.yaml::$.probes[0]: 'Description' is a required property
  examples/docker.yaml::$.probes[0]: 'Script' is a required property
  examples/docker.yaml::$.probes[0]: 'Hint' is a required property

But the rest of it is just about adding "nullable".

---

To test this feature: (hidden command: limactl generate-jsonschema)

make schema-limayaml.json

And use it to test: (https://check-jsonschema.readthedocs.io/)

make check-jsonschema

[mac2net]

Don't you think this universal rejection of “none” is more an opinionated code base rather than a validation error?

[afbjorklund]

Don't you think this universal rejection of “none” is more an opinionated code base rather than a validation error?

It's not a rejection (as such), there is a patch for it (to not have to decorate the go struct) but it has not been merged yet

• feat: Add NullableFromType option invopop/jsonschema#110

With the patch, one could (perhaps) mark all string pointers and arrays as accepting null as a valid value (by default).

• feat: Add NullableFromJSONSchemaTags option invopop/jsonschema#106

---

But it does catch some issues, with string being fed with null - which is valid in JavaScript but not really in Go...

Having to decorate the pointers and arrays I think is not so intrusive, so it seems like a valid workaround to me.

                "vmType": {
                    "type": "string"
                },

examples/default.yaml::$.os: None is not of type 'string'

With "nullable", the following json schema is generated instead:

                "vmType": {
                    "oneOf": [
                        {
                            "type": "string"
                        },
                        {
                            "type": "null"
                        }
                    ]
                },

[afbjorklund]

$ make check-jsonschema 
# The hostagent must be compiled with CGO_ENABLED=1 so that net.LookupIP() in the DNS server
# calls the native resolver library and not the simplistic version in the Go library.
CGO_ENABLED=1 go build -ldflags="-s -w -X github.com/lima-vm/lima/pkg/version.Version=v0.21.0-51-ga177cd8" -tags "" -o _output/bin/limactl ./cmd/limactl
_output/bin/limactl generate-schema >schema-limayaml.json
check-jsonschema --schemafile schema-limayaml.json examples/default.yaml
ok -- validation done

These arrays have to allow null, since they are given that way in the default yaml:

additionalDisks:
# disks should either be a list of disk name strings, for example:
# - "data"
# or a list of disk objects with extra parameters, for example:
# - name: "data"
#   format: true
#   fsType: "ext4"

caCerts:
  ...

  # A list of trusted CA certificate files. The files will be read and passed to cloud-init.
  files:
  # - examples/hello.crt

  # A list of trusted CA certificates. These are directly passed to cloud-init.
  certs:
  # - |
  #   -----BEGIN CERTIFICATE-----
  #   YOUR-ORGS-TRUSTED-CA-CERT-HERE
  #   -----END CERTIFICATE-----
  # - |
  #   -----BEGIN CERTIFICATE-----
  #   YOUR-ORGS-TRUSTED-CA-CERT-HERE
  #   -----END CERTIFICATE-----

networks:
# Lima can manage daemons for networks defined in $LIMA_HOME/_config/networks.yaml
# automatically. The socket_vmnet binary must be installed into
# secure locations only alterable by the "root" user.
# The same applies to vde_switch and vde_vmnet for the deprecated VDE mode.
# - lima: shared
#   # MAC address of the instance; lima will pick one based on the instance name,
#   # so DHCP assigned ip addresses should remain constant over instance restarts.
#   macAddress: ""
#   # Interface name, defaults to "lima0", "lima1", etc.
#   interface: ""
#
# Lima can also connect to "unmanaged" networks addressed by "socket". This
# means that the daemons will not be controlled by Lima, but must be started
# before the instance.  The interface type (host, shared, or bridged) is
# configured in socket_vmnet and not in lima.
# - socket: "/var/run/socket_vmnet"

hostResolver:

  ...
  hosts:
    # guest.name: 127.1.1.1
    # host.name: host.lima.internal

A better alternative would be to comment out the key as well, and leave them undefined?

The other keys are only allowing null values, when they explicitly are given that way:

vmType: null
os: null
arch: null
images:
  - location: https://cloud-images.ubuntu.com/releases/24.04/release-20240423/ubuntu-24.04-server-cloudimg-amd64.img
    arch: x86_64
    digest: sha256:32a9d30d18803da72f5936cf2b7b9efcb4d0bb63c67933f17e3bdfd1751de3f3
  - location: https://cloud-images.ubuntu.com/releases/24.04/release-20240423/ubuntu-24.04-server-cloudimg-arm64.img
    arch: aarch64
    digest: sha256:c841bac00925d3e6892d979798103a867931f255f28fefd9d5e07e3e22d0ef22
  - location: https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-amd64.img
    arch: x86_64
  - location: https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-arm64.img
    arch: aarch64
cpus: null
memory: null
disk: null
mounts:
  - location: "~"
    mountPoint: null
    writable: null
    sshfs:
      cache: null
      followSymlinks: null
      sftpDriver: null
    9p:
      securityModel: null
      protocolVersion: null
      msize: null
      cache: null
  - location: /tmp/lima
    writable: true
mountType: null
mountInotify: null
additionalDisks: null
ssh:
  localPort: 0
  loadDotSSHPubKeys: null
  forwardAgent: null
  forwardX11: null
  forwardX11Trusted: null
caCerts:
  removeDefaults: null
  files: null
  certs: null
upgradePackages: null
containerd:
  system: null
  user: null
cpuType:
  aarch64: null
  armv7l: null
  x86_64: null
rosetta:
  enabled: null
  binfmt: null
timezone: null
firmware:
  legacyBIOS: null
audio:
  device: null
video:
  display: null
  vnc:
    display: null
networks: null
propagateProxyEnv: null
hostResolver:
  enabled: null
  ipv6: null
  hosts: null
guestInstallPrefix: null
plain: null

Currently the only required key is "images", but even that might be optional (for ext).

[afbjorklund]

Here is the empty yaml:

images: []

And here is default yaml:

images:
- location: https://cloud-images.ubuntu.com/releases/24.04/release-20240423/ubuntu-24.04-server-cloudimg-amd64.img
  arch: x86_64
  digest: sha256:32a9d30d18803da72f5936cf2b7b9efcb4d0bb63c67933f17e3bdfd1751de3f3
- location: https://cloud-images.ubuntu.com/releases/24.04/release-20240423/ubuntu-24.04-server-cloudimg-arm64.img
  arch: aarch64
  digest: sha256:c841bac00925d3e6892d979798103a867931f255f28fefd9d5e07e3e22d0ef22
- location: https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-amd64.img
  arch: x86_64
- location: https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-arm64.img
  arch: aarch64
mounts:
- location: ~
- location: /tmp/lima
  writable: true

Hopefully the images can be made more similar to the containerd archives, and available from the code instead.

Or maybe not the go code, but to keep the nerdctl-full and the distro images in some kind of secondary config ?

---
"ubuntu:24.04":
  images:
# Try to use release-yyyyMMdd image if available. Note that release-yyyyMMdd will be removed after several months.
  - location: "https://cloud-images.ubuntu.com/releases/24.04/release-20240423/ubuntu-24.04-server-cloudimg-amd64.img"
    arch: "x86_64"
    digest: "sha256:32a9d30d18803da72f5936cf2b7b9efcb4d0bb63c67933f17e3bdfd1751de3f3"
  - location: "https://cloud-images.ubuntu.com/releases/24.04/release-20240423/ubuntu-24.04-server-cloudimg-arm64.img"
    arch: "aarch64"
    digest: "sha256:c841bac00925d3e6892d979798103a867931f255f28fefd9d5e07e3e22d0ef22"
  # Fallback to the latest release image.
  # Hint: run `limactl prune` to invalidate the cache
  - location: "https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-amd64.img"
    arch: "x86_64"
  - location: "https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-arm64.img"
    arch: "aarch64"
---
"nerdctl-full:1.7.6":
  images:
  - location: "https://github.com/containerd/nerdctl/releases/download/1.7.6/nerdctl-full-1.7.6-linux-amd64.tar.gz"
    arch: "x86_64"
    digest: "sha256:2c841e097fcfb5a1760bd354b3778cb695b44cd01f9f271c17507dc4a0b25606"
  - location: "https://github.com/containerd/nerdctl/releases/download/1.7.6/nerdctl-full-1.7.6-linux-arm64.tar.gz"
    arch: "aarch64"
    digest: "sha256:77c747f09853ee3d229d77e8de0dd3c85622537d82be57433dc1fca4493bab95"
    # No arm-v7
    # No riscv64

And then expanded by lima.

EDIT: Added:

• PoC: Allow loading images by name from imagestore #2404
• Move containerd archives from code to yaml #2403

Now the only duplication is the default mounts... Hmm.

images:
- default
mounts:
- default

• Provide standard values "default" and "none" for most options #2458

[afbjorklund]

The resulting JSON is supposed to be uploaded to some external URL under https://schemastore.org/

Perhaps hosted on the lima-vm.io site, or as a fallback loaded from the git repo directly (like cloud-config)

Then it can be used while editing the lima.yaml, even though it still needs to pass limactl validate.

But the hidden command and the make target is mostly for developers / while changing default.yaml

[AkihiroSuda]

Needs rebase

[AkihiroSuda]

Thanks

[AkihiroSuda]

Sorry, needs rebase again

[AkihiroSuda]

Thanks

[AkihiroSuda]

Still needs another rebase

[AkihiroSuda]

Needs rebase

[afbjorklund]

I left out the information about which arrays are null in the default template, I don't think anyone cares...

But left the "nullable" in, so that we don't get validation errors:

  examples/default.yaml::$.additionalDisks: None is not of type 'array'
  examples/default.yaml::$.mountTypesUnsupported: None is not of type 'array'
  examples/default.yaml::$.networks: None is not of type 'array'
  examples/default.yaml::$.caCerts.files: None is not of type 'array'
  examples/default.yaml::$.caCerts.certs: None is not of type 'array'

Otherwise they could commented out, and handled by the "omitempty"

[AkihiroSuda]

Can we have a test? What happens when this is out of the sync with the YAML struct definition?