Configuration file formats for JSON, INI, YAML and TOML

[infinisil]

Motivation for this change

This is a main part of the implementation of NixOS/rfcs#42 (see https://github.com/Infinisil/rfcs/blob/config-option/rfcs/0042-config-option.md#part-1-1).

Since that part of the RFC is fairly uncontroversial, I think we can merge this before the RFC is eventually accepted. This allows people to use such types in their NixOS options already.

Ping @Profpatsch @rycee @aanderse @kampka

Things done

• Tested that these types and generator combinations indeed can be used in modules.
• Write some tests
• Wrote documentation on how to use them..

Here's a simple example module though:

{ lib, config, pkgs, ... }:
let
  format = pkgs.formats.json {};
in {
  options.settings = lib.mkOption {
    type = format.type;
    default = {};
  };

  config = {
    settings.foo = 10;
    settings.bar.baz = "Hello";
    environment.etc.test.source = format.generate "config.json" config.settings;
  };
}

[infinisil]

Potentially/Eventually these formats might be implemented using lazyAttrsOf (or what becomes of it) from #70138, such that self-dependent assignments of them will be possible. Edit: This is now done in this PR :)

[aanderse]

@infinisil awesome stuff!

[infinisil]

(This still needs a bit of docs probably and a history cleanup at least btw, before anybody merges)

[infinisil]

I've now rebased this ontop of #70138 to implement types where values can depend on other values. E.g. this previously threw an infinite recursion error, but now it doesn't anymore:

{ lib, config, ... }:
let format = lib.formats.json;
in {
  options.settings = lib.mkOption {
    type = format.type;
    default = {};
  };

  config = {
    settings.foo = true;
    settings.bar = lib.mkIf config.settings.foo "Foo is true!";

    environment.etc.test.text = format.generate config.settings;
  };
}

Due to how the implementation works, it's however a bit more cumbersome if you aren't sure whether an attribute exists at all. This is how that can now be handled gracefully:

{
  # To make sure the `.name` attribute exists
  settings.name = format.empty;
  # To handle the case of `.name` being empty
  settings.greeting = "Hello, ${format.get "nobody" config.settings.name}!";
}

This will generate { greeting = "Hello, nobody!"; }

Alternatively if you don't mind name being set:

{
  settings.name = lib.mkDefault "nobody";
  settings.greeting = "Hello, ${config.settings.name}!";
}

Which will generate { name = "nobody"; greeting = "Hello, nobody!"; }

[infinisil]

It might be a good idea to only have a single empty value for all

[infinisil]

On IRC @adisbladis brought to my attention that to support more formats like TOML, we could use builtins.toJSON and convert that to toml with a build-time tool, meaning the generated strings won't be available to nix at eval time anymore and the generate functions have the signature generate :: Value -> Derivation. This doesn't seem like a problem: Nobody should need the raw Nix strings anyways, and this will speed up evaluation too.

[domenkozar]

I'd review it if it was written with terminal length in mind, but currently it's not feasible via github ui.

[roberth]

It should be noted in the docs that some of these functions may write to the store, which is not safe when Nix is handling secrets. This is in contrast to builtins.toJSON etc, which operate entirely in memory.

[roberth]

Suggested change

	# setting format types and generators. These would fit in lib/types.nix, but
	# setting format types and generators. These do not fit in lib/types.nix, because

[infinisil]

Done with https://github.com/NixOS/nixpkgs/compare/76e8e4a26fa451bbc1832e0cd599080795227593..83b16885f526d6ab7b39e98159e2b48024f3238c#diff-fa126a2182114f76a8e8df7ed256395d

[roberth]

Suggested change

	# pkgs-lib is for things that should be in lib, but can't beceause
	# pkgs-lib is for functions that can't be in lib, because

[infinisil]

Done with adjustments in https://github.com/NixOS/nixpkgs/compare/76e8e4a26fa451bbc1832e0cd599080795227593..83b16885f526d6ab7b39e98159e2b48024f3238c#diff-fa126a2182114f76a8e8df7ed256395d

[roberth]

An overlay is unnecessary because the same could be achieved by putting formats in all-packages.nix.

Categorizing functions separately from packages is beyond the scope of this change. It can be discussed in a separate issue/pr, although I doubt its usefulness.

[infinisil]

Doesn't have to be just functions though, can be arbitrary values, and other functions like fetchgit wouldn't fit in a pkgs-lib. I guess it won't be a problem to just put formats in all-packages.nix, but imo having a separation between "things needed to create packages" and "things that just happen to depend on pkgs" would make sense. Could still introduce that in the future though

[infinisil]

Not using an overlay anymore now

[roberth]

All of formats, fetchgit, dockerTools depend on Nixpkgs, take arguments, produce derivations, are maintained and released in the same way. What makes formats so architecturally different that it needs to be treated differently?

imo having a separation between "things needed to create packages" and "things that just happen to depend on pkgs" would make sense. Could still introduce that in the future though

If only one attribute is separate it doesn't make sense though. For it to make sense it must be applied to the whole of Nixpkgs and it must have practical benefits. The prior can be done in the future and I'm skeptical about the benefits, but would love to be proven wrong in that same future.

For now please follow the example of, say, dockerTools.

[roberth]

Looks like my view of the PR was outdated (maybe caused by the current service disruption?) anyway, thanks for removing the overlay. Currently build-support has the role of pkgs-lib so you may want to move the files to build-support/formats. I don't really like that name either so I'm going to leave this change up to you to decide.

[infinisil]

The difference between e.g. fetchgit and formats.json is that fetchgit is needed for building many of the pkgs.* derivations, whereas formats.json doesn't have any use for any pkgs.*. Instead formats are mostly just useful for NixOS modules (really, the module system in general, so it wouldn't fit in nixos/lib)

This is also why I don't think build-support would make sense, because formats.json doesn't support any package building.

[worldofpeace]

I'd review it if it was written with terminal length in mind, but currently it's not feasible via github ui.

this is useful https://github.com/StylishThemes/GitHub-code-wrap

[rycee]

An alternative that doesn't need any installations is to use the "Split" diff option. For whatever reason, it seems to wrap lines while "Unified" doesn't.

[infinisil]

It should be noted in the docs that some of these functions may write to the store, which is not safe when Nix is handling secrets. This is in contrast to builtins.toJSON etc, which operate entirely in memory.

For NixOS modules at least it doesn't make much of a difference, because the only way for a string (which toJSON returns) to influence a system build is by putting it in a derivation. With the interface this PR adds, this step is just done a little earlier. But yeah it can't hurt to point this out, I'll add a note

[infinisil]

@roberth Looking good now?

[Ericson2314]

I wonder if pkgs-lib should be pkgs/lib to match nixos/lib.

[infinisil]

Awesome, thanks for merging!

@Ericson2314 I intentionally didn't name it lib because it could easily be confused with pkgs.lib.

[fzakaria]

@infinisil you might want to update this documentation section as well https://nixos.org/manual/nixpkgs/unstable/#sec-generators
It seems out of date considering the new NixOS doc you added.

[fzakaria]

Actually reading lib/generators.nix it looks like it should be consolidated with the new formats option introduced here.