Skip to content

lib/modules: improve errors for options/config-mixups#131205

Merged
lheckemann merged 6 commits intoNixOS:masterfrom
Ma27:showdefs-overflow
Sep 29, 2021
Merged

lib/modules: improve errors for options/config-mixups#131205
lheckemann merged 6 commits intoNixOS:masterfrom
Ma27:showdefs-overflow

Conversation

@Ma27
Copy link
Member

@Ma27 Ma27 commented Jul 23, 2021

Motivation for this change

I realized that the error-message I implemented in fa30c9a isn't very helpful if config and options isn't mixed up in a submodule, but in a "top-level" declaration. While working on that, I also added a depthLimit-parameter to generators.toPretty to avoid weird stack overflow-errors.

The details can be found in the commit-messages.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • 21.11 Release Notes (or backporting 21.05 Relase notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

@Ma27 Ma27 requested a review from infinisil July 23, 2021 09:22
@Ma27 Ma27 requested review from Profpatsch, edolstra and nbp as code owners July 23, 2021 09:22
@Ma27 Ma27 changed the title lib/modules: improve errors lib/modules: improve errors for options/config-mixups Jul 23, 2021
@ofborg ofborg bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux. labels Jul 23, 2021
Copy link
Member

@lheckemann lheckemann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like it! Some suggestions here.

@infinisil
Copy link
Member

I already tried to add this in both #98477 and #98761, but @Profpatsch didn't want it

@Ma27
Copy link
Member Author

Ma27 commented Jul 23, 2021

I already tried to add this in both #98477 and #98761, but @Profpatsch didn't want it

My main problem is that it's relatively easy to (accidentally) write a configuration which results in a stack overflow as shown in d844d41.

I don't have much of a problem to implement toPretty as an actual Nix-builtin, but until we can use this in nixpkgs, we'd still have to use lib.generators.toPretty. To me, this seems like an OK tradeoff to eliminate another source of a weird error :)

@Ma27 Ma27 force-pushed the showdefs-overflow branch from fba35bd to fe52892 Compare July 23, 2021 14:14
@infinisil
Copy link
Member

@Ma27 This isn't what @Profpatsch has been arguing in above threads. He argues that things like recursion limiting should be a separate function that first transforms the input, then passes it to toPretty. I don't really agree personally, I think having such recursion limiting built into toPretty would be very useful (hence my two PR's which added that (and more)).

@Ma27
Copy link
Member Author

Ma27 commented Jul 26, 2021

Whoops, sorry, thanks for pointing this out!

I guess we have (more or less) the same opinion on this, so I'd say, let's wait for @Profpatsch to chime in and see if we can agree on something mergeable :)

@Ma27
Copy link
Member Author

Ma27 commented Aug 20, 2021

@infinisil @Profpatsch any updates here regarding this discussion? :)

@Profpatsch
Copy link
Member

I’m pretty sure this should be a separate function indeed. nix is lazy for a reason, we can separate these things logically and then compose them together without incurring extra costs.

@Ma27
Copy link
Member Author

Ma27 commented Aug 23, 2021

OK, thanks for the feedback :)
Not sure when I'll get to it, but I'll see how this can be reasonably implemented :)

Ma27 added 4 commits August 25, 2021 23:18
When having e.g. recursive attr-set, it cannot be printed which is
solved by Nix itself like this:

    $ nix-instantiate --eval -E 'let a.b = 1; a.c = a; in builtins.trace a 1'
    trace: { b = 1; c = <CYCLE>; }
    1

However, `generators.toPretty` tries to evaluate something until it's
done which can result in a spurious `stack-overflow`-error:

    $ nix-instantiate --eval -E 'with import <nixpkgs/lib>; generators.toPretty {  } (mkOption { type = types.str; })'
    error: stack overflow (possible infinite recursion)

Those attr-sets are in fact rather common, one example is shown above, a
`types.<type>`-declaration is such an example. By adding an optional
`depthLimit`-argument, `toPretty` will stop evaluating as soon as the
limit is reached:

    $ nix-instantiate --eval -E 'with import ./Projects/nixpkgs-update-int/lib; generators.toPretty { depthLimit = 2; } (mkOption { type = types.str; })' |xargs -0 echo -e
    "{
      _type = \"option\";
      type = {
        _type = \"option-type\";
        check = <function>;
        deprecationMessage = null;
        description = \"string\";
        emptyValue = { };
        functor = {
          binOp = <unevaluated>;
          name = <unevaluated>;
          payload = <unevaluated>;
          type = <unevaluated>;
          wrapped = <unevaluated>;
        };
        getSubModules = null;
        getSubOptions = <function>;
        merge = <function>;
        name = \"str\";
        nestedTypes = { };
        substSubModules = <function>;
        typeMerge = <function>;
      };
    }"

Optionally, it's also possible to let `toPretty` throw an error if the
limit is exceeded.
When having a bogus declaration such as

    { lib, ... }:
    {
      foo.bar = mkOption {
        type = types.str;
      };
    }

the evaluation will terminate with a not-so helpful

    error: stack overflow (possible infinite recursion)

To make sure a useful error is still provided, I added a `depthLimit` of
`10` which should be perfectly sufficient to `toPretty` when it's used
in an error-case for `showDefs`.
The message I originally implemented here was to catch a mixup of
`config' and `options' in a `types.submodule'[1]. However it looks
rather weird for a wrongly declared top-level option.

So I decided to throw

    error: The option `foo' does not exist. Definition values:
           - In `<unknown-file>':
               {
                 bar = {
                   _type = "option";
                   type = {
                     _type = "option-type";
               ...

           It seems as you're trying to declare an option by placing it into `config' rather than `options'!

for an expression like

    with import ./lib;

    evalModules {
      modules = [
        {
          foo.bar = mkOption {
            type = types.str;
          };
        }
      ];
    }

[1]  fa30c9a
As suggested in NixOS#131205.

Now it's possible to pretty-print a value with `lib.generators` like
this:

    with lib.generators;
    toPretty { }
      (withRecursion { depthLimit = 10; } /* arbitrarily complex value */)

Also, this can be used for any other pretty-printer now if needed.
@Ma27 Ma27 force-pushed the showdefs-overflow branch from fe52892 to 5773ae9 Compare August 25, 2021 22:33
@Ma27
Copy link
Member Author

Ma27 commented Aug 25, 2021

I added a function called withRecursion where you can do something like lib.generators.whatever-you-like {} (lib.generators.withRecursion { depthLimit = 10; } your-value). I'm not entirely sure, but is this the direction you'd like to go in? I'm open for further suggestions, of course :)

@Ma27
Copy link
Member Author

Ma27 commented Aug 31, 2021

cc @Profpatsch would you mind re-reviewing this in case you currently have time&energy for that? :)

@Ma27
Copy link
Member Author

Ma27 commented Sep 14, 2021

cc @Profpatsch any chance to get another review? :)

@Ma27
Copy link
Member Author

Ma27 commented Sep 28, 2021

cc @infinisil @Profpatsch is there anything else I can do to get this merged? :)

Copy link
Member

@lheckemann lheckemann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Text nits, but I'm all for merging if these are resolved.

@lheckemann lheckemann merged commit a3df3d0 into NixOS:master Sep 29, 2021
@Ma27 Ma27 deleted the showdefs-overflow branch September 29, 2021 09:04
roberth added a commit to hercules-ci/nixpkgs that referenced this pull request Dec 9, 2023
Ma27 added a commit that referenced this pull request Dec 10, 2023
lib/modules: Test optionless module errors from #131205
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants