nixos/httpd: support overridable virtual hosts

[aanderse]

I have split this PR into multiple commits to make it easier to review. Ignoring white space also helps in a few places. Before merging all commits should be squashed as the commits individually break various things.

Motivation for this change

• the virtualHosts option is of type listOf instead of attrsOf which doesn't allow users to modify a virtualHost once created
• each virtualHost defined is specific to either http or https which makes it extremely difficult to implement integration with security.acme.certs
• httpd module code is overly complex
• httpd module has fallen far behind the nginx module and as a result isn't used by many people on nixos

Things not yet done

• merge a PR to rewrite of the awstats module, with offer to do so generously provided by @aristaeus
• rewrite examples in the nixos manual under the abstractions section of the manual which reference services.httpd, something which @samueldr may have some ideas on
• write release notes
• squash all commits into a single before merging this PR

Things done

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Ensured that relevant documentation is up to date
• Fits CONTRIBUTING.md.

Notify maintainers

cc @

[aanderse]

Example of current test configuration:

{
  imports = [ /home/aaron/nixpkgs/nixos/modules/services/web-servers/apache-httpd/default.nix ];
  disabledModules = [
    "services/logging/awstats.nix"
    "services/web-servers/apache-httpd/default.nix"
  ];

  services.httpd = {
    enable = true;
    enablePHP = true;
    adminAddr = "webmaster@example.org";
    logPerVirtualHost = true;
    virtualHosts = {
      "test1.example.org" = {
        documentRoot = "/var/www/test1.example.org";
        addSSL = true;
        enableACME = true;
      };
      "test2.example.org" = {
        documentRoot = "/var/www/test2.example.org";
        forceSSL = true;
        useACMEHost = "test2.example.org";
      };
    };
  };
  
  security.acme.certs."test2.example.org" = {
    user = "wwwrun";
    group = "wwwrun";
    webroot = "/var/lib/acme/acme-challenges";
  };
}

[aanderse]

ping @redvers, @FPtje, and @Izorkin as users of the httpd module. Any review, testing, and feedback is appreciated.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/what-are-your-goals-for-20-03/4773/3

[aanderse]

cc @vanyaklimenko in case they have interest in testing this PR.

[aanderse]

@GrahamcOfBorg test haproxy proxy upnp

[aanderse]

@GrahamcOfBorg test limesurvey mediawiki
@GrahamcOfBorg test moodle wordpress

[ivan]

I applied this to nixpkgs master, converted my configuration, and merged my http:// and https:// virtualhosts into a single virtualhost. That broke https:// because I had

        listen = [
          { ip = "1.2.3.4"; port = 80; }
          { ip = "1.2.3.4"; port = 443; }
        ];

which caused Apache to run HTTP without SSL on port 443.

From the implementation in nginx, I correctly guessed that I could do this here too:

        listen = [
          { ip = "1.2.3.4"; port = 80; }
          { ip = "1.2.3.4"; port = 443; ssl = true; }
        ];

but this is perplexingly undocumented in apache-httpd/per-server-options.nix. (Yes, it really does work, but I cannot figure out why.)

[ivan]

Sorry! I was looking at the wrong branch locally and see that it is documented.

[ivan]

Migrating my configuration wasn't very painful and the migrated configuration appears to behave correctly.

I think this should be merged once someone else looks over it, or confirms that those six Apache-using modules in nixpkgs still work.

[aanderse]

@ivan I just need to test awstats PR and then merge it, and get a few more people testing this then I'm satisfied. At this point though I'm looking for review and testing, so thank you very much!

[aanderse]

@GrahamcOfBorg test limesurvey mediawiki
@GrahamcOfBorg test moodle wordpress
@GrahamcOfBorg test haproxy hitch proxy upnp

[aanderse]

I've been running this on 3 machines for a while now, @ivan has given it a look over and migrated their configuration to use it, all outstanding issues/requirements addressed... so merging.

[immae]

@aanderse
I’m a little surprised by the invasive enforced

        <Directory "${documentRoot}">
            Options Indexes FollowSymLinks
            AllowOverride None
            ${allGranted}
        </Directory> 

part of a vhost, is there no more way to define a "private" vhost? (like one where we want authenticated users only)

EDIT: seems like it’s not recent (I just never noticed it before) and it can be overriden by a subsequent Directory directive, so I guess it’s nothing new there..

[aanderse]

@immae your edit sums up the situation nicely. Not something I put in place, and easily overridden if you read through apache documentation to understand how merging happens.

That being said... the situation in this module is less than perfect. I have spent some time trying to work through a solution to this but unfortunately ran into some roadblocks. I'm hoping to come up with a solution to improve things at some point in the future.