nixos/sourcehut: updates, fixes, hardening

[ju1m]

This PR is still baking a bit, but is ready for testing and reviewing.

Motivation for this change

Be able to install and use latest https://sr.ht/~sircmpwn/sourcehut/ .
Beware, this is a huge change, and it may or may not cause troubles for people using the previous NixOS module. Which was no longer functional when I started this PR (due to a wrong --loglevel=info passed to celery instead of --loglevel INFO).
Fixes #126090

Things done

• Fix update.sh to handle Go modules.
• Add a working metasrht-manager into systemPackages.
• Repurpose the services option to only serve as selecting what (local or remote) services are displayed, and use an enable option for each local service of sourcehut.
• Factorize configuration of Sourcehut services.
• Type-check and describe many settings.
• Configuring of credentials outside the Nix store.
• systemd hardening of Sourcehut services.
• Use systemd timers instead of cron.
• Use a dedicated (and thus hardenable) gitsrht-fcgiwrap.service instead of services.fcgiwrap.
• Manual check services of timers work as expected.
• Only metasrht.service has access to the stripe-secret-key now.
• Only enable docker when needed.
• Use mkEnableOption.
• Manual check webhooks are working as expected.
• Per service config.ini.
• Better working integration into sshd.
• Now configuring of local nginx, postfix, postgresql, and redis is made optional.
• Better nginx integration.
• Fix celery calls by specifying --loglevel INFO instead of --loglevel=info.
• Configure celery to use the eventlet concurrent networking library.
• Configure celery services to share a single Redis database.
• Manual testing of buildssrht. I don't use it, but the service and the worker both run.
• Manual testing of dispatchsrht. I don't use it.
• Manual testing of gitsrht.
• Manual testing of hgsrht. I don't use it.
• Manual testing of hubsrht.
• Manual testing of listssrht.
• Manual testing of metasrht.
• Manual testing of mansrht.
• Manual testing of pagessrht. The service is running, but I don't know how I should put content in it.
• Manual testing of pastesrht.
• Manual testing of todosrht.
• Update the meta.doc. But there may be more to do.
• Add myself in maintainers.
• Add a release entry. I'm exhausted for now.

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• 21.11 Release Notes (or backporting 21.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

Example of config:

{ pkgs, lib, config, ... }:
let
  inherit (config.services) sourcehut;
  inherit (config.users) users groups;
  # This `gnupg` module is from https://github.com/NixOS/nixpkgs/pull/93659
  # but use whatever secret scheme you prefer.
  inherit (config.security) gnupg;
  domain = "example.com";
in
{
services.sourcehut = {
  enable = true;
  listenAddress = "localhost";

  # Enable locally some Sourcehut services
  builds = {
    enable = true;
    enableWorker = true;
    images.nixos.unstable.x86_64 =
      let
        systemConfig = { pkgs, ... }: {
          # passwordless ssh server
          services.openssh = {
            enable = true;
            permitRootLogin = "yes";
            extraConfig = "PermitEmptyPasswords yes";
          };

          users = {
            mutableUsers = false;
            # build user
            extraUsers."build" = {
              isNormalUser = true;
              uid = 1000;
              extraGroups = [ "wheel" ];
              password = "";
            };
            users.root.password = "";
          };

          security.sudo.wheelNeedsPassword = false;
          nix.trustedUsers = [ "root" "build" ];
          documentation.nixos.enable = false;

          # builds.sr.ht-image-specific network settings
          networking = {
            hostName = "build";
            dhcpcd.enable = false;
            defaultGateway.address = "10.0.2.2";
            usePredictableInterfaceNames = false; # so that we just get eth0 and not some weird id
            interfaces."eth0".ipv4.addresses = [{
              address = "10.0.2.15";
              prefixLength = 25;
            }];
            enableIPv6 = false;
            nameservers = [
              # OpenNIC anycast
              "185.121.177.177"
              "169.239.202.202"
              # Google as a fallback :(
              "8.8.8.8"
            ];
            firewall.allowedTCPPorts = [ 22 ]; # allow ssh
          };

          environment.systemPackages = [
            pkgs.gitMinimal
            #pkgs.mercurial
            pkgs.curl
            pkgs.gnupg
          ];
        };
        qemuConfig = { pkgs, ... }: {
          imports = [ systemConfig ];
          fileSystems."/".device = "/dev/disk/by-label/nixos";
          boot.initrd.availableKernelModules = [
            "ahci"
            "ehci_pci"
            "sd_mod"
            "usb_storage"
            "usbhid"
            "virtio_balloon"
            "virtio_blk"
            "virtio_pci"
            "virtio_ring"
            "xhci_pci"
          ];
          boot.loader = {
            grub = {
              version = 2;
              device = "/dev/vda";
            };
            timeout = 0;
          };
        };
        config = (import (pkgs.path + "/nixos/lib/eval-config.nix") {
          inherit pkgs; modules = [ qemuConfig ];
          system = "x86_64-linux";
        }).config;
      in
      import (pkgs.path + "/nixos/lib/make-disk-image.nix") {
        inherit pkgs lib config;
        diskSize = 16000;
        format = "qcow2-compressed";
        contents = [
          { source = pkgs.writeText "gitconfig" ''
              [user]
                name = builds.sr.ht
                email = build@sr.ht
            '';
            target = "/home/build/.gitconfig";
            user = "build";
            group = "users";
            mode = "644";
          }
        ];
      };
  };
  #dispatch.enable = true;
  git.enable = true;
  hub.enable = true;
  lists.enable = true;
  meta.enable = true;
  man.enable = true;
  pages.enable = true;
  paste.enable = true;
  todo.enable = true;

  # Integration into local services
  postgresql.enable = true;
  postfix.enable = true;
  redis.enable = true;
  nginx.enable = true;

  settings = {
    "sr.ht" = {
      environment = "production";
      global-domain = domain;
      origin = "https://${domain}";
      owner-email = "root+srht@${domain}";
      owner-name = "Owner Name";
      site-blurb = "blurb";
      site-info = "https://${domain}";
      site-name = "Site Name";
      # nix shell nixpkgs#sourcehut.coresrht -c srht-keygen network
      network-key = gnupg.secrets."sourcehut/network-key".path;
      # nix shell nixpkgs#sourcehut.coresrht -c srht-keygen service
      service-key = gnupg.secrets."sourcehut/service-key".path;
    };

    # This is only needed for the pages.sr.ht service.
    # But may also be used to offer release storage in git.sr.ht,
    # and build artifact storage in builds.sr.ht
    objects = {
      s3-upstream = "localhost";
      s3-access-key = "12345";
      s3-secret-key = pkgs.writeText "s3-secret-key" "12345678";
    };

    # oauth-client-id of services are automatically authorized when the `metasrht-api.service` starts
    # TODO: I think it's better if oauth-client-id are different from one another.
    "builds.sr.ht" = {
      oauth-client-secret = gnupg.secrets."sourcehut/oauth-client-secret".path;
      oauth-client-id = "299db9f9c2013170";
    };
    "dispatch.sr.ht" = {
      oauth-client-secret = gnupg.secrets."sourcehut/oauth-client-secret".path;
      oauth-client-id = "299db9f9c2013170";
    };
    "hub.sr.ht" = {
      oauth-client-secret = gnupg.secrets."sourcehut/oauth-client-secret".path;
      oauth-client-id = "299db9f9c2013170";
    };
    "git.sr.ht" = {
      outgoing-domain = "http://git.${domain}";
      oauth-client-secret = gnupg.secrets."sourcehut/oauth-client-secret".path;
      oauth-client-id = "299db9f9c2013170";
    };
    "lists.sr.ht" = {
      oauth-client-secret = gnupg.secrets."sourcehut/oauth-client-secret".path;
      oauth-client-id = "299db9f9c2013170";
    };
    "lists.sr.ht::worker" = {
    };
    "man.sr.ht" = {
      oauth-client-secret = gnupg.secrets."sourcehut/oauth-client-secret".path;
      oauth-client-id = "299db9f9c2013170";
    };
    "meta.sr.ht" = {
    };
    "meta.sr.ht::settings" = {
      onboarding-redirect = "https://meta.${domain}";
      registration = true;
      # Or register users with something like:
      # metasrht-manageuser -t admin -e john@example.com john
    };
    "meta.sr.ht::api" = {
    };
    "pages.sr.ht" = {
      oauth-client-secret = gnupg.secrets."sourcehut/oauth-client-secret".path;
      oauth-client-id = "299db9f9c2013170";
      s3-bucket = "pagesbuck";
    };
    "paste.sr.ht" = {
      oauth-client-secret = gnupg.secrets."sourcehut/oauth-client-secret".path;
      oauth-client-id = "299db9f9c2013170";
    };
    "todo.sr.ht" = {
      oauth-client-secret = gnupg.secrets."sourcehut/oauth-client-secret".path;
      oauth-client-id = "299db9f9c2013170";
    };

    # nix shell nixpkgs#sourcehut.coresrht -c srht-keygen webhook
    webhooks.private-key = gnupg.secrets."sourcehut/webhook-key".path;

    mail = {
      smtp-host = "localhost";
      smtp-port = 25;
      smtp-user = null;
      smtp-password = null;
      smtp-from = "sourcehut@example.com";
      error-to = "root+sourcehut+error@example.com";
      error-from = "root+sourcehut+error@example.com";
      pgp-privkey = null;
      pgp-pubkey = null;
      pgp-key-id = null;
    };
  };
};

# Minio provides an S3 bucket, it is needed for the `pages` service
# and can be used by git.sr.ht and builds.sr.ht
# to offer release storage and build artifact storage, respectively
services.minio = {
  enable = true;
  accessKey = "12345";
  secretKey = "12345678";
  #region = "";
  browser = true;
};
environment.systemPackages = [ pkgs.minio-client ];
}

Note that other settings are needed for enabling and properly configuring local nginx, postfix, postgresql, and redis services, but I don't have the energy to put a minimal example here right now.

Beware that origin= settings are recorded into the SQL databases and not updated if you later change them (eg. by switching from HTTP to HTTPS). See for instance:

meta.sr.ht=# select * from user_webhook_subscription;
 id |          created           |                          url                           |           events           | user_id | token_id 
----+----------------------------+--------------------------------------------------------+----------------------------+---------+----------
  1 | 2021-08-14 01:37:05.252564 | http://git.sourcephile.wg/oauth/webhook/profile-update | profile:update             |       1 |        1
  2 | 2021-08-14 01:37:05.288726 | http://git.sourcephile.wg/webhook/notify/keys          | ssh-key:remove,ssh-key:add |       1 |        1
(2 rows)

Note that AFAIK those ssh-key:remove,ssh-key:add user webhooks will ultimately be phased out of sourcehut and authorized SSH public keys no longer cached into a Redis database.

[tomberek]

This is amazing... it will take me a bit of time to review and test.

[misuzu]

the hgsrht's user has read and write access to it.

[ju1m]

AFAICS the chowning done by systemd on StateDirectory= happens before BindPaths=are applied, hence if repos is changed from the default to some path outside StateDirectory= its ownership will be preserved.

[SuperSandro2000]

This only works on unstable.

[ju1m]

@SuperSandro2000, could you elaborate please? What is failing? And does it have to be fixed?

[SuperSandro2000]

nix build is a experimental feature behind the nix-command flag and does not work by default IIRC.

[tomberek]

Should the restrictions allow the pre-start services to write to /var/lib/sourcehut/*srht/*.oauth?

[tomberek]

This might have been a problem with my own setup, but I had to wipe the entire previous installation, DB+state. Note: clear cookies after changing secrets.

When pushing to git. Should check if the warnings/permissions for logs need to be fixed.

2021/08/15 02:58:24 repo ID 1; name 'thing'; owner ID 1; owner name 'tomberek'; visibility 'public'; pusher type 'admin'; pusher suspension notice ''; access grant ''
2021/08/15 02:58:24 Executing command: [git-receive-pack /var/lib/sourcehut/gitsrht/repos/~tomberek/thing]
Enumerating objects: 3, done.
Counting objects: 100% (3/3), done.
Writing objects: 100% (3/3), 203 bytes | 203.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
remote: 2021/08/15 02:58:24 Warning: unable to open log file: open /var/log/gitsrht-update-hook: permission denied (using stderr instead)
remote: hooks/pre-receive 2021/08/15 02:58:24 [hooks/pre-receive]
remote: hooks/pre-receive 2021/08/15 02:58:24 Running pre-receive for push f04eb06d-7eb0-4289-a2fe-cb1a9ae9d915
remote: 2021/08/15 02:58:24 Warning: unable to open log file: open /var/log/gitsrht-update-hook: permission denied (using stderr instead)
remote: hooks/update 2021/08/15 02:58:24 [hooks/update refs/heads/master 0000000000000000000000000000000000000000 a98c53622b5ad4f79e4158cd7e6aabc13f510c73]
remote: hooks/update 2021/08/15 02:58:24 Running update for push f04eb06d-7eb0-4289-a2fe-cb1a9ae9d915
remote: 2021/08/15 02:58:24 Warning: unable to open log file: open /var/log/gitsrht-update-hook: permission denied (using stderr instead)
remote: hooks/post-update 2021/08/15 02:58:24 [hooks/post-update refs/heads/master]
remote: hooks/post-update 2021/08/15 02:58:24 Running post-update for push f04eb06d-7eb0-4289-a2fe-cb1a9ae9d915
remote: hooks/post-update 2021/08/15 02:58:25 Skipping stage 3, no work
To git.srht.tomberek.info:~tomberek/thing
 * [new branch]      master -> master
Branch 'master' set up to track remote branch 'master' from 'origin'.

Edit: I just read the sshsrht comment: https://github.com/NixOS/nixpkgs/pull/133984/files#diff-4a251309b594fdd697eb23c8e2d64191de7e00668e928df0247cdab24cb71dfeR693-R705

[r-rmcgibbo]

Result of nixpkgs-review pr 133984 at 749d922f run on x86_64-linux 1

1 package failed to build:

• nixos-install-tools

12 packages built successfully:

• sourcehut.buildsrht
• sourcehut.coresrht
• sourcehut.dispatchsrht
• sourcehut.gitsrht
• sourcehut.hgsrht
• sourcehut.hubsrht
• sourcehut.listssrht
• sourcehut.mansrht
• sourcehut.metasrht
• sourcehut.pagessrht
• sourcehut.pastesrht
• sourcehut.todosrht

11 suggestions:

• warning: no-python-tests

  Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
  Near pkgs/applications/version-management/sourcehut/builds.nix:61:0:

     |
  61 |     description = "Continuous integration service for the sr.ht network";
     | ^
  

• warning: unused-argument

  Unused argument: writeText.
  Near pkgs/applications/version-management/sourcehut/core.nix:28:3:

     |
  28 | , writeText
     |   ^
  

• warning: no-python-tests

  Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
  Near pkgs/applications/version-management/sourcehut/git.nix:77:0:

     |
  77 |     description = "Git repository hosting service for the sr.ht network";
     | ^
  

• warning: no-python-tests

  Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
  Near pkgs/applications/version-management/sourcehut/dispatch.nix:36:0:

     |
  36 |     description = "Task dispatcher and service integration tool for the sr.ht network";
     | ^
  

• warning: no-python-tests

  Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
  Near pkgs/applications/version-management/sourcehut/man.nix:34:0:

     |
  34 |     description = "Wiki service for the sr.ht network";
     | ^
  

• warning: unclear-gpl

  agpl3 is a deprecated license, please check if project uses agpl3Plus or agpl3Only and change meta.license accordingly.

  Near pkgs/applications/version-management/sourcehut/pages.nix:34:5:

     |
  34 |     license = licenses.agpl3;
     |     ^
  

• warning: no-python-tests

  Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
  Near pkgs/applications/version-management/sourcehut/paste.nix:34:0:

     |
  34 |     description = "Ad-hoc text file hosting service for the sr.ht network";
     | ^
  

• warning: no-python-tests

  Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
  Near pkgs/applications/version-management/sourcehut/lists.nix:42:0:

     |
  42 |     description = "Mailing list service for the sr.ht network";
     | ^
  

• warning: no-python-tests

  Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
  Near pkgs/applications/version-management/sourcehut/meta.nix:71:0:

     |
  71 |     description = "Account management service for the sr.ht network";
     | ^
  

• warning: missing-patch-comment

  Consider adding a comment explaining the purpose of this patch on the line preceeding.
  Near pkgs/applications/version-management/sourcehut/core.nix:49:5:

     |
  49 |     ./disable-npm-install.patch
     |     ^
  

• warning: no-python-tests

  Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
  Near pkgs/applications/version-management/sourcehut/hg.nix:37:0:

     |
  37 |     description = "Mercurial repository hosting service for the sr.ht network";
     | ^
  

Note that build failures may predate this PR, and could be nondeterministic or hardware dependent.
Please exercise your independent judgement. Does something look off? Please file an issue or reach out on IRC.

---

Result of nixpkgs-review pr 133984 at 749d922f run on aarch64-linux 1

1 package failed to build:

• nixos-install-tools

12 packages built successfully:

• sourcehut.buildsrht
• sourcehut.coresrht
• sourcehut.dispatchsrht
• sourcehut.gitsrht
• sourcehut.hgsrht
• sourcehut.hubsrht
• sourcehut.listssrht
• sourcehut.mansrht
• sourcehut.metasrht
• sourcehut.pagessrht
• sourcehut.pastesrht
• sourcehut.todosrht

11 suggestions:

• warning: no-python-tests

  Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
  Near pkgs/applications/version-management/sourcehut/dispatch.nix:36:0:

     |
  36 |     description = "Task dispatcher and service integration tool for the sr.ht network";
     | ^
  

• warning: missing-patch-comment

  Consider adding a comment explaining the purpose of this patch on the line preceeding.
  Near pkgs/applications/version-management/sourcehut/core.nix:49:5:

     |
  49 |     ./disable-npm-install.patch
     |     ^
  

• warning: no-python-tests

  Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
  Near pkgs/applications/version-management/sourcehut/meta.nix:71:0:

     |
  71 |     description = "Account management service for the sr.ht network";
     | ^
  

• warning: no-python-tests

  Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
  Near pkgs/applications/version-management/sourcehut/builds.nix:61:0:

     |
  61 |     description = "Continuous integration service for the sr.ht network";
     | ^
  

• warning: no-python-tests

  Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
  Near pkgs/applications/version-management/sourcehut/hg.nix:37:0:

     |
  37 |     description = "Mercurial repository hosting service for the sr.ht network";
     | ^
  

• warning: unclear-gpl

  agpl3 is a deprecated license, please check if project uses agpl3Plus or agpl3Only and change meta.license accordingly.

  Near pkgs/applications/version-management/sourcehut/pages.nix:34:5:

     |
  34 |     license = licenses.agpl3;
     |     ^
  

• warning: unused-argument

  Unused argument: writeText.
  Near pkgs/applications/version-management/sourcehut/core.nix:28:3:

     |
  28 | , writeText
     |   ^
  

• warning: no-python-tests

  Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
  Near pkgs/applications/version-management/sourcehut/lists.nix:42:0:

     |
  42 |     description = "Mailing list service for the sr.ht network";
     | ^
  

• warning: no-python-tests

  Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
  Near pkgs/applications/version-management/sourcehut/man.nix:34:0:

     |
  34 |     description = "Wiki service for the sr.ht network";
     | ^
  

• warning: no-python-tests

  Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
  Near pkgs/applications/version-management/sourcehut/git.nix:77:0:

     |
  77 |     description = "Git repository hosting service for the sr.ht network";
     | ^
  

• warning: no-python-tests

  Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
  Near pkgs/applications/version-management/sourcehut/paste.nix:34:0:

     |
  34 |     description = "Ad-hoc text file hosting service for the sr.ht network";
     | ^
  

Note that build failures may predate this PR, and could be nondeterministic or hardware dependent.
Please exercise your independent judgement. Does something look off? Please file an issue or reach out on IRC.

[ju1m]

Should the restrictions allow the pre-start services to write to /var/lib/sourcehut/*srht/*.oauth?

@tomberek, AFAICS metasrht-api is the only service using those *.oauth files, owned by metasrht, so I don't see a problem here.
Thinking about it, I don't see the need for such caching, avoiding a few psql runs in preStart= is not worth the added complexity IMHO. I've removed the code for these *.oauth files.
I'm also making the psql command to use connexion-string so it will also work with a remote postgresql.

[ju1m]

I had to rebase with master because of fbafeb7, this messes up the diff shown by GitHub, but nothing has changed except what I've mentioned here, and the commits to fix the manual and appease nixpkgs-review.

[ju1m]

Pushed some more work on the buildsrht and buildsrht-worker front.
Those should now be configured. I've put in the demo in the intro of this PR @tomberek's example.

[ju1m]

Pushed some more work to fix and harden the integration of git.sr.ht into nginx.