nixos/crowdsec: init module

[TornaxO7]

"Continue-PR" of #387625.
Note: I'm not really experienced with nix but I can try to help to maintain this.

Things done

• Built on platform:
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• Tested, as applicable:
  • NixOS tests in nixos/tests.
  • Package tests at passthru.tests.
  • Tests in lib/tests or pkgs/test for functions and "core" functionality.
• Ran nixpkgs-review on this PR. See nixpkgs-review usage.
• Tested basic functionality of all binary files, usually in ./result/bin/.
• Nixpkgs Release Notes
  • Package update: when the change is major or breaking.
• NixOS Release Notes
  • Module addition: when adding a new NixOS module.
  • Module update: when the change is significant.
• Fits CONTRIBUTING.md, pkgs/README.md, maintainers/README.md and others READMEs.

---

Add a 👍 reaction to pull requests you find important.

[rharish101]

I've tested this (from the previous PR, but nothing substantial has changed since then) for both log processors and local API servers using systemd-nspawn containers on my home server, and it works well. The CrowdSec console is also able to see my setup.

Here's a diagram of my setup:

Legend:

1. Blue box: A host/NixOS installation (either a physical machine or a VM).
2. Red box: A systemd-nspawn container that has a veth pair with the host.
3. Yellow box: A systemd-nspawn container that only has bridge networking (with other containers), i.e. it can only access the internet through red boxes.
4. Green box: A systemd service.

For this setup, I only had to change the following in this crowdsec.nix file:

1. I changed RestartSec of the crowdsec systemd service to 5, since I observed that after syncing with the CrowdSec hub, it would shut itself down and the logs say to reload the service. So now I just have to wait 5s for the service to reload, and not 60s. Although, this issue is probably because the containers are ephemeral, and whatever it downloads from the hub is expected to be persisted.
2. I had to remove the ~@privileged syscall limit (i.e. allowing @privileged syscalls) from the service config of the crowdsec systemd service. Otherwise, the log processors would crash from a SIGSYS, which is presumably systemd killing it because it tried to use that syscall. Note that I'm running the service as root (which should be fine, since the container is using user namespacing, so root in the container is some random user in the host), which explains the issue. I'm running it as root, since otherwise I don't know how to allow the crowdsec service in the container to read my SOPS-nix secrets.
3. EDIT: I also had to add the crowdsecurity/linux collection to the hub collections whenever I want to use a log processor. For example, if I wanted to use the LePresidente/authelia collection, just adding that to the collections lists didn't work (as in, the authelia parser crashed). I had to also add the Linux collection.
4. EDIT: openFirewall only opens 6060 and 8080. If I change the port via settings.general.api.server.listen_uri, then it doesn't open the custom port. IDK if this is the usual behaviour in NixOS modules, but it's something that I noticed.

I'm a noob at self-hosting, so some of these issues might be just misconfigurations/stupid decisions from my side.

[acuteaangle]

@rharish101 Perhaps off-topic(?), but systemd credentials can be used to pass secrets into services.

---

(I’m not able to directly test this PR at the moment, but I should be able to try it out when I get home.)

[rharish101]

@rharish101 Perhaps off-topic(?), but systemd credentials can be used to pass secrets into services.

(off-topic) Wow, yet another reason why systemd rocks. Thanks a lot!

With this, my second point about the ~@privileged syscall filter is now moot.

[acuteaangle]

Reviewing now

[acuteaangle]

I’m out of time to keep looking through this tonight.
I didn’t get a chance to set it up and run it locally tonight, so I may have missed an obvious lint/formatting issue somewhere.

dynamicUser will likely require changing a bunch of the directory constants.

systemd-confinement (source) is something we could also potentially look into. Not many nixpkgs modules use it currently, as it was incompatible with dynamicUser in the past.

I’m unfamiliar with the nixos tests, so I don’t know what sort of testing would be expected of a service like crowdsec. ‘internetworked intrusion prevention system’ feels like a tricky category of module to test, but is perhaps doable with some sort of container setup.

[acuteaangle]

Again, would be nice if we can use DynamicUser

[TornaxO7]

I found a different opinion from another active NixOS maintainer: https://c3d2.social/@sandro/114922075808595518

@SuperSandro2000 may I ask if your contra-arguments for including DynamicUser are significant enough to not use it or not?

[acuteaangle]

(I’m not the mentioned user, but I’d like to make my case for DynamicUser.)

why does every other new #NixOS module need to use DynamicUser

RFC 0052

It is not compatible with easily providing sops secrets

systemd credentials may be used

and using LoadCredentials is just a pain in the ass.

fair enough.
In my opinion, the module should—if possible—abstract this.
forgejo does, as do nextcloud, paperless, lemmy, and many others.

it usually has negligible benefits.

Using dynamicUser means we don’t need to persist state in the form of a UID mapping, and is recommended for NixOS modules unless ‘problematic’. It may indeed be problematic for crowdsec—I haven’t yet had a chance to check what capabilities it needs to acquire logs. If it does work, I believe it is the best option.

[acuteaangle]

Would wrapProgram (full docs) from make(Binary)Wrapper be a simpler solution here?

[TornaxO7]

May I ask how that should look like?

      cscli = pkgs.writeShellScriptBin "cscli" ''
        wrapProgram $out/bin/cscli
      ''

like this?

[acuteaangle]

Whoops, forgot I was looking at a module, not a package.

https://github.com/NixOS/nixpkgs/blob/3e9c7e5c0dabf27e365d553630990a64a1f340ab/nixos/modules/services/security/crowdsec.nix#L469-L470

This feels like something that should be handled in the package.

https://github.com/NixOS/nixpkgs/blob/3e9c7e5c0dabf27e365d553630990a64a1f340ab/nixos/modules/services/security/crowdsec.nix#L471-L475

This looks fine.
Of note: I believe sudo will drop most environment variables (by default); that may or may not be desirable.

I’d probably take after paperless and try to make it a little more robust,

        sudo=exec
        if [ "$USER" != "${cfg.user}" ]; then
          ${
             if config.security.sudo.enable then
               "sudo='exec ${config.security.wrapperDir}/sudo -u ${cfg.user}"
             else
               ">&2 echo 'Aborting, cscli must be run as user `${cfg.user}`!'; exit 2"
           }
        fi     
        $sudo ${lib.getExe' cfg.package "cscli"} -c=${configFile} "$@"

but most modules use the pattern exactly as already written, so I have no objections either way.

[06kellyjac]

Yeah, we can adjust cscli to have crowdsec in the path

[TornaxO7]

This feels like something that should be handled in the package.

So you mean that I should create a PR first to apply this to the package?

[priegger]

Typo in the pr title: crowsec - > crowdsec.

[TornaxO7]

Typo in the pr title: crowsec - > crowdsec.

Thanks! Fixed it

[TornaxO7]

Wrote some responses to the review.

[TornaxO7]

Do you mean as the default username? I'd prefer to stick to crowdsec here since it's also more intuitive in my opinion.

[TornaxO7]

May I ask how that should look like?

      cscli = pkgs.writeShellScriptBin "cscli" ''
        wrapProgram $out/bin/cscli
      ''

like this?

[TornaxO7]

Now this is interesting. I did execute nix fmt but I'm still getting the lint error. Does anyone know what I'm doing wrong?

[acuteaangle]

I haven’t found it yet, but it’s almost always trailing whitespace: the autoformatter doesn’t fix it, but the lint does complain about it.

[acuteaangle]

Hmm. Unable to reproduce locally. Cloning the PR and running nix-shell --run treefmt returns successfully.
I can’t tell if it’s grabbing the same commit. I have 3e9c7e5c0dabf27e365d553630990a64a1f340ab locally, but I don’t see that hash in the logs.
Rebase on master, run nix fmt, force push again.
If it doesn’t resolve itself from that, ping @NixOS/nix-formatting

[TornaxO7]

Although I don't know why it's working now.

[TornaxO7]

nvm... nix-shell --run treefmt did some changes but it looks like as if it's not enough...

[TornaxO7]

@NixOS/nix-formatting I'm currently having the problem that after nix-shell -p treefmt I'm still not passing the CI although the command doesn't change anything:

I pushed the last formatting changes with git push --force-with-lease.
I'm a bit out of ideas what I could do. May I ask if you have any suggestions what I could try next? (I also tried nix fmt).

[acuteaangle]

Oh! I think maybe I see it. Line 509–510.
maybe move that [ up to be on the same line as the = and then format again?

I’m on a mobile device at the moment, so I can’t try it out.

[MattSturgeon]

I'm currently having the problem that after nix-shell -p treefmt

I assume you mean nix-shell --run treefmt?

Using -p would enter a shell with the unwrapped treefmt package installed; but we want to use the treefmt wrapped with nixpkgs' config, so we enter the current directory's shell and run treefmt.

As @piegamesde said, the shell's pinned version was recently bumped, so your branch is likely using a different version of nixfmt to CI.

Rebasing, then re-running treefmt should help.

[TornaxO7]

Yay, rebasing worked indeed. Thank you for the help :)

[TornaxO7]

I'm a bit unsure which reviews I should apply in which way. Any suggestions what I have to do to remove all "blockers"?

[acuteaangle]

Can anyone confirm that they have been able to successfully test this module?

I’ve been trying to setup a minimal test case equivalent to the stock fail2ban rule (watch logs via journald, monitor for failed SSH logins, ban via iptables).

However, I have not been successful so far.
I don’t understand how the api options work, as there doesn’t appear to be an api option definition.

Additionally, doesn’t crowdsec-firewall-bouncer also need a module?
Should it be handled as a part of this module? services.crowdsec.firewall-bouncer.enable?

EDIT:
I forgot that @rharish101 reported testing the module.
Did you need to create a custom service definition for crowdsec-firewall-bouncer?

[rharish101]

Can anyone confirm that they have been able to successfully test this module?

I’ve been trying to setup a minimal test case equivalent to the stock fail2ban rule (watch logs via journald, monitor for failed SSH logins, ban via iptables).

Yup, here is my config for an SSH log processor (that doesn't act like a security engine):

services.crowdsec = {
  enable = true;
  autoUpdateService = true;
  name = "${config.networking.hostName}-sshd";

  localConfig.acquisitions = [
    {
      source = "journalctl";
      journalctl_filter = [ "_SYSTEMD_UNIT=sshd.service" ];
      labels.type = "syslog";
      use_time_machine = true;
    }
  ];
  hub.collections = [ "crowdsecurity/linux" ];
  settings.lapi.credentialsFile = config.sops.secrets."crowdsec/sshd-creds".path;
};

For the CrowdSec security engine, this is my config:

# Add secrets using an environment file.
systemd.services.crowdsec.serviceConfig.EnvironmentFile = envFile;
services.crowdsec = {
  enable = true;
  autoUpdateService = true;
  name = "${config.networking.hostName}-lapi";

  # XXX: CrowdSec refuses to start unless some acquisitions are specified.
  localConfig.acquisitions = [
    {
      source = "journalctl";
      journalctl_filter = [ "_SYSTEMD_UNIT=ssh.service" ];
      labels.type = "syslog";
    }
  ];

  settings.general = {
    db_config = {
      type = "postgres";
      user = "crowdsec";
      password = "\${DB_PASSWORD}";
      db_name = "crowdsec";
      host = constants.bridges.csec-pg.pg.ip4;
      port = constants.ports.postgres;
    };
    api.server = {
      enable = true;
      listen_uri = "0.0.0.0:${toString constants.ports.crowdsec}";
      console_path = "/var/lib/crowdsec/credentials/console.yaml";
      online_client.credentials_path = "/var/lib/crowdsec/credentials/capi.yaml";
    };
  };
  settings.lapi.credentialsFile = "/var/lib/crowdsec/credentials/lapi.yaml";
};

Additionally, doesn’t crowdsec-firewall-bouncer also need a module? Should it be handled as a part of this module? services.crowdsec.firewall-bouncer.enable?

EDIT: I forgot that @rharish101 reported testing the module. Did you need to create a custom service definition for crowdsec-firewall-bouncer?

I wrote a custom module (just for my own use case) for a crowdsec-firewall-bouncer systemd service:

{
  config,
  lib,
  pkgs,
  ...
}:
{
  options.modules.crowdsec-bouncer = {
    enable = lib.mkEnableOption "Enable CrowdSec firewall bouncer";
  };

  config =
    let
      package = pkgs.crowdsec-firewall-bouncer;
      configFmt = pkgs.formats.yaml { };
      configFile = configFmt.generate "crowdsec-firewall-bouncer.yaml" {
        mode = "ipset";
        log_mode = "stdout";
        api_url = "\${API_URL}";
        api_key = "\${API_KEY}";
      };
      ip4List = "crowdsec-blacklists";
      ip6List = "crowdsec6-blacklists";
    in
    lib.mkIf config.modules.crowdsec-bouncer.enable {
      networking.firewall.extraPackages = [ pkgs.ipset ];
      networking.firewall.extraCommands = ''
        ipset -exist create ${ip4List} hash:net timeout 3600
        ipset -exist create ${ip6List} hash:net family inet6 timeout 3600
        iptables -A INPUT -m set --match-set ${ip4List} src -j DROP
        iptables -A FORWARD -m set --match-set ${ip4List} src -j DROP
        ip6tables -A INPUT -m set --match-set ${ip6List} src -j DROP
        ip6tables -A FORWARD -m set --match-set ${ip6List} src -j DROP
      '';

      environment.systemPackages = [
        package
        pkgs.ipset
      ];

      sops.secrets."crowdsec/bouncer-env".restartUnits = [ "crowdsec-firewall-bouncer.service" ];

      # Reference: https://github.com/crowdsecurity/cs-firewall-bouncer/blob/main/config/crowdsec-firewall-bouncer.service
      systemd.services.crowdsec-firewall-bouncer = {
        description = "The firewall bouncer for CrowdSec";
        after = [
          "syslog.target"
          "network.target"
          "remote-fs.target"
          "nss-lookup.target"
          "crowdsec.service"
        ];
        wantedBy = [ "multi-user.target" ];
        path = with pkgs; [
          iptables
          ipset
        ];
        serviceConfig = {
          Type = "notify";
          ExecStart = [ "${lib.getExe package} -c ${configFile}" ];
          ExecStartPre = [ "${lib.getExe package} -c ${configFile} -t" ];
          ExecStartPost = [ "${lib.getExe' pkgs.coreutils-full "sleep"} 0.1" ];
          Restart = "always";
          RestartSec = 10;
          LimitNOFILE = 65536;
          KillMode = "mixed";
          EnvironmentFile = config.sops.secrets."crowdsec/bouncer-env".path;
        };
      };
    };
}

[TornaxO7]

I'm going to clean up the commit history if everything is fine.

[TornaxO7]

@acuteaangle I'm a bit unsure if I'm understanding the DynamicUser thingy correctly:
So I just need to apply DynamicUser = true, right?
Should I give the crowdsec-update-hub also the DynamicUser = true option? I don't see why I sholdn't.

[rharish101]

Is there a clean way of handling the hub update branch? Right now, it's hard-coded to v${cfg.package.version} (unless lib.mkForce is used, I guess) in line 634 here: https://github.com/TornaxO7/nixpkgs/blob/cf632eed832bc3e5e1929d56b9fa0ac151821c49/nixos/modules/services/security/crowdsec.nix#L634

I updated nixpkgs, and CrowdSec is now at 1.6.11. This led to the service created by this module to stop working, since the CrowdSec hub doesn't have a v1.6.11 branch yet. Here's the link to an issue I opened upstream: crowdsecurity/hub#1444.

The maintainer suggested I change the branch to master, and I verified that it worked (by changing that line to say master by default).

[TornaxO7]

The maintainer suggested I change the branch to master, and I verified that it worked (by changing that line to say master by default).

I'd say it's fine to set it to master.
But what do others think about this?

Hm... maybe we can let the user set the version on their own and give it the default value master?

[TornaxO7]

(Just a reminder that I'm still open for discussion/feedback 👀 )

[rharish101]

I'd say it's fine to set it to master. But what do others think about this?

Hm... maybe we can let the user set the version on their own and give it the default value master?

I agree, we could expose an option to set it, say with services.crowdsec.hub.branch, that has a default value of master.

@TornaxO7 If you want to test with DynamicUser, you can also let me know/send me patches. I can try it out on weekends with my setup and report back.

[TornaxO7]

What. Why did it got closed?

[TornaxO7]

Aw come on.

[acuteaangle]

There should be a button to reopen.

[TornaxO7]

I can't reopen it. Somehow I messed it really up. However, I've created a new PR: #437310

[MattSturgeon]

What. Why did it got closed?

Looks like it was closed as part of you pushing? Maybe you force-pushed a history that had no diff against master?

IIRC, GitHub usually closes PRs when the branch is deleted or has no diff with the base branch. And until the branch returns to a state where it has changes not in the base branch, it can't be re-opened either.

[TornaxO7]

Hm... probably because I had an outdated version on my main machine...