nixos/crowdsec: refactor architecture#446307
Conversation
nixpkgs-ci
bot
added
10.rebuild-linux: 1-10
|
Alright, I added my changes for now. The only option which I currently have is 
and I'm unsure what the cause may be. Another big change is, that there's now only @nicomem may I ask if you can try out my branch if it fixes #445342 for you? Do you get the same error as I do? |
|
Thanks, I'll try your branch and get back to you on my results. I also got this error when I was trying to make it work on my machine but I don't remember exactly what fixed it. |
nicomem
left a comment
nicomem
left a comment
There was a problem hiding this comment.
Some feedback from me, looking at the changes before testing it
|
After applying #446307 (comment) and #446307 (comment), the However, it does not work correctly as if I call |
|
From what I have seen, the idiomatic way to make systemd.services.crowdsec.serviceConfig = {
StateDirectory = "crowdsec";
StateDirectoryMode = "0750";
ConfigurationDirectory = "crowdsec";
ConfigurationDirectoryMode = "0750";
}; |
|
People who use impermanence might appreciate |
Hm... should I add an option where the user can enable/disable it or should I just remove the |
After skimming through systemd.exec it looks like you're right. Nice one. Thanks! |
Hmm, I admit that I wasn't aware of this context. I have just been disabling anywhere it bothered me. I'd like some other people to way in on this. The impermanence issue is a bug and will hopefully be fixed sometime in the future, so that shouldn't be the only reason to forgo |
|
Anything I can do to help here? These options seem much more reasonable than the current ones. I'd also like to mention I'm getting this error when trying to run it: This is my current configuration: https://gist.github.com/poperigby/63ef3e5b6205b6a5e4de320bcb63838a |
Aye, testing! What you already did:
Thx! I'll see when I'll get back to this |
|
just tested this and ran into the same error as @poperigby. i looked in the journal and it seems like it needed the api.server.online_client.credentials_path set if anything needs to pull from crowdsec servers (anything on the hub, and possibly acquisitions) in addition the console setup fails and is a bit non intuitive. you need to go to the console and grab just the token out of the command it gives you and paste just that in the console.tokenFile. it then fails as it tries to write the console.yaml into /nix/store, which of course fails due to RO filesystem here is my configuration repo with link pointed to my system flake and the line where the crowdsec specific bits start: https://gitlab.com/Amethyst_Nightshade/nixos-flakes/-/blob/main/nixosConfigurations/tailnerd-agh.nix#L304 |
|
Alright, I'm back!
Hm... this doesn't sound nice.
It looks like as if you've changed your repository. |
|
Alright, I'd say we can start testing this again. I don't know exactly when I'll have time to test this on my own so I'd be happy if there would be enough testers here 👀 |
TornaxO7
force-pushed
the
crowdsec
branch
3 times, most recently
from
15604fe to
4b22b83
Compare
|
So the main blocker - the need to manually run Trying out every Completions on the environment = {
systemPackages =
let
cscliWrapper = pkgs.symlinkJoin {
name = "cscli";
paths = [
(pkgs.writeShellScriptBin "cscli" ''
exec systemd-run \
--quiet \
--pty \
--wait \
--collect \
--pipe \
--property=ExecPaths="${cfg.settings.config.config_paths.plugin_dir}" \
--property=User=${cfg.user} \
--property=Group=${cfg.group} \
--property=DynamicUser=true \
--property=StateDirectory="crowdsec crowdsec/hub" \
--property=StateDirectoryMode="0750" \
--property=ConfigurationDirectory="crowdsec crowdsec/acquis.d" \
--property=ConfigurationDirectoryMode="0750" \
-- \
${lib.getExe configuredCscli} "$@"
'')
(pkgs.runCommand "cscli-completions" { } ''
mkdir -p $out/share
ln -s ${cfg.package}/share/bash-completion $out/share/bash-completion
ln -s ${cfg.package}/share/zsh $out/share/zsh
ln -s ${cfg.package}/share/fish $out/share/fish
'')
];
};
in
[ cscliWrapper ];but it asks for the sudo password on each tab press, and displays every completion in duplicate... Which isn't ideal. Other than that, I've been using it for a month with all optional features and it works really well. |
|
Thank you for trying out the changes and reporting any issues :)
That's really weird. Why does it want to write to
This should be fixeable by just creating the file.
Hm... one idea would be to introduce an option where you can set a group name which will then be used in the sudoers-file to disable passwords for all users which are in this group to invoke the |
|
But really, is it really worthwile to allow Completions: I tried allowing versus normal root elevation prompt: If I do |
Depending on what is necessary and what not, I'd say Regarding the authentification: I have no idea yet on how to fix that |
|
Some documentation on cscli setup: https://docs.crowdsec.net/docs/next/log_processor/service-discovery-setup/ |
|
Tested on: NixOS 26.05, x86_64-linux, commit 675f491 Great work on this refactor — the module is a big improvement in clarity over the current upstream version. Engine, hub collections, acquisitions (journalctl + file), CAPI community blocklist (15k IPs pulled successfully), and the iptables firewall bouncer all work well together. Declarative hub management via hub.collections is especially nice. Issues required manual workarounds: CAPI registration: Setting settings.config.api.server.online_client.credentials_path is not enough on a fresh install — the credentials file doesn't exist yet and the setup service doesn't auto-register. I had to run cscli capi register --file /var/lib/crowdsec/online_api_credentials.yaml manually. Note that running cscli capi register without --file silently fails on NixOS because it tries to write to read-only /etc/crowdsec/. Firewall bouncer: registerBouncer.enable = true fails with Failed to set up mount namespacing: /var/lib/private/crowdsec: No such file or directory due to DynamicUser timing. Had to register manually with cscli bouncers add firewall-bouncer, save the key to a file, and use secrets.apiKeyPath. Also, crowdsec-firewall-bouncer.nix:93 still references the old settings.general.api.server.listen_uri instead of settings.config.api.server.listen_uri, so settings.api_url must be set explicitly. Caddy log permissions: CrowdSec file acquisition couldn't read Caddy logs because they default to 0600 caddy:caddy. Fixed by adding users.users.crowdsec.extraGroups = [ "caddy" ] and setting systemd.services.caddy.serviceConfig.UMask = "0027" so new logs are created as 0640 (group-readable). Existing logs need a one-time chmod 640 /var/log/caddy/*.log |
|
That's odd, @peterkmet as I don't have any issues with capi registration nor firewall bouncer registration, starting from a clean slate. And I tested that many many times. Did you delete all prior crowdsec config and state before using the new commit ? Would you mind sharing your config (cs and csfwb) and the logs of crowsec-setup and crowdsec-firewall-bouncer-register? Ideally would you mind reinstalling on a cleaned system ? after deleting /etc/crowdsec, /etc/var/private/crowdsec and /etc/var/private/crowdsec-firewall-bouncer, run systemctl start crowsec-setup and systemctl start crowdsec-firewall-bouncer-register. Finally run journalctl -xeu on these 2 services and post the output in this thread if they still fail. EDIT The commit you're using dates from 3 weeks from now. Current latest commit is 4b22b83. So that explains it: |
|
Testing PR #446307 - commit 4b22b83 (verified via nix-prefetch-url hash match) CrowdSec engine starts and runs correctly Issue 1: Firewall bouncer register - credentials permissions Overall the refactor is a big improvement - declarative hub management and structured settings are great. Just the two registration issues need fixing for a fully automated fresh deployment. |
|
Issue 1 The Can you please try this version of the firewall-bouncer https://pastebin.com/LYXVGU7U (or if you prefer, it's on my fork Issue 2 For EDIT: You also need to provide the path to a file containing the enroll key that you get from app.crowdsec.net. For example: services.crowdsec.settings.console.enrollKeyFile = config.age.secrets.${crowdSecEnrollKeySecretName}.path;Issue 3 Yes crowdsec is going to need specific access to any logfiles it is expected to read. You might even need to add them to |
|
@peterkmet since the pending changes I had have been merged in ba52755, and considering my remarks above, do you still have any problems ? Thanks for any feedback. It would be nice to get this merged before the next NixOS release. |
Hi James, today I spent some time playing with new module, here are my findings: Disclaimer I am relatively new to NixOS, so much of my testing and current workarounds were developed through trial and error. Some of the solutions I am using might not follow NixOS best practices, and I am open to suggestions on how these should be implemented "the right way" within the module's architecture. Tested Version: PR: NixOS/nixpkgs#311140 1. The "Cold Start" Issue (CAPI & Community Blocklists) Observation: On a fresh installation (tested on a clean Hetzner VM), the engine successfully registers with CAPI (cscli capi status is OK), but no community decisions are downloaded, even after significant wait time. Enrollment Context: I noticed that the community list is pulled immediately only if the instance is manually enrolled in the CrowdSec Console. While I see the recommendation to use services.crowdsec.settings.console.enrollKeyFile, I did not use it during this test to see the "out-of-the-box" behavior for a new user without a console account. Problem: If enrollment is effectively mandatory to receive the community blocklist, it should be clearly documented. Ideally, the module should still provide protection for "anonymous" instances, as they are currently left unprotected by global data for an extended period after installation. 2. Caddy Logs Acquisition & Permissions Trap Technical Detail: Caddy creates logs with 600 permissions (read/write for owner only). Since files are owned by caddy and group permissions are 0, group membership for crowdsec provides no access. Proposed Solution: The module could implement an ACL (Access Control List) via systemd.tmpfiles.rules to grant the crowdsec user read access (e.g., setfacl -m u:crowdsec:r /var/log/caddy/*.log). Alternatively, supporting log acquisition via systemd-journald might bypass these permission issues entirely. 3. Installation Resource Exhaustion (Hetzner 4GB RAM) Observation: Attempting to install NixOS with the CrowdSec module enabled in a single initial step caused the system to hang/freeze. Workaround: I had to perform a base NixOS install first and then add CrowdSec in a second nixos-rebuild step. This might indicate a resource spike during the initial service activation on lower-spec machines. 4. UX Improvement: Simplified Whitelisting Current Trial/Error Method: I am currently forcing a whitelist by manually injecting YAML into environment.etc, which feels overly complex and brittle for a NixOS module: Proposal: A native Nix option to abstract this (e.g., services.crowdsec.whitelist.ips = [ "..." ];) would be much more user-friendly. 5. Feature Suggestion: Native 3rd Party Blacklist Support Current Trial/Error Method: I’ve implemented a custom oneshot systemd service to fetch the Turris Sentinel Greylist and import it via cscli. It works, but it’s a "hacky" way to handle it: Proposal: Adding a declarative way to define external blacklist URLs directly in the module configuration would significantly improve the module's value as a security orchestrator. Final Verdict The module is a great step forward and the core services work well. However, the "out-of-the-box" experience regarding community data and log permissions needs refinement to be truly production-ready for the NixOS community. here is my crowdsec.nix https://pastebin.com/Pbqj8swE |
|
Hi thanks for the feedback. Some quick answers from my phone.
|
|
Hi, I just came across this PR. I saw that all the services.crowdsec.settings.general.api.client.credentials_path = lib.mkForce "\${CREDENTIALS_DIRECTORY}/csec-creds";
systemd.services.crowdsec.serviceConfig.LoadCredential = [ "csec-creds:csec-creds" ];This way, CrowdSec should pick up the path to the credentials using the |
From this line: systemd.services.crowdsec.serviceConfig.LoadCredential = [ "csec-creds:csec-creds" ];you are setting the path to I'd really like to stick to |
Nope, it's actually the name for the system-level systemd credential, which is set up by sops. It's hard to expose that path, because only root can read it, while crowdsec can't. Also, all of this runs in a NixOS native container (i.e. systemd-spawn). Here's the whole file: https://github.com/rharish101/nix-configs/blob/main/modules/containers/authelia.nix |
You can use |
|
Hm... fair enough. I'll see what I can do |
|
Perhaps you could do something like what the Immich module does? See this option's description: https://search.nixos.org/options?channel=25.11&query=services.immich.settings&show=services.immich.settings. I can set the Honestly, they should just create a |
| overwriteInstallConfigDir = | ||
| p: | ||
| lib.optionalString (p != null) '' | ||
| cp -a ${cfg.package}/share/crowdsec/config/${p} ${config_paths.config_dir} |
There was a problem hiding this comment.
The Nix store is read-only, so directories in the package are read-only. Something attempts to create files in the patterns directory, and this fails with a "permissions denied" message. As the "Updating hub..." print message isn't reached, I assume that cp first creates a directory without write permissions, and then attempts to copy inside it.
There was a problem hiding this comment.
I'm going to be attempting
mkdir ${config_paths.config_dir}/${p}
cp -a ${cfg.package}/share/crowdsec/config/${p} ${config_paths.config_dir}/${p}/
as a replacement for this cp line. This will leave the patterns themselves read only, which we'll see if that's fine.
There was a problem hiding this comment.
Let me quickly replace this with install.
There was a problem hiding this comment.
Alright, I just pushed some changes. Feel free to try it out again.
On my system, I'm getting a fully clean installation.
There was a problem hiding this comment.
Thanks for updating this even while you're busy!
There was a problem hiding this comment.
You're welcome. Didn't require a big change and I try to have a little window time to code :)
|
Just a little info: I'm currently in my exam-session (hence don't have that much free-time and energy left).
and create a PR to my branch (Reviewing shouldn't take too much time). |
Co-authored-by: Nicolas Mémeint <git@nicomem.com> Co-authored-by: Jarno Pelkonen <23549925+jarppiko@users.noreply.github.com> Co-authored-by: Yash Garg <me@yashgarg.dev> Co-authored-by: Gaël James <gaeljames@gmail.com> Co-authored-by: Azat Bahawi <azat@bahawi.net>
nixpkgs-ci
bot
added
the
2.status: merge conflict
17 participants
This PR should just refactor the
crowdsecmodule. It's almost a total rewrite. So I recommend just to look at the new file instead of the diff.Alright, so after looking a bit through the code, I've got the following suggestion:
configattribute or something similar because in my opinion it's really confusing to have to "config" attributes.The generated config file should bewrittensymlinked to the default path (/etc/crowdsec/config.yaml) which should fix nixos/crowdsec: fatal error when usingcscli explain#445337services.crowdsec.settings.capiandservices.crowdsec.settings.lapisince the user can set them throughservices.crowdsec.settings.general.Feel free to share your thoughts on my suggestions!
Closes #473707
Closes #469519
Closes #459224
Closes #458206
Closes #445342
Closes #445337
Things done
passthru.tests.nixpkgs-reviewon this PR. See nixpkgs-review usage../result/bin/.Add a 👍 reaction to pull requests you find important.