roles/watch-store: always restart

[zowoq]

No description provided.

[Mic92]

bors merge
Also always good to keep @domenkozar in touch about potential issues we encounter with the code.

[bors]

Build succeeded:

• ci/hercules/evaluation
• ci/hercules/onPush/default

[domenkozar]

Where there any errors you encountered?

[Mic92]

Can't tell for sure but I guess @zowoq had his reason for changing the restart type.

[zowoq]

This PR was incorrect and didn't help, I only skimmed the log initially and overlooked the actual error that happens after reboot:

systemd[1]: Started Cachix watch store Agent.
cachix-watch-store-agent-start[3586]: cachix: ConnectionError (HttpExceptionRequest Request {
cachix-watch-store-agent-start[3586]:   host                 = "cachix.org"
cachix-watch-store-agent-start[3586]:   port                 = 443
cachix-watch-store-agent-start[3586]:   secure               = True
cachix-watch-store-agent-start[3586]:   requestHeaders       = [("Accept","application/json;charset=utf-8,application/json"),("Authorization","<REDACTED>"),("User-Agent","cachix 1.3.3")]
cachix-watch-store-agent-start[3586]:   path                 = "/api/v1/cache/nix-community"
cachix-watch-store-agent-start[3586]:   queryString          = "?"
cachix-watch-store-agent-start[3586]:   method               = "GET"
cachix-watch-store-agent-start[3586]:   proxy                = Nothing
cachix-watch-store-agent-start[3586]:   rawBody              = False
cachix-watch-store-agent-start[3586]:   redirectCount        = 10
cachix-watch-store-agent-start[3586]:   responseTimeout      = ResponseTimeoutDefault
cachix-watch-store-agent-start[3586]:   requestVersion       = HTTP/1.1
cachix-watch-store-agent-start[3586]:   proxySecureMode      = ProxySecureWithConnect
cachix-watch-store-agent-start[3586]: }
cachix-watch-store-agent-start[3586]:  (ConnectionFailure Network.Socket.getAddrInfo (called with preferred socket type/protocol: AddrInfo {addrFlags = [AI_ADDRCONFIG], addrFamily = AF_UNSPEC, addrSocketType = Stream, addrProtocol = 0, addrAddress = 0.0.0.0:0, addrCanonName = Nothing}, host name: Just "cachix.org", service name: Just "443"): does not exist (Temporary failure in name resolution)))
systemd[1]: cachix-watch-store-agent.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: cachix-watch-store-agent.service: Failed with result 'exit-code'.
systemd[1]: cachix-watch-store-agent.service: Consumed 26ms CPU time, received 536B IP traffic, sent 536B IP traffic.
systemd[1]: cachix-watch-store-agent.service: Scheduled restart job, restart counter is at 5.
systemd[1]: Stopped Cachix watch store Agent.
systemd[1]: cachix-watch-store-agent.service: Consumed 26ms CPU time, received 536B IP traffic, sent 536B IP traffic.
systemd[1]: cachix-watch-store-agent.service: Start request repeated too quickly.
systemd[1]: cachix-watch-store-agent.service: Failed with result 'exit-code'.
systemd[1]: Failed to start Cachix watch store Agent.

Also we've got cachix pinned to 1.3.3, both push and watch-store in 1.4.2 wasn't working, I saw multi-user was mentioned in NixOS/nixpkgs#223827 (comment) so that may be why 1.4 is an issue for us.

This was the error on 1.4:

Warning: /nix/store/*** is not valid, skipping
cachix: FatalError {fatalErrorMessage = "no such path /nix/store/***"}

[domenkozar]

To achieve endless restarts, you need to disable burst limiting and increase the restart period:

diff --git a/nixos/modules/services/system/cachix-watch-store.nix b/nixos/modules/services/system/cachix-watch-store.nix
index ec73c0bcdcf..63713f0d3bd 100644
--- a/nixos/modules/services/system/cachix-watch-store.nix
+++ b/nixos/modules/services/system/cachix-watch-store.nix
@@ -62,7 +62,13 @@ in
       after = [ "network-online.target" ];
       path = [ config.nix.package ];
       wantedBy = [ "multi-user.target" ];
+      unitConfig = {
+        # allow to restart indefinately 
+        StartLimitIntervalSec = 0;
+      };
       serviceConfig = {
+        # don't put too much stress on the machine when restarting
+        RestartSec = 1;
         # we don't want to kill children processes as those are deployments
         KillMode = "process";
         Restart = "on-failure";

I'll look into the errors you're getting with watch-store on 1.4.2, thanks for letting me know!

[domenkozar]

NixOS/nixpkgs#226831