daemon: fix deadlock when SSH client disconnects during remote builds

[Mic92]

When a remote SSH client disconnects during a long-running operation like addToStore(), the nix-daemon can deadlock in a circular wait:

• Process A (SSH daemon): blocked reading from downstream store socket, waiting for response from local daemon
• Process B (local daemon): blocked reading from upstream socket, waiting for more NAR data from SSH daemon

The existing interrupt mechanism (ReceiveInterrupts + MonitorFdHup) correctly detects the SSH disconnect and sets _isInterrupted, but the daemon remains blocked in read() on the downstream store connection. Even though SIGUSR1 causes read() to return EINTR, the circular dependency prevents forward progress.

Fix this by adding shutdownConnections() to RemoteStore that calls shutdown(fd, SHUT_RDWR) on all tracked connection file descriptors. Register an interrupt callback in processConnection() that invokes this method when the store is a RemoteStore. This causes any blocking read() to return 0 (EOF), breaking the circular wait and allowing both processes to exit cleanly.

The fix tracks connection FDs in a synchronized set, populated when connections are created by the Pool factory. On interrupt, all FDs are shut down regardless of whether they're idle or in-use.

I have this setup now deployed for a while in my two CI setups (one of them with higher load, that triggered this issue more reliable) and haven't had any lockup since for a couple of weeks. I wish the solution was a bit cleaner, maybe someone can think of a better way, but I couldn't get the interrupt code to actually leave the daemon protocol loop without this code.
The key to produce this deadlock seem to be to build the same store path from different nix build instances in a concurrent way.

Motivation

Context

---

Add 👍 to pull requests you find important.

The Nix maintainer team uses a GitHub project board to schedule and track reviews.

[xokdvium]

Overall makes sense to me. I've also seen clients lock-up on a blocking write to a local daemon socket in FdSink in the destructor (it does a blocking flush in the destructor). Will this also fix it?

[Mic92]

Overall makes sense to me. I've also seen clients lock-up on a blocking write to a local daemon socket in FdSink in the destructor (it does a blocking flush in the destructor). Will this also fix it?

Only if MonitorFdHup is triggered. You might need to check if this thread is still active in your case.

[xokdvium]

I see, then that's a separate fix we should do.