--build-on-remote flag hangs up on "Building disko script" step and never finishes on MacOS

[onnimonni]

Issue

Even on the latest version of nixos-anywhere I can't get deployments working with --build-on-remote. I already tried to mention about this on #371 but it's probably cleaner to open new issue instead than continuing there.

Example logs

$ nix --extra-experimental-features "flakes nix-command" run github:nix-community/nixos-anywhere/ -- --debug --print-build-logs --build-on-remote --flake .#myHost [email protected]

...

### Building disko script ###
+ nixCopy --to ssh://[email protected] '.#nixosConfigurations."myHost".config.system.build.diskoScript' --derivation --no-check-sigs
+ NIX_SSHOPTS='-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i /tmp/tmp.ZiSqUZEHAN/nixos-anywhere '
+ nix copy --extra-experimental-features 'nix-command flakes' --no-write-lock-file -L --to ssh://[email protected] '.#nixosConfigurations."myHost".config.system.build.diskoScript' --derivation --no-check-sigs
Warning: Permanently added 'X.Y.Z.W' (ED25519) to the list of known hosts.

You can see my flake and diskoscripts in this example repo: https://github.com/onnimonni/hetzner-auction-nixos-example.

I can get this working if I kexec the server and then deploy directly there as I documented on the example git repo but not directly from my MacOS.

MacOS version

$ sw_vers
ProductName:		macOS
ProductVersion:		14.6.1
BuildVersion:		23G93

[Mic92]

Did it work with ssh-ng://?

[onnimonni]

Probably a stupid question but how would I use that? By using a copy of the script with my own mods?

[Mic92]

You can even take the nix copy command that you see in the debug output

[onnimonni]

So this is what I tried:

$ nix copy --extra-experimental-features 'nix-command flakes' --no-write-lock-file -L --to ssh-ng://[email protected] '.#nixosConfigurations."myHost".config.system.build.diskoScript' --derivation --no-check-sigs --debug --print-build-logs 2> nix-copy-debug.log

I redacted my ip-address and changed /Users/$USER to ~ in the log.

It doesn't do anything after the server accepts my public key and just waits. I pressed ctrl+c to stop the process after 15minutes.

I attached the logs here:
nix-copy-debug.log

Here's also my nix version:

$ nix --version
nix (Nix) 2.24.5

[sedlund]

log missing context of stdout -- looks like ssh issue

[onnimonni]

I tried this again today with the ssh provided by my mac:

$ ssh -V
OpenSSH_9.7p1, LibreSSL 3.3.6

and also again with homebrew installed ssh:

$ ssh -V
OpenSSH_9.9p1, OpenSSL 3.3.2 3 Sep 2024

I didn't get anything printed to stderr and just redirected all from stdout to this log file:

$ nix copy --extra-experimental-features 'nix-command flakes' --no-write-lock-file -L --substitute-on-destination --to ssh://[email protected] '.#nixosConfigurations."utm".config.system.build.diskoScript' --derivation --no-check-sigs --debug  --print-build-logs --verbose &> stdout-log.txt

In this case I was using nixos-anywhere with an UTM virtual machine and it's still the same case as with remote servers.

ssh to the UTM machine works perfectly without any issues:

$ ssh [email protected] whoami
root

Here are the logs which I saved:
stdout-log.txt

This is most likely caused by my own setup since it seems to be working well for everyone else. How could one debug what's going on with the ssh connection since this only happens with the nix copy command 🤔