Support building fully remotely

[wmertens]

If you're running nixops on your OS X laptop and deploy, it will first download a bunch of things to your laptop and fail at some point because you're on Darwin. Next, you log in on the deployed remote host and add your own ssh key so that nix-build can use the host. Then, you deploy again and it starts uploading what it just downloaded. This is not a nice user story.

It would be great if there was support for a buildhost, which runs the deployment build and then pushes to the other nodes in the network. One or more of the machines in the network are marked as workers and one is marked as nixops master. If the nixops master is defined it will perform all downloads and coordinate builds, and all other machines are populated from it.

Something like that? Or at the very least documenting a workflow like "deploy a VM from this image, do this to make it into a regular nixos machine with nixops and copy your network definitions on there" would be nice.

[copumpkin]

👍 ran into the exact same issue

[domenkozar]

I have a patch somewhere...

[wmertens]

@domenkozar about that patch... send it to me and I'll clean it up? Or is it ready to go?

[edolstra]

In theory this is already supposed to work. From the code:

        # If we're not running on Linux, then perform the build on the
        # target machines.  FIXME: Also enable this if we're on 32-bit
        # and want to deploy to 64-bit.
        if platform.system() != 'Linux' and os.environ.get('NIX_REMOTE') != 'daemon':
            if os.environ.get('NIX_REMOTE_SYSTEMS') == None:
                remote_machines = []
                for m in sorted(selected, key=lambda m: m.index):
                    key_file = m.get_ssh_private_key_file()
                    if not key_file: raise Exception("do not know private SSH key for machine ‘{0}’".format(m.name))
                    # FIXME: Figure out the correct machine type of ‘m’ (it might not be x86_64-linux).
                    remote_machines.append("root@{0} {1} {2} 2 1\n".format(m.get_ssh_name(), 'i686-linux,x86_64-linux', key_file))
                    # Use only a single machine for now (issue #103).
                    break
                remote_machines_file = "{0}/nix.machines".format(self.tempdir)
                with open(remote_machines_file, "w") as f:
                    f.write("".join(remote_machines))
                os.environ['NIX_REMOTE_SYSTEMS'] = remote_machines_file
            else:
                self.logger.log("using predefined remote systems file: {0}".format(os.environ['NIX_REMOTE_SYSTEMS']))

            # FIXME: Use ‘--option use-build-hook true’ instead of setting
            # $NIX_BUILD_HOOK, once Nix supports that.
            os.environ['NIX_BUILD_HOOK'] = os.path.dirname(os.path.realpath(nixops.util.which("nix-build"))) + "/../libexec/nix/build-remote.pl"

            load_dir = "{0}/current-load".format(self.tempdir)
            if not os.path.exists(load_dir): os.makedirs(load_dir, 0700)
            os.environ['NIX_CURRENT_LOAD'] = load_dir

[domenkozar]

Yeah the work that needs to be done is to check if platforms match and provide a way to override the default to on or off.

[domenkozar]

Found it, it was written by @goodwillcoding https://gist.github.com/goodwillcoding/9c536748ba80da2eeebd

[edolstra]

Hm, calling uname on every machine seems unnecessary. Presumably NixOps already knows the architecture of the remote machines (e.g. from the nixpkgs.system option).

[wmertens]

@edolstra actually there are two problems and my report above doesn't highlight them properly.

• Remote building doesn't work out of the box, due to nix-build not getting SSH credentials from nixops
• When building remotely does work, all packages are still downloaded locally and pushed from local. This is very slow.

[wmertens]

How about designating a push host, from which other systems get populated? Defaulting to localhost but if set to the build host and if SSH behaviour is fixed, that would behave as I'd prefer.

[edolstra]

The problem with a push host is that there is no guarantee that it has connectivity to the other machines (e.g. if you deploy a network with EC2 machines in different regions).

[wmertens]

@edolstra But in that case you simply keep the push host as localhost and
then things are as they are now.

I'm now wondering how to make that work. The push host needs to initiate
the builds so it needs to run nix-daemon and the .drv files need to make it
over there, but other than that it doesn't need anything right?

So deploying from scratch with a remote push host would be:

• nixops scans config for hosts, brings up servers
• if remote push host
  • build .drv files with correct platform
  • once push host up, copy .drv files over, run nix-daemon and
    instantiate

Would that work?

On Wed, Apr 15, 2015 at 2:27 PM Eelco Dolstra notifications@github.com
wrote:

The problem with a push host is that there is no guarantee that it has
connectivity to the other machines (e.g. if you deploy a network with EC2
machines in different regions).

—
Reply to this email directly or view it on GitHub
#260 (comment).

[wmertens]

Forgot to add:

• Once everything built, use the push host to push everything and
  activate like normally

On Wed, Apr 15, 2015 at 3:12 PM Wout Mertens wout.mertens@gmail.com wrote:

@edolstra But in that case you simply keep the push host as localhost and
then things are as they are now.

I'm now wondering how to make that work. The push host needs to initiate
the builds so it needs to run nix-daemon and the .drv files need to make it
over there, but other than that it doesn't need anything right?

So deploying from scratch with a remote push host would be:

• nixops scans config for hosts, brings up servers
• if remote push host
  • build .drv files with correct platform
  • once push host up, copy .drv files over, run nix-daemon and
    instantiate

Would that work?

On Wed, Apr 15, 2015 at 2:27 PM Eelco Dolstra notifications@github.com
wrote:

The problem with a push host is that there is no guarantee that it has
connectivity to the other machines (e.g. if you deploy a network with EC2
machines in different regions).

—
Reply to this email directly or view it on GitHub
#260 (comment).

[domenkozar]

Just for reference, this also addresses #195

[ryanartecona]

I just downloaded and started playing with nixops, starting with examples from the manual, and hit this issue pretty quickly. I'm also too new to Nix{,OS,ops} to unstick myself even by manually fixing things up as described in this issue description. No fun!