NixOps: How to deploy to an existing NixOS VM?

Question

I have almost the same problem as in this question, but it was never answered:

nixops: how to use local ssh key when deploying on machine with existing nixos (targetEnv is none)?

I'm not using Terraform though. Just NixOS + NixOps. So far, I:

• Created a new VM on Vultr
• Did a standard NixOS install from the current iso (20.09 pre something), setting a root password
• Enabled ssh with root password authentication and did a nixos-rebuild switch
• Manually generated an ssh keypair on my laptop
• sshed into the VM with the password and added the public key to /root/.ssh/authorized_keys

Now I can ssh into the VM manually with the new key, as expected:

ssh -i .secrets/vultrtest1_rsa [email protected]

Cool. Next, I copied the existing NixOS config files to my laptop and tried to wire them up to NixOps. I tried a minimal test1.nix, as well as adding the deployment."none" and/or users.users.root.openssh sections below.

vultrtest1
├── configuration.nix
└── hardware-configuration.nix
test1.nix

# test1.nix
{
  network.description = "vultr test 1";
  network.enableRollback = true;

  vultrtest1 = { config, pkgs, ... } : {
    deployment.targetHost = "XXX.XXX.XXX.XXX";
    imports = [ ./vultrtest1/configuration.nix ];

    # deployment.targetEnv = "none"; # existing nixos vm

    # same result with or without this section:
    deployment."none" = {
      sshPrivateKey = builtins.readFile ./secrets/vultrtest1_rsa;
      sshPublicKey  = builtins.readFile ./secrets/vultrtest1_rsa.pub;
      sshPublicKeyDeployed = true;
    };

    # same result with or without this:
    users.users.root.openssh.authorizedKeys.keyFiles = [ ./secrets/vultrtest1_rsa.pub ];
  };

}

In all cases, when I try to create and deploy the network NixOps tries to generate another SSH key, then fails to log in with it:

$ nixops create test1.nix -d test1
created deployment ‘b4ac25fa-c842-11ea-9a84-00163e5e6c00’
b4ac25fa-c842-11ea-9a84-00163e5e6c00

$ nixops list
+--------------------------------------+-------+------------------------+------------+------+
| UUID                                 | Name  | Description            | # Machines | Type |
+--------------------------------------+-------+------------------------+------------+------+
| b4ac25fa-c842-11ea-9a84-00163e5e6c00 | test1 | Unnamed NixOps network |          0 |      |
+--------------------------------------+-------+------------------------+------------+------+

$ nixops deploy -d test1                                                             
vultrtest1> generating new SSH keypair... done
[email protected]: Permission denied (publickey,keyboard-interactive).
vultrtest1> could not connect to ‘[email protected]’, retrying in 1 seconds...
[email protected]: Permission denied (publickey,keyboard-interactive).
vultrtest1> could not connect to ‘[email protected]’, retrying in 2 seconds...
[email protected]: Permission denied (publickey,keyboard-interactive).
vultrtest1> could not connect to ‘[email protected]’, retrying in 4 seconds...
[email protected]: Permission denied (publickey,keyboard-interactive).
vultrtest1> could not connect to ‘[email protected]’, retrying in 8 seconds...
[email protected]: Permission denied (publickey,keyboard-interactive).
Traceback (most recent call last):
  File "/nix/store/kybdy5m979h4kvswq2gx3la3rpw5cq5k-nixops-1.7/bin/..nixops-wrapped-wrapped", line 991, in <module>
    args.op()
  File "/nix/store/kybdy5m979h4kvswq2gx3la3rpw5cq5k-nixops-1.7/bin/..nixops-wrapped-wrapped", line 412, in op_deploy
    max_concurrent_activate=args.max_concurrent_activate)
  File "/nix/store/kybdy5m979h4kvswq2gx3la3rpw5cq5k-nixops-1.7/lib/python2.7/site-packages/nixops/deployment.py", line 1063, in deploy
    self.run_with_notify('deploy', lambda: self._deploy(**kwargs))
  File "/nix/store/kybdy5m979h4kvswq2gx3la3rpw5cq5k-nixops-1.7/lib/python2.7/site-packages/nixops/deployment.py", line 1052, in run_with_notify
    f()
  File "/nix/store/kybdy5m979h4kvswq2gx3la3rpw5cq5k-nixops-1.7/lib/python2.7/site-packages/nixops/deployment.py", line 1063, in <lambda>
    self.run_with_notify('deploy', lambda: self._deploy(**kwargs))
  File "/nix/store/kybdy5m979h4kvswq2gx3la3rpw5cq5k-nixops-1.7/lib/python2.7/site-packages/nixops/deployment.py", line 996, in _deploy
    nixops.parallel.run_tasks(nr_workers=-1, tasks=self.active_resources.itervalues(), worker_fun=worker)
  File "/nix/store/kybdy5m979h4kvswq2gx3la3rpw5cq5k-nixops-1.7/lib/python2.7/site-packages/nixops/parallel.py", line 44, in thread_fun
    result_queue.put((worker_fun(t), None, t.name))
  File "/nix/store/kybdy5m979h4kvswq2gx3la3rpw5cq5k-nixops-1.7/lib/python2.7/site-packages/nixops/deployment.py", line 979, in worker
    os_release = r.run_command("cat /etc/os-release", capture_stdout=True)
  File "/nix/store/kybdy5m979h4kvswq2gx3la3rpw5cq5k-nixops-1.7/lib/python2.7/site-packages/nixops/backends/__init__.py", line 337, in run_command
    return self.ssh.run_command(command, self.get_ssh_flags(), **kwargs)
  File "/nix/store/kybdy5m979h4kvswq2gx3la3rpw5cq5k-nixops-1.7/lib/python2.7/site-packages/nixops/ssh_util.py", line 280, in run_command
    master = self.get_master(flags, timeout, user)
  File "/nix/store/kybdy5m979h4kvswq2gx3la3rpw5cq5k-nixops-1.7/lib/python2.7/site-packages/nixops/ssh_util.py", line 200, in get_master
    compress=self._compress)
  File "/nix/store/kybdy5m979h4kvswq2gx3la3rpw5cq5k-nixops-1.7/lib/python2.7/site-packages/nixops/ssh_util.py", line 57, in __init__
    "‘{0}’".format(target)
nixops.ssh_util.SSHConnectionFailed: unable to start SSH master connection to ‘[email protected]’

What am I missing? Perhaps I can manually add the key NixOps just generated?

Update: I used SQLiteBrowser to look inside the NixOps state database and pasted the generated public key into authorized_keys. Now I can ssh in with the newly generated key manually, but NixOps still fails to deploy.

Answer 1

Solved it temporarily, in a not-very-satisfying way:

• browsed the database for the public + private key NixOps generated
• manually added those to authorized_keys on the VM
• also added the old key to the local ~/.ssh with an entry in ~/.ssh/config

No idea why NixOps uses the local ssh config, or how to prevent that. The entry that works looks like:

Host XXX.XXX.XXX.XXX
  HostName XXX.XXX.XXX.XXX
  Port 22
  User root
  IdentityFile ~/.ssh/vultrtest1_rsa

Will wait a couple days, then mark this as the solution unless anyone can explain how to tell NixOps to use the local key from .secrets instead of ~/.ssh.

Answer 2

Looking at the source at

https://github.com/NixOS/nixops/blob/master/nix/options.nix

there is deployment.provisionSSHKey option

which says.

  deployment.provisionSSHKey = mkOption {
  type = types.bool;
  default = true;
  description = ''
    This option specifies whether to let NixOps provision SSH deployment keys.
    NixOps will by default generate an SSH key, store the private key in its state file,
    and add the public key to the remote host.
    Setting this option to <literal>false</literal> will disable this behaviour
    and rely on you to manage your own SSH keys by yourself and to ensure
    that <command>ssh</command> has access to any keys it requires.
  '';
};

Maybe this can help? Once i'll get back to my Nixops machine, I'll give it a try.