Can't Signing and pushing trust metadata in Notary

Question

I am implementing Notary in a virtual machine. To have a reference, I have a docker registry on host A and I want to deploy Notary Server, Signer and CLI on host B to get push images to registry and sign them from different machine. However, the problem happens when I try to sign an image on host B of Notary with role targets. The following error message appears:

[root@HostB ~]# docker push my.registry:443/galera-leader-proxy:v1.0.0
The push refers to a repository [my.registry:443/galera-leader-proxy]
5f70bf18a086: Layer already exists
1de59669c563: Layer already exists
17dd9fb03617: Layer already exists
26093688fdcb: Layer already exists
e08be57f5919: Layer already exists
v1.0.0: digest: sha256:6e48967416ea76ba2825511da7b05107a41f585629009d18ccbf30a1e1ce0e5a size: 2179
Signing and pushing trust metadata
ERRO[0000] couldn't add target to targets: could not find necessary signing keys, at least one of these keys must be available: b92334936cf0a0f0e3fb9dce459212537387847ee288ce27762fd54850f89e6f
Failed to sign "my.registry:443/galera-leader-proxy":v1.0.0 - could not find necessary signing keys, at least one of these keys must be available: b92334936cf0a0f0e3fb9dce459212537387847ee288ce27762fd54850f89e6f
Error: could not find signing keys for remote repository my.registry:443/galera-leader-proxy, or could not decrypt signing key: could not find necessary signing keys, at least one of these keys must be available: b92334936cf0a0f0e3fb9dce459212537387847ee288ce27762fd54850f89e6f

Docker image is pushed to the registry but at the time of signing I get the error message that does not find the "keys" to sign. However, if I see the keys of notary, the key that can not be found to sign if it is available. Then I do not know why this happens or that I have configured badly:

[root@HostB ~]# dockernotary key list

ROLE        GUN                          KEY ID                                                              LOCATION
----        ---                          ------                                                              --------
root                                     7b8139837e3bf8b013f69bf0750d46ba0f70a6a6d9640eadcb592ae8a5ae2c0d    /home/gmaurelia/.docker/trust/private
snapshot    ...43/galera-leader-proxy    92cf3f72d573cab7b6045f72fe224a4ccf786e9ddd29c89b3a542b610061c763    /home/gmaurelia/.docker/trust/private
targets     ...43/galera-leader-proxy    b92334936cf0a0f0e3fb9dce459212537387847ee288ce27762fd54850f89e6f    /home/gmaurelia/.docker/trust/private


PD: alias dockernotary="notary -c 
/home/gmaurelia/.docker/trust/config.json -d 
/home/gmaurelia/.docker/trust/ -s https://notary-server:4443"

I can not even sign under the role: targets or targets/releases

Answer 1

For notary on multiple hosts, you need to perform a delegation step on your first host. This is a multi-step process documented by docker that involves the following:

• generate a TLS key pair on host B (the below includes a self signed step, you could also sign by a trusted CA):

  openssl genrsa -out delegation.key 2048

  openssl req -new -sha256 -key delegation.key -out delegation.csr

  openssl x509 -req -sha256 -days 365 -in delegation.csr -signkey delegation.key -out delegation.crt

• Copy the crt file from host B to host A and add the new certificate delegation with a notary command on host A. Then publish that change up to the server (the below assumes docker.io is your server):

  notary delegation add docker.io/<username>/<imagename> targets/releases delegation.crt --all-paths

  notary publish docker.io/<username>/<imagename>

• Import the new TLS key on host B to be used by notary:

  notary key import delegation.key --role user

Now you should be able to generate signatures on host B.

With notary, you should take care to protect and backup the root certificate that was generated on host A. This is often referred to as the offline certificate. If security of your two hosts is not a concern (you fully trust them), you could simply copy the $HOME/.docker/trust folder between the two.

Answer 2

The problem I had was that before I docker push, I applied the command: notary init my.registry:443/collection so notary generated a collection with different keys and in this way I could not do push docker of any image under any role nor even targets.

Once I did it the right way, I applied the steps you mentioned to me and the problem was solved. The notary configuration is the following:

command: tree $HOME/.docker/trust/

.docker/trust
├── certs
│   ├── delegation.crt
│   └── proof
│       ├── delegation.crt
│       ├── delegation.csr
│       └── delegation.key
├── config.json
├── private
│   ├── root_keys
│   │   └── 4e46a197de40621094f86e0cea4aa892d7c3cfb1b3400c64f6d7d82e4b97a470.key
│   └── tuf_keys
│       ├── 3269a0858ca91001c543435d0242e747bd08e68b52533f1b42028388ed02c7e6.key
│       └── my.registry:443
│           └── galera-leader-proxy
│               └── 
|           873ba8267df2be149fba2230441961812159c35537b18c133247239f4bafa989.key
├── root-ca.crt
├── tls
│   └── my.registry:443
│       └── root-ca.crt
└── tuf
    └── my.registry:443
        └── galera-leader-proxy
            ├── changelist
            └── metadata
                ├── root.json
                ├── snapshot.json
                ├── targets
                │   ├── kube1.json
                │   └── releases.json
                ├── targets.json
                └── timestamp.json

On the other hand, to configure the client correctly I defined the following alias:

alias dockernotary="notary -c $HOME/.docker/trust/config.json -d $HOME/.docker/trust/ -s https://notary-server:4443"

Saludos.