fork/exec /usr/bin/rkt: permission denied

Question

I'm trying to install Kubernetes.

configuration details:

Controler

• OS: Container Linux by CoreOS alpha (1339.0.0)
• LAN IP: 192.168.1.2

• FQDN: coreos-2.tux-in.com

• environment variables used for installation:

  export MASTER_FQDN=coreos-2.tux-in.com
  export MASTER_IP=192.168.1.2
  export WORKER_IP=192.168.1.3
  export WORKER_FQDN=coreos-3.tux-in.com
  export ETCD_ENDPOINTS=http://127.0.0.1:4001
  export ADVERTISE_IP=192.168.1.2
  export K8S_VER=v1.5.4_coreos.0
  export HYPERKUBE_IMAGE_REPO=quay.io/coreos/hyperkube
  export POD_NETWORK=10.2.0.0/16
  export SERVICE_IP_RANGE=10.3.0.0/24
  export K8S_SERVICE_IP=10.3.0.1
  export DNS_SERVICE_IP=10.3.0.10
  export USE_CALICO=true
  export CONTAINER_RUNTIME=rkt
  export EMAIL="[email protected]"
  export uuid_file="/var/run/kubelet-pod.uuid"
  

worker

• OS: Container Linux by CoreOS alpha (1339.0.0)
• LAN IP: 192.168.1.3

• FQDN: coreos-3.tux-in.com

• environment variables used for installation

  export ETCD_ENDPOINTS=http://127.0.0.1:4001
  export CONTROLLER_ENDPOINT=https://coreos-2.tux-in.com
  export ADVERTISE_IP=192.168.1.3
  export K8S_VER=v1.5.4_coreos.0
  export HYPERKUBE_IMAGE_REPO=quay.io/coreos/hyperkube
  export POD_NETWORK=10.2.0.0/16
  export DNS_SERVICE_IP=10.3.0.10
  export USE_CALICO=true
  export CONTAINER_RUNTIME=rkt
  export EMAIL="[email protected]"
  export uuid_file="/var/run/kubelet-pod.uuid"
  export  CALICO_OPTS="--volume cni-bin,kind=host,source=/opt/cni/bin \
              --mount volume=cni-bin,target=/opt/cni/bin"
  

I used my fork of coreos-kubernetes to install the controller and worker (https://github.com/kfirufk/coreos-kubernetes).

the controller node is running fine, but the worker node complains permission denied when trying to run rkt on every image that kubernetes tells it to execute.

for example:

Mar 11 21:34:58 coreos-3.tux-in.com kubelet-wrapper[10203]: E0311 21:34:58.654536   10203 pod_workers.go:184] Error syncing pod ac3615637dcb614ace172dc5c5dd85f3, skipping: failed to SyncPod: failed to run [prepare --quiet --pod-manifest /tmp/manifest-kube-proxy-192.168.1.3-986457484 --stage1-name=coreos.com/rkt/stage1-fly]: fork/exec /usr/bin/rkt: permission denied

any ideas ?

Answer 1

it was a simple problem.

I point kubelet.service to run rkt with /opt/bin/host-rkt that contains:

#!/bin/sh
# This is bind mounted into the kubelet rootfs and all rkt shell-outs go
# through this rkt wrapper. It essentially enters the host mount namespace
# (which it is already in) only for the purpose of breaking out of the chroot
# before calling rkt. It makes things like rkt gc work and avoids bind mounting
# in certain rkt filesystem dependancies into the kubelet rootfs. This can
# eventually be obviated when the write-api stuff gets upstream and rkt gc is
# through the api-server. Related issue:
# https://github.com/coreos/rkt/issues/2878
exec nsenter -m -u -i -n -p -t 1 -- /usr/bin/rkt "$@"

and it wasn't with executable permission. that's it.. a simple chmod +x /opt/bin/host-rkt solved it :)