Starting open-iscsi service in rancher logs error

Question

I'm testing Prometheus, with a cTor Storage Class with OpenEBS, in a Rancher 2.1 Cluster (1 node) with RancherOS running in Hyper-v. The cStor volume is not mounting, and the pod is launching a event:

Unable to mount volumes for pod "prometheus-server-595746cc5-7xrmt_prometheus(46f03eb4-630f-11e9-b3e5-00155daf3d1f)": timeout expired 
waiting for volumes to attach or mount for pod "prometheus"/"prometheus-server-595746cc5-7xrmt". list of unmounted volumes=[storage-volume]. list of 
unattached volumes=[config-volume storage-volume prometheus-server-token-2w8ng] a minute ago

I'm thinking that this is related with the output from command:

sudo ros s up open-iscsi

That is:

[rancher@kub-master ~]$ sudo ros s up open-iscsi
ERRO[0002] non-200 http response: 404
ERRO[0002] Failed to load rancher.docker.engine=(docker-18.09.5-ce): non-200 
http response: 404
INFO[0002] Project [os]: Starting project
INFO[0002] [0/21] [open-iscsi]: Starting
INFO[0092] [1/21] [open-iscsi]: Started
INFO[0092] Project [os]: Project started

So, the question is...the ERRO[0002] is important? where is documented how can i fix that?

I can't find how to fix that, and I tried:

• Reinstalling the open-iscsi service.
• As i'm unsure if the container cannot mount the volume because this error, then i have tried recreating the Pool, the Storage class, ..., and the error persists.

Answer 1

The error from the pod (unable to mount) is not enough information. Check the logs and events for the OpenEBS provisioner or any pod that starts with pvc in the openebs namespace. That should tell you the actual problem.

If you're running a 1-node cluster, did you tell OpenEBS not to make 3 replicas? It might be waiting for the other 2 replicas to come online and thus is not able to make the volume available for the prometheus Pod to use.

These are just thoughts off the top of my head.

The 404 error is strange, but it seems to have loaded the open-iscsi drivers without error. It might not matter. Check Github for an issue about it, and if there isn't one, please open one and document how to reproduce the error.

Answer 2

Can you check the prerequisites in the RancherOS for the OpenEBS installation from the OpenEBS documentation- docs.openebs.io? Copying the content from there.

If you are using RancherOS as the operating system for your Kubernetes cluster, you simply need to enable the iSCSI service and start it on all the hosts or nodes. If you are using Ubuntu or RHEL as the operating system for your Kubernetes cluster, you need to

• Verify iSCSI initiators are installed on all nodes (and )
• Add the extra_binds under Kubelet service in cluster YAML file to mount the iSCSI binary and configuration inside the Kubelet.

iSCSI services On RancherOS

To run iSCSI services, execute the following commands on each of the cluster hosts or nodes.

sudo ros s enable open-iscsi sudo ros s up open-iscsi

Run the below commands on all the nodes to make sure the below directories are persistent, by default these directories are ephemeral.

ros config set rancher.services.user-volumes.volumes [/home:/home,/opt:/opt,/var/lib/kubelet:/var/lib/kubelet,/etc/kubernetes:/etc/kubernetes,/var/openebs] system-docker rm all-volumes reboot

iSCSI services on RHEL or Ubuntu Step1: Verify iSCSI initiator is installed and services are running

OPERATING SYSTEM ISCSI PACKAGE COMMANDS RHEL / CentOS iscsi-initiator-utils yum install iscsi-initiator-utils -y sudo systemctl enable iscsid && sudo systemctl start iscsid modprobe iscsi_tcp Ubuntu/Debian open-iscsi sudo apt install open-iscsi sudo systemctl enable iscsid && sudo systemctl start iscsid modprobe iscsi_tcp

Step2: Add extra_binds under kubelet service in cluster YAML

After installing the initiator tool on your nodes, edit the YAML for your cluster, editing the kubelet configuration to mount the iSCSI binary and configuration, as shown in the sample below.

services: kubelet: extra_binds: - "/etc/iscsi:/etc/iscsi" - "/sbin/iscsiadm:/sbin/iscsiadm" - "/var/lib/iscsi:/var/lib/iscsi" - "/lib/modules"