How to include controller nodes as OSDs in ceph when installed with rook?

Question

I have a Kubernetes cluster with three control and three worker nodes. To install Ceph I'm using rook. After the installation, only the worker nodes are included in the Ceph cluster - but not the controller nodes.

The configuration snippet:

storage:                                                                                                                                                                       
  useAllNodes: true
  useAllDevices: true

Is there an easy way to also include the controller nodes in the Ceph cluster? Or: do I need to explicitly list all controller and worker nodes in the storage nodes list?

Answer 1

The root cause for the problem was, that the master nodes were tainted in a way that no 'normal' pod was allowed to run there.

Adapting the cluster.yaml configuration of rook and adding the following lines which explicitly allowed the ceph pods run also on master nodes solved the problem:

placement:
  all:
    tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/master

Thanks a lot to Hackaholic who pointed me in this direction.

Update 2024-03: The above solution stopped working for me using kubernetes 1.26.3 and kubespray 1.10.13. IMHO one of the reasons is, that the master label is deprecated. Using the deprecated master and new control-plane label works for me - specifying only one does not. So the current solution looks like:

placement:                                                                                                                                       
  all:                                                                                                                                           
    tolerations:                                                                                                                                 
      - effect: NoSchedule                                                                                                                       
        key: node-role.kubernetes.io/control-plane                                                                                               
      - effect: NoSchedule                                                                                                                       
        key: node-role.kubernetes.io/master