how to seperate traffic in Rook

Question

AFAIK ceph have 2 specific trafic path:

1. Traffic between client and ceph nodes,
2. Traffic between ceph nodes (Inter Ceph-node).

So, let say my network is like this

Note :

1. Kube-node-4 is a kubernet worker that do not take part as rook node. Just a ceph-client
2. Red, Green and blue line is a seperate ethernet network.

Can I do trafic seperation like this using Rook? Is there any documentation on how to do it?

Sincerely

-bino-

Answer 1

Check out the ceph docs, what you describe is the separation of public and cluster networks. Cluster network is used for OSD <--> OSD traffic only (replication of PGs) while the public network is for Ceph clients as well as the other Ceph daemons (MON, MGR, etc). I'm not familiar with rook but according to the guide you have to override the config, to get the current config map run:

kubectl -n rook-ceph get ConfigMap rook-config-override -o yaml

Enable the hostNetwork setting in the Ceph Cluster CRD configuration. For example,

  network:
    provider: host

and then

Define the subnets to use for public and private OSD networks. Edit the rook-config-override configmap to define the custom network configuration:

kubectl -n rook-ceph edit configmap rook-config-override

In the editor, add a custom configuration to instruct ceph which subnet is the public network and which subnet is the private network. For example:

apiVersion: v1
data:
  config: |
    [global]
    public network =  10.0.7.0/24
    cluster network = 10.0.10.0/24
    public addr = ""
    cluster addr = ""

After applying the updated rook-config-override configmap, it will be necessary to restart the OSDs by deleting the OSD pods in order to apply the change. Restart the OSD pods by deleting them, one at a time, and running ceph -s between each restart to ensure the cluster goes back to "active/clean" state.