CockroachDB distributed workload on all nodes

Question

I've deployed a CockroachDB cluster on Kubernetes using this guide:

https://github.com/cockroachlabs-field/kubernetes-examples/blob/master/SECURE.md

I deployed it with

$ helm install k8crdb --set Secure.Enabled=true cockroachdb/cockroachdb --namespace=thesis-crdb

Here is how it looks when I list it with $ helm list --namespace=thesis-crdb

NAME    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                  APP VERSION
k8crdb  thesis-crdb     1               2021-01-29 20:18:25.5710691 +0100 CET   deployed        cockroachdb-5.0.4      20.2.4

Here is how it looks when I list it with $ kubectl get all --namespace=thesis-crdb

NAME                                READY   STATUS      RESTARTS   AGE
pod/k8crdb-cockroachdb-0            1/1     Running     0          3h1m
pod/k8crdb-cockroachdb-1            1/1     Running     0          3h1m
pod/k8crdb-cockroachdb-2            1/1     Running     0          3h1m
pod/k8crdb-cockroachdb-init-j2h7t   0/1     Completed   0          3h1m

NAME                                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)              AGE
service/k8crdb-cockroachdb          ClusterIP   None            <none>        26257/TCP,8080/TCP   3h1m
service/k8crdb-cockroachdb-public   ClusterIP   10.99.163.201   <none>        26257/TCP,8080/TCP   3h1m

NAME                                  READY   AGE
statefulset.apps/k8crdb-cockroachdb   3/3     3h1m

NAME                                COMPLETIONS   DURATION   AGE
job.batch/k8crdb-cockroachdb-init   1/1           33s        3h1m

Now I wanna simulate traffic to this cluster. First I access the pod with: $ kubectl exec -i -t -n thesis-crdb k8crdb-cockroachdb-0 -c db "--" sh -c "clear; (bash || ash || sh)"

Which gets me inside the first pod/node.

From here I initiate the workload

[root@k8crdb-cockroachdb-0 cockroach]# cockroach workload init movr 'postgresql://root@localhost:26257?sslmode=disable' 

And then I run the workload for 5 minutes

[root@k8crdb-cockroachdb-0 cockroach]# cockroach workload run movr --duration=5m 'postgresql://root@localhost:26257?sslmode=disable' 

I am aware that I'm running the workload on one node, but I was under the expression that the workload would be distributed among all nodes? Because when I monitor the performance with the cockroachDB console I see that it's only the first node that is doing all the work, and the other nodes are idle.

As you can see the second (and third node) haven't had any workload at all. Is this just a visual glitch in the console? Or how can I run the workload so it get distributed evenly among all nodes in the cluster?

-UPDATE-

Yes, glad you brought up the cockroachdb-client-secure pod, because that's where I no longer could follow the guide. I tried as they did in the guide by doing: $ curl https://raw.githubusercontent.com/cockroachdb/cockroach/master/cloud/kubernetes/client-secure.yaml | sed -e 's/serviceAccountName\: cockroachdb/serviceAccountName\: k8crdb-cockroachdb/g' | kubectl create -f -

But it throws this error:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1638  100  1638    0     0   4136      0 --:--:-- --:--:-- --:--:--  4146
Error from server (Forbidden): error when creating "STDIN": pods "cockroachdb-client-secure" is forbidden: error looking up service account default/k8crdb-cockroachdb: serviceaccount "k8crdb-cockroachdb" not found

I also don't know if my certificates have been approved, because when I try this: $ kubectl get csr k8crdb-cockroachdb-0 --namespace=thesis-crdb

I throws this:

Error from server (NotFound): certificatesigningrequests.certificates.k8s.io "k8crdb-cockroachdb-0" not found

And when I try to approve certificate: $ kubectl certificate approve k8crdb-cockroachdb-0 --namespace=thesis-crdb

It throws:

Error from server (NotFound): certificatesigningrequests.certificates.k8s.io "k8crdb-cockroachdb-0" not found

Any idea how to proceed from here?

Answer 1

This is not a glitch. Nodes will only receive SQL traffic if clients connect to them and issue SQL statements. It seems like you're running the workload by logging in to one of the cockroach pods and directing it to connect to that pod on its local port. That means only that pod is going to receive queries. The cockroach workload subcommand takes an arbitrary number of pgurl strings and will balance load over all of them. Note also that k8crdb-cockroachdb-public represents a load-balancer over all o

If you look at the guide you posted, it continues to describe how to deploy the cockroachdb-client-secure pod. Th If you were to run the workload there pointed at the load balancer, with something like:

'postgres://root@k8crdb-cockroachdb-public?sslcert=cockroach-certs%2Fclient.root.crt&sslkey=cockroach-certs%2Fclient.root.key&sslrootcert=cockroach-certs%2Fca.crt&sslmode=verify-full'

---

UPDATE

I'm not an expert in the k8s here but I think your issue creating the client pod relates to the namespace. It's currently assuming that everything is in the default namespace but it appears that you're working in the --namespace=thesis-crdb. Consider adding a namespace flag to the kubectl create -f - command. Or, potentially consider setting the namespace for the session:

kubectl config set-context --current --namespace=thesis-crdb