forbidden: User , "code": 403 error while accessing kube-apiserver using curl

Question

Kubernetes version : v1.19.0

I have created a user and performed clusterrolebinding with a role cluster-admin.

[root@project1-master ~]# kubectl describe clusterrole cluster-admin
Name:         cluster-admin
Labels:       kubernetes.io/bootstrapping=rbac-defaults
Annotations:  rbac.authorization.kubernetes.io/autoupdate: true
PolicyRule:
  Resources  Non-Resource URLs  Resource Names  Verbs
  ---------  -----------------  --------------  -----
  *.*        []                 []              [*]
             [*]                []              [*]

[root@project1-master ~]# kubectl describe clusterrolebinding sajeesh  cluster-admin
Name:         sajeesh
Labels:       <none>
Annotations:  <none>
Role:
  Kind:  ClusterRole
  Name:  cluster-admin
Subjects:
  Kind  Name     Namespace
  ----  ----     ---------
  User  sajeesh


Name:         cluster-admin
Labels:       kubernetes.io/bootstrapping=rbac-defaults
Annotations:  rbac.authorization.kubernetes.io/autoupdate: true
Role:
  Kind:  ClusterRole
  Name:  cluster-admin
Subjects:
  Kind   Name            Namespace
  ----   ----            ---------
  Group  system:masters

I am able to run kubectl with this useraccount and get pods information :

[root@project1-master ~]# kubectl get pods --as sajeesh
NAME    READY   STATUS    RESTARTS   AGE
busyb   1/1     Running   3          21h

But when i try to access kube-apiserver using curl it show forbidden error as following :

[root@project1-master ~]# curl --cacert /etc/kubernetes/pki/ca.crt --cert sajeesh.crt --key sajeesh.key https://$IP:6443/api/v1/namespaces/default/pods/busyb
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "pods \"busyb\" is forbidden: User \"system:anonymous\" cannot get resource \"pods\" in API group \"\" in the namespace \"default\"",
  "reason": "Forbidden",
  "details": {
    "name": "busyb",
    "kind": "pods"
  },
  "code": 403

I have re-verified the cacert , cert & key i am providing with that user account .They are correct.

Any suggestions why this is happening and how to fix it.

Answer 1

finally managed to findout the problem.it is not related to kubernetes but with curl command i am using .

curl --cacert /etc/kubernetes/pki/ca.crt --cert sajeesh.crt --key sajeesh.key https://$IP:6443/api/v1/namespaces/default/pods/busyb 

when i used -v switch along with command , It showed :

* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/kubernetes/pki/ca.crt
  CApath: none
* warning: certificate file name "sajeesh.crt" handled as nickname; please use "./sajeesh.crt" to force file name
* NSS: client certificate not found: newsajeesh.crt
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

So basically it was looking for absolute path as input for arguments --cert & --key

curl --cacert /etc/kubernetes/pki/ca.crt --cert ./sajeesh.crt --key ./sajeesh.key https://$IP:6443/api/v1/namespaces/default/pods/busyb 

after giving the absolute path it worked fine and i am able to get the output.

Answer 2

The clusterrolebinding sajeesh has name sajeesh and Group system:masters.Hence the certificate need to have sajeesh as Common Name(CN) and system:masters as Organization(O). When the request goes to kubernetes API server it will inspect the certificate and match CN with name and O with Group in clusterrolebinding and if it does not match you get 403 Forbidden error.

You can verify above by running below command

openssl x509 -in sajeesh.crt -text -noout