Checking API Access

January 2024

This blog post describes using the kubectl command line tool to check API access in Kubernetes. It especially focuses on the difference between the --user and --as options of the kubectl command line tool.

Pre-requisites

You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using minikube.

OpenSSL command line utility will be used to view the x509 certificates.

Role-based access control

Role-based access control (RBAC) is a method of regulating access to computer or network resources based on the roles of individual users within your organization.
It is advised to go through the Kubernetes RBAC documentation before reading ahead.

Checking API Access

The kubectl auth can-i command can be used to determine whether a user has permissions to execute a certain action.

Default Admin User

In the first example, we will work with the default admin user.

Check the contexts available in the minikube cluster
kubectl config get-contexts minikube
Output
CURRENT   NAME       CLUSTER    AUTHINFO   NAMESPACE
*         minikube   minikube   minikube   default
Check the name of the admin user that is created with our basic minikube installation.
kubectl config view -o jsonpath='{.contexts[?(@.name=="minikube")].context.user}'
Output
minikube
A user in Kubernetes is nothing but a key and certificate pair issued by the Kubernetes cluster and presented to the Kubernetes API.
Check the certificate of the minikube user
kubectl config view -o jsonpath='{.users[?(@.name=="minikube")].user.client-certificate}'
You should see an output with the path to the minikube user’s certificate
/Users/adityasamant/.minikube/profiles/minikube/client.crt
View the Subject of this certificate (use the path generated in the previous command)
openssl x509 -in /Users/adityasamant/.minikube/profiles/minikube/client.crt -text -noout | grep Subject | grep -v "Public Key Info"
Output
Subject: O=system:masters, CN=minikube-user

As can be seen above, the minikube admin user is part of the system:masters group.

system:masters is a group which is hardcoded into the Kubernetes API server source code as having unrestricted rights to the Kubernetes API server. Any user who is a member of this group has full cluster-admin rights to the cluster. Even if every cluster role and role is deleted from the cluster, users who are members of this group retain full access to the cluster.

Use the kubectl auth can-i command to verify a few scenarios.

Check permissions to create pods
kubectl auth can-i create pods
yes
Check permissions to create deployments
kubectl auth can-i create deployments
yes
Check permissions to delete secrets
kubectl auth can-i delete secrets
yes

Normal User

Configure a normal user and verify how the kubectl auth can-i commands can be used to check the access. To do this we need to issue a certificate for the user.

Create a private key and a csr file
openssl genrsa -out jane.key 2048
openssl req -new -key jane.key -out jane.csr -subj "/CN=jane"

This will generate a private key named jane.key and a certificate signing request named jane.csr.

Get the base64 encoded value of the CSR file content
cat jane.csr | base64 | tr -d "\n"
Create a CertificateSigningRequest
cat <<EOF | kubectl apply -f -
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: csr-name (1)
spec:
  request: base64 encoded csr (2)
  signerName: kubernetes.io/kube-apiserver-client
  expirationSeconds: 86400  # one day
  usages:
  - client auth
EOF
1 Provide the name as jane
2 Provide the base64 encoded value of the CSR
Get the CSR
kubectl get csr jane
NAME   AGE   SIGNERNAME                            REQUESTOR       REQUESTEDDURATION   CONDITION
jane   55s   kubernetes.io/kube-apiserver-client   minikube-user   24h                 Pending
Approve the CSR
kubectl certificate approve jane
kubectl get csr jane
NAME   AGE     SIGNERNAME                            REQUESTOR       REQUESTEDDURATION   CONDITION
jane   2m17s   kubernetes.io/kube-apiserver-client   minikube-user   24h                 Approved,Issued

Granting permissions via RBAC

Create a clusterrole granting permissions to only create pods
kubectl create clusterrole createpods --verb=create --resource=pods
clusterrole.rbac.authorization.k8s.io/createpods created
Create a clusterrolebinding to bind the clusterrole with user jane
kubectl create clusterrolebinding createpods --clusterrole=createpods --user=jane
clusterrolebinding.rbac.authorization.k8s.io/createpods created

Difference between 'as' and 'user' options of kubectl

kubectl has a number of global options that can be passed as an argument to any kubectl command. The list can be found with the following command:

kubectl options

Two of the options are --user and --as. It is important to understand the difference between them.

--user='':
The name of the kubeconfig user to use
--as='':
Username to impersonate for the operation. User could be a regular user or a service account in a namespace.

The --user option is used when you want to trigger the kubectl command under the context of a user which is configured in the kubeconfig file. This option throws an error if the user is not present in the kubeconfig file.

The --as option is used to impersonate any user or serviceaccount. --as can be used for a user irrespective of whether that user is present in the kubeconfig file or not.

Let’s put the theory into action with the help of the user we created.

Using --as to Check Access

Check permissions to create pods
kubectl auth can-i create pods --as=jane
yes
Check permissions to create deployments
kubectl auth can-i create deployments --as=jane
no
Check permissions to delete secrets
kubectl auth can-i delete secrets --as=jane
no

Due to the fact that we explicitly assigned the clusterrole createpods to user jane, we see that jane has access to create pods, but no access to create deployments or delete secrets. Great, this is as expected.

Using --user to Check Access

Try the same commands, but this time using the --user option

kubectl auth can-i create pods --user=jane
error: auth info "jane" does not exist

The command leads to an error. This is because we have not configured the user jane in the kubeconfig file.

Fix that by adding the user jane to the kubeconfig file.

Adding the user to kubeconfig

Get the user’s certificate
kubectl get csr jane -o jsonpath='{.status.certificate}'| base64 -d > jane.crt
Add the new credentials to kubeconfig
kubectl config set-credentials jane --client-key=jane.key --client-certificate=jane.crt --embed-certs=true
User "jane" set.
Add the context to kubeconfig
kubectl config set-context jane --cluster=minikube --user=jane
Context "jane" created.

Let’s try the kubectl auth can-i command once again to verify the permissions on user jane:

Check permissions to create pods
kubectl auth can-i create pods --user=jane
yes
Check permissions to create deployments
kubectl auth can-i create deployments --user=jane
no
Check permissions to delete secrets
kubectl auth can-i delete secrets --user=jane
no

Now everything works as expected.

The --as option does not check the actual presence of the user in the kubeconfig. It only checks the explicitly configured roles and bindings that are bound to a user and returns a response based on that.
Example for a non-existent user
kubectl auth can-i create pods --as=nobody
no

Custom Admin User

Configure a new admin user and verify the behaviour of the kubectl auth can-i commands.
This time we will check the permissions that a user inherits via the group it is attached to.
Issue a certificate for the new admin user.

Create a private key and a csr file
openssl genrsa -out poweruser.key 2048
openssl req -new -key poweruser.key -out poweruser.csr -subj "/CN=poweruser/O=system:masters" (1)
1 CN is the name of the user and O is the group that this user will belong to.
Create a CertificateSigningRequest
cat <<EOF | kubectl apply -f -
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: csr-name (1)
spec:
  request: base64 encoded csr (2)
  signerName: kubernetes.io/kube-apiserver-client
  expirationSeconds: 86400  # one day
  usages:
  - client auth
EOF
1 Provide the name as poweruser
2 Provide the base64 encoded value of the CSR

The above command throws an error as the kube-apiserver blocks any CertificateSigningRequest that attempts to add a user as part of the system:masters group.

Error from server (Forbidden): error when creating "STDIN": certificatesigningrequests.certificates.k8s.io "poweruser" is forbidden: use of kubernetes.io/kube-apiserver-client signer with system:masters group is not allowed

Granting permissions via RBAC through groups

In order to create a new admin user we will create a custom admin group that replicates the behaviour of the system:masters group. Let’s call it example:masters

To do this, create a new clusterrolebinding as below:

cat <<EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: example-cluster-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: example:masters
EOF

Add the new admin user to the example:masters group

Delete the previous files created for poweruser
rm poweruser.key poweruser.csr
Create a new private key and a csr file
openssl genrsa -out poweruser.key 2048
openssl req -new -key poweruser.key -out poweruser.csr -subj "/CN=poweruser/O=example:masters"
Create a CertificateSigningRequest
cat <<EOF | kubectl apply -f -
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: csr-name (1)
spec:
  request: base64 encoded csr (2)
  signerName: kubernetes.io/kube-apiserver-client
  expirationSeconds: 86400  # one day
  usages:
  - client auth
EOF
1 Provide the name as poweruser
2 Provide the base64 encoded value of the CSR
Approve the CSR
kubectl certificate approve poweruser
certificatesigningrequest.certificates.k8s.io/poweruser approved
Get the certificate
kubectl get csr poweruser -o jsonpath='{.status.certificate}'| base64 -d > poweruser.crt
View the Subject of this certificate
openssl x509 -in poweruser.crt -text -noout | grep Subject | grep -v "Public Key Info"
Subject: O=example:masters, CN=poweruser (1)
1 The user is associated with the example:masters group

Add the new admin user to kubeconfig

Add the new credentials
kubectl config set-credentials poweruser --client-key=poweruser.key --client-certificate=poweruser.crt --embed-certs=true
User "poweruser" set.
Add the context
kubectl config set-context poweruser --cluster=minikube --user=poweruser
Context "poweruser" created.

Try the kubectl auth can-i command to verify the permissions on the new admin user:

Check permissions to create pods
kubectl auth can-i create pods --user=poweruser
yes
Check permissions to create deployments
kubectl auth can-i create deployments --user=poweruser
yes
Check permissions to delete secrets
kubectl auth can-i delete secrets --user=poweruser
yes

Try the same commands but with using the --as option.

Check permissions to create pods
kubectl auth can-i create pods --as=poweruser
no
Check permissions to create deployments
kubectl auth can-i create deployments --as=poweruser
no
Check permissions to delete secrets
kubectl auth can-i delete secrets --as=poweruser
no

Strange. The output expected was yes for all 3 commands as poweruser is an admin user with full access to the cluster.

We can prove this as follows:

Switch the context to work with the poweruser
kubectl config use-context poweruser
Create a pod
kubectl run nginx --image=nginx
pod/nginx created
Create the deployment
kubectl create deployment nginx-deploy --image=nginx
deployment.apps/nginx-deploy created
Create and delete a secret
kubectl create secret generic test-secret --from-literal=secret=1234
secret/test-secret created
kubectl delete secrets test-secret
secret "test-secret" deleted

My first thought was that this is a defect in Kubernetes.
I raised an issue to Kubernetes for confirmation.[1]

The reality is that the behaviour is as-expected. The API server has no knowledge of group membership apart from what is encoded directly in the credential or provided by a token webhook. To overcome this you need to pass the group you want to impersonate with the --as-group flag.

The 'as-group' option of kubectl

Try the same commands but this time append the --as-group option as well.

Check permissions to create pods
kubectl auth can-i create pods --as=poweruser --as-group=example:masters
yes
Check permissions to create deployments
kubectl auth can-i create deployments --as=poweruser --as-group=example:masters
yes
Check permissions to delete secrets
kubectl auth can-i delete secrets --as=poweruser --as-group=example:masters
yes

Now the results are as expected.

Summary

The kubectl auth can-i command behaves differently for the --user and --as options.

The --user option checks for the actual presence of the user in the kubeconfig file and has the ability[.nobr] to check permissions derived from the group of the user.

The --as option can be used to check permissions for any user irrespective of its presence in the kubeconfig file. It checks permissions which are directly bound to the user through RBAC, and does not check permissions that are derived from the user’s group. The API server has no knowledge of group membership apart from whatever is encoded directly in the credential or provided by a token webhook.

The --as-group option should be used to check for permissions that are derived from the user’s group.

Cleaning up

Delete the resources created during this lab.

rm jane*
rm poweruser*
kubectl delete pod nginx
kubectl delete deployment nginx-deploy

Optionally, you can delete the entire minikube cluster.

minikube delete --all