Contents

1. RBAC授权体系
2. 使用RBAC
3. dashboard
4. 后记

RBAC授权体系

RBAC是K8s的一种授权体系，通过角色（role、clusterrole），对象（user、group、sa），绑定关系（rolebinding、clusterrolebinding）三种定义组合去灵活描述对K8s资源的操作权限。
K8s还要其他授权模式，默认是AlwaysAllow，还有Node、ABAC等。

角色

角色有role、clusterrole，主要描述规则和权限，只有授予权限，不能禁止权限，可以通过Role定义在一个命名空间中的角色，或者可以使用ClusterRole定义集群范围的角色。

对象

对象主要是user、group、serviceaccount三种，其中system:被保留作为用来Kubernetes系统使用，第一次使用RBAC启动apiserver的时候创建，因此不能作为用户的前缀。

绑定关系

把角色和对象绑定，就能描述user能做什么，sa能做什么，完成整个授权动作

更具体的操作可以参考网上资料：
Using RBAC Authorization
Kubernetes RBAC
Kubernetes-基于RBAC的授权

使用RBAC

RBAC在1.8版已经稳定，使用rbac.authorization.k8s.io/v1 API进行定义，启用RBAC需要在apiserver启动时指定参数

1	--authorization-mode=RBAC

kubeconfig授权

kubeconfig通常使用x509证书认证访问apiserver，k8s会读取证书的CN作为user，O作为group
下面给一个例子，用这个json生成的x509证书，其中system:kube-proxy是user，k8s是group

{
  "CN": "system:kube-proxy",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Guangdong",
      "L": "Zhongshan",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

给这个x509证书授权可用更新bxr-dev namespace上的deployment的image版本（kubectl set image）

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: updateimg
  namespace: bxr-dev
rules:
  - apiGroups: ["extensions"]
    resources: ["deployments"]
    verbs: ["get","patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: updateimg
  namespace: bxr-dev
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: updateimg
subjects:
- kind: User
  name: system:kube-proxy

SA授权

下面的yaml描述了readpod sa只能对pods资源进行get、list、watch操作：

apiVersion: v1
kind: ServiceAccount
metadata:
  name: readpod
  namespace: bxr-dev
---  
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: readpod
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: readpod 
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: readpod
subjects:
- kind: ServiceAccount
  name: readpod
  namespace: bxr-dev

验证

创建一个测试pod，使用readpod sa

如果不指定serviceAccountName，默认使用default sa

apiVersion: v1
kind: Pod
metadata:
  name: nettool 
  labels:
    role: tools
spec:
  serviceAccountName: readpod 
  containers:
  - name: nettool
    image: registry.bxr.cn/nettool:v2
    command:
      - sleep
      - "3600"

访问pod正常：

1
2
3

TOKEN=`cat /var/run/secrets/kubernetes.io/serviceaccount/token`
curl -H "Authorization: Bearer $TOKEN"  --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt https://kubernetes.default/api/v1/pods
结果太长忽略

访问nodes显示403：

# curl -H "Authorization: Bearer $TOKEN"  --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt https://kubernetes.default/api/v1/nodes     
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "nodes is forbidden: User \"system:serviceaccount:bxr-dev:readpod\" cannot list resource \"nodes\" in API group \"\" at the cluster scope",
  "reason": "Forbidden",
  "details": {
    "kind": "nodes"
  },
  "code": 403
}

踩坑

我们的环境之前没有做RBAC，一直是AlwaysAllow，不知是不是这个原因，配置RBAC遇到一些问题，下面记录一下

坑1

第一次配置碰到启用RBAC后node节点不能获取权限的报错：

E0722 14:26:55.781794    7156 kubelet.go:2266] node "k8s-dev-node2" not found
E0722 14:26:55.882538    7156 kubelet.go:2266] node "k8s-dev-node2" not found
E0722 14:26:55.956837    7156 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: services is forbidden: User "system:node:k8s-dev-node2" cannot list resource "services" in API group "" at the cluster scope
E0722 14:26:55.956991    7156 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: pods is forbidden: User "system:node:k8s-dev-node2" cannot list resource "pods" in API group "" at the cluster scope
E0722 14:26:55.957667    7156 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: nodes "k8s-dev-node2" is forbidden: User "system:node:k8s-dev-node2" cannot list resource "nodes" in API group "" at the cluster scope
E0722 14:26:55.983322    7156 kubelet.go:2266] node "k8s-dev-node2" not found

解决方法是添加group到system:nodes绑定：

https://github.com/kubernetes/kops/issues/3551

1	kubectl set subject clusterrolebinding system:node --group=system:nodes

坑2

重启容器时报错:

Events:
  Type     Reason                  Age               From                    Message
  ----     ------                  ----              ----                    -------
  Normal   Scheduled               21s               default-scheduler       Successfully assigned bxr-dev/nginx-static-6s578 to k8s-dev-node2
  Warning  FailedCreatePodSandBox  20s               kubelet, k8s-dev-node2  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "872e50de75eabb3754b5abdd0fe1d8c85f97af82c278e26b3b1b40760a1032cf" network for pod "nginx-static-6s578": NetworkPlugin cni failed to set up pod "nginx-static-6s578_bxr-dev" network: pods "nginx-static-6s578" is forbidden: User "system:kube-proxy" cannot get resource "pods" in API group "" in the namespace "bxr-dev"
  Warning  FailedCreatePodSandBox  18s               kubelet, k8s-dev-node2  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "d5a46e02c897a50a959d1b98a593f253fcce4a824808af62300f9d8a3b2cd852" network for pod "nginx-static-6s578": NetworkPlugin cni failed to set up pod "nginx-static-6s578_bxr-dev" network: pods "nginx-static-6s578" is forbidden: User "system:kube-proxy" cannot get resource "pods" in API group "" in the namespace "bxr-dev"

user system:kube-proxy绑定clusterrole system:node-proxier
解决方法是添加get pods权限给system:node-proxier clusterrole

- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get

坑3

过了一段时间kubelet重启失败，自动续签证书的坑：

F1113 18:34:54.883386   26329 server.go:261] failed to run Kubelet: cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User "kubelet-bootstrap" cannot create resource "certificatesigningrequests" in API group "certificates.k8s.io" at the cluster scope

解决方法：

1	kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap

dashboard

打开rbac后，kubernetes dashboard就需要登录才能使用了，可以选择令牌方式登录，令牌就用admin sa，也可以自己根据权限新建一个sa和role绑定

ubuntu@k8s-m1:~/k8s$ kubectl get sa -n kube-system
NAME                         SECRETS   AGE
admin                        1         480d

ubuntu@k8s-m1:~/k8s$ kubectl -n kube-system get secret
NAME                                     TYPE                                  DATA   AGE
admin-token-xjlwp                        kubernetes.io/service-account-token   3      479d

kubectl -n kube-system describe secret admin-token-xjlwp
Name:         admin-token-xjlwp
Namespace:    kube-system
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: admin
              kubernetes.io/service-account.uid: 9ba77c46-5f30-11e8-b2c9-525400dbc2de

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     2045 bytes
namespace:  11 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3Mi略。。。  这一大串就是令牌

后记

为什么要打开rbac？因为如果使用AllwaysAllow，任意一个pod都能访问master并操作数据，显然安全性大大降低，所以墙裂建议打开rbac的。

Zong's blog

日常积累，技术分享

K8s RBAC授权初探