Contents

k8s集群某个节点出现不能启动pod的情况，其他节点正常。
从错误信息很容易联想到rbac的权限问题，查询节点syslog、kubelet日志、calico日志等但没有明显指示。
重新对比环境的calico rbac配置和官方rbac配置，没有发现异常。
完整报错信息：

Events:
  Type     Reason                  Age                    From                    Message
  ----     ------                  ----                   ----                    -------
  Normal   Scheduled               10m                    default-scheduler       Successfully assigned bxr-dev/core-service-clazzalbum-7886f68fbc-547b2 to k8s-dev-node5
  Warning  FailedCreatePodSandBox  10m                    kubelet, k8s-dev-node5  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "d77730a242c3da53c5263e86e595d104b53da9e3b4526fdbbab2404fa76678ec" network for pod "core-service-clazzalbum-7886f68fbc-547b2": NetworkPlugin cni failed to set up pod "core-service-clazzalbum-7886f68fbc-547b2_bxr-dev" network: Unauthorized
  Warning  FailedCreatePodSandBox  10m                    kubelet, k8s-dev-node5  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "66ff75fde18b59ecc336042a18003d1f03968a3c5f50e988d74d52e536d86ddb" network for pod "core-service-clazzalbum-7886f68fbc-547b2": NetworkPlugin cni failed to set up pod "core-service-clazzalbum-7886f68fbc-547b2_bxr-dev" network: Unauthorized
  Warning  FailedCreatePodSandBox  10m                    kubelet, k8s-dev-node5  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "d1f4d7910a872b90ecf2080719985db3b869dd74ee684c3fa5770c8968abe06f" network for pod "core-service-clazzalbum-7886f68fbc-547b2": NetworkPlugin cni failed to set up pod "core-service-clazzalbum-7886f68fbc-547b2_bxr-dev" network: Unauthorized
  Warning  FailedCreatePodSandBox  10m                    kubelet, k8s-dev-node5  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "960dc8a43831ca4c6589fe2a8e1566444aaf69a3961933989eff6bcea862bc27" network for pod "core-service-clazzalbum-7886f68fbc-547b2": NetworkPlugin cni failed to set up pod "core-service-clazzalbum-7886f68fbc-547b2_bxr-dev" network: Unauthorized
  Warning  FailedCreatePodSandBox  10m                    kubelet, k8s-dev-node5  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "08bde655d956d48be9285438121f7f7e0d2eb354a8b797166445af0f67803142" network for pod "core-service-clazzalbum-7886f68fbc-547b2": NetworkPlugin cni failed to set up pod "core-service-clazzalbum-7886f68fbc-547b2_bxr-dev" network: Unauthorized
  Warning  FailedCreatePodSandBox  10m                    kubelet, k8s-dev-node5  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "cb7368a1cbabc8f1043a2cedbaa5bda01cce02e86f8323bcf9c76313ec736ac2" network for pod "core-service-clazzalbum-7886f68fbc-547b2": NetworkPlugin cni failed to set up pod "core-service-clazzalbum-7886f68fbc-547b2_bxr-dev" network: Unauthorized
  Warning  FailedCreatePodSandBox  10m                    kubelet, k8s-dev-node5  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "2f6d8ac6aec39dcca1645fefd5988ea78129efc1464acd1b05b97cb429bdd26d" network for pod "core-service-clazzalbum-7886f68fbc-547b2": NetworkPlugin cni failed to set up pod "core-service-clazzalbum-7886f68fbc-547b2_bxr-dev" network: Unauthorized
  Warning  FailedCreatePodSandBox  10m                    kubelet, k8s-dev-node5  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "064b477ce9998c9ceb47e3fb35f97cacacf273c6553bfa797e68a6bfdc8a453b" network for pod "core-service-clazzalbum-7886f68fbc-547b2": NetworkPlugin cni failed to set up pod "core-service-clazzalbum-7886f68fbc-547b2_bxr-dev" network: Unauthorized
  Warning  FailedCreatePodSandBox  10m                    kubelet, k8s-dev-node5  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "da4842bec87e9677e5aa9f0e028c33875e7a16562b232c16d234be666431028f" network for pod "core-service-clazzalbum-7886f68fbc-547b2": NetworkPlugin cni failed to set up pod "core-service-clazzalbum-7886f68fbc-547b2_bxr-dev" network: Unauthorized
  Warning  FailedCreatePodSandBox  10m (x4 over 10m)      kubelet, k8s-dev-node5  (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "818e2a24df475c2fccf1d5c6003fdf5283e9d31ac409eb6c67d188f054db632d" network for pod "core-service-clazzalbum-7886f68fbc-547b2": NetworkPlugin cni failed to set up pod "core-service-clazzalbum-7886f68fbc-547b2_bxr-dev" network: Unauthorized

解决方法：
该节点进行过kubelet证书轮换，有可能这个kubelet还是使用了旧证书，重启kubelet和kube-proxy后pod能正常启动。
检查了kubelet配置，已经有–rotate-certificates参数，资料上说轮换证书后会自动加载，出现这个情况需要后续继续观察。