Contents
  1. 1. 一 介绍
  2. 2. 二 对比docker
    1. 2.1. 1 比docker简洁
    2. 2.2. 2 命令精简
  3. 3. 三 安装配置
    1. 3.1. cri-o
    2. 3.2. cri-tools
  4. 4. 四 整合kubelet
  5. 5. 五 可能不足的地方
    1. 5.1. 版本问题
    2. 5.2. 稳定性
  6. 6. 六 出现的问题
    1. 6.1. 获取不到容器监控指标
    2. 6.2. 不能创建容器
    3. 6.3. kubelet重复报Failed to create existing container

一 介绍

CRI-O - OCI-based implementation of Kubernetes Container Runtime Interface
这是cri-o的github标题,符合OCI基准实现的Kubernetes容器运行时接口,从这个标题很容易看出,这是一个专门服务k8s的容器实现。从他的版本发布规律也很容易得出这个结论,因为cri-o的版本号和k8s版本基本一致。
这里面有2个概念比较容易混淆

  • CRI Container Runtime Interface这是k8s提出的一个概念,在容器界除了最出名的docker和本文介绍的cri-o,还有很多不同的容器实现,例如contrainerd, frakti,rkt等,这些容器实现各有特色,只要支持CRI就可以被k8s支持
  • OCI Open Container Initiative这是开放容器标准,也叫容器runtime,简单来说只要符合这个标准运行的就是容器,例如runC,Kata,gVisor这些runtime创建出来的都是OCI标准容器容器标准,也叫容器runtime,简单来说只要符合这个标准运行的,就是容器,例如runC,Kata,gVisor这些runtime创建出来的都是OCI标准容器

白话 Kubernetes Runtime 这篇文章写得挺有趣,可以读一下加深了解
为Kubernetes选择合适的容器运行时 这篇更详细一点

二 对比docker

1 比docker简洁

cri-o对比docker的一大特点就是直接,调用链条短。熟悉docker架构就知道,docker是传统的守护进程架构,docker把命令发送到docker daemon,docker daemon调用contrainerd进程,创建containerd-shim进程调用runC启动容器。整个架构显得繁琐,因为docker本身也有一套集群编排架构swarm,所以很多规则和调用放在k8s架构下显得重。
而cri-o就直接调用runC创建容器,而且容器并不会下挂到crio进程下面,天生没有docker重启容器的尴尬。
下面这个图直观描述了调用链

2 命令精简

cri-o本身实现的命令没有docker全,其中一个考虑是够用就好,例如没有push,build这些生产环境不需要的命令,所以他本身适合做单纯的容器环境。

三 安装配置

cri-o

安装挺简单,ubuntu可以用apt

1
2
3
sudo apt-add-repository ppa:projectatomic/ppa
sudo apt-get update -qq
sudo apt-get install cri-o-[REQUIRED VERSION]

版本可以和k8s环境一致,也可以高一个版本

配置文件修改一下 /etc/crio/crio.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
# The CRI-O configuration file specifies all of the available configuration
# options and command-line flags for the crio(8) OCI Kubernetes Container Runtime
# daemon, but in a TOML format that can be more easily modified and versioned.
#
# Please refer to crio.conf(5) for details of all configuration options.

# CRI-O reads its storage defaults from the containers-storage.conf(5) file
# located at /etc/containers/storage.conf. Modify this storage configuration if
# you want to change the system's defaults. If you want to modify storage just
# for CRI-O, you can change the storage configuration options here.
[crio]

# Path to the "root directory". CRI-O stores all of its data, including
# containers images, in this directory.
#root = "/home/lsm5/.local/share/containers/storage"

# Path to the "run directory". CRI-O stores all of its state in this directory.
#runroot = "/tmp/1000"

# Storage driver used to manage the storage of images and containers. Please
# refer to containers-storage.conf(5) to see all available storage drivers.
#storage_driver = "vfs"

# List to pass options to the storage driver. Please refer to
# containers-storage.conf(5) to see all available storage options.
#storage_option = [
#]

# If set to false, in-memory locking will be used instead of file-based locking.
file_locking = false

# Path to the lock file.
file_locking_path = "/run/crio.lock"


# The crio.api table contains settings for the kubelet/gRPC interface.
[crio.api]

# Path to AF_LOCAL socket on which CRI-O will listen.
listen = "/var/run/crio/crio.sock"

# IP address on which the stream server will listen.
stream_address = "127.0.0.1"

# The port on which the stream server will listen.
stream_port = "0"

# Enable encrypted TLS transport of the stream server.
stream_enable_tls = false

# Path to the x509 certificate file used to serve the encrypted stream. This
# file can change, and CRI-O will automatically pick up the changes within 5
# minutes.
stream_tls_cert = ""

# Path to the key file used to serve the encrypted stream. This file can
# change, and CRI-O will automatically pick up the changes within 5 minutes.
stream_tls_key = ""

# Path to the x509 CA(s) file used to verify and authenticate client
# communication with the encrypted stream. This file can change, and CRI-O will
# automatically pick up the changes within 5 minutes.
stream_tls_ca = ""

# Maximum grpc send message size in bytes. If not set or <=0, then CRI-O will default to 16 * 1024 * 1024.
grpc_max_send_msg_size = 16777216

# Maximum grpc receive message size. If not set or <= 0, then CRI-O will default to 16 * 1024 * 1024.
grpc_max_recv_msg_size = 16777216

# The crio.runtime table contains settings pertaining to the OCI runtime used
# and options for how to set up and manage the OCI runtime.
[crio.runtime]

# A list of ulimits to be set in containers by default, specified as
# "<ulimit name>=<soft limit>:<hard limit>", for example:
# "nofile=1024:2048"
# If nothing is set here, settings will be inherited from the CRI-O daemon
default_ulimits = [
"nofile=5120:5120"
]

# default_runtime is the _name_ of the OCI runtime to be used as the default.
# The name is matched against the runtimes map below.
default_runtime = "runc"

# If true, the runtime will not use pivot_root, but instead use MS_MOVE.
no_pivot = false

# Path to the conmon binary, used for monitoring the OCI runtime.
conmon = "/usr/bin/conmon"

# Environment variable list for the conmon process, used for passing necessary
# environment variables to conmon or the runtime.
conmon_env = [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
]

# If true, SELinux will be used for pod separation on the host.
selinux = false

# Path to the seccomp.json profile which is used as the default seccomp profile
# for the runtime.
seccomp_profile = "/etc/crio/seccomp.json"

# Used to change the name of the default AppArmor profile of CRI-O. The default
# profile name is "crio-default-" followed by the version string of CRI-O.
apparmor_profile = "crio-default"

# Cgroup management implementation used for the runtime.
cgroup_manager = "systemd"

# List of default capabilities for containers. If it is empty or commented out,
# only the capabilities defined in the containers json file by the user/kube
# will be added.
default_capabilities = [
"CHOWN",
"DAC_OVERRIDE",
"FSETID",
"FOWNER",
"NET_RAW",
"SETGID",
"SETUID",
"SETPCAP",
"NET_BIND_SERVICE",
"SYS_CHROOT",
"KILL",
]

# List of default sysctls. If it is empty or commented out, only the sysctls
# defined in the container json file by the user/kube will be added.
default_sysctls = [
]

# List of additional devices. specified as
# "<device-on-host>:<device-on-container>:<permissions>", for example: "--device=/dev/sdc:/dev/xvdc:rwm".
#If it is empty or commented out, only the devices
# defined in the container json file by the user/kube will be added.
additional_devices = [
]

# Path to OCI hooks directories for automatically executed hooks.
hooks_dir = [
]

# List of default mounts for each container. **Deprecated:** this option will
# be removed in future versions in favor of default_mounts_file.
default_mounts = [
]

# Path to the file specifying the defaults mounts for each container. The
# format of the config is /SRC:/DST, one mount per line. Notice that CRI-O reads
# its default mounts from the following two files:
#
# 1) /etc/containers/mounts.conf (i.e., default_mounts_file): This is the
# override file, where users can either add in their own default mounts, or
# override the default mounts shipped with the package.
#
# 2) /usr/share/containers/mounts.conf: This is the default file read for
# mounts. If you want CRI-O to read from a different, specific mounts file,
# you can change the default_mounts_file. Note, if this is done, CRI-O will
# only add mounts it finds in this file.
#
#default_mounts_file = ""

# Maximum number of processes allowed in a container.
pids_limit = 1024

# Maximum sized allowed for the container log file. Negative numbers indicate
# that no size limit is imposed. If it is positive, it must be >= 8192 to
# match/exceed conmon's read buffer. The file is truncated and re-opened so the
# limit is never exceeded.
log_size_max = 536870912

# Whether container output should be logged to journald in addition to the kuberentes log file
log_to_journald = false

# Path to directory in which container exit files are written to by conmon.
container_exits_dir = "/var/run/crio/exits"

# Path to directory for container attach sockets.
container_attach_socket_dir = "/var/run/crio"

# If set to true, all containers will run in read-only mode.
read_only = false

# Changes the verbosity of the logs based on the level it is set to. Options
# are fatal, panic, error, warn, info, and debug.
log_level = "error"

# The UID mappings for the user namespace of each container. A range is
# specified in the form containerUID:HostUID:Size. Multiple ranges must be
# separated by comma.
uid_mappings = ""

# The GID mappings for the user namespace of each container. A range is
# specified in the form containerGID:HostGID:Size. Multiple ranges must be
# separated by comma.
gid_mappings = ""

# The minimal amount of time in seconds to wait before issuing a timeout
# regarding the proper termination of the container.
ctr_stop_timeout = 0

# The "crio.runtime.runtimes" table defines a list of OCI compatible runtimes.
# The runtime to use is picked based on the runtime_handler provided by the CRI.
# If no runtime_handler is provided, the runtime will be picked based on the level
# of trust of the workload.

[crio.runtime.runtimes.runc]
runtime_path = "/usr/lib/cri-o-runc/sbin/runc"
runtime_type = ""



# The crio.image table contains settings pertaining to the management of OCI images.
#
# CRI-O reads its configured registries defaults from the system wide
# containers-registries.conf(5) located in /etc/containers/registries.conf. If
# you want to modify just CRI-O, you can change the registries configuration in
# this file. Otherwise, leave insecure_registries and registries commented out to
# use the system's defaults from /etc/containers/registries.conf.
[crio.image]

# Default transport for pulling images from a remote container storage.
default_transport = "docker://"

# The path to a file containing credentials necessary for pulling images from
# secure registries. The file is similar to that of /var/lib/kubelet/config.json
global_auth_file = ""

# The image used to instantiate infra containers.
pause_image = "ccr.ccs.tencentyun.com/google_container/pause-amd64:3.1"

# The path to a file containing credentials specific for pulling the pause_image from
# above. The file is similar to that of /var/lib/kubelet/config.json
pause_image_auth_file = ""

# The command to run to have a container stay in the paused state.
pause_command = "/pause"

# Path to the file which decides what sort of policy we use when deciding
# whether or not to trust an image that we've pulled. It is not recommended that
# this option be used, as the default behavior of using the system-wide default
# policy (i.e., /etc/containers/policy.json) is most often preferred. Please
# refer to containers-policy.json(5) for more details.
signature_policy = ""

# Controls how image volumes are handled. The valid values are mkdir, bind and
# ignore; the latter will ignore volumes entirely.
image_volumes = "mkdir"

# List of registries to be used when pulling an unqualified image (e.g.,
# "alpine:latest"). By default, registries is set to "docker.io" for
# compatibility reasons. Depending on your workload and usecase you may add more
# registries (e.g., "quay.io", "registry.fedoraproject.org",
# "registry.opensuse.org", etc.).
registries = [
"quay.io",
"docker.io"
]

insecure_registries = ["registry.bxr.cn"]


# The crio.network table containers settings pertaining to the management of
# CNI plugins.
[crio.network]

# Path to the directory where CNI configuration files are located.
network_dir = "/etc/cni/net.d/"

# Paths to directories where CNI plugin binaries are located.
plugin_dirs = [
"/opt/cni/bin",
]
  • cgroup_manager 这个要和kubelet配置一致,推荐设置systemd
  • log_size_max 建议配一下,不要让log塞满磁盘
  • pause_image 换一个国内的吧
  • registries 取消掉注释,不然要写完整image路径
  • insecure_registries 内部registry

配置systemd启动
/etc/systemd/system/crio.service

1
2
3
4
5
6
7
8
9
10
11
[Unit]
Description=OCI-based implementation of Kubernetes Container Runtime Interface
Documentation=https://github.com/kubernetes-incubator/cri-o

[Service]
ExecStart=/usr/bin/crio --log-level info
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

cri-tools

cri-tools是一个管理cri-o的命令工具,也适用其他CRI标准的容器。
下载对应版本的二进制包解压就能用,注意配置一下 /etc/crictl.yaml 里面的runtime-endpoint
该有的命令都有了,当然前面提到比docker命令少,有个特色是能管理pod维度

1
2
3
4
5
ubuntu@k8s-dev-node4:/etc/kubernetes$ sudo crictl pods
POD ID CREATED STATE NAME NAMESPACE ATTEMPT
dbdb04f7200a1 2 hours ago Ready metrics-server-549bdfcbf7-kvz56 kube-system 0
8944933980db5 2 hours ago Ready coredns-9d458994c-bgxqm kube-system 0
0dd7b784b65a3 6 hours ago Ready calico-node-ssg6f kube-system 0

四 整合kubelet

如果之前有docker,需要把运行的容器停止。在kubelet启动参数加上:

1
2
3
--container-runtime=remote
--container-runtime-endpoint=/var/run/crio/crio.sock
--runtime-request-timeout=10m

网上包括官方doc都是–container-runtime-endpoint=unix:///var/run/crio/crio.sock,但是这样会引起一个问题,看第五章

如果之前kubelet的systemd配置了docker前置条件,需要改一下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=crio.service
Requires=crio.service

[Service]
EnvironmentFile=/etc/kubernetes/kubelet
WorkingDirectory=/var/lib/kubelet
ExecStart=/usr/bin/kubelet $KUBELET_ARGS
Restart=on-failure

[Install]
WantedBy=multi-user.target

然后重启kubelet

验证:

1
2
3
4
5
6
ubuntu@k8s-dev-m1:~$ kubectl get no -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8s-dev-node1 Ready k8s-node 22d v1.13.5 10.22.0.2 <none> Ubuntu 16.04.1 LTS 4.4.0-91-generic docker://18.3.1
k8s-dev-node3.bxr.cn Ready k8s-node 306d v1.13.4 10.22.0.40 <none> Container Linux by CoreOS 1745.5.0 (Rhyolite) 4.14.44-coreos-r1 docker://18.3.1
k8s-dev-node4 Ready k8s-node 418d v1.13.4 10.22.1.5 <none> Ubuntu 16.04.1 LTS 4.4.0-130-generic cri-o://1.14.10
k8s-dev-node5 Ready k8s-node 418d v1.13.0 10.22.0.23 <none> Ubuntu 16.04.1 LTS 4.4.0-130-generic docker://18.6.1

可以看到k8s-dev-node4的CONTAINER-RUNTIME已经换成cri-o

五 可能不足的地方

版本问题

crio版本和k8s版本的发布保持一致,官方也说使用版本最好和k8s保持1个小版差异,这样在升级k8s版本的时候可能要多考虑一下crio的版本,而k8s一般能兼容多个版本的docker

稳定性

稳定性方面好像还没经过大厂使用验证,也没有docker使用广泛,社区肯定不如docker活跃

六 出现的问题

获取不到容器监控指标

1
2
Jan  9 09:53:25 localhost kubelet[6482]: E0109 09:53:25.371839    6482 handler.go:306] HTTP InternalServerError serving /stats/summary: Internal Error: failed to get imageFs stats: failed to get imageFs info: no imagefs label for configured runtime
Jan 9 09:53:45 localhost kubelet[6482]: E0109 09:53:45.569308 6482 kubelet.go:1304] Image garbage collection failed multiple times in a row: failed to get imageFs info: no imagefs label for configured runtime

这个报错是kubelet的10255端口获取不到容器指标,测试:

1
2
ubuntu@k8s-dev-node4:~$ curl localhost:10255/stats/summary
Internal Error: failed to get imageFs stats: failed to get imageFs info: no imagefs label for configured runtimeubuntu@k8s-dev-node4:~$

参考github的issue https://github.com/kubernetes/kubernetes/issues/71712
kubelet 1.13 配置runtime要去掉unix://

1
--container-runtime-endpoint=/var/run/crio/crio.sock

不能创建容器

1
Jan  9 11:07:26 localhost kubelet[18240]: E0109 11:07:26.818320   18240 manager.go:1147] Failed to create existing container: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cecfa7f_e89f_11e8_8d00_525400df7e25.slice/crio-0dd7b784b65a3b7fef40c0634b92b5663f49b53c0a9b176f674ab7517efa717f.scope: invalid character 'c' looking for beginning of value

重启cri-o后,kubelet一直报Failed to create existing container,观察集群pod和node节点pods状态正常,暂未发现实质影响

kubelet重复报Failed to create existing container

1
E0224 18:17:35.083060   10955 manager.go:1086] Failed to create existing container: /kubepods/besteffort/pode26e7d50-4521-4962-a988-d646bb0fe953/crio-b3b80307234c60c4cc9ed419f9f402977abb253304ef3d6755dff85943f5dd5e: Error finding container b3b80307234c60c4cc9ed419f9f402977abb253304ef3d6755dff85943f5dd5e: Status 404 returned error &{%!s(*http.body=&{0xc001883fe0 <nil> <nil> false false {0 0} false false false <nil>}) {%!s(int32=0) %!s(uint32=0)} %!s(bool=false) <nil> %!s(func(error) error=0x74f5f0) %!s(func() error=0x74f580)}

cgroup_manager设置systemd
在节点查看是否有NotReady的pods并删掉 sudo crictl rmp podid

Contents
  1. 1. 一 介绍
  2. 2. 二 对比docker
    1. 2.1. 1 比docker简洁
    2. 2.2. 2 命令精简
  3. 3. 三 安装配置
    1. 3.1. cri-o
    2. 3.2. cri-tools
  4. 4. 四 整合kubelet
  5. 5. 五 可能不足的地方
    1. 5.1. 版本问题
    2. 5.2. 稳定性
  6. 6. 六 出现的问题
    1. 6.1. 获取不到容器监控指标
    2. 6.2. 不能创建容器
    3. 6.3. kubelet重复报Failed to create existing container