By Zong

Contents
  1. 1. 搭建kubeDNS
  2. 2. 修改kubelet参数
  3. 3. 测试

Kubernetes把一个实际的服务抽象为pod和service资源，pod是后端，负责处理业务逻辑，后端的pod可以有很多，有些pod可能异常被controller回收或者动态缩放，pod的ip都是动态分配的，所以需要抽象出service把pod提供的服务开放出来，这样我们就不需要管后端pod的ip变化。那service本身的ip我们怎么知道呢？k8s使用DNS解决这个问题。
K8s的DNS服务之前使用etcd + kube2sky + skydns搭建，在之后改为kubedns + dnsmasq，本文当然基于kubedns搭建。

ps:最新版推荐CoreDNS

搭建kubeDNS

kubeDNS容器监听apiserver的服务变化，但service和endpoints出现更新时，把名字和IP在内存中存储起来，对外使用dnsmasq提供DNS服务。
在kubernetes源码包cluster/addons/dns目录中有DNS服务需要的yaml文件，可以使用这些文件略加修改搭建DNS。
kubedns-sa.yaml

1
2
3
4
5
6
7
8
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-dns
  namespace: kube-system
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile

kubedns-cm.yaml

1
2
3
4
5
6
7
apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-dns
  namespace: kube-system
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists

ConfigMap文件中可以配置自定义DNS和上游DNS，自定义DNS指对特定域名查询特定的DNS服务器，上游DNS指外部的DNS服务器，不指定的话就从主机/etc/resolv.conf继承。
下面这个文件设置了域名out-of.k8s的查询会被发送到10.11.33.44服务器。上游DNS指定了8.8.8.8和114.114.114.114

1
2
3
4
5
6
7
8
9
10
11
12
apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-dns
  namespace: kube-system
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
data：
   stubDomains: |
      {"out-of.k8s": ["10.11.33.44"]}
   upstreamNameservers: |
      ["8.8.8.8","114.114.114.114"]

kubedns-deployment.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
apiVersion: extensions/v1beta2
kind: Deployment
metadata:
  name: kube-dns
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  # replicas: not specified here:
  # 1. In order to make Addon Manager do not reconcile this replicas parameter.
  # 2. Default is 1.
  # 3. Will be tuned in real time if DNS horizontal auto-scaling is turned on.
  strategy:
    rollingUpdate:
      maxSurge: 10%
      maxUnavailable: 0
  selector:
    matchLabels:
      k8s-app: kube-dns
  template:
    metadata:
      labels:
        k8s-app: kube-dns
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      tolerations:
      - key: "CriticalAddonsOnly"
        operator: "Exists"
      volumes:
      - name: kube-dns-config
        configMap:
          name: kube-dns
          optional: true
      containers:
      - name: kubedns
        image: registry.cn-hangzhou.aliyuncs.com/szss_k8s/k8s-dns-kube-dns-amd64:1.14.5
        resources:
          # TODO: Set memory limits when we've profiled the container for large
          # clusters, then set request = limit to keep this container in
          # guaranteed class. Currently, this container falls into the
          # "burstable" category so the kubelet doesn't backoff from restarting it.
          limits:
            memory: 170Mi
          requests:
            cpu: 100m
            memory: 70Mi
        livenessProbe:
          httpGet:
            path: /healthcheck/kubedns
            port: 10054
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
        readinessProbe:
          httpGet:
            path: /readiness
            port: 8081
            scheme: HTTP
          # we poll on pod startup for the Kubernetes master service and
          # only setup the /readiness HTTP server once that's available.
          initialDelaySeconds: 3
          timeoutSeconds: 5
        args:
        - --domain=wx-dev.local.
        - --dns-port=10053
        - --config-dir=/kube-dns-config
        - --v=2
        env:
        - name: PROMETHEUS_PORT
          value: "10055"
        ports:
        - containerPort: 10053
          name: dns-local
          protocol: UDP
        - containerPort: 10053
          name: dns-tcp-local
          protocol: TCP
        - containerPort: 10055
          name: metrics
          protocol: TCP
        volumeMounts:
        - name: kube-dns-config
          mountPath: /kube-dns-config
      - name: dnsmasq
        image: registry.cn-hangzhou.aliyuncs.com/szss_k8s/k8s-dns-dnsmasq-nanny-amd64:1.14.5
        livenessProbe:
          httpGet:
            path: /healthcheck/dnsmasq
            port: 10054
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
        args:
        - -v=2
        - -logtostderr
        - -configDir=/etc/k8s/dns/dnsmasq-nanny
        - -restartDnsmasq=true
        - --
        - -k
        - --cache-size=1000
        - --log-facility=-
        - --server=/wx-dev.local/127.0.0.1#10053
        - --server=/in-addr.arpa/127.0.0.1#10053
        - --server=/ip6.arpa/127.0.0.1#10053
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        # see: https://github.com/kubernetes/kubernetes/issues/29055 for details
        resources:
          requests:
            cpu: 150m
            memory: 20Mi
        volumeMounts:
        - name: kube-dns-config
          mountPath: /etc/k8s/dns/dnsmasq-nanny
      - name: sidecar
        image: registry.cn-hangzhou.aliyuncs.com/szss_k8s/k8s-dns-sidecar-amd64:1.14.5
        livenessProbe:
          httpGet:
            path: /metrics
            port: 10054
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
        args:
        - --v=2
        - --logtostderr
        - --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.wx-dev.local,5,A
        - --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.wx-dev.local,5,A
        ports:
        - containerPort: 10054
          name: metrics
          protocol: TCP
        resources:
          requests:
            memory: 20Mi
            cpu: 10m
      dnsPolicy: Default  # Don't use cluster DNS.
      serviceAccountName: kube-dns
  • 把配置中wx-dev.local改成你的域名
  • 这个配置中镜像地址都换成了阿里的registry.cn-hangzhou.aliyuncs.com/szss_k8s

创建出的deployment：

1
2
3
ubuntu@ubuntu1:~$ sudo kubectl get deployments --namespace=kube-system
NAME       DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
kube-dns   1         1         1            1           9d

kubedns-svc.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
apiVersion: v1
kind: Service
metadata:
  name: kube-dns
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/name: "KubeDNS"
spec:
  selector:
    k8s-app: kube-dns
  clusterIP: 169.169.0.2
  ports:
  - name: dns
    port: 53
    protocol: UDP
  - name: dns-tcp
    port: 53
    protocol: TCP
  • clusterIP是DNS服务端ip，与集群同网段

创建出的service：

1
2
3
ubuntu@ubuntu1:~$ sudo kubectl get svc --namespace=kube-system
NAME       TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)         AGE
kube-dns   ClusterIP   169.169.0.2   <none>        53/UDP,53/TCP   9d

修改kubelet参数

在集群中每个node节点的kubelet服务添加启动参数指定DNS地址。

1
--cluster-dns=169.169.0.2 --cluster-domain=wx-dev.local

测试

我们之前已经创建好一个nginx的pod和service了

1
2
3
4
ubuntu@ubuntu1:~$ sudo kubectl get svc
NAME            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)       AGE
kubernetes      ClusterIP   169.169.0.1      <none>        443/TCP       20d
nginx-service   NodePort    169.169.181.54   <none>        80:8081/TCP   10d

现在我们要使用dns方式，通过查询nginx-service得到对应的ip。
先创建一个基础的busybox运行nslookup命令
busybox.yaml:

1
2
3
4
5
6
7
8
9
10
11
12
apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
  - name: busybox
    image: registry.cn-hangzhou.aliyuncs.com/qinyujia-test/busybox
    command:
	  - sleep
	  - "3600"

运行命令得到结果

1
2
3
4
5
ubuntu@ubuntu1:~$ sudo kubectl exec busybox -- nslookup nginx-service
Server:    169.169.0.2
Address 1: 169.169.0.2 kube-dns.kube-system.svc.wx-dev.local
Name:      nginx-service
Address 1: 169.169.181.54 nginx-service.default.svc.wx-dev.local

证明在同一个k8s集群内，dns能准确找出service name和ip的对应关系。

Contents
  1. 1. 搭建kubeDNS
  2. 2. 修改kubelet参数
  3. 3. 测试