一 部署master節點
1.1 master節點服務
kubernetes master 節點運行如下組件：
- kube-apiserver
- kube-scheduler
- kube-controller-manager
- kube-nginx
kube-apiserver、kube-scheduler 和 kube-controller-manager 均以多實例模式運行：
kube-scheduler 和 kube-controller-manager 會自動選舉產生一個 leader 實例，其它實例處於阻塞模式，當 leader 掛了后，重新選舉產生新的 leader，從而保證服務可用性。
kube-apiserver 是無狀態的，需要通過 kube-nginx 進行代理訪問，從而保證服務可用性。
1.2 安裝Kubernetes
1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# wget https://dl.k8s.io/v1.14.2/kubernetes-server-linux-amd64.tar.gz
3 [root@k8smaster01 work]# tar -xzvf kubernetes-server-linux-amd64.tar.gz
4 [root@k8smaster01 work]# cd kubernetes
5 [root@k8smaster01 kubernetes]# tar -xzvf kubernetes-src.tar.gz
1.3 分發Kubernetes
1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
3 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]}
4 do
5 echo ">>> ${master_ip}"
6 scp kubernetes/server/bin/{apiextensions-apiserver,cloud-controller-manager,kube-apiserver,kube-controller-manager,kube-proxy,kube-scheduler,kubeadm,kubectl,kubelet,mounter} root@${master_ip}:/opt/k8s/bin/
7 ssh root@${master_ip} "chmod +x /opt/k8s/bin/*"
8 done
二 部署高可用kube-apiserver
2.1 高可用apiserver介紹
本實驗部署一個三實例 kube-apiserver 集群的步驟，它們通過 kube-nginx 進行代理訪問，從而保證服務可用性。
2.2 創建Kubernetes證書和私鑰
1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# cat > kubernetes-csr.json <<EOF
3 {
4 "CN": "kubernetes",
5 "hosts": [
6 "127.0.0.1",
7 "172.24.8.71",
8 "172.24.8.72",
9 "172.24.8.73",
10 "${CLUSTER_KUBERNETES_SVC_IP}",
11 "kubernetes",
12 "kubernetes.default",
13 "kubernetes.default.svc",
14 "kubernetes.default.svc.cluster",
15 "kubernetes.default.svc.cluster.local."
16 ],
17 "key": {
18 "algo": "rsa",
19 "size": 2048
20 },
21 "names": [
22 {
23 "C": "CN",
24 "ST": "Shanghai",
25 "L": "Shanghai",
26 "O": "k8s",
27 "OU": "System"
28 }
29 ]
30 }
31 EOF
32 #創建Kubernetes的CA證書請求文件
解釋：
hosts 字段指定授權使用該證書的 IP 和域名列表，這裏列出了 master 節點 IP、kubernetes 服務的 IP 和域名；
kubernetes 服務 IP 是 apiserver 自動創建的，一般是 –service-cluster-ip-range 參數指定的網段的第一個IP，後續可以通過下面命令獲取：
1 # kubectl get svc kubernetes
1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# cfssl gencert -ca=/opt/k8s/work/ca.pem \
3 -ca-key=/opt/k8s/work/ca-key.pem -config=/opt/k8s/work/ca-config.json \
4 -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes #生成CA密鑰（ca-key.pem）和證書（ca.pem）
2.3 分發證書和私鑰
1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
3 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]}
4 do
5 echo ">>> ${master_ip}"
6 ssh root@${master_ip} "mkdir -p /etc/kubernetes/cert"
7 scp kubernetes*.pem root@${master_ip}:/etc/kubernetes/cert/
8 done
2.4 創建加密配置文件
1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
3 [root@k8smaster01 work]# cat > encryption-config.yaml <<EOF
4 kind: EncryptionConfig
5 apiVersion: v1
6 resources:
7 - resources:
8 - secrets
9 providers:
10 - aescbc:
11 keys:
12 - name: key1
13 secret: ${ENCRYPTION_KEY}
14 - identity: {}
15 EOF
2.5 分發加密配置文件
1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
3 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]}
4 do
5 echo ">>> ${master_ip}"
6 scp encryption-config.yaml root@${master_ip}:/etc/kubernetes/
7 done
2.6 創建審計策略文件
1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
3 [root@k8smaster01 work]# cat > audit-policy.yaml <<EOF
4 apiVersion: audit.k8s.io/v1beta1
5 kind: Policy
6 rules:
7 # The following requests were manually identified as high-volume and low-risk, so drop them.
8 - level: None
9 resources:
10 - group: ""
11 resources:
12 - endpoints
13 - services
14 - services/status
15 users:
16 - 'system:kube-proxy'
17 verbs:
18 - watch
19
20 - level: None
21 resources:
22 - group: ""
23 resources:
24 - nodes
25 - nodes/status
26 userGroups:
27 - 'system:nodes'
28 verbs:
29 - get
30
31 - level: None
32 namespaces:
33 - kube-system
34 resources:
35 - group: ""
36 resources:
37 - endpoints
38 users:
39 - 'system:kube-controller-manager'
40 - 'system:kube-scheduler'
41 - 'system:serviceaccount:kube-system:endpoint-controller'
42 verbs:
43 - get
44 - update
45
46 - level: None
47 resources:
48 - group: ""
49 resources:
50 - namespaces
51 - namespaces/status
52 - namespaces/finalize
53 users:
54 - 'system:apiserver'
55 verbs:
56 - get
57
58 # Don't log HPA fetching metrics.
59 - level: None
60 resources:
61 - group: metrics.k8s.io
62 users:
63 - 'system:kube-controller-manager'
64 verbs:
65 - get
66 - list
67
68 # Don't log these read-only URLs.
69 - level: None
70 nonResourceURLs:
71 - '/healthz*'
72 - /version
73 - '/swagger*'
74
75 # Don't log events requests.
76 - level: None
77 resources:
78 - group: ""
79 resources:
80 - events
81
82 # node and pod status calls from nodes are high-volume and can be large, don't log responses for expected updates from nodes
83 - level: Request
84 omitStages:
85 - RequestReceived
86 resources:
87 - group: ""
88 resources:
89 - nodes/status
90 - pods/status
91 users:
92 - kubelet
93 - 'system:node-problem-detector'
94 - 'system:serviceaccount:kube-system:node-problem-detector'
95 verbs:
96 - update
97 - patch
98
99 - level: Request
100 omitStages:
101 - RequestReceived
102 resources:
103 - group: ""
104 resources:
105 - nodes/status
106 - pods/status
107 userGroups:
108 - 'system:nodes'
109 verbs:
110 - update
111 - patch
112
113 # deletecollection calls can be large, don't log responses for expected namespace deletions
114 - level: Request
115 omitStages:
116 - RequestReceived
117 users:
118 - 'system:serviceaccount:kube-system:namespace-controller'
119 verbs:
120 - deletecollection
121
122 # Secrets, ConfigMaps, and TokenReviews can contain sensitive & binary data,
123 # so only log at the Metadata level.
124 - level: Metadata
125 omitStages:
126 - RequestReceived
127 resources:
128 - group: ""
129 resources:
130 - secrets
131 - configmaps
132 - group: authentication.k8s.io
133 resources:
134 - tokenreviews
135 # Get repsonses can be large; skip them.
136 - level: Request
137 omitStages:
138 - RequestReceived
139 resources:
140 - group: ""
141 - group: admissionregistration.k8s.io
142 - group: apiextensions.k8s.io
143 - group: apiregistration.k8s.io
144 - group: apps
145 - group: authentication.k8s.io
146 - group: authorization.k8s.io
147 - group: autoscaling
148 - group: batch
149 - group: certificates.k8s.io
150 - group: extensions
151 - group: metrics.k8s.io
152 - group: networking.k8s.io
153 - group: policy
154 - group: rbac.authorization.k8s.io
155 - group: scheduling.k8s.io
156 - group: settings.k8s.io
157 - group: storage.k8s.io
158 verbs:
159 - get
160 - list
161 - watch
162
163 # Default level for known APIs
164 - level: RequestResponse
165 omitStages:
166 - RequestReceived
167 resources:
168 - group: ""
169 - group: admissionregistration.k8s.io
170 - group: apiextensions.k8s.io
171 - group: apiregistration.k8s.io
172 - group: apps
173 - group: authentication.k8s.io
174 - group: authorization.k8s.io
175 - group: autoscaling
176 - group: batch
177 - group: certificates.k8s.io
178 - group: extensions
179 - group: metrics.k8s.io
180 - group: networking.k8s.io
181 - group: policy
182 - group: rbac.authorization.k8s.io
183 - group: scheduling.k8s.io
184 - group: settings.k8s.io
185 - group: storage.k8s.io
186
187 # Default level for all other requests.
188 - level: Metadata
189 omitStages:
190 - RequestReceived
191 EOF
2.7 分發策略文件
1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
3 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]}
4 do
5 echo ">>> ${master_ip}"
6 scp audit-policy.yaml root@${master_ip}:/etc/kubernetes/audit-policy.yaml
7 done
2.8 創建訪問 metrics-server的證書和密鑰
1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# cat > proxy-client-csr.json <<EOF
3 {
4 "CN": "aggregator",
5 "hosts": [],
6 "key": {
7 "algo": "rsa",
8 "size": 2048
9 },
10 "names": [
11 {
12 "C": "CN",
13 "ST": "Shanghai",
14 "L": "Shanghai",
15 "O": "k8s",
16 "OU": "System"
17 }
18 ]
19 }
20 EOF
21 #創建metrics-server的CA證書請求文件
解釋：
CN 名稱需要位於 kube-apiserver 的 –requestheader-allowed-names 參數中，否則後續訪問 metrics 時會提示權限不足。
1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# cfssl gencert -ca=/opt/k8s/work/ca.pem \
3 -ca-key=/opt/k8s/work/ca-key.pem -config=/opt/k8s/work/ca-config.json \
4 -profile=kubernetes proxy-client-csr.json | cfssljson -bare proxy-client #生成CA密鑰（ca-key.pem）和證書（ca.pem）
2.9 分發證書和私鑰
1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
3 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]}
4 do
5 echo ">>> ${master_ip}"
6 scp proxy-client*.pem root@${master_ip}:/etc/kubernetes/cert/
7 done
2.10 創建kube-apiserver的systemd
1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
3 [root@k8smaster01 work]# cat > kube-apiserver.service.template <<EOF
4 [Unit]
5 Description=Kubernetes API Server
6 Documentation=https://github.com/GoogleCloudPlatform/kubernetes
7 After=network.target
8
9 [Service]
10 WorkingDirectory=${K8S_DIR}/kube-apiserver
11 ExecStart=/opt/k8s/bin/kube-apiserver \\
12 --advertise-address=##MASTER_IP## \\
13 --default-not-ready-toleration-seconds=360 \\
14 --default-unreachable-toleration-seconds=360 \\
15 --feature-gates=DynamicAuditing=true \\
16 --max-mutating-requests-inflight=2000 \\
17 --max-requests-inflight=4000 \\
18 --default-watch-cache-size=200 \\
19 --delete-collection-workers=2 \\
20 --encryption-provider-config=/etc/kubernetes/encryption-config.yaml \\
21 --etcd-cafile=/etc/kubernetes/cert/ca.pem \\
22 --etcd-certfile=/etc/kubernetes/cert/kubernetes.pem \\
23 --etcd-keyfile=/etc/kubernetes/cert/kubernetes-key.pem \\
24 --etcd-servers=${ETCD_ENDPOINTS} \\
25 --bind-address=##MASTER_IP## \\
26 --secure-port=6443 \\
27 --tls-cert-file=/etc/kubernetes/cert/kubernetes.pem \\
28 --tls-private-key-file=/etc/kubernetes/cert/kubernetes-key.pem \\
29 --insecure-port=0 \\
30 --audit-dynamic-configuration \\
31 --audit-log-maxage=15 \\
32 --audit-log-maxbackup=3 \\
33 --audit-log-maxsize=100 \\
34 --audit-log-mode=batch \\
35 --audit-log-truncate-enabled \\
36 --audit-log-batch-buffer-size=20000 \\
37 --audit-log-batch-max-size=2 \\
38 --audit-log-path=${K8S_DIR}/kube-apiserver/audit.log \\
39 --audit-policy-file=/etc/kubernetes/audit-policy.yaml \\
40 --profiling \\
41 --anonymous-auth=false \\
42 --client-ca-file=/etc/kubernetes/cert/ca.pem \\
43 --enable-bootstrap-token-auth \\
44 --requestheader-allowed-names="aggregator" \\
45 --requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \\
46 --requestheader-extra-headers-prefix="X-Remote-Extra-" \\
47 --requestheader-group-headers=X-Remote-Group \\
48 --requestheader-username-headers=X-Remote-User \\
49 --service-account-key-file=/etc/kubernetes/cert/ca.pem \\
50 --authorization-mode=Node,RBAC \\
51 --runtime-config=api/all=true \\
52 --enable-admission-plugins=NodeRestriction \\
53 --allow-privileged=true \\
54 --apiserver-count=3 \\
55 --event-ttl=168h \\
56 --kubelet-certificate-authority=/etc/kubernetes/cert/ca.pem \\
57 --kubelet-client-certificate=/etc/kubernetes/cert/kubernetes.pem \\
58 --kubelet-client-key=/etc/kubernetes/cert/kubernetes-key.pem \\
59 --kubelet-https=true \\
60 --kubelet-timeout=10s \\
61 --proxy-client-cert-file=/etc/kubernetes/cert/proxy-client.pem \\
62 --proxy-client-key-file=/etc/kubernetes/cert/proxy-client-key.pem \\
63 --service-cluster-ip-range=${SERVICE_CIDR} \\
64 --service-node-port-range=${NODE_PORT_RANGE} \\
65 --logtostderr=true \\
66 --v=2
67 Restart=on-failure
68 RestartSec=10
69 Type=notify
70 LimitNOFILE=65536
71
72 [Install]
73 WantedBy=multi-user.target
74 EOF
解釋：
- –advertise-address：apiserver 對外通告的 IP（kubernetes 服務後端節點 IP）；
- –default-*-toleration-seconds：設置節點異常相關的閾值；
- –max-*-requests-inflight：請求相關的最大閾值；
- –etcd-*：訪問 etcd 的證書和 etcd 服務器地址；
- –experimental-encryption-provider-config：指定用於加密 etcd 中 secret 的配置；
- –bind-address： https 監聽的 IP，不能為 127.0.0.1，否則外界不能訪問它的安全端口 6443；
- –secret-port：https 監聽端口；
- –insecure-port=0：關閉監聽 http 非安全端口(8080)；
- –tls-*-file：指定 apiserver 使用的證書、私鑰和 CA 文件；
- –audit-*：配置審計策略和審計日誌文件相關的參數；
- –client-ca-file：驗證 client (kue-controller-manager、kube-scheduler、kubelet、kube-proxy 等)請求所帶的證書；
- –enable-bootstrap-token-auth：啟用 kubelet bootstrap 的 token 認證；
- –requestheader-*：kube-apiserver 的 aggregator layer 相關的配置參數，proxy-client & HPA 需要使用；
- –requestheader-client-ca-file：用於簽名 –proxy-client-cert-file 和 –proxy-client-key-file 指定的證書；在啟用了 metric aggregator 時使用；
- –requestheader-allowed-names：不能為空，值為逗號分割的 –proxy-client-cert-file 證書的 CN 名稱，這裏設置為 “aggregator”；
- –service-account-key-file：簽名 ServiceAccount Token 的公鑰文件，kube-controller-manager 的 –service-account-private-key-file 指定私鑰文件，兩者配對使用；
- –runtime-config=api/all=true： 啟用所有版本的 APIs，如 autoscaling/v2alpha1；
- –authorization-mode=Node,RBAC、–anonymous-auth=false： 開啟 Node 和 RBAC 授權模式，拒絕未授權的請求；
- –enable-admission-plugins：啟用一些默認關閉的 plugins；
- –allow-privileged：運行執行 privileged 權限的容器；
- –apiserver-count=3：指定 apiserver 實例的數量；
- –event-ttl：指定 events 的保存時間；
- –kubelet-*：如果指定，則使用 https 訪問 kubelet APIs；需要為證書對應的用戶(上面 kubernetes*.pem 證書的用戶為 kubernetes) 用戶定義 RBAC 規則，否則訪問 kubelet API 時提示未授權；
- –proxy-client-*：apiserver 訪問 metrics-server 使用的證書；
- –service-cluster-ip-range： 指定 Service Cluster IP 地址段；
- –service-node-port-range： 指定 NodePort 的端口範圍。
提示：如果 kube-apiserver 機器沒有運行 kube-proxy，則還需要添加 –enable-aggregator-routing=true 參數。
注意：requestheader-client-ca-file 指定的 CA 證書，必須具有 client auth and server auth；
如果 –requestheader-allowed-names 為空，或者 –proxy-client-cert-file 證書的 CN 名稱不在 allowed-names 中，則後續查看 node 或 pods 的 metrics 失敗，提示：
1 [root@zhangjun-k8s01 1.8+]# kubectl top nodes
2 Error from server (Forbidden): nodes.metrics.k8s.io is forbidden: User "aggregator" cannot list resource "nodes" in API group "metrics.k8s.io" at the cluster scope
3
2.11 分發systemd
1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
3 [root@k8smaster01 work]# for (( i=0; i < 3; i++ ))
4 do
5 sed -e "s/##MASTER_NAME##/${MASTER_NAMES[i]}/" -e "s/##MASTER_IP##/${MASTER_IPS[i]}/" kube-apiserver.service.template > kube-apiserver-${MASTER_IPS[i]}.service
6 done
7 [root@k8smaster01 work]# ls kube-apiserver*.service #替換相應的IP
8 [root@k8smaster01 ~]# cd /opt/k8s/work
9 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
10 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]}
11 do
12 echo ">>> ${master_ip}"
13 scp kube-apiserver-${master_ip}.service root@${master_ip}:/etc/systemd/system/kube-apiserver.service
14 done #分發systemd
三 啟動並驗證
3.1 啟動kube-apiserver服務
1 [root@k8smaster01 ~]# source /opt/k8s/bin/environment.sh
2 [root@k8smaster01 ~]# for master_ip in ${MASTER_IPS[@]}
3 do
4 echo ">>> ${master_ip}"
5 ssh root@${master_ip} "mkdir -p ${K8S_DIR}/kube-apiserver"
6 ssh root@${master_ip} "systemctl daemon-reload && systemctl enable kube-apiserver && systemctl restart kube-apiserver"
7 done
3.2 檢查kube-apiserver服務
1 [root@k8smaster01 ~]# source /opt/k8s/bin/environment.sh
2 [root@k8smaster01 ~]# for master_ip in ${MASTER_IPS[@]}
3 do
4 echo ">>> ${master_ip}"
5 ssh root@${master_ip} "systemctl status kube-apiserver |grep 'Active:'"
6 done
3.3 查看kube-apiserver寫入etcd的數據
1 [root@k8smaster01 ~]# source /opt/k8s/bin/environment.sh
2 [root@k8smaster01 ~]# ETCDCTL_API=3 etcdctl \
3 --endpoints=${ETCD_ENDPOINTS} \
4 --cacert=/opt/k8s/work/ca.pem \
5 --cert=/opt/k8s/work/etcd.pem \
6 --key=/opt/k8s/work/etcd-key.pem \
7 get /registry/ --prefix --keys-only
3.4 檢查集群信息
1 [root@k8smaster01 ~]# kubectl cluster-info
2 [root@k8smaster01 ~]# kubectl get all --all-namespaces
3 [root@k8smaster01 ~]# kubectl get componentstatuses
4 [root@k8smaster01 ~]# sudo netstat -lnpt|grep kube #檢查 kube-apiserver 監聽的端口
提示：
如果執行 kubectl 命令式時輸出如下錯誤信息，則說明使用的 ~/.kube/config 文件不對，請切換到正確的賬戶后再執行該命令：
The connection to the server localhost:8080 was refused – did you specify the right host or port?
執行 kubectl get componentstatuses 命令時，apiserver 默認向 127.0.0.1 發送請求。當 controller-manager、scheduler 以集群模式運行時，有可能和 kube-apiserver 不在一台機器上，這時 controller-manager 或 scheduler 的狀態為 Unhealthy，但實際上它們工作正常；
6443: 接收 https 請求的安全端口，對所有請求做認證和授權；
由於關閉了非安全端口，故沒有監聽 8080。
3.5 授權
授予 kube-apiserver 訪問 kubelet API 的權限。
在執行 kubectl exec、run、logs 等命令時，apiserver 會將請求轉發到 kubelet 的 https 端口。本實驗定義 RBAC 規則，授權 apiserver 使用的證書（kubernetes.pem）用戶名（CN：kuberntes）訪問 kubelet API 的權限：
1 [root@k8smaster01 ~]# kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes
本站聲明:網站內容來源於博客園,如有侵權,請聯繫我們,我們將及時處理
【其他文章推薦】
※想知道網站建置、網站改版該如何進行嗎?將由專業工程師為您規劃客製化網頁設計及後台網頁設計
※不管是台北網頁設計公司、台中網頁設計公司，全省皆有專員為您服務
※Google地圖已可更新顯示潭子電動車充電站設置地點!!
※帶您來看台北網站建置，台北網頁設計，各種案例分享
