作者：李毓
k8s的adm安装方式有一个巨坑，就是证书过期问题。其中涉及到的证书有apiserver,kubelet,etcd,proxy等等证书。这个问题在二进制安装方式是不存在的，因为可以手动更改证书。但是由于adm是自动安装，所以需要后期处理。
目前的解决方式一般有三种，第一种是集群升级，通过升级k8s，间接的把证书也升级了。第二种是修改源代码，也就是对kubeadm重新编译。第三种就是重新生成证书。
由于k8s1.18.8版本刚刚更新了。所以我想趁着这次机会演绎一下集群升级和证书更新的一石二鸟操作。
约定：
三台机器，一台master，二台server。版本是1.18.6
1[root@adm-master ~]# kubectl get nodes 2NAME STATUS ROLES AGE VERSION 3adm-master Ready master 37d v1.18.6 4adm-node1 Ready <none> 37d v1.18.6 5adm-node2 Ready <none> 37d v1.18.6
查看一下证书的有效期
1[root@adm-master ~]# openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text |grep ' Not ' 2 Not Before: Aug 1 13:41:05 2020 GMT 3 Not After : Aug 1 13:41:05 2021 GMT
我们先来看一下kubeadm的版本
1[root@adm-master ~]# kubeadm version 2kubeadm version: &version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:56:34Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
可以看见，和集群的版本是相同的。
通过kbueadm upgrade plan 方式查看升级计划
1[root@adm-master ~]# kubeadm upgrade plan 2[upgrade/config] Making sure the configuration is correct: 3[upgrade/config] Reading configuration from the cluster... 4[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' 5[preflight] Running pre-flight checks. 6[upgrade] Running cluster health checks 7[upgrade] Fetching available versions to upgrade to 8[upgrade/versions] Cluster version: v1.18.0 9[upgrade/versions] kubeadm version: v1.18.6 10I0907 23:36:36.404303 26584 version.go:252] remote version is much newer: v1.19.0; falling back to: stable-1.18 11[upgrade/versions] Latest stable version: v1.18.8 12[upgrade/versions] Latest stable version: v1.18.8 13W0907 23:36:52.285681 26584 version.go:102] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.18.txt": Get https://storage.googleapis.com/kubernetes-release/release/stable-1.18.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 14W0907 23:36:52.285711 26584 version.go:103] falling back to the local client version: v1.18.6 15[upgrade/versions] Latest version in the v1.18 series: v1.18.6 16[upgrade/versions] Latest version in the v1.18 series: v1.18.6 17 18Upgrade to the latest version in the v1.18 series: 19 20COMPONENT CURRENT AVAILABLE 21API Server v1.18.0 v1.18.6 22Controller Manager v1.18.0 v1.18.6 23Scheduler v1.18.0 v1.18.6 24Kube Proxy v1.18.0 v1.18.6 25CoreDNS 1.6.7 1.6.7 26Etcd 3.4.3 3.4.3-0 27 28You can now apply the upgrade by executing the following command: 29 30 kubeadm upgrade apply v1.18.6 31 32Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply': 33COMPONENT CURRENT AVAILABLE 34Kubelet 3 x v1.18.6 v1.18.8 35 36Upgrade to the latest stable version: 37 38COMPONENT CURRENT AVAILABLE 39API Server v1.18.0 v1.18.8 40Controller Manager v1.18.0 v1.18.8 41Scheduler v1.18.0 v1.18.8 42Kube Proxy v1.18.0 v1.18.8 43CoreDNS 1.6.7 1.6.7 44Etcd 3.4.3 3.4.3-0 45 46You can now apply the upgrade by executing the following command: 47 48 kubeadm upgrade apply v1.18.8 49 50Note: Before you can perform this upgrade, you have to update kubeadm to v1.18.8.
开始升级
1[root@adm-master ~]# kubeadm upgrade apply v1.18.8 2[upgrade/config] Making sure the configuration is correct: 3[upgrade/config] Reading configuration from the cluster... 4[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' 5[preflight] Running pre-flight checks. 6[upgrade] Running cluster health checks 7[upgrade/version] You have chosen to change the cluster version to "v1.18.8" 8[upgrade/versions] Cluster version: v1.18.0 9[upgrade/versions] kubeadm version: v1.18.6 10[upgrade/version] FATAL: the --version argument is invalid due to these errors: 11 12 - Specified version to upgrade to "v1.18.8" is higher than the kubeadm version "v1.18.6". Upgrade kubeadm first using the tool you used to install kubeadm 13 14Can be bypassed if you pass the --force flag 15To see the stack trace of this error execute with --v=5 or higher 16[root@adm-master ~]# kubeadm upgrade apply v1.18.8 --force 17[upgrade/config] Making sure the configuration is correct: 18[upgrade/config] Reading configuration from the cluster... 19[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' 20[preflight] Running pre-flight checks. 21[upgrade] Running cluster health checks 22[upgrade/version] You have chosen to change the cluster version to "v1.18.8" 23[upgrade/versions] Cluster version: v1.18.0 24[upgrade/versions] kubeadm version: v1.18.6 25[upgrade/version] Found 1 potential version compatibility errors but skipping since the --force flag is set: 26 27 - Specified version to upgrade to "v1.18.8" is higher than the kubeadm version "v1.18.6". Upgrade kubeadm first using the tool you used to install kubeadm 28[upgrade/prepull] Will prepull images for components [kube-apiserver kube-controller-manager kube-scheduler etcd] 29[upgrade/prepull] Prepulling image for component etcd. 30[upgrade/prepull] Prepulling image for component kube-apiserver. 31[upgrade/prepull] Prepulling image for component kube-controller-manager. 32[upgrade/prepull] Prepulling image for component kube-scheduler. 33[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-controller-manager 34[apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler 35[apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-etcd 36[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-apiserver 37[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler 38[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-etcd 39[upgrade/prepull] Prepulled image for component etcd. 40[upgrade/prepull] Prepulled image for component kube-apiserver. 41[upgrade/prepull] Prepulled image for component kube-controller-manager. 42[upgrade/prepull] Prepulled image for component kube-scheduler. 43[upgrade/prepull] Successfully prepulled the images for all the control plane components 44[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.18.8"... 45Static pod: kube-apiserver-adm-master hash: 6861528d68248c9d1178280d3594d8db 46Static pod: kube-controller-manager-adm-master hash: 871d07b6ec226107d162a636bad7f0aa 47Static pod: kube-scheduler-adm-master hash: 35d86d4a27d4ed2186b3ab641e946a02 48[upgrade/etcd] Upgrading to TLS for etcd 49[upgrade/etcd] Non fatal issue encountered during upgrade: the desired etcd version for this Kubernetes version "v1.18.8" is "3.4.3-0", but the current etcd version is "3.4.3". Won't downgrade etcd, instead just continue 50[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests642601366" 51W0907 23:42:33.736477 28017 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC" 52[upgrade/staticpods] Preparing for "kube-apiserver" upgrade 53[upgrade/staticpods] Renewing apiserver certificate 54[upgrade/staticpods] Renewing apiserver-kubelet-client certificate 55[upgrade/staticpods] Renewing front-proxy-client certificate 56[upgrade/staticpods] Renewing apiserver-etcd-client certificate 57[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2020-09-07-23-42-25/kube-apiserver.yaml" 58[upgrade/staticpods] Waiting for the kubelet to restart the component 59[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s) 60Static pod: kube-apiserver-adm-master hash: 6861528d68248c9d1178280d3594d8db 61Static pod: kube-apiserver-adm-master hash: 253cf8d5be40058a076cd11584613b96 62[apiclient] Found 1 Pods for label selector component=kube-apiserver 63[upgrade/staticpods] Component "kube-apiserver" upgraded successfully! 64[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade 65[upgrade/staticpods] Renewing controller-manager.conf certificate 66[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2020-09-07-23-42-25/kube-controller-manager.yaml" 67[upgrade/staticpods] Waiting for the kubelet to restart the component 68[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s) 69Static pod: kube-controller-manager-adm-master hash: 871d07b6ec226107d162a636bad7f0aa 70Static pod: kube-controller-manager-adm-master hash: 628316be9d303769769e096bfd3537e4 71[apiclient] Found 1 Pods for label selector component=kube-controller-manager 72[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully! 73[upgrade/staticpods] Preparing for "kube-scheduler" upgrade 74[upgrade/staticpods] Renewing scheduler.conf certificate 75[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2020-09-07-23-42-25/kube-scheduler.yaml" 76[upgrade/staticpods] Waiting for the kubelet to restart the component 77[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s) 78Static pod: kube-scheduler-adm-master hash: 35d86d4a27d4ed2186b3ab641e946a02 79Static pod: kube-scheduler-adm-master hash: 7363eeef53899d60c792412f58124026 80[apiclient] Found 1 Pods for label selector component=kube-scheduler 81[upgrade/staticpods] Component "kube-scheduler" upgraded successfully! 82[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace 83[kubelet] Creating a ConfigMap "kubelet-config-1.18" in namespace kube-system with the configuration for the kubelets in the cluster 84[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.18" ConfigMap in the kube-system namespace 85[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 86[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes 87[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials 88[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token 89[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster 90[addons] Applied essential addon: CoreDNS 91[addons] Applied essential addon: kube-proxy 92 93[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.18.8". Enjoy! 94 95[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.
查看镜像版本和集群版本
1[root@adm-master ~]# kubectl get nodes 2NAME STATUS ROLES AGE VERSION 3adm-master Ready master 37d v1.18.6 4adm-node1 Ready <none> 37d v1.18.6 5adm-node2 Ready <none> 37d v1.18.6 6[root@adm-master ~]# docker images 7REPOSITORY TAG IMAGE ID CREATED SIZE 8registry.aliyuncs.com/google_containers/kube-proxy v1.18.8 0fb7201f92d0 3 weeks ago 117MB 9registry.aliyuncs.com/google_containers/kube-controller-manager v1.18.8 6a979351fe5e 3 weeks ago 162MB 10registry.aliyuncs.com/google_containers/kube-apiserver v1.18.8 92d040a0dca7 3 weeks ago 173MB 11registry.aliyuncs.com/google_containers/kube-scheduler v1.18.8 6f7135fb47e0 3 weeks ago 95.3MB 12registry.aliyuncs.com/google_containers/kube-proxy v1.18.0 43940c34f24f 5 months ago 117MB 13registry.aliyuncs.com/google_containers/kube-controller-manager v1.18.0 d3e55153f52f 5 months ago 162MB 14registry.aliyuncs.com/google_containers/kube-apiserver v1.18.0 74060cea7f70 5 months ago 173MB 15registry.aliyuncs.com/google_containers/kube-scheduler v1.18.0 a31f78c7c8ce 5 months ago 95.3MB 16registry.cn-shenzhen.aliyuncs.com/carp/flannel v0.11 f60e29a33f27 6 months ago 52.6MB 17registry.aliyuncs.com/google_containers/pause 3.2 80d28bedfe5d 6 months ago 683kB 18registry.aliyuncs.com/google_containers/coredns 1.6.7 67da37a9a360 7 months ago 43.8MB 19registry.aliyuncs.com/google_containers/etcd 3.4.3-0 303ce5db0e90 10 months ago 288MB
发现没有升级，因为kubectl和kubelet也需要升级。
1[root@adm-master ~]# yum install -y kubelet-1.18.8-0 kubeadm-1.18.8-0 kubectl-1.18.8-0 2 3systemctl daemon-reload 4systemctl restart kubelet 5 6 7[root@adm-master ~]# kubectl get nodes 8NAME STATUS ROLES AGE VERSION 9adm-master Ready master 37d v1.18.8 10adm-node1 Ready <none> 37d v1.18.6 11adm-node2 Ready <none> 37d v1.18.6
maser已经更新了，node还没有更新，去node执行一下.
1yum install -y kubelet-1.18.8-0 2systemctl daemon-reload 3systemctl restart kubelet
再查看一下
1[root@adm-master ~]# kubectl get nodes 2NAME STATUS ROLES AGE VERSION 3adm-master Ready master 37d v1.18.8 4adm-node1 Ready <none> 37d v1.18.8 5adm-node2 Ready <none> 37d v1.18.8
集群升级成功
顺便查看一下证书日期
1[root@adm-master ~]# openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text |grep ' Not ' 2 Not Before: Aug 1 13:41:05 2020 GMT 3 Not After : Sep 7 15:42:34 2021 GMT
到期日已经变了。
这样一石二鸟计划就顺利完成咯。