作者：郝建伟

背景

面对更多项目现场交付，偶而会遇到客户环境不具备公网条件，完全内网部署，这就需要有一套完善且高效的离线部署方案。

系统资源

参数配置

注：在全部节点执行以下操作

系统基础设置

工作、日志及数据存储目录设定

1$ mkdir -p /export/servers
2$ mkdir -p /export/logs
3$ mkdir -p /export/data
4$ mkdir -p /export/upload

内核及网络参数优化

1$ vim /etc/sysctl.conf
2
3# 设置以下内容
4fs.file-max = 1048576
5net.ipv4.tcp_syncookies = 1
6net.ipv4.tcp_tw_reuse = 1
7net.ipv4.tcp_fin_timeout = 5
8net.ipv4.neigh.default.gc_stale_time = 120
9net.ipv4.conf.default.rp_filter = 0
10net.ipv4.conf.all.rp_filter = 0
11net.ipv4.conf.all.arp_announce = 2
12net.ipv4.conf.lo.arp_announce = 2 
13vm.max_map_count = 262144
14
15# 及时生效
16sysctl -w vm.max_map_count=262144

ulimit优化

1$ vim /etc/security/limits.conf 
2
3# 设置以下内容
4* soft memlock unlimited
5* hard memlock unlimited
6* soft nproc 102400
7* hard nproc 102400
8* soft nofile 1048576
9* hard nofile 1048576

基础环境准备

ansible安装

1. 环境说明

2. 部署说明

物联管理平台机器数量繁多，需要ansible进行批量操作机器，节省时间，需要从部署节点至其他节点root免密。

1注：在不知道root密码情况下，可以手动操作名密，按以下操作步骤执行：
2# 需要在部署机器上执行以下命令生成公钥
3$ ssh-keygen -t rsa
4# 复制~/.ssh/id_rsa.pub内容，并粘贴至其他节点~/.ssh/authorized_keys文件里面
5# 如果没有authorized_keys文件，可先执行创建创建在进行粘贴操作
6$ touch ~/.ssh/authorized_keys

3. 部署步骤

1） 在线安装

$ yum -y install https://releases.ansible.com/ansible/rpm/release/epel-7-x86_64/ansible-2.9.27-1.el7.ans.noarch.rpm

2） 离线安装

1# 提前上传ansible及所有依赖rpm包，并切换至rpm包目录
2$ yum -y ./*rpm

3） 查看版本

1$ ansible --version
2ansible 2.9.27
3  config file = /etc/ansible/ansible.cfg
4  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
5  ansible python module location = /usr/lib/python2.7/site-packages/ansible
6  executable location = /usr/bin/ansible
7  python version = 2.7.5 (default, Apr  2 2020, 13:16:51) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]

4） 设置管理主机列表

1$ vim /etc/ansible/hosts
2
3[master]
410.132.10.91    node_name=k8s-master1
510.132.10.92    node_name=k8s-master2
610.132.10.93    node_name=k8s-master3
7
8[worker]
910.132.10.94    node_name=k8s-worker1
1010.132.10.95    node_name=k8s-worker2
1110.132.10.96    node_name=k8s-worker3
1210.132.10.97    node_name=k8s-worker4
1310.132.10.98    node_name=k8s-worker5
1410.132.10.99    node_name=k8s-worker6
15
16[etcd]
1710.132.10.91    etcd_name=etcd1
1810.132.10.92    etcd_name=etcd2
1910.132.10.93    etcd_name=etcd3
20
21[k8s:children]
22master
23worker

5） 禁用ssh主机检查

1$ vi /etc/ansible/ansible.cfg
2# 修改以下设置
3# uncomment this to disable SSH key host checking
4host_key_checking = False

6） 取消SELINUX设定及放开防火墙

1$ ansible k8s -m command -a "setenforce 0"
2$ ansible k8s -m command -a "sed --follow-symlinks -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config"
3$ ansible k8s -m command -a "firewall-cmd --set-default-zone=trusted"
4$ ansible k8s -m command -a "firewall-cmd --complete-reload"
5$ ansible k8s -m command -a "swapoff -a"

7）hosts设置

1$ cd /export/upload && vim hosts_set.sh
2#设置以下脚本内容
3#!/bin/bashcat > /etc/hosts << EOF
4127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
5::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
610.132.10.100 deploy harbor
710.132.10.91 master01
810.132.10.92 master02
910.132.10.93 master03
1010.132.10.94 worker01
1110.132.10.95 worker02
1210.132.10.96 worker03
1310.132.10.97 worker04
1410.132.10.98 worker05
1510.132.10.99 worker06
16EOF
17
18$ ansible new_worker -m copy -a 'src=/export/upload/hosts_set.sh dest=/export/upload'
19$ ansible new_worker -m command -a 'sh /export/upload/hosts_set.sh'

docker安装

1. 环境说明

2. 部署说明

此处的docker主要用于harbor针对镜像的操作，包括镜像的打tag以及push，用于后期部署pod的时候直接通过harbor私有镜像库拉取。

3. 部署方法

1） 在线安装

$ yum -y install docker-ce-20.10.17

2） 离线安装

1# 提前上传docker及所有依赖rpm包，并切换至rpm包目录
2$ yum -y ./*rpm

3） 重新加载配置文件，启动并查看状态

1$ systemctl start docker
2$ systemctl status docker

4） 设置开机自启

$ systemctl enable docker 

5） 查看版本

1$ docker version
2Client: Docker Engine - Community
3 Version:           20.10.17 
4 API version:       1.41 
5 Go version:        go1.17.11 
6 Git commit:        100c701 
7 Built:             Mon Jun  6 23:05:12 2022 
8 OS/Arch:           linux/amd64 
9 Context:           default 
10 Experimental:      true
11 
12Server: Docker Engine - Community
13 Engine:  
14  Version:          20.10.17  
15  API version:      1.41 (minimum version 1.12)  
16  Go version:       go1.17.11  
17  Git commit:       a89b842  
18  Built:            Mon Jun  6 23:03:33 2022  
19  OS/Arch:          linux/amd64  
20  Experimental:     false 
21 containerd:  
22  Version:          1.6.8  
23  GitCommit:        9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6 
24 runc: 
25  Version:          1.1.4  
26  GitCommit:        v1.1.4-0-g5fd4c4d 
27 docker-init:  
28 Version:          0.19.0  
29 GitCommit:        de40ad0

docker-compose安装

1. 环境说明

2. 部署说明

harbor私有镜像库依赖。

3. 部署方法

1） 下载docker-compose并上传至服务器

$ curl -L https://github.com/docker/compose/releases/download/v2.9.0/docker-compose-linux-x86_64 -o docker-compose

2） 修改docker-compose执行权限

1$ mv docker-compose /usr/local/bin/
2$ chmod +x /usr/local/bin/docker-compose
3$ docker-compose version

3） 查看版本

1$ docker-compose version
2Docker Compose version v2.9.0

harbor安装

1. 环境说明

2. 部署说明

私有镜像仓库。

3. 下载harbor离线安装包并上传至服务器

$ wget https://github.com/goharbor/harbor/releases/download/v2.4.3/harbor-offline-installer-v2.4.3.tgz

4. 解压安装包

1$ tar -xzvf harbor-offline-installer-v2.4.3.tgz -C /export/servers/
2$ cd /export/servers/harbor

5. 修改配置文件

1$ mv harbor.yml.tmpl harbor.yml
2$ vim harbor.yml

6. 设置以下内容

1hostname: 10.132.10.100
2http.port: 8090
3data_volume: /export/data/harbor
4log.location: /export/logs/harbor

7. 导入harbor镜像

1$ docker load -i harbor.v2.4.3.tar.gz 
2# 等待导入harbor依赖镜像文件
3$ docker images
4REPOSITORY                      TAG       IMAGE ID       CREATED       SIZE
5goharbor/harbor-exporter        v2.4.3    776ac6ee91f4   4 weeks ago   81.5MB
6goharbor/chartmuseum-photon     v2.4.3    f39a9694988d   4 weeks ago   172MB
7goharbor/redis-photon           v2.4.3    b168e9750dc8   4 weeks ago   154MB
8goharbor/trivy-adapter-photon   v2.4.3    a406a715461c   4 weeks ago   251MB
9goharbor/notary-server-photon   v2.4.3    da89404c7cf9   4 weeks ago   109MB
10goharbor/notary-signer-photon   v2.4.3    38468ac13836   4 weeks ago   107MB
11goharbor/harbor-registryctl     v2.4.3    61243a84642b   4 weeks ago   135MB
12goharbor/registry-photon        v2.4.3    9855479dd6fa   4 weeks ago   77.9MB
13goharbor/nginx-photon           v2.4.3    0165c71ef734   4 weeks ago   44.4MB
14goharbor/harbor-log             v2.4.3    57ceb170dac4   4 weeks ago   161MB
15goharbor/harbor-jobservice      v2.4.3    7fea87c4b884   4 weeks ago   219MB
16goharbor/harbor-core            v2.4.3    d864774a3b8f   4 weeks ago   197MB
17goharbor/harbor-portal          v2.4.3    85f00db66862   4 weeks ago   53.4MB
18goharbor/harbor-db              v2.4.3    7693d44a2ad6   4 weeks ago   225MB
19goharbor/prepare                v2.4.3    c882d74725ee   4 weeks ago   268MB

8. 启动harbor

1./prepare  # 如果有二次修改harbor.yml文件，请执行使配置文件生效
2./install.sh --help # 查看启动参数
3./install.sh --with-chartmuseum

运行环境搭建

docker安装

1. 环境说明

2. 部署说明

k8s容器运行环境docker部署

3. 部署方法

1） 上传docker及依赖rpm包

$ ls /export/upload/docker-rpm.tgz 

2） 分发安装包

1$ ansible k8s -m copy -a "src=/export/upload/docker-rpm.tgz dest=/export/upload/"
2# 全部节点返回以下信息
3CHANGED => {
4    "ansible_facts": {
5        "discovered_interpreter_python": "/usr/bin/python"
6    }, 
7    "changed": true, 
8    "checksum": "acd3897edb624cd18a197bcd026e6769797f4f05", 
9    "dest": "/export/upload/docker-rpm.tgz", 
10    "gid": 0, 
11    "group": "root", 
12    "md5sum": "3ba6d9fe6b2ac70860b6638b88d3c89d", 
13    "mode": "0644", 
14    "owner": "root", 
15    "secontext": "system_u:object_r:usr_t:s0", 
16    "size": 103234394, 
17    "src": "/root/.ansible/tmp/ansible-tmp-1661836788.82-13591-17885284311930/source", 
18    "state": "file", 
19    "uid": 0
20}

3） 执行解压并安装

$ ansible k8s -m shell -a "tar xzvf /export/upload/docker-rpm.tgz -C /export/upload/ && yum -y install /export/upload/docker-rpm/*"

4） 设置开机自启并启动

$ ansible k8s -m shell -a "systemctl enable docker && systemctl start docker"

5） 查看版本

1$ ansible k8s -m shell -a "docker version"
2# 全部节点返回以下信息
3CHANGED | rc=0 >>
4Client: Docker Engine - Community
5 Version:           20.10.17
6 API version:       1.41
7 Go version:        go1.17.11
8 Git commit:        100c701
9 Built:             Mon Jun  6 23:05:12 2022
10 OS/Arch:           linux/amd64
11 Context:           default
12 Experimental:      true
13
14Server: Docker Engine - Community
15 Engine:
16  Version:          20.10.17
17  API version:      1.41 (minimum version 1.12)
18  Go version:       go1.17.11
19  Git commit:       a89b842
20  Built:            Mon Jun  6 23:03:33 2022
21  OS/Arch:          linux/amd64
22  Experimental:     false
23 containerd:
24  Version:          1.6.8
25  GitCommit:        9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
26 runc:
27  Version:          1.1.4
28  GitCommit:        v1.1.4-0-g5fd4c4d
29 docker-init:
30  Version:          0.19.0
31  GitCommit:        de40ad0

kubernetes安装

有网环境安装

1# 添加阿里云YUM的软件源：
2cat > /etc/yum.repos.d/kubernetes.repo << EOF
3[kubernetes]
4name=Kubernetes
5baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
6enabled=1
7gpgcheck=0
8repo_gpgcheck=0
9gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
10EOF

下载离线安装包

1# 创建rpm软件存储目录：
2mkdir -p /export/download/kubeadm-rpm
3
4# 执行命令：
5yum install -y kubelet-1.22.4 kubeadm-1.22.4 kubectl-1.22.4 --downloadonly --downloaddir /export/download/kubeadm-rpm

无网环境安装

1） 上传kubeadm及依赖rpm包

1$ ls /export/upload/
2kubeadm-rpm.tgz 

2） 分发安装包

1$ ansible k8s -m copy -a "src=/export/upload/kubeadm-rpm.tgz dest=/export/upload/"
2# 全部节点返回以下信息
3CHANGED => {
4    "ansible_facts": {
5        "discovered_interpreter_python": "/usr/bin/python"
6    }, 
7    "changed": true, 
8    "checksum": "3fe96fe1aa7f4a09d86722f79f36fb8fde69facb", 
9    "dest": "/export/upload/kubeadm-rpm.tgz", 
10    "gid": 0, 
11    "group": "root", 
12    "md5sum": "80d5bda420db6ea23ad75dcf0f76e858", 
13    "mode": "0644", 
14    "owner": "root", 
15    "secontext": "system_u:object_r:usr_t:s0", 
16    "size": 67423355, 
17    "src": "/root/.ansible/tmp/ansible-tmp-1661840257.4-33361-139823848282879/source", 
18    "state": "file", 
19    "uid": 0
20}

3） 执行解压并安装

$ ansible k8s -m shell -a "tar xzvf /export/upload/kubeadm-rpm.tgz -C /export/upload/ && yum -y install /export/upload/kubeadm-rpm/*"

4） 设置开机自启并启动

1$ ansible k8s -m shell -a "systemctl enable kubelet && systemctl start kubelet"
2注：此时kubelet启动失败，会进入不断重启，这个是正常现象，执行init或join后问题会自动解决，对此官网有如下描述，也就是此时不用理会kubelet.service，可执行发下命令查看kubelet状态。
3$ journalctl -xefu kubelet

4） 分发依赖镜像至集群节点

1# 可以在有公网环境提前下载镜像
2$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.8.4
3$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.5.0-0
4$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.22.4
5$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.22.4
6$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.22.4
7$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.22.4
8$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.5
9$ docker pull rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.0
10$ docker pull rancher/mirrored-flannelcni-flannel:v0.19.1
11# 导出镜像文件，上传部署节点并导入镜像库
12$ ls /export/upload
13
14$ docker load -i google_containers-coredns-v1.8.4.tar
15$ docker load -i google_containers-etcd:3.5.0-0.tar
16$ docker load -i google_containers-kube-apiserver:v1.22.4.tar
17$ docker load -i google_containers-kube-controller-manager-v1.22.4.tar
18$ docker load -i google_containers-kube-proxy-v1.22.4.tar
19$ docker load -i google_containers-kube-scheduler-v1.22.4.tar
20$ docker load -i google_containers-pause-3.5.tar
21$ docker load -i rancher-mirrored-flannelcni-flannel-cni-plugin-v1.1.0.tar
22$ docker load -i rancher-mirrored-flannelcni-flannel-v0.19.1.tar
23
24# 镜像打harbor镜像库tag
25$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.8.4 10.132.10.100:8090/community/coredns:v1.8.4
26$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.5.0-0 10.132.10.100:8090/community/etcd:3.5.0-0
27$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.22.4 10.132.10.100:8090/community/kube-apiserver:v1.22.4
28$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.22.4 10.132.10.100:8090/community/kube-controller-manager:v1.22.4
29$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.22.4 10.132.10.100:8090/community/kube-proxy:v1.22.4
30$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.22.4 10.132.10.100:8090/community/kube-scheduler:v1.22.4
31$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.5 10.132.10.100:8090/community/pause:3.5
32$ docker tag rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.0 10.132.10.100:8090/community/mirrored-flannelcni-flannel-cni-plugin:v1.1.0
33$ docker tag rancher/mirrored-flannelcni-flannel:v0.19.1 10.132.10.100:8090/community/mirrored-flannelcni-flannel:v0.19.1
34
35# 推送至harbor镜像库
36$ docker push 192.168.186.120:8090/community/coredns:v1.8.4
37$ docker push 192.168.186.120:8090/community/etcd:3.5.0-0
38$ docker push 192.168.186.120:8090/community/kube-apiserver:v1.22.4
39$ docker push 192.168.186.120:8090/community/kube-controller-manager:v1.22.4
40$ docker push 192.168.186.120:8090/community/kube-proxy:v1.22.4
41$ docker push 192.168.186.120:8090/community/kube-scheduler:v1.22.4
42$ docker push 192.168.186.120:8090/community/pause:3.5
43$ docker push 192.168.186.120:8090/community/mirrored-flannelcni-flannel-cni-plugin:v1.1.0
44$ docker push 192.168.186.120:8090/community/mirrored-flannelcni-flannel:v0.19.1

5）部署首个master

1$ kubeadm init \
2--control-plane-endpoint "10.132.10.91:6443" \
3--image-repository 10.132.10.100/community \
4--kubernetes-version v1.22.4 \
5--service-cidr=172.16.0.0/16 \
6--pod-network-cidr=10.244.0.0/16 \
7--token "abcdef.0123456789abcdef" \
8--token-ttl "0" \
9--upload-certs
10
11# 显示以下信息
12[init] Using Kubernetes version: v1.22.4
13[preflight] Running pre-flight checks
14	[WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly
15[preflight] Pulling images required for setting up a Kubernetes cluster
16[preflight] This might take a minute or two, depending on the speed of your internet connection
17[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
18[certs] Using certificateDir folder "/etc/kubernetes/pki"
19[certs] Generating "ca" certificate and key
20[certs] Generating "apiserver" certificate and key
21[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local master01] and IPs [172.16.0.1 10.132.10.91]
22[certs] Generating "apiserver-kubelet-client" certificate and key
23[certs] Generating "front-proxy-ca" certificate and key
24[certs] Generating "front-proxy-client" certificate and key
25[certs] Generating "etcd/ca" certificate and key
26[certs] Generating "etcd/server" certificate and key
27[certs] etcd/server serving cert is signed for DNS names [localhost master01] and IPs [10.132.10.91 127.0.0.1 ::1]
28[certs] Generating "etcd/peer" certificate and key
29[certs] etcd/peer serving cert is signed for DNS names [localhost master01] and IPs [10.132.10.91 127.0.0.1 ::1]
30[certs] Generating "etcd/healthcheck-client" certificate and key
31[certs] Generating "apiserver-etcd-client" certificate and key
32[certs] Generating "sa" key and public key
33[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
34[kubeconfig] Writing "admin.conf" kubeconfig file
35[kubeconfig] Writing "kubelet.conf" kubeconfig file
36[kubeconfig] Writing "controller-manager.conf" kubeconfig file
37[kubeconfig] Writing "scheduler.conf" kubeconfig file
38[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
39[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
40[kubelet-start] Starting the kubelet
41[control-plane] Using manifest folder "/etc/kubernetes/manifests"
42[control-plane] Creating static Pod manifest for "kube-apiserver"
43[control-plane] Creating static Pod manifest for "kube-controller-manager"
44[control-plane] Creating static Pod manifest for "kube-scheduler"
45[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
46[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
47[apiclient] All control plane components are healthy after 11.008638 seconds
48[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
49[kubelet] Creating a ConfigMap "kubelet-config-1.22" in namespace kube-system with the configuration for the kubelets in the cluster
50[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
51[upload-certs] Using certificate key:
529151ea7bb260a42297f2edc486d5792f67d9868169310b82ef1eb18f6e4c0f13
53[mark-control-plane] Marking the node master01 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
54[mark-control-plane] Marking the node master01 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
55[bootstrap-token] Using token: abcdef.0123456789abcdef
56[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
57[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
58[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
59[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
60[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
61[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
62[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
63[addons] Applied essential addon: CoreDNS
64[addons] Applied essential addon: kube-proxy
65
66Your Kubernetes control-plane has initialized successfully!
67
68To start using your cluster, you need to run the following as a regular user:
69
70  mkdir -p $HOME/.kube
71  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
72  sudo chown $(id -u):$(id -g) $HOME/.kube/config
73
74Alternatively, if you are the root user, you can run:
75
76  export KUBECONFIG=/etc/kubernetes/admin.conf
77
78You should now deploy a pod network to the cluster.
79Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
80  https://kubernetes.io/docs/concepts/cluster-administration/addons/
81
82You can now join any number of the control-plane node running the following command on each as root:
83
84  kubeadm join 10.132.10.91:6443 --token abcdef.0123456789abcdef \
85	--discovery-token-ca-cert-hash sha256:4884a98b0773bc89c36dc5fa51569293103ff093e9124431c4c8c2d5801a96a2 \
86	--control-plane --certificate-key 9151ea7bb260a42297f2edc486d5792f67d9868169310b82ef1eb18f6e4c0f13
87
88Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
89As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
90"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.
91
92Then you can join any number of worker nodes by running the following on each as root:
93
94kubeadm join 10.132.10.91:6443 --token abcdef.0123456789abcdef \
95	--discovery-token-ca-cert-hash sha256:4884a98b0773bc89c36dc5fa51569293103ff093e9124431c4c8c2d5801a96a2

6）生成kubelet环境配置文件

1# 执行命令
2$ mkdir -p $HOME/.kube
3$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
4$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

7）配置网络插件flannel

1# 创建flannel.yml文件
2$ touch /export/servers/kubernetes/flannel.yml
3$ vim /export/servers/kubernetes/flannel.yml
4# 设置以下内容，需要关注有网无网时对应的地址切换
5---
6kind: Namespace
7apiVersion: v1
8metadata:
9  name: kube-flannel
10  labels:
11    pod-security.kubernetes.io/enforce: privileged
12---
13kind: ClusterRole
14apiVersion: rbac.authorization.k8s.io/v1
15metadata:
16  name: flannel
17rules:
18- apiGroups:
19  - ""
20  resources:
21  - pods
22  verbs:
23  - get
24- apiGroups:
25  - ""
26  resources:
27  - nodes
28  verbs:
29  - list
30  - watch
31- apiGroups:
32  - ""
33  resources:
34  - nodes/status
35  verbs:
36  - patch
37---
38kind: ClusterRoleBinding
39apiVersion: rbac.authorization.k8s.io/v1
40metadata:
41  name: flannel
42roleRef:
43  apiGroup: rbac.authorization.k8s.io
44  kind: ClusterRole
45  name: flannel
46subjects:
47- kind: ServiceAccount
48  name: flannel
49  namespace: kube-flannel
50---
51apiVersion: v1
52kind: ServiceAccount
53metadata:
54  name: flannel
55  namespace: kube-flannel
56---
57kind: ConfigMap
58apiVersion: v1
59metadata:
60  name: kube-flannel-cfg
61  namespace: kube-flannel
62  labels:
63    tier: node
64    app: flannel
65data:
66  cni-conf.json: |
67    {
68      "name": "cbr0",
69      "cniVersion": "0.3.1",
70      "plugins": [
71        {
72          "type": "flannel",
73          "delegate": {
74            "hairpinMode": true,
75            "isDefaultGateway": true
76          }
77        },
78        {
79          "type": "portmap",
80          "capabilities": {
81            "portMappings": true
82          }
83        }
84      ]
85    }
86  net-conf.json: |
87    {
88      "Network": "10.244.0.0/16",
89      "Backend": {
90        "Type": "vxlan"
91      }
92    }
93---
94apiVersion: apps/v1
95kind: DaemonSet
96metadata:
97  name: kube-flannel-ds
98  namespace: kube-flannel
99  labels:
100    tier: node
101    app: flannel
102spec:
103  selector:
104    matchLabels:
105      app: flannel
106  template:
107    metadata:
108      labels:
109        tier: node
110        app: flannel
111    spec:
112      affinity:
113        nodeAffinity:
114          requiredDuringSchedulingIgnoredDuringExecution:
115            nodeSelectorTerms:
116            - matchExpressions:
117              - key: kubernetes.io/os
118                operator: In
119                values:
120                - linux
121      hostNetwork: true
122      priorityClassName: system-node-critical
123      tolerations:
124      - operator: Exists
125        effect: NoSchedule
126      serviceAccountName: flannel
127      initContainers:
128      - name: install-cni-plugin
129        # 在有网环境下可以切换下面地址
130        # image: docker.io/rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.0
131        # 在无网环境下需要使用私有harbor地址
132        image: 10.132.10.100:8090/community/mirrored-flannelcni-flannel-cni-plugin:v1.1.0
133        command:
134        - cp
135        args:
136        - -f
137        - /flannel
138        - /opt/cni/bin/flannel
139        volumeMounts:
140        - name: cni-plugin
141          mountPath: /opt/cni/bin
142      - name: install-cni
143        # 在有网环境下可以切换下面地址
144        # image: docker.io/rancher/mirrored-flannelcni-flannel:v0.19.1
145        # 在无网环境下需要使用私有harbor地址
146        image: 10.132.10.100:8090/community/mirrored-flannelcni-flannel:v0.19.1
147        command:
148        - cp
149        args:
150        - -f
151        - /etc/kube-flannel/cni-conf.json
152        - /etc/cni/net.d/10-flannel.conflist
153        volumeMounts:
154        - name: cni
155          mountPath: /etc/cni/net.d
156        - name: flannel-cfg
157          mountPath: /etc/kube-flannel/
158      containers:
159      - name: kube-flannel
160        # 在有网环境下可以切换下面地址
161        # image: docker.io/rancher/mirrored-flannelcni-flannel:v0.19.1
162        # 在无网环境下需要使用私有harbor地址
163        image: 10.132.10.100:8090/community/mirrored-flannelcni-flannel:v0.19.1
164        command:
165        - /opt/bin/flanneld
166        args:
167        - --ip-masq
168        - --kube-subnet-mgr
169        resources:
170          requests:
171            cpu: "100m"
172            memory: "50Mi"
173          limits:
174            cpu: "100m"
175            memory: "50Mi"
176        securityContext:
177          privileged: false
178          capabilities:
179            add: ["NET_ADMIN", "NET_RAW"]
180        env:
181        - name: POD_NAME
182          valueFrom:
183            fieldRef:
184              fieldPath: metadata.name
185        - name: POD_NAMESPACE
186          valueFrom:
187            fieldRef:
188              fieldPath: metadata.namespace
189        - name: EVENT_QUEUE_DEPTH
190          value: "5000"
191        volumeMounts:
192        - name: run
193          mountPath: /run/flannel
194        - name: flannel-cfg
195          mountPath: /etc/kube-flannel/
196        - name: xtables-lock
197          mountPath: /run/xtables.lock
198      volumes:
199      - name: run
200        hostPath:
201          path: /run/flannel
202      - name: cni-plugin
203        hostPath:
204          path: /opt/cni/bin
205      - name: cni
206        hostPath:
207          path: /etc/cni/net.d
208      - name: flannel-cfg
209        configMap:
210          name: kube-flannel-cfg
211      - name: xtables-lock
212        hostPath:
213          path: /run/xtables.lock
214          type: FileOrCreate

8）安装网络插件flannel

1# 生效yml配置文件
2$ kubectl apply -f kube-flannel.yml
3
4# 查看pods状态
5$ kubectl get pods -A
6NAMESPACE      NAME                               READY   STATUS    RESTARTS   AGE
7kube-flannel   kube-flannel-ds-kjmt4              1/1     Running   0          148m
8kube-system    coredns-7f84d7b4b5-7qr8g           1/1     Running   0          4h18m
9kube-system    coredns-7f84d7b4b5-fljws           1/1     Running   0          4h18m
10kube-system    etcd-master01                      1/1     Running   0          4h19m
11kube-system    kube-apiserver-master01            1/1     Running   0          4h19m
12kube-system    kube-controller-manager-master01   1/1     Running   0          4h19m
13kube-system    kube-proxy-wzq2t                   1/1     Running   0          4h18m
14kube-system    kube-scheduler-master01            1/1     Running   0          4h19m

9）加入其他master节点

1# 在master01执行如下操作
2# 查看token列表
3$ kubeadm token list
4
5# master01执行init操作后生成加入命令如下
6$ kubeadm join 10.132.10.91:6443 \
7--token abcdef.0123456789abcdef \
8--discovery-token-ca-cert-hash sha256:4884a98b0773bc89c36dc5fa51569293103ff093e9124431c4c8c2d5801a96a2 \
9--control-plane --certificate-key 9151ea7bb260a42297f2edc486d5792f67d9868169310b82ef1eb18f6e4c0f13
10
11# 在其他master节点执行如下操作
12# 分别执行上一步的加入命令，加入master节点至集群
13$ kubeadm join 10.132.10.91:6443 \
14--token abcdef.0123456789abcdef \
15--discovery-token-ca-cert-hash sha256:4884a98b0773bc89c36dc5fa51569293103ff093e9124431c4c8c2d5801a96a2 \
16--control-plane --certificate-key 9151ea7bb260a42297f2edc486d5792f67d9868169310b82ef1eb18f6e4c0f13
17
18# 此处如果报错，一般是certificate-key过期，可以在master01执行如下命令更新
19$ kubeadm init phase upload-certs --upload-certs
203b647155b06311d39faf70cb094d9a5e102afd1398323e820cfb3cfd868ae58f
21
22# 将上面生成的值替换certificate-key值再次在其他master节点执行如下命令
23$ kubeadm join 10.132.10.91:6443 \
24--token abcdef.0123456789abcdef \
25--discovery-token-ca-cert-hash sha256:4884a98b0773bc89c36dc5fa51569293103ff093e9124431c4c8c2d5801a96a2     
26--control-plane 
27--certificate-key 3b647155b06311d39faf70cb094d9a5e102afd1398323e820cfb3cfd868ae58f
28
29# 生成kubelet环境配置文件
30$ mkdir -p $HOME/.kube
31$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
32$ sudo chown $(id -u):$(id -g) $HOME/.kube/config
33
34# 在任意master节点执行查看节点状态命令
35$ kubectl get nodes
36NAME       STATUS   ROLES                  AGE     VERSION
37master01   Ready    control-plane,master   5h58m   v1.22.4
38master02   Ready    control-plane,master   45m     v1.22.4
39master03   Ready    control-plane,master   44m     v1.22.4

9）加入worker节点

1# 在其他worker节点执行master01执行init操作后生成的加入命令如下
2# 分别执行上一步的加入命令，加入master节点至集群
3$ kubeadm join 10.132.10.91:6443 \
4--token abcdef.0123456789abcdef \
5--discovery-token-ca-cert-hash sha256:4884a98b0773bc89c36dc5fa51569293103ff093e9124431c4c8c2d5801a96a2
6
7# 此处如果报错，一般是token过期，可以在master01执行如下命令重新生成加入命令
8$ kubeadm token create --print-join-command
9kubeadm join 10.132.10.91:6443 \
10--token abcdef.0123456789abcdef \
11--discovery-token-ca-cert-hash sha256:cf30ddd3df1c6215b886df1ea378a68ad5a9faad7933d53ca9891ebbdf9a1c3f
12
13# 将上面生成的加入命令再次在其他worker节点执行
14# 查看集成状态
15$ kubectl get nodes
16NAME       STATUS   ROLES                  AGE     VERSION
17master01   Ready    control-plane,master   6h12m   v1.22.4
18master02   Ready    control-plane,master   58m     v1.22.4
19master03   Ready    control-plane,master   57m     v1.22.4
20worker01   Ready    <none>                 5m12s   v1.22.4
21worker02   Ready    <none>                 4m10s   v1.22.4
22worker03   Ready    <none>                 3m42s   v1.22.4

10）配置kubernetes dashboard

1apiVersion: v1
2kind: Namespace
3metadata:
4  name: kubernetes-dashboard
5
6---
7
8apiVersion: v1
9kind: ServiceAccount
10metadata:
11  labels:
12    k8s-app: kubernetes-dashboard
13  name: kubernetes-dashboard
14  namespace: kubernetes-dashboard
15
16---
17
18kind: Service
19apiVersion: v1
20metadata:
21  labels:
22    k8s-app: kubernetes-dashboard
23  name: kubernetes-dashboard
24  namespace: kubernetes-dashboard
25spec:
26  type: NodePort
27  ports:
28    - port: 443
29      targetPort: 8443
30      nodePort: 31001
31  selector:
32    k8s-app: kubernetes-dashboard
33
34---
35
36apiVersion: v1
37kind: Secret
38metadata:
39  labels:
40    k8s-app: kubernetes-dashboard
41  name: kubernetes-dashboard-certs
42  namespace: kubernetes-dashboard
43type: Opaque
44
45---
46
47apiVersion: v1
48kind: Secret
49metadata:
50  labels:
51    k8s-app: kubernetes-dashboard
52  name: kubernetes-dashboard-key-holder
53  namespace: kubernetes-dashboard
54type: Opaque
55
56---
57
58kind: ConfigMap
59apiVersion: v1
60metadata:
61  labels:
62    k8s-app: kubernetes-dashboard
63  name: kubernetes-dashboard-settings
64  namespace: kubernetes-dashboard
65
66---
67
68kind: Role
69apiVersion: rbac.authorization.k8s.io/v1
70metadata:
71  labels:
72    k8s-app: kubernetes-dashboard
73  name: kubernetes-dashboard
74  namespace: kubernetes-dashboard
75rules:
76  # Allow Dashboard to get, update and delete Dashboard exclusive secrets.
77  - apiGroups: [""]
78    resources: ["secrets"]
79    resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs", "kubernetes-dashboard-csrf"]
80    verbs: ["get", "update", "delete"]
81    # Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map.
82  - apiGroups: [""]
83    resources: ["configmaps"]
84    resourceNames: ["kubernetes-dashboard-settings"]
85    verbs: ["get", "update"]
86    # Allow Dashboard to get metrics.
87  - apiGroups: [""]
88    resources: ["services"]
89    resourceNames: ["heapster", "dashboard-metrics-scraper"]
90    verbs: ["proxy"]
91  - apiGroups: [""]
92    resources: ["services/proxy"]
93    resourceNames: ["heapster", "http:heapster:", "https:heapster:", "dashboard-metrics-scraper", "http:dashboard-metrics-scraper"]
94    verbs: ["get"]
95
96---
97
98kind: ClusterRole
99apiVersion: rbac.authorization.k8s.io/v1
100metadata:
101  labels:
102    k8s-app: kubernetes-dashboard
103  name: kubernetes-dashboard
104rules:
105  # Allow Metrics Scraper to get metrics from the Metrics server
106  - apiGroups: ["metrics.k8s.io"]
107    resources: ["pods", "nodes"]
108    verbs: ["get", "list", "watch"]
109
110---
111
112apiVersion: rbac.authorization.k8s.io/v1
113kind: RoleBinding
114metadata:
115  labels:
116    k8s-app: kubernetes-dashboard
117  name: kubernetes-dashboard
118  namespace: kubernetes-dashboard
119roleRef:
120  apiGroup: rbac.authorization.k8s.io
121  kind: Role
122  name: kubernetes-dashboard
123subjects:
124  - kind: ServiceAccount
125    name: kubernetes-dashboard
126    namespace: kubernetes-dashboard
127
128---
129
130apiVersion: rbac.authorization.k8s.io/v1
131kind: ClusterRoleBinding
132metadata:
133  name: kubernetes-dashboard
134roleRef:
135  apiGroup: rbac.authorization.k8s.io
136  kind: ClusterRole
137  name: kubernetes-dashboard
138subjects:
139  - kind: ServiceAccount
140    name: kubernetes-dashboard
141    namespace: kubernetes-dashboard
142
143---
144
145kind: Deployment
146apiVersion: apps/v1
147metadata:
148  labels:
149    k8s-app: kubernetes-dashboard
150  name: kubernetes-dashboard
151  namespace: kubernetes-dashboard
152spec:
153  replicas: 1
154  revisionHistoryLimit: 10
155  selector:
156    matchLabels:
157      k8s-app: kubernetes-dashboard
158  template:
159    metadata:
160      labels:
161        k8s-app: kubernetes-dashboard
162    spec:
163      securityContext:
164        seccompProfile:
165          type: RuntimeDefault
166      containers:
167        - name: kubernetes-dashboard
168          image: kubernetesui/dashboard:v2.5.0
169          imagePullPolicy: Always
170          ports:
171            - containerPort: 8443
172              protocol: TCP
173          args:
174            - --auto-generate-certificates
175            - --namespace=kubernetes-dashboard
176            # Uncomment the following line to manually specify Kubernetes API server Host
177            # If not specified, Dashboard will attempt to auto discover the API server and connect
178            # to it. Uncomment only if the default does not work.
179            # - --apiserver-host=http://my-address:port
180          volumeMounts:
181            - name: kubernetes-dashboard-certs
182              mountPath: /certs
183              # Create on-disk volume to store exec logs
184            - mountPath: /tmp
185              name: tmp-volume
186          livenessProbe:
187            httpGet:
188              scheme: HTTPS
189              path: /
190              port: 8443
191            initialDelaySeconds: 30
192            timeoutSeconds: 30
193          securityContext:
194            allowPrivilegeEscalation: false
195            readOnlyRootFilesystem: true
196            runAsUser: 1001
197            runAsGroup: 2001
198      volumes:
199        - name: kubernetes-dashboard-certs
200          secret:
201            secretName: kubernetes-dashboard-certs
202        - name: tmp-volume
203          emptyDir: {}
204      serviceAccountName: kubernetes-dashboard
205      nodeSelector:
206        "kubernetes.io/os": linux
207      # Comment the following tolerations if Dashboard must not be deployed on master
208      tolerations:
209        - key: node-role.kubernetes.io/master
210          effect: NoSchedule
211
212---
213
214kind: Service
215apiVersion: v1
216metadata:
217  labels:
218    k8s-app: dashboard-metrics-scraper
219  name: dashboard-metrics-scraper
220  namespace: kubernetes-dashboard
221spec:
222  ports:
223    - port: 8000
224      targetPort: 8000
225  selector:
226    k8s-app: dashboard-metrics-scraper
227
228---
229
230kind: Deployment
231apiVersion: apps/v1
232metadata:
233  labels:
234    k8s-app: dashboard-metrics-scraper
235  name: dashboard-metrics-scraper
236  namespace: kubernetes-dashboard
237spec:
238  replicas: 1
239  revisionHistoryLimit: 10
240  selector:
241    matchLabels:
242      k8s-app: dashboard-metrics-scraper
243  template:
244    metadata:
245      labels:
246        k8s-app: dashboard-metrics-scraper
247    spec:
248      securityContext:
249        seccompProfile:
250          type: RuntimeDefault
251      containers:
252        - name: dashboard-metrics-scraper
253          image: kubernetesui/metrics-scraper:v1.0.7
254          ports:
255            - containerPort: 8000
256              protocol: TCP
257          livenessProbe:
258            httpGet:
259              scheme: HTTP
260              path: /
261              port: 8000
262            initialDelaySeconds: 30
263            timeoutSeconds: 30
264          volumeMounts:
265          - mountPath: /tmp
266            name: tmp-volume
267          securityContext:
268            allowPrivilegeEscalation: false
269            readOnlyRootFilesystem: true
270            runAsUser: 1001
271            runAsGroup: 2001
272      serviceAccountName: kubernetes-dashboard
273      nodeSelector:
274        "kubernetes.io/os": linux
275      # Comment the following tolerations if Dashboard must not be deployed on master
276      tolerations:
277        - key: node-role.kubernetes.io/master
278          effect: NoSchedule
279      volumes:
280        - name: tmp-volume
281          emptyDir: {}