kubespray离线安装k8s集群

来源: DevOpSec公众号 作者: DevOpSec

背景

国内安装k8s集群，下载镜像往往失败或者半天下载不下来，导致安装不成功，影响摸鱼心情。 是否可以通过kubespray离线部署k8s集群。

完全可以的，准备就绪后，只需10多分钟即可搭建出一套高可用的k8s集群，心情畅悦。

准备工作

1. 首先需要一台科学上网机器，用于下载k8s相关镜像和安装包
2. 搭建nginx下载站，用于存放k8s集群安装所需的安装包。
3. 搭建harbor镜像服务，存储安装k8s集群所需的镜像。
4. 需要一台独立<font color="red">管理机器</font>，安装docker环境用于部署k8s集群。

开始实施

资源

在科学上网机器上做如下操作

1. 下载kubespray包，选最新最稳定版

1a. 检出kubespray代码
2git clone git@github.com:kubernetes-sigs/kubespray.git
3
4b. 此时对外release版本v.2.21.0
5cd kubespray
6git clone v.2.21.0

2. 离线下载需要的安装包

1a. 生成下载包和镜像列表
2yum install ansible -y #安装ansible
3cd contrib/offline
4sh generate_list.sh
5
6b. 查看目录结构
7tree temp/
8temp/
9├── files.list          #依赖包的列表
10├── files.list.template
11├── images.list         #依赖镜像的列表
12└── images.list.template
13
14c. 下载软件包到目录 temp/files
15wget -x -P temp/files -i temp/files.list
16

3. 配置nginx代理，代理可以不在科学上网机器上

1a. nginx配置块
2server {
3        listen 80 yourdownload.domian.com;
4        location /k8s/ {
5                alias /path/kubespray/contrib/offline/temp/files/;
6        }
7}
8
9reload nginx 进程
10
11b. 测试，注意版本，如果能正常下载则代表代理搭建正常
12wget http://yourdomain.domian.com/k8s/etcd-io/etcd/releases/download/v3.5.6/etcd-v3.5.6-linux-amd64.tar.gz 
13

4. 下载依赖镜像

1a. 安装并启动docker-ce 
2yum install docker-ce
3service docker start
4
5b. 安装 skopeo 将依赖镜像同步到自建私有镜像仓库
6yum install  skopeo   ## 注意centos操作系统版本要在7.x
7
8c. 创建docker公共的项目k8s，harbor域名：youharbor.domain.com
9如果需要登录则执行
10docker login youharbor.domain.com
11
12d. 同步镜像
13for image in $(cat temp/images.list); do skopeo copy docker://${image} docker://youharbor.domain.com/k8s/${image#*/}; done
14

5. 下载kubespray镜像，省去安装kubespray各种依赖

1下载镜像
2docker pull quay.io/kubespray/kubespray:v2.21.0 
3
4镜像打tag
5docker tag quay.io/kubespray/kubespray:v2.21.0 youharbor.domain.com/k8s/quay.io/kubespray/kubespray:v2.21.0 
6
7推送到自己的harbor仓库
8docker push youharbor.domain.com/k8s/quay.io/kubespray/kubespray:v2.21.0 

在<font color="red">管理机器</font>上操作

1. 安装docker，下载镜像

1a. 安装docker-ce并启动
2b. 下载kubespray image
3docker pull youharbor.domain.com/k8s/quay.io/kubespray/kubespray:v2.21.0

2. 免密登录k8s集群机器

11. 生产公私钥对
2ssh-keygen  -b 4096
3遇到输密码直接回车
4d. 公钥copy到master和node机器
5ssh-copy-id master1
6ssh-copy-id master2
7ssh-copy-id master3
8ssh-copy-id node1

3. 下载kubespray包选最新最稳定版，或者从科学上网的机器上copy

1
2a. 检出kubespray代码
3git clone git@github.com:kubernetes-sigs/kubespray.git
4
5b. 此时对外release版本v.2.21.0
6cd kubespray
7git clone v.2.21.0
8
9或者从`科学上网的机器`上copy
10
11cp -rp 192.168.2.111:/path/kubespray .

4. 配置离线安装

1cp -rp /path/kubespray/inventory/sample/  /path/kubespray/inventory/mycluster/
2vim /path/kubespray/inventory/mycluster/group_vars/all/offline.yml
3
4
5# 替换镜像地址
6registry_host: "yourharbor.domian.com/k8s"
7kube_image_repo: "{{ registry_host }}"
8gcr_image_repo: "{{ registry_host }}"
9github_image_repo: "{{ registry_host }}"
10docker_image_repo: "{{ registry_host }}"
11quay_image_repo: "{{ registry_host }}"
12
13# 替换软件包地址
14files_repo: "http://yourdownload.domian.com/k8s"
15kubeadm_download_url: "{{ files_repo }}/storage.googleapis.com/kubernetes-release/release/{{ kube_version }}/bin/linux/{{ image_arch }}/kubeadm"
16kubectl_download_url: "{{ files_repo }}/storage.googleapis.com/kubernetes-release/release/{{ kube_version }}/bin/linux/{{ image_arch }}/kubectl"
17kubelet_download_url: "{{ files_repo }}/storage.googleapis.com/kubernetes-release/release/{{ kube_version }}/bin/linux/{{ image_arch }}/kubelet"
18cni_download_url: "{{ files_repo }}/github.com/containernetworking/plugins/releases/download/{{ cni_version }}/cni-plugins-linux-{{ image_arch }}-{{ cni_version }}.tgz"
19crictl_download_url: "{{ files_repo }}/github.com/kubernetes-sigs/cri-tools/releases/download/{{ crictl_version }}/crictl-{{ crictl_version }}-{{ ansible_system | lower }}-{{ image_arch }}.tar.gz"
20etcd_download_url: "{{ files_repo }}/github.com/etcd-io/etcd/releases/download/{{ etcd_version }}/etcd-{{ etcd_version }}-linux-{{ image_arch }}.tar.gz"
21calicoctl_download_url: "{{ files_repo }}/github.com/projectcalico/calico/releases/download/{{ calico_ctl_version }}/calicoctl-linux-{{ image_arch }}"
22calico_crds_download_url: "{{ files_repo }}/github.com/projectcalico/calico/archive/{{ calico_version }}.tar.gz"
23flannel_cni_download_url: "{{ files_repo }}/github.com/flannel-io/cni-plugin/releases/download/{{ flannel_cni_version }}/flannel-{{ image_arch }}"
24helm_download_url: "{{ files_repo }}/get.helm.sh/helm-{{ helm_version }}-linux-{{ image_arch }}.tar.gz"
25crun_download_url: "{{ files_repo }}/github.com/containers/crun/releases/download/{{ crun_version }}/crun-{{ crun_version }}-linux-{{ image_arch }}"
26kata_containers_download_url: "{{ files_repo }}/github.com/kata-containers/kata-containers/releases/download/{{ kata_containers_version }}/kata-static-{{ kata_containers_version }}-{{ ansible_architecture }}.tar.xz"
27runc_download_url: "{{ files_repo }}/github.com/opencontainers/runc/releases/download/{{ runc_version }}/runc.{{ image_arch }}"
28containerd_download_url: "{{ files_repo }}/github.com/containerd/containerd/releases/download/v{{ containerd_version }}/containerd-{{ containerd_version }}-linux-{{ image_arch }}.tar.gz"
29nerdctl_download_url: "{{ files_repo }}/github.com/containerd/nerdctl/releases/download/v{{ nerdctl_version }}/nerdctl-{{ nerdctl_version }}-{{ ansible_system | lower }}-{{ image_arch }}.tar.gz"
30krew_download_url: "{{ files_repo }}/github.com/kubernetes-sigs/krew/releases/download/{{ krew_version }}/krew-{{ host_os }}_{{ image_arch }}.tar.gz"
31cri_dockerd_download_url: "{{ files_repo }}/github.com/Mirantis/cri-dockerd/releases/download/{{ cri_dockerd_version }}/cri-dockerd-{{ cri_dockerd_version }}-linux-{{ image_arch }}.tar.gz"
32gvisor_runsc_download_url: "{{ files_repo }}/storage.googleapis.com/gvisor/releases/release/{{ gvisor_version }}/{{ ansible_architecture }}/runsc"
33gvisor_containerd_shim_runsc_download_url: "{{ files_repo }}/storage.googleapis.com/gvisor/releases/release/{{ gvisor_version }}/{{ ansible_architecture }}/containerd-shim-runsc-v1"
34youki_download_url: "{{ files_repo }}/github.com/containers/youki/releases/download/v{{ youki_version }}/youki_v{{ youki_version | regex_replace('\\.', '_') }}_linux.tar.gz"
35

5. 配置机器资源

1vim  /path/kubespray/inventory/mycluster/inventory.ini
2
3[all]
4master1 ansible_host=192.168.1.11
5master2 ansible_host=192.168.1.12
6master3 ansible_host=192.168.1.13
7node1 ansible_host=192.168.1.14
8
9[kube_control_plane]
10master1
11master2
12master3
13
14[etcd]
15master1
16master2
17master3
18
19[kube_node]
20node1
21
22[calico_rr]
23
24[k8s_cluster:children]
25kube_control_plane
26kube_node
27calico_rr

6. 配置安装集群信息

1vim  /path/kubespray/inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
2
3a. 配置pod网段
4kube_service_addresses: 10.233.0.0/18  ## 有1.6万多个ip，更具自己环境配置
5
6b. 配置pod网段
7kube_service_addresses: 10.233.0.0/18  ## 有1.6万多个ip，更具自己环境配置
8
9c. 选择网络插件，默认calico
10kube_network_plugin: calico
11
12d. 容器运行时
13container_manager: containerd  #推荐containerd
14
15e. 是否开启kata容器，建议开启
16kata_containers_enabled: true
17
18f. 证书是否自动续期，建议开启
19auto_renew_certificates: true
20
21也可以通过手动续期，在所有的master节点执行 
22sh /usr/local/bin/k8s-certs-renew.sh
23

以上工作准备完之后，就开始我们的集群安装之旅，内网环境大概10多分钟就能搞定集群安装。

kubespray运维k8s集群，在<font color="red">管理机器</font>上操作

0. 启动kubespray容器 kubespray容器启动脚本如下

1cat runc.sh
2
3# 把我们刚刚准备的配置挂载到容器里
4docker run --rm -it --mount type=bind,source=/path/kubespray/inventory/mycluster,dst=/inventory \
5  --mount type=bind,source=/path/kubespray,dst=/apps/kubespray \
6  --mount type=bind,source="${HOME}"/.ssh/id_rsa,dst=/root/.ssh/id_rsa \
7  yourharbor.domian.com/k8s/quay.io/kubespray/kubespray:v2.21.0 bash

1. k8s集群安装

1a. 启动kubespray容器
2sh runc.sh
3
4b. 在容器里执行命令安装集群
5# ANSIBLE_HOST_KEY_CHECKING=False 作用是首次访问集群机器不用输入yes验证
6# 注意集群节点都能访问yourharbor.domian.com和yourdownload.domian.com域名
7ANSIBLE_HOST_KEY_CHECKING=False  ansible-playbook -i /inventory/inventory.ini --private-key /root/.ssh/id_rsa cluster.yml

如果安装过程中有报jmespath依赖包的错误，见下文问题部分

2. k8s集群新增节点

1a. 修改/path/kubespray/inventory/mycluster/inventory.ini
2[kube_node]
3node1
4node2 #新增节点
5
6b. 启动容器
7sh runc.sh
8
9b. 执行扩容
10ANSIBLE_HOST_KEY_CHECKING=False  ansible-playbook -i /inventory/inventory.ini --private-key /root/.ssh/id\_rsa scale.yml -b -v

3. k8s集群删除节点

1a. 修改/path/kubespray/inventory/mycluster/inventory.ini
2[kube_node]
3node1
4node2 #删除节点
5node3 #删除节点
6
7b. 启动容器
8sh runc.sh
9
10b. 执行删除
11ANSIBLE_HOST_KEY_CHECKING=False  ansible-playbook -i /inventory/inventory.ini --private-key /root/.ssh/id_rsa remove-node.yml -b -v --extra-vars "node=node2,node3"

4. k8s集群升级

1a. 下载文件和对应的镜像，具体操作见上文
2
3b. 启动容器
4sh runc.sh
5
6b. 执行升级
7ANSIBLE_HOST_KEY_CHECKING=False  ansible-playbook -b -i /inventory/inventory.ini --private-key /root/.ssh/id_rsa upgrade-cluster.yml  -e kube\_version=v1.26.3

5. 卸载k8s集群

1a. 启动容器
2sh runc.sh
3
4b. ANSIBLE_HOST_KEY_CHECKING=False  ansible-playbook -i /inventory/inventory.ini --private-key /root/.ssh/id\_rsa reset.yml

问题

1. jmespath 依赖包没有

1如果安装过程中有报jmespath依赖包的错误，重新打镜像
2cat Dockerfile
3FROM youharbor.domain.com/k8s/quay.io/kubespray/kubespray:v2.21.0 
4RUN pip3 install jmespath -i http://pypi.douban.com/simple/ --trusted-host=pypi.douban.com/simple
5
6docker build youharbor.domain.com/k8s/quay.io/kubespray/kubespray:v2.21.0 .
7docker push youharbor.domain.com/k8s/quay.io/kubespray/kubespray:v2.21.0 

2. node节点上kubelet报错

1error:
2Apr 27 13:06:37 plat-data-mysql-sh-001 kubelet[26809]: E0427 13:06:37.782074   26809 summary_sys_containers.go:48] "Failed to get system container stats" err="failed to get cgroup stats for "/kube.slice/containerd.service": failed to get container info for "/kube.slice/containerd.service": unknown container "/kube.slice/containerd.service"" containerName="/kube.slice/containerd.service"
3
4解决：
5sed -i 's#/kube.slice/containerd.service#/system.slice/kubelet.service#g' /etc/kubernetes/kubelet.env ; service kubelet restart