3、交付Dubbo微服务到kubernetes集群

1.基础架构

1.1.架构图

Zookeeper是Dubbo微服务集群的注册中心
它的高可用机制和k8s的etcd集群一致
java编写，需要jdk环境

1.2.节点规划

主机名

角色

hdss7-11.host.com

k8s代理节点1，zk1

10.4.7.11

hdss7-12.host.com

k8s代理节点2，zk2

10.4.7.12

hdss7-21.host.com

k8s运算节点1，zk3

10.4.7.21

hdss7-22.host.com

k8s运算节点2，jenkins

10.4.7.22

hdss7-200.host.com

k8s运维节点（docker仓库）

10.4.7.200

2.部署zookeeper

2.1.安装jdk 1.8（3台zk节点都要安装）

1//解压、创建软链接
2[root@hdss7-11 src]# mkdir /usr/java
3[root@hdss7-11 src]# tar xf jdk-8u221-linux-x64.tar.gz  -C /usr/java/
4[root@hdss7-11 src]# ln -s /usr/java/jdk1.8.0_221/ /usr/java/jdk
5[root@hdss7-11 src]# cd /usr/java/
6[root@hdss7-11 java]# ll
7total 0
8lrwxrwxrwx 1 root root  23 Nov 30 17:38 jdk -> /usr/java/jdk1.8.0_221/
9drwxr-xr-x 7   10  143 245 Jul  4 19:37 jdk1.8.0_221
10
11//创建环境变量
12[root@hdss7-11 java]# vi /etc/profile
13export JAVA_HOME=/usr/java/jdk
14export PATH=$JAVA_HOME/bin:$JAVA_HOME/bin:$PATH
15export CLASSPATH=$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/lib/tools.jar
16
17//source并检查
18[root@hdss7-11 java]# source /etc/profile
19[root@hdss7-11 java]# java -version
20java version "1.8.0_221"
21Java(TM) SE Runtime Environment (build 1.8.0_221-b11)
22Java HotSpot(TM) 64-Bit Server VM (build 25.221-b11, mixed mode)

2.2.安装zk（3台节点都要安装）

zookeeper官方地址

2.2.1.解压，创建软链接

1[root@hdss7-11 src]# tar xf zookeeper-3.4.14.tar.gz  -C /opt/
2[root@hdss7-11 src]# ln -s /opt/zookeeper-3.4.14/ /opt/zookeeper

1[root@hdss7-11 opt]# mkdir  -pv /data/zookeeper/data /data/zookeeper/logs
2mkdir: created directory ‘/data’
3mkdir: created directory ‘/data/zookeeper’
4mkdir: created directory ‘/data/zookeeper/data’
5mkdir: created directory ‘/data/zookeeper/logs’

2.2.3.配置

1//各节点相同
2[root@hdss7-11 opt]# vi /opt/zookeeper/conf/zoo.cfg
3tickTime=2000
4initLimit=10
5syncLimit=5
6dataDir=/data/zookeeper/data
7dataLogDir=/data/zookeeper/logs
8clientPort=2181
9server.1=zk1.od.com:2888:3888
10server.2=zk2.od.com:2888:3888
11server.3=zk3.od.com:2888:3888

myid

1//各节点不同
2[root@hdss7-11 opt]# vi /data/zookeeper/data/myidvi
31
4[root@hdss7-12 opt]# vi /data/zookeeper/data/myid
52
6[root@hdss7-21 opt]# vi /data/zookeeper/data/myid
73

2.2.4.做dns解析

1[root@hdss7-11 opt]# vi /var/named/od.com.zone
2
3$ORIGIN od.com.
4$TTL 600        ; 10 minutes
5@               IN SOA  dns.od.com. dnsadmin.od.com. (
6                                2019111006 ; serial                        //序列号前滚1
7                                10800      ; refresh (3 hours)
8                                900        ; retry (15 minutes)
9                                604800     ; expire (1 week)
10                                86400      ; minimum (1 day)
11                                )
12                                NS   dns.od.com.
13$TTL 60 ; 1 minute
14dns                A    10.4.7.11
15harbor             A    10.4.7.200
16k8s-yaml           A    10.4.7.200
17traefik            A    10.4.7.10
18dashboard          A    10.4.7.10
19zk1                A    10.4.7.11
20zk2                A    10.4.7.12
21zk3                A    10.4.7.21
22
23[root@hdss7-11 opt]# systemctl restart named
24[root@hdss7-11 opt]# dig -t A zk1.od.com @10.4.7.11 +short
2510.4.7.11

2.2.4.依次启动并检查

启动

1[root@hdss7-11 opt]# /opt/zookeeper/bin/zkServer.sh start
2ZooKeeper JMX enabled by default
3Using config: /opt/zookeeper/bin/../conf/zoo.cfg
4Starting zookeeper ... STARTED
5[root@hdss7-12 opt]# /opt/zookeeper/bin/zkServer.sh start
6
7[root@hdss7-21 opt]# /opt/zookeeper/bin/zkServer.sh start
8

检查

1[root@hdss7-11 opt]# netstat -ntlup|grep 2181
2tcp6       0      0 :::2181                 :::*                    LISTEN      69157/java   
3
4[root@hdss7-11 opt]# zookeeper/bin/zkServer.sh  status  
5ZooKeeper JMX enabled by default
6Using config: /opt/zookeeper/bin/../conf/zoo.cfg
7Mode: follower
8
9[root@hdss7-12 opt]# zookeeper/bin/zkServer.sh  status
10ZooKeeper JMX enabled by default
11Using config: /opt/zookeeper/bin/../conf/zoo.cfg
12Mode: leader

3.部署jenkins

jenkins官网 jenkins 镜像

3.1.准备镜像

hdss7-200上

1[root@hdss7-200 ~]# docker pull jenkins/jenkins:2.190.3
2[root@hdss7-200 ~]# docker images |grep jenkins
3[root@hdss7-200 ~]# docker tag 22b8b9a84dbe harbor.od.com/public/jenkins:v2.190.3
4[root@hdss7-200 ~]# docker push  harbor.od.com/public/jenkins:v2.190.3

3.2.制作自定义镜像

3.2.1.生成ssh秘钥对

[root@hdss7-200 ~]# ssh-keygen -t rsa -b 2048 -C "8614610@qq.com" -N "" -f /root/.ssh/id_rsa

此处用自己的邮箱

3.2.2.准备get-docker.sh文件

1[root@hdss7-200 ~]#  curl -fsSL get.docker.com -o get-docker.sh
2[root@hdss7-200 ~]# chmod +x  get-docker.sh

3.2.3.准备config.json文件

1cp /root/.docker/config.json  .
2cat  /root/.docker/config.json
3{
4	"auths": {
5		"harbor.od.com": {
6			"auth": "YWRtaW46SGFyYm9yMTIzNDU="
7		}
8	},
9	"HttpHeaders": {
10		"User-Agent": "Docker-Client/19.03.4 (linux)"
11	}

3.2.4.创建目录并准备Dockerfile

1[root@hdss7-200 ~]# mkdir /data/dockerfile/jenkins -p
2[root@hdss7-200 ~]# cd /data/dockerfile/jenkins/
3
4[root@hdss7-200 jenkins]# vi Dockerfile
5FROM harbor.od.com/public/jenkins:v2.190.3
6USER root
7RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\ 
8    echo 'Asia/Shanghai' >/etc/timezone
9ADD id_rsa /root/.ssh/id_rsa
10ADD config.json /root/.docker/config.json
11ADD get-docker.sh /get-docker.sh
12RUN echo "    StrictHostKeyChecking no" >> /etc/ssh/ssh_config &&\
13    /get-docker.sh

设置容器用户为root
设置容器内的时区
将创建的ssh私钥加入（使用git拉代码是要用，配对的公钥配置在gitlab中）
加入了登陆自建harbor仓库的config文件
修改了ssh客户端的配置，不做指纹验证
安装一个docker的客户端 //build如果失败，在get-docker.sh 后加--mirror=Aliyun

3.3.制作自定义镜像

1//准备所需文件，拷贝至/data/dockerfile/jenkins
2[root@hdss7-200 jenkins]# pwd
3/data/dockerfile/jenkins
4[root@hdss7-200 jenkins]# ll
5total 32
6-rw------- 1 root root   151 Nov 30 18:35 config.json
7-rw-r--r-- 1 root root   349 Nov 30 18:31 Dockerfile
8-rwxr-xr-x 1 root root 13216 Nov 30 18:31 get-docker.sh
9-rw------- 1 root root  1675 Nov 30 18:35 id_rsa
10
11//执行build
12 docker build . -t harbor.od.com/infra/jenkins:v2.190.3 
13
14//公钥上传到gitee测试此镜像是否可以成功连接
15[root@hdss7-200 harbor]# docker run --rm harbor.od.com/infra/jenkins:v2.190.3 ssh -i /root/.ssh/id_rsa -T git@gitee.com
16Warning: Permanently added 'gitee.com,212.64.62.174' (ECDSA) to the list of known hosts.
17Hi StanleyWang (DeployKey)! You've successfully authenticated, but GITEE.COM does not provide shell access.
18Note: Perhaps the current use is DeployKey.
19Note: DeployKey only supports pull/fetch operations
20

3.4.创建infra仓库

3.5.创建kubernetes名称空间并在此创建secret

1[root@hdss7-21 ~]# kubectl create namespace infra
2[root@hdss7-21 ~]# kubectl create secret docker-registry harbor --docker-server=harbor.od.com --docker-username=admin --docker-password=Harbor12345 -n infra

3.6.推送镜像

[root@hdss7-200 jenkins]# docker push harbor.od.com/infra/jenkins:v2.190.3

3.7.准备共享存储

运维主机hdss7-200和所有运算节点上，这里指hdss7-21、22

3.7.1.安装nfs-utils -y

[root@hdss7-200 jenkins]# yum install nfs-utils -y

3.7.2.配置NFS服务

运维主机hdss7-200上

1[root@hdss7-200 jenkins]# vi /etc/exports
2/data/nfs-volume 10.4.7.0/24(rw,no_root_squash)

3.7.3.启动NFS服务

运维主机hdss7-200上

1[root@hdss7-200 ~]# mkdir -p /data/nfs-volume
2[root@hdss7-200 ~]# systemctl start nfs
3[root@hdss7-200 ~]# systemctl enable nfs

3.8.准备资源配置清单

运维主机hdss7-200上

1[root@hdss7-200 ~]# cd /data/k8s-yaml/
2[root@hdss7-200 k8s-yaml]# mkdir /data/k8s-yaml/jenkins && mkdir /data/nfs-volume/jenkins_home && cd  jenkins

dp.yaml

1[root@hdss7-200 jenkins]# vi dp.yaml
2kind: Deployment
3apiVersion: extensions/v1beta1
4metadata:
5  name: jenkins
6  namespace: infra
7  labels: 
8    name: jenkins
9spec:
10  replicas: 1
11  selector:
12    matchLabels: 
13      name: jenkins
14  template:
15    metadata:
16      labels: 
17        app: jenkins 
18        name: jenkins
19    spec:
20      volumes:
21      - name: data
22        nfs: 
23          server: hdss7-200
24          path: /data/nfs-volume/jenkins_home
25      - name: docker
26        hostPath: 
27          path: /run/docker.sock
28          type: ''
29      containers:
30      - name: jenkins
31        image: harbor.od.com/infra/jenkins:v2.190.3
32        imagePullPolicy: IfNotPresent
33        ports:
34        - containerPort: 8080
35          protocol: TCP
36        env:
37        - name: JAVA_OPTS
38          value: -Xmx512m -Xms512m
39        volumeMounts:
40        - name: data
41          mountPath: /var/jenkins_home
42        - name: docker
43          mountPath: /run/docker.sock
44      imagePullSecrets:
45      - name: harbor
46      securityContext: 
47        runAsUser: 0
48  strategy:
49    type: RollingUpdate
50    rollingUpdate: 
51      maxUnavailable: 1
52      maxSurge: 1
53  revisionHistoryLimit: 7
54  progressDeadlineSeconds: 600

svc.yaml

1[root@hdss7-200 jenkins]# vi dp.yaml
2kind: Service
3apiVersion: v1
4metadata: 
5  name: jenkins
6  namespace: infra
7spec:
8  ports:
9  - protocol: TCP
10    port: 80
11    targetPort: 8080
12  selector:
13    app: jenkins

ingress.yaml

1kind: Ingress
2apiVersion: extensions/v1beta1
3metadata: 
4  name: jenkins
5  namespace: infra
6spec:
7  rules:
8  - host: jenkins.od.com
9    http:
10      paths:
11      - path: /
12        backend: 
13          serviceName: jenkins
14          servicePort: 80

3.9.应用资源配置清单

任意运算节点上

1[root@hdss7-21 etcd]# kubectl apply -f http://k8s-yaml.od.com/jenkins/dp.yaml
2[root@hdss7-21 etcd]# kubectl apply -f http://k8s-yaml.od.com/jenkins/svc.yaml
3[root@hdss7-21 etcd]# kubectl apply -f http://k8s-yaml.od.com/jenkins/ingress.yaml
4
5[root@hdss7-21 etcd]# kubectl get pods -n infra
6NAME                       READY   STATUS    RESTARTS   AGE
7jenkins-54b8469cf9-v46cc   1/1     Running   0          168m
8[root@hdss7-21 etcd]# kubectl get all  -n infra
9NAME                           READY   STATUS    RESTARTS   AGE
10pod/jenkins-54b8469cf9-v46cc   1/1     Running   0          169m
11
12
13NAME              TYPE        CLUSTER-IP        EXTERNAL-IP   PORT(S)   AGE
14service/jenkins   ClusterIP   192.168.183.210   <none>        80/TCP    2d21h
15
16
17NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
18deployment.apps/jenkins   1/1     1            1           2d21h
19
20NAME                                 DESIRED   CURRENT   READY   AGE
21replicaset.apps/jenkins-54b8469cf9   1         1         1       2d18h
22replicaset.apps/jenkins-6b6d76f456   0         0         0       2d21h

3.10.解析域名

hdss7-11上

1[root@hdss7-11 ~]# vi /var/named/od.com.zone
2$ORIGIN od.com.
3$TTL 600        ; 10 minutes
4@               IN SOA  dns.od.com. dnsadmin.od.com. (
5                                2019111007 ; serial
6                                10800      ; refresh (3 hours)
7                                900        ; retry (15 minutes)
8                                604800     ; expire (1 week)
9                                86400      ; minimum (1 day)
10                                )
11                                NS   dns.od.com.
12$TTL 60 ; 1 minute
13dns                A    10.4.7.11
14...
15...
16jenkins            A    10.4.7.10
17
18[root@hdss7-11 ~]# systemctl restart named
19[root@hdss7-11 ~]# dig -t A jenkins.od.com @10.4.7.11 +short
2010.4.7.10

3.11. 浏览器访问

访问：http://jenkins.od.com 需要输入初始密码：

初始密码查看（也可在log里查看）：

[root@hdss7-200 jenkins]# cat /data/nfs-volume/jenkins_home/secrets/initialAdminPassword

3.12.页面配置jenkins

3.12.1.配置用户名密码

用户名：admin 密码：admin123 //后续依赖此密码，请务必设置此密码

3.12.2.设置configure global security

允许匿名用户访问

阻止跨域请求，勾去掉

3.12.3.安装好流水线插件Blue-Ocean

注意安装插件慢的话可以设置清华大学加速 hdss-200上

1cd /data/nfs-volume/jenkins_home/updates
2sed -i 's/http:\/\/updates.jenkins-ci.org\/download/https:\/\/mirrors.tuna.tsinghua.edu.cn\/jenkins/g' default.json && sed -i 's/http:\/\/www.google.com/https:\/\/www.baidu.com/g' default.json

4.最后的准备工作

4.1.检查jenkins容器里的docker客户端

验证当前用户，时区，sock文件是否可用

验证kubernetes名称空间创建的secret是否可登陆到harbor仓库

4.2.检查jenkins容器里的SSH key

验证私钥，是否能登陆到gitee拉代码

4.3.部署maven软件

编译java，早些年用javac-->ant -->maven-->Gragle 在运维主机hdss7-200上二进制部署，这里部署maven-3.6.2版本 mvn命令是一个脚本，如果用jdk7，可以在脚本里修改

4.3.1.下载安装包

maven官方下载地址

4.3.2.创建目录并解压

目录8u232是根据docker容器里的jenkins的jdk版本命名，请严格按照此命名

1[root@hdss7-200 src]# mkdir /data/nfs-volume/jenkins_home/maven-3.6.1-8u232
2[root@hdss7-200 src]# tar xf apache-maven-3.6.1-bin.tar.gz  -C /data/nfs-volume/jenkins_home/maven-3.6.1-8u232
3[root@hdss7-200 src]# cd /data/nfs-volume/jenkins_home/maven-3.6.1-8u232
4[root@hdss7-200 maven-3.6.1-8u232]# ls
5apache-maven-3.6.1
6[root@hdss7-200 maven-3.6.1-8u232]# mv apache-maven-3.6.1/ ../ && mv ../apache-maven-3.6.1/* .
7[root@hdss7-200 maven-3.6.1-8u232]# ll
8total 28
9drwxr-xr-x 2 root root     97 Dec  3 19:04 bin
10drwxr-xr-x 2 root root     42 Dec  3 19:04 boot
11drwxr-xr-x 3  501 games    63 Apr  5  2019 conf
12drwxr-xr-x 4  501 games  4096 Dec  3 19:04 lib
13-rw-r--r-- 1  501 games 13437 Apr  5  2019 LICENSE
14-rw-r--r-- 1  501 games   182 Apr  5  2019 NOTICE
15-rw-r--r-- 1  501 games  2533 Apr  5  2019 README.txt

4.3.3.设置settings.xml国内镜像源

1搜索替换：
2[root@hdss7-200 maven-3.6.1-8u232]# vi /data/nfs-volume/jenkins_home/maven-3.6.1-8u232/conf/settings.xml
3<mirror>
4  <id>nexus-aliyun</id>
5  <mirrorOf>*</mirrorOf>
6  <name>Nexus aliyun</name>
7  <url>http://maven.aliyun.com/nexus/content/groups/public</url>
8</mirror>

4.3.制作dubbo微服务的底包镜像

运维主机hdss7-200上

4.3.1.自定义Dockerfile

1root@hdss7-200 dockerfile]# mkdir jre8
2[root@hdss7-200 dockerfile]# cd jre8/
3[root@hdss7-200 jre8]# pwd
4/data/dockerfile/jre8
5
6[root@hdss7-200 jre8]# vi Dockfile
7FROM harbor.od.com/public/jre:8u112
8RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\
9    echo 'Asia/Shanghai' >/etc/timezone
10ADD config.yml /opt/prom/config.yml
11ADD jmx_javaagent-0.3.1.jar /opt/prom/
12WORKDIR /opt/project_dir
13ADD entrypoint.sh /entrypoint.sh
14CMD ["/entrypoint.sh"]

普罗米修斯的监控匹配规则
java agent 收集jvm的信息，采集jvm的jar包
docker运行的默认启动脚本entrypoint.sh

4.3.2.准备jre底包（7版本有一个7u80）

1[root@hdss7-200 jre8]# docker pull docker.io/stanleyws/jre8:8u112
2[root@hdss7-200 jre8]# docker images |grep jre
3stanleyws/jre8                     8u112                      fa3a085d6ef1        2 years ago         363MB
4[root@hdss7-200 jre8]# docker tag fa3a085d6ef1 harbor.od.com/public/jre:8u112
5[root@hdss7-200 jre8]# docker push  harbor.od.com/public/jre:8u112

4.3.3.准备java-agent的jar包

[root@hdss7-200 jre8]# wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.3.1/jmx_prometheus_javaagent-0.3.1.jar -O jmx_javaagent-0.3.1.jar

4.3.3.准备config.yml和entrypoint.sh

1[root@hdss7-200 jre8]# vi config.yml
2---
3rules:
4  - pattern: '.*'
5
6[root@hdss7-200 jre8]# vi entrypoint.sh
7#!/bin/sh
8M_OPTS="-Duser.timezone=Asia/Shanghai -javaagent:/opt/prom/jmx_javaagent-0.3.1.jar=$(hostname -i):${M_PORT:-"12346"}:/opt/prom/config.yml"
9C_OPTS=${C_OPTS}
10JAR_BALL=${JAR_BALL}
11exec java -jar ${M_OPTS} ${C_OPTS} ${JAR_BALL}
12
13[root@hdss7-200 jre8]# chmod +x entrypoint.sh 
14

4.3.4.harbor创建base项目

4.3.5.构建dubbo微服务的底包并推到harbor仓库

1[root@hdss7-200 jre8]# docker build . -t harbor.od.com/base/jre8:8u112
2
3[root@hdss7-200 jre8]# docker push  harbor.od.com/base/jre8:8u112

5.交付dubbo微服务至kubenetes集群

5.1.配置New job流水线

添加构建参数： //以下配置项是王导根据多年生产经验总结出来的--运维甩锅大法（姿势要帅，动作要快）

登陆jenkins----->选择NEW-ITEM----->item name ：dubbo-demo----->Pipeline------>ok

需要保留多少此老的构建，这里设置，保留三天，30个

点击：“This project is parameterized”使用参数化构建jenkins

添加参数String Parameter：8个------Trim the string都勾选 app_name

image_name

git_repo

git_ver

add_tag

mvn_dir

target_dir

mvn_cmd

添加Choice Parameter：2个 base_image

maven

5.2..Pipeline Script

1pipeline {
2  agent any 
3    stages {
4      stage('pull') { //get project code from repo 
5        steps {
6          sh "git clone ${params.git_repo} ${params.app_name}/${env.BUILD_NUMBER} && cd ${params.app_name}/${env.BUILD_NUMBER} && git checkout ${params.git_ver}"
7        }
8      }
9      stage('build') { //exec mvn cmd
10        steps {
11          sh "cd ${params.app_name}/${env.BUILD_NUMBER}  && /var/jenkins_home/maven-${params.maven}/bin/${params.mvn_cmd}"
12        }
13      }
14      stage('package') { //move jar file into project_dir
15        steps {
16          sh "cd ${params.app_name}/${env.BUILD_NUMBER} && cd ${params.target_dir} && mkdir project_dir && mv *.jar ./project_dir"
17        }
18      }
19      stage('image') { //build image and push to registry
20        steps {
21          writeFile file: "${params.app_name}/${env.BUILD_NUMBER}/Dockerfile", text: """FROM harbor.od.com/${params.base_image}
22ADD ${params.target_dir}/project_dir /opt/project_dir"""
23          sh "cd  ${params.app_name}/${env.BUILD_NUMBER} && docker build -t harbor.od.com/${params.image_name}:${params.git_ver}_${params.add_tag} . && docker push harbor.od.com/${params.image_name}:${params.git_ver}_${params.add_tag}"
24        }
25      }
26    }
27}

5.3.harbor创建app项目，把dubbo服务镜像管理起来

5.4.创建app名称空间，并添加secret资源

任意运算节点上 因为要去拉app私有仓库的镜像，所以添加secret资源

1[root@hdss7-21 bin]# kubectl create ns app
2namespace/app created
3[root@hdss7-21 bin]# kubectl create secret docker-registry harbor --docker-server=harbor.od.com --docker-username=admin --docker-password=Harbor12345 -n app
4secret/harbor created

5.5.交付dubbo-demo-service

5.5.1.jenkins传参，构建dubbo-demo-service镜像，传到harbor

5.5.2.创建dubbo-demo-service的资源配置清单

特别注意：dp.yaml的image替换成自己打包的镜像名称 hdss7-200上

1[root@hdss7-200 dubbo-demo-service]# vi dp.yaml
2kind: Deployment
3apiVersion: extensions/v1beta1
4metadata:
5  name: dubbo-demo-service
6  namespace: app
7  labels: 
8    name: dubbo-demo-service
9spec:
10  replicas: 1
11  selector:
12    matchLabels: 
13      name: dubbo-demo-service
14  template:
15    metadata:
16      labels: 
17        app: dubbo-demo-service
18        name: dubbo-demo-service
19    spec:
20      containers:
21      - name: dubbo-demo-service
22        image: harbor.od.com/app/dubbo-demo-service:master_191201_1200
23        ports:
24        - containerPort: 20880
25          protocol: TCP
26        env:
27        - name: JAR_BALL
28          value: dubbo-server.jar
29        imagePullPolicy: IfNotPresent
30      imagePullSecrets:
31      - name: harbor
32      restartPolicy: Always
33      terminationGracePeriodSeconds: 30
34      securityContext: 
35        runAsUser: 0
36      schedulerName: default-scheduler
37  strategy:
38    type: RollingUpdate
39    rollingUpdate: 
40      maxUnavailable: 1
41      maxSurge: 1
42  revisionHistoryLimit: 7
43  progressDeadlineSeconds: 600

5.5.3.应用dubbo-demo-service资源配置清单

任意运算节点上

1[root@hdss7-21 bin]# kubectl apply -f http://k8s-yaml.od.com/dubbo-demo-service/dp.yaml
2deployment.extensions/dubbo-demo-service created

5.5.4.检查启动状态

dashboard查看日志

zk注册中心查看

5.6.交付dubbo-Monitor

dubbo-monitor实际上就是从注册中心registry去数据出来然后展示的工具 两个开源软件：1、dubbo-admin 2、dubbo-monitor。此处我们用dubbo-minitor

5.6.1.准备docker镜像

5.6.1.1.下载源码包、解压

1[root@hdss7-200 src]# ll|grep dubbo
2-rw-r--r-- 1 root root  23468109 Dec  4 11:50 dubbo-monitor-master.zip
3[root@hdss7-200 src]# unzip  dubbo-monitor-master.zip 
4[root@hdss7-200 src]# ll|grep dubbo
5drwxr-xr-x 3 root root        69 Jul 27  2016 dubbo-monitor-master
6-rw-r--r-- 1 root root  23468109 Dec  4 11:50 dubbo-monitor-master.zip

5.6.1.2.修改以下项配置

1[root@hdss7-200 conf]# pwd
2/opt/src/dubbo-monitor-master/dubbo-monitor-simple/conf
3
4[root@hdss7-200 conf]# vi dubbo_origin.properties 
5dubbo.registry.address=zookeeper://zk1.od.com:2181?backup=zk2.od.com:2181,zk3.od.com:2181
6dubbo.protocol.port=20880
7dubbo.jetty.port=8080
8dubbo.jetty.directory=/dubbo-monitor-simple/monitor
9dubbo.statistics.directory=/dubbo-monitor-simple/statistics
10dubbo.charts.directory=/dubbo-monitor-simple/charts
11dubbo.log4j.file=logs/dubbo-monitor.log

5.6.1.3.制作镜像

5.6.1.3.1.准备环境

由于是虚拟机环境，这里java给的内存太大，需要给小一些，nohup 替换成exec，要在前台跑，去掉结尾&符，删除nohup 行下所有行

[root@hdss7-200 bin]# vi /opt/src/dubbo-monitor-master/dubbo-monitor-simple/bin/start.sh JAVA_MEM_OPTS=" -server -Xmx2g -Xms2g -Xmn256m -XX:PermSize=128m -Xss256k -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -XX:LargePageSizeInBytes=128m -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 " else JAVA_MEM_OPTS=" -server -Xms1g -Xmx1g -XX:PermSize=128m -XX:SurvivorRatio=2 -XX:+UseParallelGC " fi

echo -e "Starting the $SERVER_NAME ...\c" nohup java $JAVA_OPTS $JAVA_MEM_OPTS $JAVA_DEBUG_OPTS $JAVA_JMX_OPTS -classpath $CONF_DIR:$LIB_JARS com.alibaba.dubbo.container.Main > $STDOUT_FILE 2>&1 &
sed命令替换，用到了sed模式空间

sed -r -i -e '/^nohup/{p;:a;N;$!ba;d}' ./dubbo-monitor-simple/bin/start.sh && sed -r -i -e "s%^nohup(.*)%exec \1%" /opt/src/dubbo-monitor-master/dubbo-monitor-simple/bin/start.sh

//调小内存，然后nohup行结尾的&去掉！！！

为了规范，复制到data下

[root@hdss7-200 src]# mv dubbo-monitor-master dubbo-monitor [root@hdss7-200 src]# cp -a dubbo-monitor /data/dockerfile/ [root@hdss7-200 src]# cd /data/dockerfile/dubbo-monitor/

5.6.1.3.2.准备Dockerfile

1[root@hdss7-200 dubbo-monitor]# pwd
2/data/dockerfile/dubbo-monitor
3
4[root@hdss7-200 dubbo-monitor]# cat Dockerfile 
5FROM jeromefromcn/docker-alpine-java-bash
6MAINTAINER Jerome Jiang
7COPY dubbo-monitor-simple/ /dubbo-monitor-simple/
8CMD /dubbo-monitor-simple/bin/start.sh

5.6.1.3.3.build镜像并push到harbor仓库

1[root@hdss7-200 dubbo-monitor]# docker build . -t harbor.od.com/infra/dubbo-monitor:latest
2[root@hdss7-200 dubbo-monitor]# docker push harbor.od.com/infra/dubbo-monitor:latest

5.6.2.解析域名

1[root@hdss7-11 ~]# vi /var/named/od.com.zone 
2$ORIGIN od.com.
3$TTL 600        ; 10 minutes
4@               IN SOA  dns.od.com. dnsadmin.od.com. (
5                                2019111008 ; serial
6                                10800      ; refresh (3 hours)
7                                900        ; retry (15 minutes)
8                                604800     ; expire (1 week)
9                                86400      ; minimum (1 day)
10                                )
11                                NS   dns.od.com.
12  。。。略                              
13dubbo-monitor      A    10.4.7.10
14
15[root@hdss7-11 ~]# systemctl restart named
16[root@hdss7-11 ~]# dig -t A dubbo-monitor.od.com @10.4.7.11 +short
1710.4.7.10

5.6.3.准备k8s资源配置清单

dp.yaml

[root@hdss7-200 k8s-yaml]# pwd /data/k8s-yaml [root@hdss7-200 k8s-yaml]# pwd /data/k8s-yaml [root@hdss7-200 k8s-yaml]# mkdir dubbo-monitor [root@hdss7-200 k8s-yaml]# cd dubbo-monitor/ kind: Deployment apiVersion: extensions/v1beta1 metadata: name: dubbo-monitor namespace: infra labels: name: dubbo-monitor spec: replicas: 1 selector: matchLabels: name: dubbo-monitor template: metadata: labels: app: dubbo-monitor name: dubbo-monitor spec: containers: - name: dubbo-monitor image: harbor.od.com/infra/dubbo-monitor:latest ports: - containerPort: 8080 protocol: TCP - containerPort: 20880 protocol: TCP imagePullPolicy: IfNotPresent imagePullSecrets: - name: harbor restartPolicy: Always terminationGracePeriodSeconds: 30 securityContext: runAsUser: 0 schedulerName: default-scheduler strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 maxSurge: 1 revisionHistoryLimit: 7 progressDeadlineSeconds: 600
svc.yaml

kind: Service apiVersion: v1 metadata: name: dubbo-monitor namespace: infra spec: ports:
- protocol: TCP port: 8080 targetPort: 8080 selector: app: dubbo-monitor
ingress.yaml

kind: Ingress apiVersion: extensions/v1beta1 metadata: name: dubbo-monitor namespace: infra spec: rules:
- host: dubbo-monitor.od.com http: paths:
  - path: / backend: serviceName: dubbo-monitor servicePort: 8080

5.6.4.应用资源配置清单

1[root@hdss7-21 bin]# kubectl apply -f http://k8s-yaml.od.com/dubbo-monitor/dp.yaml
2deployment.extensions/dubbo-monitor created
3[root@hdss7-21 bin]# kubectl apply -f http://k8s-yaml.od.com/dubbo-monitor/svc.yaml
4service/dubbo-monitor created
5[root@hdss7-21 bin]# kubectl apply -f http://k8s-yaml.od.com/dubbo-monitor/ingress.yaml
6ingress.extensions/dubbo-monitor created

5.6.5.浏览器访问

5.7.交付dubbo-demo-consumer

5.7.1.jenkins传参，构建dubbo-demo-service镜像，传到harbor

jenkins的jar包本地缓存

5.7.2.创建dubbo-demo-consumer的资源配置清单

运维主机hdss7-200上 特别注意：dp.yaml的image替换成自己打包的镜像名称 dp.yaml

1[root@hdss7-200 k8s-yaml]# pwd
2/data/k8s-yaml
3[root@hdss7-200 k8s-yaml]# mkdir dubbo-demo-consumer
4[root@hdss7-200 k8s-yaml]# cd dubbo-demo-consumer/
5[root@hdss7-200 k8s-yaml]# vi dp.yaml
6
7kind: Deployment
8apiVersion: extensions/v1beta1
9metadata:
10  name: dubbo-demo-consumer
11  namespace: app
12  labels: 
13    name: dubbo-demo-consumer
14spec:
15  replicas: 1
16  selector:
17    matchLabels: 
18      name: dubbo-demo-consumer
19  template:
20    metadata:
21      labels: 
22        app: dubbo-demo-consumer
23        name: dubbo-demo-consumer
24    spec:
25      containers:
26      - name: dubbo-demo-consumer
27        image: harbor.od.com/app/dubbo-demo-consumer:master_191204_1307
28        ports:
29        - containerPort: 8080
30          protocol: TCP
31        - containerPort: 20880
32          protocol: TCP
33        env:
34        - name: JAR_BALL
35          value: dubbo-client.jar
36        imagePullPolicy: IfNotPresent
37      imagePullSecrets:
38      - name: harbor
39      restartPolicy: Always
40      terminationGracePeriodSeconds: 30
41      securityContext: 
42        runAsUser: 0
43      schedulerName: default-scheduler
44  strategy:
45    type: RollingUpdate
46    rollingUpdate: 
47      maxUnavailable: 1
48      maxSurge: 1
49  revisionHistoryLimit: 7
50  progressDeadlineSeconds: 600

svc.yaml

1kind: Service
2apiVersion: v1
3metadata: 
4  name: dubbo-demo-consumer
5  namespace: app
6spec:
7  ports:
8  - protocol: TCP
9    port: 8080
10    targetPort: 8080
11  selector: 
12    app: dubbo-demo-consumer

ingress.yaml

1kind: Ingress
2apiVersion: extensions/v1beta1
3metadata: 
4  name: dubbo-demo-consumer
5  namespace: app
6spec:
7  rules:
8  - host: demo.od.com
9    http:
10      paths:
11      - path: /
12        backend: 
13          serviceName: dubbo-demo-consumer
14          servicePort: 8080

5.7.3.应用dubbo-demo-consumer资源配置清单

任意运算节点上

1[root@hdss7-21 bin]# kubectl apply -f http://k8s-yaml.od.com/dubbo-demo-consumer/dp.yaml
2deployment.extensions/dubbo-demo-consumer created
3[root@hdss7-21 bin]# kubectl apply -f http://k8s-yaml.od.com/dubbo-demo-consumer/svc.yaml
4service/dubbo-demo-consumer created
5[root@hdss7-21 bin]# kubectl apply -f http://k8s-yaml.od.com/dubbo-demo-consumer/ingress.yaml
6ingress.extensions/dubbo-demo-consumer created

5.7.4.解析域名

hdss7-11上

1[root@hdss7-11 ~]# vi /var/named/od.com.zone 
2$ORIGIN od.com.
3$TTL 600        ; 10 minutes
4@               IN SOA  dns.od.com. dnsadmin.od.com. (
5                                2019111009 ; serial                      
6                                10800      ; refresh (3 hours)
7                                900        ; retry (15 minutes)
8                                604800     ; expire (1 week)
9                                86400      ; minimum (1 day)
10                                )
11                                NS   dns.od.com.
12$TTL 60 ; 1 minute
13dns                A    10.4.7.11
14...
15...
16demo               A    10.4.7.10
17
18[root@hdss7-11 ~]# systemctl restart named
19[root@hdss7-11 ~]# dig -t A demo.od.com @10.4.7.11 +short 
2010.4.7.10
21

5.7.5.检查启动状态

dashboard查看： http://dashboard.od.com
dubbo-monitor查看: http://dubbo-monitor.od.com

访问demo.od.com查看： http://demo.od.com/hello?name=wang

6.实战维护dubbo微服务集群

6.1.更新（rolling update）

修改代码提交GIT（发版）
使用jenkins进行CI（持续构建）
修改并应用k8s资源配置清单
或者在k8s上修改yaml的harbor镜像地址

6.2.扩容（scaling）

在k8s的dashboard上直接操作：登陆dashboard页面-->部署-->伸缩-->修改数量-->确定
命令行扩容，如下示例：
- Examples:
  
  Scale a replicaset named 'foo' to 3.
  
  kubectl scale --replicas=3 rs/foo
  
  Scale a resource identified by type and name specified in "foo.yaml" to 3.
  
  kubectl scale --replicas=3 -f foo.yaml
  
  If the deployment named mysql's current size is 2, scale mysql to 3.
  
  kubectl scale --current-replicas=2 --replicas=3 deployment/mysql
  
  Scale multiple replication controllers.
  
  kubectl scale --replicas=5 rc/foo rc/bar rc/baz
  
  Scale statefulset named 'web' to 3.
  
  kubectl scale --replicas=3 statefulset/web

6.3.宿主机突发故障处理

假如hdss7-21突发故障，离线

其他运算节点上操作：先删除故障节点使k8s触发自愈机制，pod在健康节点重新拉起

[root@hdss7-22 ~]# kubectl delete node hdss7-21.host.com node "hdss7-21.host.com" deleted
前端代理修改配置文件，把节点注释掉，使其不再调度到故障节点（hdss-7-21）

[root@hdss7-11 ~]# vi /etc/nginx/nginx.conf [root@hdss7-11 ~]# vi /etc/nginx/conf.d/od.com.conf [root@hdss7-11 ~]# nginx -t [root@hdss7-11 ~]# nginx -s reload
节点修好，直接启动，会自行加到集群，修改label，并把节点加回前端负载

[root@hdss7-21 bin]# kubectl label node hdss7-21.host.com node-role.kubernetes.io/master= node/hdss7-21.host.com labeled [root@hdss7-21 bin]# kubectl label node hdss7-21.host.com node-role.kubernetes.io/node= node/hdss7-21.host.com labeled [root@hdss7-21 bin]# kubectl get nodes NAME STATUS ROLES AGE VERSION hdss7-21.host.com Ready master,node 8d v1.15.4 hdss7-22.host.com Ready master,node 10d v1.15.4

6.4.FAQ

6.4.1.supervisor restart 不成功？

/etc/supervisord.d/xxx.ini 追加：

1killasgroup=true
2stopasgroup=true

3、交付Dubbo微服务到kubernetes集群

1.基础架构

1.1.架构图

1.2.节点规划

2.部署zookeeper

2.1.安装jdk 1.8（3台zk节点都要安装）

2.2.安装zk（3台节点都要安装）

2.2.1.解压，创建软链接

2.2.2.创建数据目录和日志目录

2.2.3.配置

2.2.4.做dns解析

2.2.4.依次启动并检查

3.部署jenkins

3.1.准备镜像

3.2.制作自定义镜像

3.2.1.生成ssh秘钥对

3.2.2.准备get-docker.sh文件

3.2.3.准备config.json文件

3.2.4.创建目录并准备Dockerfile

3.3.制作自定义镜像

3.4.创建infra仓库

3.5.创建kubernetes名称空间并在此创建secret

3.6.推送镜像

3.7.准备共享存储

3.7.1.安装nfs-utils -y

3.7.2.配置NFS服务

3.7.3.启动NFS服务

3.8.准备资源配置清单

3.9.应用资源配置清单

3.10.解析域名

3.11. 浏览器访问

3.12.页面配置jenkins

3.12.1.配置用户名密码

3.12.2.设置configure global security

3.12.3.安装好流水线插件Blue-Ocean

4.最后的准备工作

4.1.检查jenkins容器里的docker客户端

4.2.检查jenkins容器里的SSH key

4.3.部署maven软件

4.3.1.下载安装包

4.3.2.创建目录并解压

4.3.3.设置settings.xml国内镜像源

4.3.制作dubbo微服务的底包镜像

4.3.1.自定义Dockerfile

4.3.2.准备jre底包（7版本有一个7u80）

4.3.3.准备java-agent的jar包

4.3.3.准备config.yml和entrypoint.sh

4.3.4.harbor创建base项目

4.3.5.构建dubbo微服务的底包并推到harbor仓库

5.交付dubbo微服务至kubenetes集群

5.1.配置New job流水线

5.2..Pipeline Script

5.3.harbor创建app项目，把dubbo服务镜像管理起来

5.4.创建app名称空间，并添加secret资源

5.5.交付dubbo-demo-service

5.5.1.jenkins传参，构建dubbo-demo-service镜像，传到harbor

5.5.2.创建dubbo-demo-service的资源配置清单

5.5.3.应用dubbo-demo-service资源配置清单

5.5.4.检查启动状态

5.6.交付dubbo-Monitor

5.6.1.准备docker镜像

5.6.1.1.下载源码包、解压

5.6.1.2.修改以下项配置

5.6.1.3.制作镜像

5.6.1.3.1.准备环境

5.6.1.3.2.准备Dockerfile

5.6.1.3.3.build镜像并push到harbor仓库

5.6.2.解析域名

5.6.3.准备k8s资源配置清单

5.6.4.应用资源配置清单

5.6.5.浏览器访问

5.7.交付dubbo-demo-consumer

5.7.1.jenkins传参，构建dubbo-demo-service镜像，传到harbor

5.7.2.创建dubbo-demo-consumer的资源配置清单

5.7.3.应用dubbo-demo-consumer资源配置清单

5.7.4.解析域名

5.7.5.检查启动状态

6.实战维护dubbo微服务集群

6.1.更新（rolling update）

6.2.扩容（scaling）