IPVS vs. IPTABLES
IPVS模式在Kubernetes 1.8中被引入，在1.9中进入beta测试。 IPTABLES模式在1.1版本中被添加进来，在1.2开始就变成了默认的操作模式。IPVS 和 IPTABLES都是基于netfilter实现的。
IPVS与IPTABLES有以下几点不同的地方:
-
IPVS为大规模集群提供更好的可扩展性和性能。
-
IPVS比IPTABLES支持更复杂的负载均衡算法 (least load, least connections, locality, weighted, etc.)。
-
IPVS支持服务器健康检查和重连等等。
配置kube-proxy ipvs模式
k8s版本：1.10
默认情况下，Kube-proxy依然使用iptables来实现Service到POD之间的负载均衡。
1[root@node01 ~]# kubectl get svc -o wide 2NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR 3httpd-svc ClusterIP 10.110.73.171 <none> 8080/TCP 1d run=httpd 4kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 1d <none> 5[root@node01 ~]# iptables-save... 6-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE 7-A KUBE-SEP-6E7XQMQ4RAYOWTTM -s 10.244.0.3/32 -m comment --comment "kube-system/kube-dns:dns" -j KUBE-MARK-MASQ 8-A KUBE-SEP-6E7XQMQ4RAYOWTTM -p udp -m comment --comment "kube-system/kube-dns:dns" -m udp -j DNAT --to-destination 10.244.0.3:53 9-A KUBE-SEP-7SDFQQ3CPAUK3SXM -s 10.244.1.2/32 -m comment --comment "default/httpd-svc:" -j KUBE-MARK-MASQ 10-A KUBE-SEP-7SDFQQ3CPAUK3SXM -p tcp -m comment --comment "default/httpd-svc:" -m tcp -j DNAT --to-destination 10.244.1.2:80 11-A KUBE-SEP-FXPWU3QR6KR3AMXC -s 172.16.65.180/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ 12-A KUBE-SEP-FXPWU3QR6KR3AMXC -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-FXPWU3QR6KR3AMXC --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 172.16.65.180:6443 13-A KUBE-SEP-IT2ZTR26TO4XFPTO -s 10.244.0.2/32 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-MARK-MASQ 14-A KUBE-SEP-IT2ZTR26TO4XFPTO -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp" -m tcp -j DNAT --to-destination 10.244.0.2:53 15-A KUBE-SEP-TOSIKZVXYIAJSFHP -s 10.244.2.2/32 -m comment --comment "default/httpd-svc:" -j KUBE-MARK-MASQ 16-A KUBE-SEP-TOSIKZVXYIAJSFHP -p tcp -m comment --comment "default/httpd-svc:" -m tcp -j DNAT --to-destination 10.244.2.2:80 17-A KUBE-SEP-UO4LFRT7VSL3X4EB -s 10.244.0.4/32 -m comment --comment "default/httpd-svc:" -j KUBE-MARK-MASQ 18-A KUBE-SEP-UO4LFRT7VSL3X4EB -p tcp -m comment --comment "default/httpd-svc:" -m tcp -j DNAT --to-destination 10.244.0.4:80 19-A KUBE-SEP-YIL6JZP7A3QYXJU2 -s 10.244.0.2/32 -m comment --comment "kube-system/kube-dns:dns" -j KUBE-MARK-MASQ 20-A KUBE-SEP-YIL6JZP7A3QYXJU2 -p udp -m comment --comment "kube-system/kube-dns:dns" -m udp -j DNAT --to-destination 10.244.0.2:53 21-A KUBE-SEP-ZXMNUKOKXUTL2MK2 -s 10.244.0.3/32 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-MARK-MASQ 22-A KUBE-SEP-ZXMNUKOKXUTL2MK2 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp" -m tcp -j DNAT --to-destination 10.244.0.3:53 23-A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.96.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-MARK-MASQ 24-A KUBE-SERVICES -d 10.96.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y 25-A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.96.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-MARK-MASQ 26-A KUBE-SERVICES -d 10.96.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU 27-A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-MARK-MASQ 28-A KUBE-SERVICES -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-SVC-ERIFXISQEP7F7OF4 29-A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.110.73.171/32 -p tcp -m comment --comment "default/httpd-svc: cluster IP" -m tcp --dport 8080 -j KUBE-MARK-MASQ 30-A KUBE-SERVICES -d 10.110.73.171/32 -p tcp -m comment --comment "default/httpd-svc: cluster IP" -m tcp --dport 8080 -j KUBE-SVC-RL3JAE4GN7VOGDGP 31-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS 32-A KUBE-SVC-ERIFXISQEP7F7OF4 -m comment --comment "kube-system/kube-dns:dns-tcp" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-IT2ZTR26TO4XFPTO 33-A KUBE-SVC-ERIFXISQEP7F7OF4 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-SEP-ZXMNUKOKXUTL2MK2 34-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-FXPWU3QR6KR3AMXC --mask 255.255.255.255 --rsource -j KUBE-SEP-FXPWU3QR6KR3AMXC 35-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-FXPWU3QR6KR3AMXC 36-A KUBE-SVC-RL3JAE4GN7VOGDGP -m comment --comment "default/httpd-svc:" -m statistic --mode random --probability 0.33332999982 -j KUBE-SEP-UO4LFRT7VSL3X4EB 37-A KUBE-SVC-RL3JAE4GN7VOGDGP -m comment --comment "default/httpd-svc:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-7SDFQQ3CPAUK3SXM 38-A KUBE-SVC-RL3JAE4GN7VOGDGP -m comment --comment "default/httpd-svc:" -j KUBE-SEP-TOSIKZVXYIAJSFHP 39-A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment "kube-system/kube-dns:dns" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-YIL6JZP7A3QYXJU2 40-A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment "kube-system/kube-dns:dns" -j KUBE-SEP-6E7XQMQ4RAYOWTTM 41...
第一步，在内核中加载ip_vs模块：
1cat > /etc/sysconfig/modules/ipvs.modules <<EOF 2#!/bin/bash 3ipvs_modules="ip_vs ip_vs_lc ip_vs_wlc ip_vs_rr ip_vs_wrr ip_vs_lblc ip_vs_lblcr ip_vs_dh ip_vs_sh ip_vs_nq ip_vs_sed ip_vs_ftp nf_conntrack_ipv4" 4for kernel_module in \${ipvs_modules}; do 5 /sbin/modinfo -F filename \${kernel_module} > /dev/null 2>&1 6 if [ $? -eq 0 ]; then 7 /sbin/modprobe \${kernel_module} 8 fi 9done 10EOF 11chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep ip_vs
输出结果应该为：
1[root@node01 ~]# lsmod | grep ip_vs 2ip_vs_ftp 13079 0 3ip_vs_sed 12519 0 4ip_vs_nq 12516 0 5ip_vs_sh 12688 0 6ip_vs_dh 12688 0 7ip_vs_lblcr 12922 0 8ip_vs_lblc 12819 0 9ip_vs_wrr 12697 0 10ip_vs_rr 12600 3 11ip_vs_wlc 12519 0 12ip_vs_lc 12516 0 13nf_nat 26787 3 ip_vs_ftp,nf_nat_ipv4,nf_nat_masquerade_ipv4 14ip_vs 141092 27 ip_vs_dh,ip_vs_lc,ip_vs_nq,ip_vs_rr,ip_vs_sh,ip_vs_ftp,ip_vs_sed,ip_vs_wlc,ip_vs_wrr,ip_vs_lblcr,ip_vs_lblc 15nf_conntrack 133387 7 ip_vs,nf_nat,nf_nat_ipv4,xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_netlink,nf_conntrack_ipv4 16libcrc32c 12644 4 xfs,ip_vs,nf_nat,nf_conntrack
第二步，安装ipvs管理工具ipvsadm
yum install -y ipvsadm
第三步，修改集群配置文件
在使用kubeadm init --config config.yaml初始化集群前，修改集群配置文件
1Kubernetes v1.8 v1.9 2kind: MasterConfiguration 3apiVersion: kubeadm.k8s.io/v1alpha1 4... 5kubeProxy: 6 config: 7 featureGates: SupportIPVSProxyMode=true 8 mode: ipvs 9... 10 11Kubernetes v1.10 12kind: MasterConfiguration 13apiVersion: kubeadm.k8s.io/v1alpha1 14... 15kubeProxy: 16 config: 17 featureGates: 18 SupportIPVSProxyMode: true 19 mode: ipvs 20...
第四步，结果验证
1[root@node01 ~]# ipvsadm -L -n 2IP Virtual Server version 1.2.1 (size=4096) 3Prot LocalAddress:Port Scheduler Flags 4 -> RemoteAddress:Port Forward Weight ActiveConn InActConn 5TCP 10.96.0.1:443 rr persistent 10800 6 -> 172.16.65.180:6443 Masq 1 3 0 7TCP 10.96.0.10:53 rr 8 -> 10.244.0.2:53 Masq 1 0 0 9 -> 10.244.0.3:53 Masq 1 0 0 10UDP 10.96.0.10:53 rr 11 -> 10.244.0.2:53 Masq 1 0 0 12 -> 10.244.0.3:53 Masq 1 0 0
第五步，测试
创建http-svc service
1apiVersion: apps/v1beta1 2kind: Deployment 3metadata: 4 name: httpd 5spec: 6 replicas: 3 7 template: 8 metadata: 9 labels: 10 run: httpd 11 spec: 12 containers: 13 - name: httpd 14 image: httpd 15 ports: 16 - containerPort: 80 17 18--- 19apiVersion: v1 20kind: Service 21metadata: 22 name: httpd-svc 23spec: 24 selector: 25 run: httpd type: NodePort 26 ports: 27 - protocol: TCP 28 port: 8080 29 targetPort: 80 nodePort: 30001
查看效果
1[root@node01 yaml-store]# kubectl get pod -o wide 2NAME READY STATUS RESTARTS AGE IP NODE 3httpd-749bf8c6f4-htzgl 1/1 Running 0 11m 10.244.2.2 node03 4httpd-749bf8c6f4-vqwff 1/1 Running 0 11m 10.244.1.2 node02 5httpd-749bf8c6f4-wkv9w 1/1 Running 0 11m 10.244.0.4 node01 6[root@node01 yaml-store]# kubectl get svc -o wide 7NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR 8httpd-svc ClusterIP 10.99.129.90 <none> 8080:30001/TCP 11m run=httpd 9kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 58m <none> 10 11[root@node01 yaml-store]# ipvsadm -L -n 12IP Virtual Server version 1.2.1 (size=4096) 13Prot LocalAddress:Port Scheduler Flags 14 -> RemoteAddress:Port Forward Weight ActiveConn InActConn 15TCP 172.16.65.181:30001 rr 16 -> 10.244.0.4:80 Masq 1 0 0 17 -> 10.244.1.2:80 Masq 1 0 0 18 -> 10.244.2.2:80 Masq 1 0 0 19TCP 172.17.0.1:30001 rr 20 -> 10.244.0.4:80 Masq 1 0 0 21 -> 10.244.1.2:80 Masq 1 0 0 22 -> 10.244.2.2:80 Masq 1 0 0 23TCP 10.96.0.1:443 rr persistent 10800 24 -> 172.16.65.180:6443 Masq 1 3 0 25TCP 10.96.0.10:53 rr 26 -> 10.244.0.2:53 Masq 1 0 0 27 -> 10.244.0.3:53 Masq 1 0 0 28TCP 10.99.129.90:8080 rr 29 -> 10.244.0.4:80 Masq 1 0 0 30 -> 10.244.1.2:80 Masq 1 0 0 31 -> 10.244.2.2:80 Masq 1 0 0 32TCP 10.244.0.0:30001 rr 33 -> 10.244.0.4:80 Masq 1 0 0 34 -> 10.244.1.2:80 Masq 1 0 0 35 -> 10.244.2.2:80 Masq 1 0 0 36TCP 10.244.0.1:30001 rr 37 -> 10.244.0.4:80 Masq 1 0 0 38 -> 10.244.1.2:80 Masq 1 0 0 39 -> 10.244.2.2:80 Masq 1 0 0 40TCP 127.0.0.1:30001 rr 41 -> 10.244.0.4:80 Masq 1 0 0 42 -> 10.244.1.2:80 Masq 1 0 0 43 -> 10.244.2.2:80 Masq 1 0 0 44UDP 10.96.0.10:53 rr 45 -> 10.244.0.2:53 Masq 1 0 0 46 -> 10.244.0.3:53 Masq 1 0 0
参考文章：
https://github.com/kubernetes/kubernetes/tree/master/pkg/proxy/ipvs