本文为苏宁网络架构师陈刚的原创文章。

01准备测试机

在16G的笔记本没跑起来，就干脆拼凑了一台游戏工作室级别的机器：双路E5-2860v3 CPU，24核48线程，128G DDR4 ECC内存，NVME盘 512G。在上面开5个VM，假装是物理服务器。

· 192.16.35.110 deployer

· 192.16.35.111 tf控制器

· 192.16.35.112 openstack服务器，同时也是计算节点

· 192.16.35.113 k8s master

· 192.16.35.114 k8s的Node k01，同时也是ops的计算节点

直接使用vagrant拉镜像会很慢，就先下载下来：

https://cloud.centos.org/centos/7/vagrant/x86_64/images/

下载对应的VirtualBox.box文件。

然后使用命令, 命名为vagrant的box：

vagrant box add centos/7 CentOS-7-x86_64-Vagrant-2004_01.VirtualBox.box

1cat << EEOOFF > vagrantfile
2###start 
3#-*- mode: ruby -*-
4#vi: set ft=ruby :
5Vagrant.require_version ">=2.0.3"
6
7#All Vagrant configuration is done below. The "2" in Vagrant.configure
8#configures the configuration version (we support older styles for
9#backwards compatibility). Please don't change it unless you know what
10#you're doing.
11
12ENV["LC_ALL"] = "en_US.UTF-8"
13
14VAGRANTFILE_API_VERSION = "2"
15
16Vagrant.configure("2") do |config|
17  #The most common configuration options are documented and commented below.
18  #For a complete reference, please see the online documentation at
19  #https://docs.vagrantup.com.
20
21  #Every Vagrant development environment requires a box. You can search for
22  #boxes at https://atlas.hashicorp.com/search.
23
24  config.vm.box = "geerlingguy/centos7"
25  #config.vbguest.auto_update = false
26  #config.vbguest.no_remote = true  
27
28  config.vm.define "deployer" do | dp |
29    dp.vm.provider "virtualbox" do | v |
30      v.memory = "8000"
31      v.cpus = 2
32    end
33    dp.vm.network "private_network", ip: "192.16.35.110", auto_config: true
34    dp.vm.hostname = "deployer"
35  end
36
37  config.vm.define "tf" do | tf |
38    tf.vm.provider "virtualbox" do | v |
39      v.memory = "64000"
40      v.cpus = 16
41    end
42    tf.vm.network "private_network", ip: "192.16.35.111", auto_config: true
43    tf.vm.hostname = "tf"
44  end
45
46  config.vm.define "ops" do | os |
47    os.vm.provider "virtualbox" do | v |
48      v.memory = "16000"
49      v.cpus = 4
50    end
51    os.vm.network "private_network",ip: "192.16.35.112",  auto_config: true
52    os.vm.hostname = "ops"
53  end
54
55  config.vm.define "k8s" do | k8 |
56    k8.vm.provider "virtualbox" do | v |
57      v.memory = "8000"
58      v.cpus = 2
59    end
60    k8.vm.network "private_network", ip: "192.16.35.113", auto_config: true
61    k8.vm.hostname = "k8s"
62  end
63
64  config.vm.define "k01" do | k1 |
65    k1.vm.provider "virtualbox" do | v |
66      v.memory = "4000"
67      v.cpus = 2
68    end
69    k1.vm.network "private_network", ip: "192.16.35.114", auto_config: true
70    k1.vm.hostname = "k01"
71  end
72
73  config.vm.provision "shell", privileged: true, path: "./setup.sh"
74
75end
76
77EEOOFF
78
79cat << EEOOFF > setup.sh
80#!/bin/bash
81##Setup vagrant vms.
82#set -eu
83
84#Copy hosts info
85cat <<EOF > /etc/hosts
86127.0.0.1 localhost
87127.0.1.1 vagrant.vm vagrant
88
89192.16.35.110 deployer
90192.16.35.111 tf
91192.16.35.112 ops
92192.16.35.113 k8s
93192.16.35.114 k01
94
95# The following lines are desirable for IPv6 capable hosts
96::1     localhost ip6-localhost ip6-loopback
97ff02::1 ip6-allnodes
98ff02::2 ip6-allrouters
99EOF
100
101systemctl stop firewalld
102systemctl disable firewalld
103iptables -F && iptables -X && iptables -F -t nat && iptables -X -t nat
104iptables -P FORWARD ACCEPT
105
106swapoff -a 
107sed -i 's/.*swap.*/#&/' /etc/fstab
108# swapoff -a && sysctl -w vm.swappiness=0
109
110# setenforce  0 
111sed -i "s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/sysconfig/selinux 
112sed -i "s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config 
113sed -i "s/^SELINUX=permissive/SELINUX=disabled/g" /etc/sysconfig/selinux 
114sed -i "s/^SELINUX=permissive/SELINUX=disabled/g" /etc/selinux/config  
115
116# modprobe ip_vs_rr
117modprobe br_netfilter
118
119yum -y update
120
121# sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: No such file or directory
122# sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory
123# yum install -y bridge-utils.x86_64
124# modprobe bridge
125# modprobe br_netfilter
126# Setup system vars
127
128yum install -y epel-release
129yum install -y yum-utils device-mapper-persistent-data lvm2 net-tools vim chrony python python-setuptools python-pip iproute lrzsz tree git
130
131yum install -y libguestfs-tools libvirt-python virt-install libvirt ansible
132
133pip install wheel --upgrade -i https://mirrors.aliyun.com/pypi/simple/
134pip install pip --upgrade -i https://mirrors.aliyun.com/pypi/simple/
135pip install ansible  netaddr --upgrade -i https://mirrors.aliyun.com/pypi/simple/
136
137# python-urllib3 should be installed before "pip install requests"
138# if install failed, pip uninstall urllib3, then reinstall python-urllib3
139# pip uninstall -y urllib3 | true
140# yum install -y python-urllib3 
141pip install requests -i https://mirrors.aliyun.com/pypi/simple/
142
143systemctl disable libvirtd.service
144systemctl disable dnsmasq
145systemctl stop libvirtd.service
146systemctl stop dnsmasq
147
148if [  -d "/root/.ssh" ]; then
149      rm -rf /root/.ssh
150fi
151
152ssh-keygen -q -t rsa -N "" -f ~/.ssh/id_rsa
153
154cat ~/.ssh/id_rsa.pub > ~/.ssh/authorized_keys
155chmod go-rwx ~/.ssh/authorized_keys
156
157# 
158# timedatectl set-timezone Asia/Shanghai
159
160if [ -f "/etc/chrony.conf" ]; then
161   mv /etc/chrony.conf /etc/chrony.conf.bak
162fi
163
164cat <<EOF > /etc/chrony.conf
165      allow 192.16.35.0/24
166      server ntp1.aliyun.com iburst
167      local stratum 10
168      logdir /var/log/chrony
169      rtcsync
170      makestep 1.0 3
171      driftfile /var/lib/chrony/drift
172EOF
173
174systemctl restart chronyd.service
175systemctl enable chronyd.service
176
177echo "* soft nofile 65536" >> /etc/security/limits.conf
178echo "* hard nofile 65536" >> /etc/security/limits.conf
179echo "* soft nproc 65536"  >> /etc/security/limits.conf
180echo "* hard nproc 65536"  >> /etc/security/limits.conf
181echo "* soft  memlock  unlimited"  >> /etc/security/limits.conf
182echo "* hard memlock  unlimited"  >> /etc/security/limits.conf
183
184if [ ! -d "/var/log/journal" ]; then
185  mkdir /var/log/journal
186fi
187
188if [ ! -d "/etc/systemd/journald.conf.d" ]; then
189  mkdir /etc/systemd/journald.conf.d
190fi
191
192cat <<EOF > /etc/systemd/journald.conf.d/99-prophet.conf 
193[Journal]
194Storage=persistent
195
196Compress=yes
197
198SyncIntervalSec=5m
199RateLimitInterval=30s
200RateLimitBurst=1000
201
202SystemMaxUse=10G
203
204SystemMaxFileSize=200M
205
206ForwardToSyslog=no
207
208EOF
209
210systemctl restart systemd-journald
211
212EEOOFF

02在所有的节点上安装docker

CentOS

例如：如果pip安装软件的速度很慢，可以考虑使用基于aliyun的pip加速

· 各个节点设置pip加速

mkdir .pip && tee ~/.pip/pip.conf <<-'EOF'[global]trusted-host = mirrors.aliyun.comindex-url = https://mirrors.aliyun.com/pypi/simpleEOF

注意requests包不能在urllib3之后安装，否则会出错：

1pip uninstall urllib3
2pip uninstall chardet
3pip install requests

(这些命令应该都已经在 setup.sh 中执行过了)

1yum install -y yum-utils device-mapper-persistent-data lvm2 net-tools iproute lrzsz tree git
2yum-config-manager   --add-repo   https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
3yum makecache fast
4yum install -y docker-ce
5yum -y install epel-release
6systemctl daemon-reload
7systemctl enable docker
8systemctl restart docker
9yum install -y chrony
10systemctl start chronyd
11systemctl enable chronyd

03拉取并启动Contrail-Kolla-Ansible-Deployer容器

容器的Nightly builds 可以从这里访问： Docker Hub

https://hub.docker.com/r/opencontrailnightly/contrail-kolla-ansible-deployer/tags

例如：

1vim /etc/docker/daemon.json
2{ "registry-mirrors" : [ "https://hub-mirror.c.163.com",
3    "https://registry.docker-cn.com" ] }
4systemctl restart docker
5
6export CAD_IMAGE=opencontrailnightly/contrail-kolla-ansible-deployer:master-latest
7docker run -td --net host --name contrail_kolla_ansible_deployer $CAD_IMAGE

04 将配置文件复制到容器

instance.yaml：用于配置Tungsten Fabric集群的模板文件。

https://github.com/Juniper/contrail-ansible-deployer/wiki/Contrail-with-Openstack-Kolla#13-configure-necessary-parameters-configinstancesyaml-under-appropriate-parameters

要获得有关如何配置该文件中所有可用参数的信息，可阅读这里：

https://github.com/Juniper/contrail-ansible-deployer/blob/master/README.md#configuration

1cat << EOF > instances.yaml
2provider_config:  bms:    ssh_pwd: vagrant    ssh_user: root    ntpserver: ntp1.aliyun.com    domainsuffix: localinstances:  tf:    provider: bms    ip: 192.16.35.111    roles:      config_database:      config:      control:      analytics_database:      analytics:      webui:  ops:    provider: bms    ip: 192.16.35.112    roles:
3      openstack:
4      openstack_compute:  
5      vrouter:
6        PHYSICAL_INTERFACE: enp0s8
7  k8s:    provider: bms    ip: 192.16.35.113    roles:
8      k8s_master:
9      k8s_node:
10      kubemanager:
11      vrouter:
12        PHYSICAL_INTERFACE: enp0s8
13  k01:    provider: bms    ip: 192.16.35.114    roles:
14      openstack_compute:
15      k8s_node:
16      vrouter:
17        PHYSICAL_INTERFACE: enp0s8
18contrail_configuration:  AUTH_MODE: keystone  KEYSTONE_AUTH_URL_VERSION: /v3
19  KEYSTONE_AUTH_ADMIN_PASSWORD: vagrant
20  CLOUD_ORCHESTRATOR: openstack
21  CONTRAIL_VERSION: latest
22  UPGRADE_KERNEL: true
23  ENCAP_PRIORITY: "VXLAN,MPLSoUDP,MPLSoGRE"
24  PHYSICAL_INTERFACE: enp0s8
25global_configuration:
26  CONTAINER_REGISTRY: opencontrailnightly
27kolla_config:
28  kolla_globals:    enable_haproxy: no    enable_ironic: "no"    enable_swift: "no"
29    network_interface: "enp0s8"
30  kolla_passwords:    keystone_admin_password: vagrant
31
32EOF
33
34export INSTANCES_FILE=instances.yaml
35docker cp $INSTANCES_FILE contrail_kolla_ansible_deployer:/root/contrail-ansible-deployer/config/instances.yaml

05准备好所有节点的环境

除了deployer，我在所有节点上都做了一遍。

正常的做法是建个自己的repository放各种image，实验环境节点少，直接国内下载也很快的。

注意python和python-py这两个包是冲突的，只能安装其中之一，最好先全卸载，再安装其中一个：

1pip uninstall docker-py docker
2 pip install python
3
4yum -y install python-devel python-subprocess32 python-setuptools python-pip
5
6 pip install --upgrade pip
7
8 find / -name *subpro*.egg-info
9 find / -name *subpro*.egg-info |xargs rm -rf
10
11pip install -I sixpip install -I docker-compose

将k8s repository改成阿里的，缺省的Google源太慢或不通：vi

playbooks/roles/k8s/tasks/RedHat.yml

1yum_repository:
2name: Kubernetes
3description: k8s repo
4baseurl: https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
5gpgkey: https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
6repo_gpgcheck: yes
7gpgcheck: yes
8when: k8s_package_version is defined

playbook中安装这些需要访问海外网站，可以从国内下载，然后改个tag：

1k8s.gcr.io/kube-apiserver:v1.14.8
2k8s.gcr.io/kube-controller-manager:v1.14.8
3k8s.gcr.io/kube-scheduler:v1.14.8
4k8s.gcr.io/kube-proxy:v1.14.8
5k8s.gcr.io/pause:3.1
6k8s.gcr.io/etcd:3.3.10
7k8s.gcr.io/coredns:1.3.1

换个方法变通处理一下

1docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.14.8
2docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.14.8
3docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.14.8
4docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.14.8
5docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1
6docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.3.10
7docker pull coredns/coredns:1.3.1
8docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kubernetes-dashboard-amd64:v1.8.3

再重新给下载的打个tag

1docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.14.8 k8s.gcr.io/kube-apiserver:v1.14.8
2docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.14.8 k8s.gcr.io/kube-controller-manager:v1.14.8
3docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.14.8 k8s.gcr.io/kube-scheduler:v1.14.8
4docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.14.8 k8s.gcr.io/kube-proxy:v1.14.8
5docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1 k8s.gcr.io/pause:3.1
6docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.3.10 k8s.gcr.io/etcd:3.3.10
7docker tag docker.io/coredns/coredns:1.3.1 k8s.gcr.io/coredns:1.3.1
8docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kubernetes-dashboard-amd64:v1.8.3  k8s.gcr.io/kubernetes-dashboard-amd64:v1.8.3

06启动deployer容器，进入其中进行部署

docker start contrail_kolla_ansible_deployer

进入deployer容器：

1docker exec -it contrail_kolla_ansible_deployer bashcd /root/contrail-ansible-deployer
2ansible-playbook -i inventory/ -e orchestrator=openstack playbooks/provision_instances.yml
3ansible-playbook -i inventory/ -e orchestrator=openstack playbooks/configure_instances.yml
4ansible-playbook -i inventory/ -e orchestrator=openstack playbooks/install_openstack.yml
5ansible-playbook -i inventory/ -e orchestrator=openstack playbooks/install_k8s.yml
6ansible-playbook -i inventory/ -e orchestrator=openstack playbooks/install_contrail.yml
7
8kubectl taint nodes k8s node-role.kubernetes.io/master-

最后一次kubelet升级到最新，遇到CSI的bug，修改一下配置文件后重启kubelet即可：

1After experiencing the same issue, editing /var/lib/kubelet/config.yaml to add:
2featureGates:  CSIMigration: false

07安装完成后，建2个VM和容器测试一下

1yum install -y gcc python-devel
2pip install python-openstackclient
3pip install python-ironicclient
4
5source /etc/kolla/kolla-toolbox/admin-openrc.sh

如果openstack命令有如下“queue”的报错，是需要python3：

1File "/usr/lib/python2.7/site-packages/openstack/utils.py", line 13, in <module>
2    import queue
3ImportError: No module named queue
4
5rm -f /usr/bin/python
6ln -s /usr/bin/python3 /usr/bin/python
7pip install python-openstackclient
8pip install python-ironicclient
9yum install -y python3-pip
10
11yum install -y gcc python-devel wgetpip install --upgrade setuptoolspip install --ignore-installed python-openstackclient
12
13我每次都需要python3，所以干脆也安装了这个：
14pip3 install python-openstackclient -i https://mirrors.aliyun.com/pypi/simple/
15pip3 install python-ironicclient -i https://mirrors.aliyun.com/pypi/simple/

进入Tungsten Fabric，用浏览器：https://192.16.35.111:8143

进入openstack，用浏览器：https://192.16.35.112

在k8s master上（192.16.35.113）：

1scp root@192.16.35.114:/opt/cni/bin/contrail-k8s-cni /opt/cni/bin/
2mkdir /etc/cni/net.d
3scp root@192.16.35.114:/etc/cni/net.d/10-contrail.conf /etc/cni/net.d/10-contrail.conf

wget

https://github.com/cirros-dev/cirros/releases/download/0.4.0/cirros-0.4.0-x86_64-disk.img

官方下载地址

https://download.cirros-cloud.net/

curl -O

https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img

wget

http://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img

wget

http://download.cirros-cloud.net/daily/20161201/cirros-d161201-x86_64-disk.img

（都没有找到带tcpdump的版本）

reboot

source /etc/kolla/kolla-toolbox/admin-openrc.sh

1openstack image create cirros --disk-format qcow2 --public --container-format bare --file cirros-0.4.0-x86_64-disk.imgnova flavor-create m1.tiny auto 512 1 1openstack network create net1openstack subnet create --subnet-range 10.1.1.0/24 --network net1 mysubnet1
2NET_ID=`openstack network list | grep net1 | awk -F '|' '{print $2}' | tr -d ' '`
3
4nova boot --image cirros --flavor m1.tiny --nic net-id=${NET_ID} VM1
5nova boot --image cirros --flavor m1.tiny --nic net-id=${NET_ID} VM2

进入k8s_master, 192.16.35.113:

1yum install -y git
2git clone https://github.com/virtualhops/k8s-demo
3kubectl create -f k8s-demo/po-ubuntuapp.yml
4kubectl create -f k8s-demo/rc-frontend.yml
5kubectl expose rc/frontend
6kubectl exec -it ubuntuapp curl frontend # many times

参考方案：

https://github.com/Juniper/contrail-ansible-deployer/wiki/%5B-Container-Workflow%5D-Deploying-Contrail-with-OpenStack

推荐阅读

Tungsten Fabric实战：对接vMX虚拟路由平台填坑
Tungsten Fabric实战：基于K8s的部署踩坑
TF实战Q&A丨你不理解透，出了问题都不知道怎么弄
TF 实战Q&A丨只在此网中，云深不知处