lxcfs容器资源视图隔离 for k8s

来源: DevOpSec公众号作者: DevOpSec

背景

k8s版本1.25.6，业务k8s容器化，虚机里进程迁移到容器里后，运维在执行free -m top等命令排查问题时一脸迷惑，显示内存还有很多结果pod的容器被oom或CPU资源显示很多核且空闲很多资源进程却运行很慢，我们看到的资源视图是物理机的而非我们做了限定pod里容器的资源，这给研发和运维排查问题带来一定的干扰。

那是什么原因导致运维看到的资源视图还是物理机的呢？

我们知道容器通过cgroup对CPU、内存、交换空间等资源进行限制，但是容器并不是完全独立隔离的，它与主机共享内核，因此可以访问主机上的一些信息。在Linux系统中，/proc目录下存放了许多虚拟文件，它们提供了对系统内核和运行时信息的访问。/proc/meminfo文件包含了关于内存使用和状态的信息，例如总内存大小、可用内存、已使用内存等。当在容器里执行free -m时，实际上是在访问主机上的/proc/meminfo文件的信息，所以展示的是物理机的内存信息。

我们知道什么原因导致的容器资源视图没有隔离的问题，在实际的使用过程中除了有迷惑还会有一些痛点：

比如nginx 根据CPU核数自动设置worker数量。
jvm程序内存根据系统内存大小自动设置jvm大小，导致进程启动不了或者运行过程中经常oom。
信息的过度泄露可能会危害物理机的安全等。

那怎么解决容器资源视图隔离的问题？ Linux容器（LXC）社区早就意识到上述问题，他们开发了LXCFS（Linux Containers File System）来解决容器资源视图隔离的问题。

下面来看看LXCFS的工作原理。

`LXCFS` 工作原理

LXCFS是一个使用FUSE（Filesystem in Userspace）实现的小型虚拟文件系统，旨在让Linux容器感觉更像一个虚拟机。它最初是LXC的一个附带项目，但可由任何运行时使用。

LXCFS确保procfs中关键文件提供的信息是针对容器的，例如：

1/proc/cpuinfo
2/proc/diskstats
3/proc/meminfo
4/proc/stat
5/proc/swaps
6/proc/uptime
7/proc/slabinfo
8/sys/devices/system/cpu/online

LXCFS将这些信息适配到容器内，以便显示的值（例如/proc/uptime）真正反映容器的运行时间，而不是主机的运行时间。

即LXCFS在容器内部创建了一个虚拟的文件系统，通过挂载主机上的一些关键目录（如/proc和/sys等）到容器内部的对应目录下，使得容器内的进程可以看到主机上的资源信息，同时，LXCFS通过自己的逻辑和计算，提供了对这些资源信息的虚拟视图，使得容器内部能够看到主机上实际的资源使用情况。

容器里执行free -m，读取文件/proc/meminfo
因为/proc/meminfo文件是挂载的，所以会读取/var/lib/lxcfs/proc/meminfo文件见下文，这就触发了LXCFS的工作机制
LXCFS文件系通过gblic系统调用vfs接口然后转向Fuse内核模块
FUSE回调用户空间LXCFS文件系统实现接口，获取容器的cgroup信息
LXCFS实现根据容器id获取并计算cgroup下被限制容器的实际mem、cpu等信息，最终返回给用户看到的结果就是cgroup 限制的资源视图。

`LXCFS` 机器上部署

a. 安装 `lxcfs`

1yum install meson fuse-devel fuse cmake help2man fuse3 fuse3-devel -y
2
3git clone git://github.com/lxc/lxcfs
4cd lxcfs
5
6meson setup -Dinit-script=systemd --prefix=/usr build/
7
8meson compile -C build/
9
10meson install -C build/

b. 启动`lxcfs`

1mkdir -p /var/lib/lxcfs
2lxcfs /var/lib/lxcfs

c. 测试运行容器

1docker run -it -m 256m --memory-swap 256m --cpus=1 \
2      -v /var/lib/lxcfs/proc/cpuinfo:/proc/cpuinfo:rw \
3      -v /var/lib/lxcfs/proc/diskstats:/proc/diskstats:rw \
4      -v /var/lib/lxcfs/proc/meminfo:/proc/meminfo:rw \
5      -v /var/lib/lxcfs/proc/stat:/proc/stat:rw \
6      -v /var/lib/lxcfs/proc/swaps:/proc/swaps:rw \
7      -v /var/lib/lxcfs/proc/uptime:/proc/uptime:rw \
8      -v /var/lib/lxcfs/proc/slabinfo:/proc/slabinfo:rw \
9      -v /var/lib/lxcfs/sys/devices/system/cpu:/sys/devices/system/cpu:rw \
10      ubuntu:18.04 /bin/bash

启动容器后，执行如下命令确认是否生效

11. uptime   #容器启动时间
2
32. free -m  #内存情况
4
53. lscpu  #看online cpu 核数 或者 cat /proc/cpuinfo
6

k8s 环境下怎么为pod加上资源视图隔离呢？下面我们来看一看

`LXCFS` k8s 环境运行

解决步骤：

首先要使lxcfs进程在所有的node上运行，这个我们使用damonset解决
其次挂载node上的/sys/fs/cgroup、/usr/lib64和/usr/local 到lxcfs里，把lxcfs 容器里虚拟文件系统/var/lib/lxcfs/通过hostPath挂载到物理机上
最后创建podyaml，通过hostPath形式把node上/var/lib/lxcfs/ 挂载到pod的容器里，这样就完成了lxcfs 解决k8s 容器资源视图隔离的问题。

a. 构建`lxcfs`镜像

a.1 目录结构

1tree .                                                                             
2.
3├── Dockerfile
4├── build.sh
5└── lxcfs-lxcfs-5.0.4.tar.gz

a.2 Dockerfile

1FROM centos:7.9  #或者制定你的基础镜像
2
3#安装
4RUN yum install meson fuse-devel fuse cmake help2man fuse3 fuse3-devel git -y
5
6RUN git clone git://github.com/lxc/lxcfs && cd lxcfs
7
8RUN meson setup -Dinit-script=systemd --prefix=/usr build/
9
10RUN meson compile -C build/
11
12RUN meson install -C build/
13
14#运行
15RUN mkdir -p /var/lib/lxcfs
16
17CMD ["sh", "-c", "lxcfs /var/lib/lxcfs"]

a.3 build.sh 构建镜像

1#!/bin/bash
2
3source /etc/profile
4
5
6docker build -t yourharbor.domain.com/centos/7.9/lxcfs/5.0.4/lxcfs .
7docker push yourharbor.domain.com/centos/7.9/lxcfs/5.0.4/lxcfs

到这里lxcfs镜像就构建完了，下面看看怎么用此镜像

b. 运行`lxcfs` `daemonset` `yaml`

使用构建的lxcfs镜像，挂载node文件到pod同时挂载/var/lib/lxcfs/ 到node上,见下述yaml

1apiVersion: apps/v1
2kind: DaemonSet
3metadata:
4  annotations:
5  labels:
6    app: lxcfs
7  name: lxcfs
8  namespace: default
9spec:
10  revisionHistoryLimit: 10
11  selector:
12    matchLabels:
13      app: lxcfs
14  template:
15    metadata:
16      labels:
17        app: lxcfs
18    spec:
19      containers:
20      - yourharbor.domain.com/centos/7.9/lxcfs/5.0.4/lxcfs
21        imagePullPolicy: Always
22        name: lxcfs
23        resources: {}
24        securityContext:
25          privileged: true
26        volumeMounts:
27        - mountPath: /sys/fs/cgroup
28          name: cgroup
29        - mountPath: /var/lib/lxcfs
30          mountPropagation: Bidirectional
31          name: lxcfs
32        - mountPath: /usr/local
33          name: usr-local
34        - mountPath: /usr/lib64
35          name: usr-lib64
36      hostPID: true
37      imagePullSecrets:
38      - name: your-docker-token
39      restartPolicy: Always
40      tolerations:
41      - effect: NoSchedule
42        key: node-role.kubernetes.io/master
43      - effect: NoSchedule
44        key: your-taint-key
45        operator: Exists
46      volumes:
47      - hostPath:
48          path: /sys/fs/cgroup
49          type: ""
50        name: cgroup
51      - hostPath:
52          path: /usr/local
53          type: ""
54        name: usr-local
55      - hostPath:
56          path: /usr/lib64
57          type: ""
58        name: usr-lib64
59      - hostPath:
60          path: /var/lib/lxcfs
61          type: DirectoryOrCreate
62        name: lxcfs

apply上述yaml后可能个别node上lxcfs daemonset pod 启动报如下错误

Error: failed to generate container "974c6c0465adae1a244e3416b3e053ba2dccb0cbd123c2d02317c9301e3f83d0" spec: failed to apply OCI options: failed to stat "/var/lib/lxcfs": stat /var/lib/lxcfs: transport endpoint is not connected

解决办法

umount /var/lib/lxcfs

c. 验证 deployment pod yaml 定义

1apiVersion: apps/v1
2kind: Deployment
3metadata:
4  name: web
5spec:
6  replicas: 2
7  selector:
8    matchLabels:
9      app: web 
10  template:
11    metadata:
12      labels:
13        app: web
14    spec:
15      volumes:
16        - hostPath:
17            path: /var/lib/lxcfs/proc/cpuinfo
18            type: ""
19          name: lxcfs-proc-cpuinfo
20        - hostPath:
21            path: /var/lib/lxcfs/proc/diskstats
22            type: ""
23          name: lxcfs-proc-diskstats
24        - hostPath:
25            path: /var/lib/lxcfs/proc/meminfo
26            type: ""
27          name: lxcfs-proc-meminfo
28        - hostPath:
29            path: /var/lib/lxcfs/proc/stat
30            type: ""
31          name: lxcfs-proc-stat
32        - hostPath:
33            path: /var/lib/lxcfs/proc/swaps
34            type: ""
35          name: lxcfs-proc-swaps
36        - hostPath:
37            path: /var/lib/lxcfs/proc/uptime
38            type: ""
39          name: lxcfs-proc-uptime
40        - hostPath:
41            path: /var/lib/lxcfs/proc/loadavg
42            type: ""
43          name: lxcfs-proc-loadavg
44        - hostPath:
45            path: /var/lib/lxcfs/sys/devices/system/cpu/online
46            type: ""
47          name: lxcfs-sys-devices-system-cpu-online
48      containers:
49        - name: web
50          image: httpd:2.4.32
51          imagePullPolicy: Always
52          resources:
53            requests:
54              memory: "256Mi"
55              cpu: "500m"
56            limits:
57              memory: "256Mi"
58              cpu: "500m"
59          volumeMounts:
60            - mountPath: /proc/cpuinfo
61              name: lxcfs-proc-cpuinfo
62              readOnly: true
63            - mountPath: /proc/meminfo
64              name: lxcfs-proc-meminfo
65              readOnly: true
66            - mountPath: /proc/diskstats
67              name: lxcfs-proc-diskstats
68              readOnly: true
69            - mountPath: /proc/stat
70              name: lxcfs-proc-stat
71              readOnly: true
72            - mountPath: /proc/swaps
73              name: lxcfs-proc-swaps
74              readOnly: true
75            - mountPath: /proc/uptime
76              name: lxcfs-proc-uptime
77              readOnly: true
78            - mountPath: /proc/loadavg
79              name: lxcfs-proc-loadavg
80              readOnly: true
81            - mountPath: /sys/devices/system/cpu/online
82              name: lxcfs-sys-devices-system-cpu-online
83              readOnly: true

这样pod通过lxcfs实现了容器资源视图隔离。

但这里有一个问题一个两个容器这样复制粘贴设置还能接受，成千上万和容器这种重复操作，作为追求KISS原则的你肯定不能忍。

那有没有办法解决呢？我们可以通过实现 admission-webhook （准入控制 Admission Control）在授权后对请求做进一步的验证或添加默认参数。我们想到的前辈们都已经实现，就不用重复造轮子了。可以参考 lxcfs-admission-webhook

lxcfs-admission-webhook 注入实现容器自动挂载`/proc`、`/sys/`

lxcfs-admission-webhook实现了一个动态的准入webhook，更准确的讲是实现了一个修改性质的webhook，即监听pod的创建，然后对pod执行patch的操作，从而将lxcfs与容器内的目录映射关系植入到pod创建的yaml中从而实现自动挂载。

使用上也比较KISS，只用在资源文件里加一条注解即可。

下面我们看看怎么玩

1. 准备`lxcfs-admission-webhook`镜像

go build 二进制

1git clone git@github.com:denverdino/lxcfs-admission-webhook.git
2cd lxcfs-admission-webhook
3
4# build lxcfs-admission-webhook，因为是老的go项目需要转成支持go mod
5export GOPROXY=https://goproxy.cn,direct
6go mod init v1
7go mody tidy
8CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o lxcfs-admission-webhook
9
10chmod +x lxcfs-admission-webhook
11

Dockerfile

1FROM alpine:latest
2
3ADD lxcfs-admission-webhook /lxcfs-admission-webhook
4ENTRYPOINT ["./lxcfs-admission-webhook"]

构建镜像

1docker build -t yourharbor.domain.com/alpine/lxcfs-admission-webhook:v1 .
2
3docker push yourharbor.domain.com/alpine/lxcfs-admission-webhook:v1

2. 运行`lxcfs-admission-webhook` pod

每个集群都有自己的CA证书，所以不同集群部署lxcfs-admission-webhook，先做如下操作再应用yaml

2.1 目录结构

1tree .                                  
2.
3├── dp.yaml    #lxcfs-admission-webhook deployment
4├── mutatingwebhook.yaml   #MutatingWebhookConfiguration
5└── svc.yaml   #webhook svc
6└── webhook-create-signed-cert.sh  #创建`lxcfs-admission-webhook`依赖证书

2.2 修改`webhook-create-signed-cert.sh`

注：由于k8s版本较新，lxcfs-admission-webhook近几年没有更新，所以适配新版本k8s修改了github上的k8s的证书生成脚本webhook-create-signed-cert.sh

1#!/bin/bash
2
3set -e
4
5usage() {
6    cat <<EOF
7Generate certificate suitable for use with an sidecar-injector webhook service.
8
9This script uses k8s' CertificateSigningRequest API to a generate a
10certificate signed by k8s CA suitable for use with sidecar-injector webhook
11services. This requires permissions to create and approve CSR. See
12https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster for
13detailed explantion and additional instructions.
14
15The server key/cert k8s CA cert are stored in a k8s secret.
16
17usage: ${0} [OPTIONS]
18
19The following flags are required.
20
21       --service          Service name of webhook.
22       --namespace        Namespace where webhook service and secret reside.
23       --secret           Secret name for CA certificate and server certificate/key pair.
24EOF
25    exit 1
26}
27
28while [[ $# -gt 0 ]]; do
29    case ${1} in
30        --service)
31            service="$2"
32            shift
33            ;;
34        --secret)
35            secret="$2"
36            shift
37            ;;
38        --namespace)
39            namespace="$2"
40            shift
41            ;;
42        *)
43            usage
44            ;;
45    esac
46    shift
47done
48
49[ -z ${service} ] && service=lxcfs-admission-webhook-svc
50[ -z ${secret} ] && secret=lxcfs-admission-webhook-certs
51[ -z ${namespace} ] && namespace=default
52
53if [ ! -x "$(command -v openssl)" ]; then
54    echo "openssl not found"
55    exit 1
56fi
57
58csrName=${service}.${namespace}
59tmpdir=$(mktemp -d)
60echo "creating certs in tmpdir ${tmpdir} "
61
62cat <<EOF >> ${tmpdir}/csr.conf
63[req]
64req_extensions = v3_req
65distinguished_name = req_distinguished_name
66[req_distinguished_name]
67[ v3_req ]
68basicConstraints = CA:FALSE
69keyUsage = nonRepudiation, digitalSignature, keyEncipherment
70extendedKeyUsage = serverAuth
71subjectAltName = @alt_names
72[alt_names]
73DNS.1 = ${service}
74DNS.2 = ${service}.${namespace}
75DNS.3 = ${service}.${namespace}.svc
76EOF
77
78openssl genrsa -out ${tmpdir}/server-key.pem 2048
79#openssl req -new -key ${tmpdir}/server-key.pem -subj "/CN=${service}.${namespace}.svc" -out ${tmpdir}/server.csr -config ${tmpdir}/csr.conf
80openssl req -new -key ${tmpdir}/server-key.pem -subj "/CN=system:node:${service}.${namespace}.svc;/O=system:nodes" -out ${tmpdir}/server.csr -config ${tmpdir}/csr.conf
81
82# clean-up any previously created CSR for our service. Ignore errors if not present.
83kubectl delete csr ${csrName} -n ${namespace} 2>/dev/null || true
84
85# create  server cert/key CSR and  send to k8s API
86cat <<EOF | kubectl -n ${namespace} create -f -
87apiVersion: certificates.k8s.io/v1
88kind: CertificateSigningRequest
89metadata:
90  name: ${csrName}
91spec:
92  groups:
93  - system:authenticated
94  signerName: kubernetes.io/kubelet-serving
95  request: $(cat ${tmpdir}/server.csr | base64 | tr -d '\n')
96  usages:
97  - digital signature
98  - key encipherment
99  - server auth
100EOF
101
102# verify CSR has been created
103while true; do
104    kubectl get csr ${csrName}
105    if [ "$?" -eq 0 ]; then
106        break
107    fi
108done
109
110# approve and fetch the signed certificate
111kubectl certificate approve ${csrName}
112# verify certificate has been signed
113for x in $(seq 10); do
114    serverCert=$(kubectl get csr ${csrName} -o jsonpath='{.status.certificate}')
115    if [[ ${serverCert} != '' ]]; then
116        break
117    fi
118    sleep 1
119done
120if [[ ${serverCert} == '' ]]; then
121    echo "ERROR: After approving csr ${csrName}, the signed certificate did not appear on the resource. Giving up after 10 attempts." >&2
122    exit 1
123fi
124echo ${serverCert} | openssl base64 -d -A -out ${tmpdir}/server-cert.pem
125
126
127# create the secret with CA cert and server cert/key
128kubectl create secret generic ${secret} \
129        --from-file=key.pem=${tmpdir}/server-key.pem \
130        --from-file=cert.pem=${tmpdir}/server-cert.pem \
131        --dry-run -o yaml |
132    kubectl -n ${namespace} apply -f -

修改了证书请求命令/CN=system:node:${service}.${namespace}.svc;/O=system:nodes 和修改了--namespace 的bug

然后在k8s master 节点上运行 kubectl create ns lxcfs ; sh webhook-create-signed-cert.sh --namespace lxcfs

2.2 获取集群CA证书内容

kubectl config view --raw --flatten --minify -o jsonpath='{.clusters[].cluster.certificate-authority-data}'

2.3 更新CA证书内容到`mutatingwebhook.yaml` `caBundle` 字段

1apiVersion: admissionregistration.k8s.io/v1beta1
2kind: MutatingWebhookConfiguration
3metadata:
4  name: mutating-lxcfs-admission-webhook-cfg
5  labels:
6    app: lxcfs-admission-webhook
7webhooks:
8  - name: mutating.lxcfs-admission-webhook.aliyun.com
9    clientConfig:
10      service:
11        name: lxcfs-admission-webhook-svc
12        namespace: default
13        path: "/mutate"
14      caBundle: LS0tLS1CRUdJxiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMvakNDQWVhZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJek1EY3hOekEwTXpNek5Gb1hEVE16TURjeE5EQTBNek16TkZvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTlVZCjd4SThpcXZtbEtNN0FDTUFDY0huRWxxTXgyakR1b3JkWk81cUNGYTBNalROOXNqZHhUbHNNTlMrUHpuOUxPSkMKZ2d5TW90MGNPaW0zQTd2bllRYzFCY2I3UHFLOGpjS0U2a0E5MWVyNlpNSHU0c3ZXRXEybjVyMlIvcnY5NUR2eQpIRzlzTUJnenQrWUFJNlR6OGJNazhnMzJZR1BJejEvTTJmalBCa292bVJ3U0c1UkVIYWVFNW1TdDBRMnJheGJQCmtEU0pDSEErVlV3QThuekpFRVpwdkIxbUZ6MytXKzhrOUpIYlFtSW40TzhNaCtYYXlGc2Vab2g5SC9kVERkSXUKN0JXVG5pcmg5YkNWZzJhSDJidG03ZVpSY2s1V3IrM0QxcmUrc1FxWnpVdlhFSzBQYTk4MENGd3BYTVhsenlFdQpqNkhQRjZzOUhmV0gxOVdJMUdrQ0F3RUFBYU5aTUZjd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZBQVVicWVyaklyUDRmOFV0ZjErUzRERzVSWStNQlVHQTFVZEVRUU8KTUF5Q0NtdDFZbVZ5Ym1WMFpYTXdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBTGx0OHBELzVtMnhVclJSdUJIdQpaODFKbnpDSzB6Y2ZhbHRROXFiWkFQb2syT1R6eTQrclh6SHQ4VzVHN01YVmN6TXVoZnh0OXFSeWVLekM3bmtICnpJSnIxcmxPbkkwaXdNcHJFeDlNQkpBTnBNdWNwN3ljaE82RGlOQ01ocFAwMXdDbWVENTBsVUladlIrMHhUbHEKaGVZdTFZS3Eza3Q0dzNuWVUxUGszUGU1Q3NweFNqd0NKNVF0RHpyUFY4bE5JaHNMZjRHV2U2bDN0N2J5ck9wWApsUWJiMXovazNRTDRTU3pqcEdkQVRmUnVmRmsrbk1RVkFCSmJwVWp5aHNFMlg1TjRvLzlKWFVpZVhLNlYxOHNiCnVtVUlLYlkySGIyTHNISXEveTBHeHpITnpGTndEeEdGNnNSWFF5SkFYVS9tekNWRWczbEhaWUlpUU9wdkc2VdfsZXFVPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0twx==
15    rules:
16      - operations: [ "CREATE" ]
17        apiGroups: ["core", ""]
18        apiVersions: ["v1"]
19        resources: ["pods"]
20    namespaceSelector:
21      matchLabels:
22        lxcfs-admission-webhook: enabled

2.4 `lxcfs-admission-webhook` 的 `dp.yaml`

1apiVersion: apps/v1
2kind: Deployment
3metadata:
4  name: lxcfs-admission-webhook-deployment
5  labels:
6    app: lxcfs-admission-webhook
7  namespace: lxcfs
8spec:
9  replicas: 1
10  selector:
11    matchLabels:
12      app: lxcfs-admission-webhook
13  template:
14    metadata:
15      labels:
16        app: lxcfs-admission-webhook
17    spec:
18      imagePullSecrets:
19        - name: your-docker-token
20      containers:
21        - name: lxcfs-admission-webhook
22          image: yourharbor.domain.com/alpine/lxcfs-admission-webhook:v1
23          imagePullPolicy: IfNotPresent
24          args:
25            - -tlsCertFile=/etc/webhook/certs/cert.pem
26            - -tlsKeyFile=/etc/webhook/certs/key.pem
27            - -alsologtostderr
28            - -v=4
29            - 2>&1
30          volumeMounts:
31            - name: webhook-certs
32              mountPath: /etc/webhook/certs
33              readOnly: true
34      volumes:
35        - name: webhook-certs
36          secret:
37            secretName: lxcfs-admission-webhook-certs

2.5 svc.yaml

1apiVersion: v1
2kind: Service
3metadata:
4  namespace: lxcfs
5  name: lxcfs-admission-webhook-svc
6  labels:
7    app: lxcfs-admission-webhook
8spec:
9  ports:
10  - port: 443
11    targetPort: 443
12  selector:
13    app: lxcfs-admission-webhook

3.验证，应用注解能力

给default namespace 开启lxcfs能力

kubectl label namespace default lxcfs-admission-webhook=enabled

部署deployment

1cd lxcfs-admission-webhook
2kubectl apply -f deployment/web.yaml

登录容器执行free

1$ kubectl get pod
2
3NAME                                                 READY   STATUS    RESTARTS   AGE
4lxcfs-admission-webhook-deployment-f4bdd6f66-5wrlg   1/1     Running   0          8m29s
5lxcfs-pqs2d                                          1/1     Running   0          55m
6lxcfs-zfh99                                          1/1     Running   0          55m
7web-7c5464f6b9-6zxdf                                 1/1     Running   0          8m10s
8web-7c5464f6b9-nktff                                 1/1     Running   0          8m10s
9
10$ kubectl exec -ti web-7c5464f6b9-6zxdf sh
11# free
12             total       used       free     shared    buffers     cached
13Mem:        262144       2744     259400          0          0        312
14-/+ buffers/cache:       2432     259712
15Swap:            0          0          0
16#

总结

这里强调一下，我们实现的是容器资源视图和物理机资源视图的隔离，而非pod的。

容器资源视图隔离后，视觉上舒服很多，对定位问题，服务启动，网络安全上都有很大帮助，行动起来吧。关注DevOpSec每周分享干货内容，我们一起进步。

lxcfs容器资源视图隔离 for k8s

背景

`LXCFS` 工作原理

`LXCFS` 机器上部署

a. 安装 `lxcfs`

b. 启动`lxcfs`

c. 测试运行容器

`LXCFS` k8s 环境运行

a. 构建`lxcfs`镜像

a.1 目录结构

a.2 Dockerfile

a.3 build.sh 构建镜像

b. 运行`lxcfs` `daemonset` `yaml`

c. 验证 deployment pod yaml 定义

lxcfs-admission-webhook 注入实现容器自动挂载`/proc`、`/sys/`

1. 准备`lxcfs-admission-webhook`镜像

2. 运行`lxcfs-admission-webhook` pod

2.1 目录结构

2.2 修改`webhook-create-signed-cert.sh`

2.2 获取集群CA证书内容

2.3 更新CA证书内容到`mutatingwebhook.yaml` `caBundle` 字段

2.4 `lxcfs-admission-webhook` 的 `dp.yaml`

2.5 svc.yaml

3.验证，应用注解能力

总结

评论区

相关推荐

lxcfs容器资源视图隔离 for k8s

背景

LXCFS 工作原理

LXCFS 机器上部署

a. 安装 lxcfs

b. 启动lxcfs

c. 测试运行容器

LXCFS k8s 环境运行

a. 构建lxcfs镜像

a.1 目录结构

a.2 Dockerfile

a.3 build.sh 构建镜像

b. 运行lxcfs daemonset yaml

c. 验证 deployment pod yaml 定义

lxcfs-admission-webhook 注入实现容器自动挂载/proc、/sys/

1. 准备lxcfs-admission-webhook镜像

2. 运行lxcfs-admission-webhook pod

2.1 目录结构

2.2 修改webhook-create-signed-cert.sh

2.2 获取集群CA证书内容

2.3 更新CA证书内容到mutatingwebhook.yaml caBundle 字段

2.4 lxcfs-admission-webhook 的 dp.yaml

2.5 svc.yaml

3.验证，应用注解能力

总结

评论区

相关推荐

`LXCFS` 工作原理

`LXCFS` 机器上部署

a. 安装 `lxcfs`

b. 启动`lxcfs`

`LXCFS` k8s 环境运行

a. 构建`lxcfs`镜像

b. 运行`lxcfs` `daemonset` `yaml`

lxcfs-admission-webhook 注入实现容器自动挂载`/proc`、`/sys/`

1. 准备`lxcfs-admission-webhook`镜像

2. 运行`lxcfs-admission-webhook` pod

2.2 修改`webhook-create-signed-cert.sh`

2.3 更新CA证书内容到`mutatingwebhook.yaml` `caBundle` 字段

2.4 `lxcfs-admission-webhook` 的 `dp.yaml`