Kubernetes 增强了应用服务的横向扩容能力，在应对线上应用服务的资源使用率在高峰和低谷的时候，我们需要能够自动去感知应用的负载变化去调整 Pod 的副本数量，削峰填谷，提高集群的整体资源利用率和应用的服务质量。为此，Kubernetes 1.2 版本中引入 Horizontal Pod Autoscaling (HPA), 它与 kubectl scale 命令相似，作为 Pod 水平自动缩放的实现。
工作机制
kubernetes 通过指标适配器获取 Pod 的资源使用情况，根据内存、CPU 或自定义度量指标自动扩缩 ReplicationController、Deployment、ReplicaSet 和 StatefulSet 中的 Pod 数量。
Horizontal Pod Autoscaler 由一个控制循环实现，循环周期由 kube-controller- manager 中的 --horizontal-pod-autoscaler-sync-period 参数指定（默认是 15 秒）。 在每个周期内，kube-controller- manager 会查询 HorizontalPodAutoscaler 中定义的指标度量值，并且与创建时设定的值和指标度量值做对比，从而实现自动伸缩的功能。
API对象
HorizontalPodAutoscaler 是 Kubernetes autoscaling API 组的资源。
1[root@k8s-test-master01 ~]# kubectl api-versions | grep autoscaling 2autoscaling/v1 3autoscaling/v2beta1 4autoscaling/v2beta2
当前稳定版本（autoscaling/v1）中只支持基于 CPU 指标的扩缩。
beta 版本（autoscaling/v2beta2）引入了基于内存和自定义指标的扩缩。
Aggregator API HPA 依赖指标适配器(如 metrics-server)，要安装指标适配器需要开启 Aggregator ，Kubeadm 搭建的集群默认已经开启，如果是二进制的方式搭建的集群，需要配置kube-apiserver kube-controller-manager 。
1 --requestheader-allowed-names="front-proxy-client" \ 2 --requestheader-client-ca-file=/etc/kubernetes/pki/ca.crt \ 3 --requestheader-extra-headers-prefix="X-Remote-Extra-" \ 4 --requestheader-group-headers=X-Remote-Group \ 5 --requestheader-username-headers=X-Remote-User \ 6 --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt \ 7 --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key \
注: requestheader-allowed-names 需要与证书定义的 CN 值一致。
Metrics API
HorizontalPodAutoscaler 控制器会从 Metrics API 中检索度量值。
metrics.k8s.io 资源指标 API，一般由 metrics-server 提供。
custom.metrics.k8s.io 自定义指标 API，一般由 prometheus-adapter 提供。
external.metrics.k8s.io 外部指标 API，一般由自定义指标适配器提供。
安装指标适配器
metrics-server 安装很简单，kubernetes 源码中提供了 yaml 清单。
1# kubernetes 源码地址 cluster/addons/metrics-server 2[root@k8s-test-master01 metrics-server]# ll 3total 32 4-rw-rw-r-- 1 root root 398 Jan 13 21:19 auth-delegator.yaml 5-rw-rw-r-- 1 root root 419 Jan 13 21:19 auth-reader.yaml 6-rw-rw-r-- 1 root root 388 Jan 13 21:19 metrics-apiservice.yaml 7-rw-rw-r-- 1 root root 3352 Jan 13 21:19 metrics-server-deployment.yaml 8-rw-rw-r-- 1 root root 336 Jan 13 21:19 metrics-server-service.yaml 9-rw-rw-r-- 1 root root 188 Jan 13 21:19 OWNERS 10-rw-rw-r-- 1 root root 1227 Jan 13 21:19 README.md 11-rw-rw-r-- 1 root root 844 Jan 13 21:19 resource-reader.yaml 12[root@k8s-test-master01 metrics-server]# kubectl create -f . 13 14# 或 https://github.com/kubernetes-sigs/metrics-server/tree/master/manifests/base 15[root@k8s-test-master01 base]# ls -lrt 16total 20 17-rw-r--r-- 1 root root 1714 Feb 17 11:00 rbac.yaml 18-rw-r--r-- 1 root root 185 Feb 17 11:00 kustomization.yaml.bak 19-rw-r--r-- 1 root root 293 Feb 17 11:00 apiservice.yaml 20-rw-r--r-- 1 root root 2163 Feb 17 11:07 deployment.yaml 21-rw-r--r-- 1 root root 216 Feb 17 11:21 service.yaml
安装完成验证，正常能获取到资源使用情况。
1[root@k8s-test-master01 ~]# kubectl top nodes 2NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% 3k8s-test-master01 273m 6% 4027Mi 52% 4k8s-test-node01 207m 5% 2361Mi 30% 5k8s-test-node02 180m 4% 1833Mi 23% 6kubeedge-raspberrypi01 195m 4% 719Mi 19% 7[root@k8s-test-master01 ~]#
Prometheus-adapter 顾名思义是基于普罗米修斯的自定义指标适配器，需要先安装普罗米修斯，如集群已经有普罗米修斯了，可以通过 prometheus.url prometheus.port 参数指定即可。
1# 创建 namespace 2kubectl create ns monitoring 3# 安装 prometheus 4helm repo add prometheus-community https://prometheus-community.github.io/helm-charts 5helm search repo prometheus-community 6helm install my-release prometheus-community/prometheus --namespace monitoring --values https://bit.ly/2RgzDtg --version 13.2.1 \ 7--set alertmanager.persistentVolume.enabled=false \ 8--set server.persistentVolume.enabled=false 9 10# 导出 prometheus-adapter yaml 清单，方便后续添加自定义指标 11helm template my-adapter prometheus-community/prometheus-adapter --namespace monitoring \ 12--set prometheus.url=http://my-release-prometheus-server \ 13--set prometheus.port=80 \ 14--set rules.default=true >adapter.yaml 15 16# 安装 prometheus-adapter 17kubectl create -f adapter.yaml
安装完成验证
1[root@k8s-test-master01 ~]# kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq . 2{ 3 "kind": "APIResourceList", 4 "apiVersion": "v1", 5 "groupVersion": "custom.metrics.k8s.io/v1beta1", 6 "resources": [ 7 { 8 "name": "namespaces/kube_statefulset_status_observed_generation", 9 "singularName": "", 10 "namespaced": false, 11 "kind": "MetricValueList", 12 "verbs": [ 13 "get" 14 ] 15 }, 16...
基于 CPU
使用 Deployment 来创建一个测试的 Pod，然后利用 HPA 来进行自动扩缩容。
1--- 2apiVersion: apps/v1 3kind: Deployment 4metadata: 5 name: demo 6spec: 7 selector: 8 matchLabels: 9 app: demo 10 replicas: 1 11 template: 12 metadata: 13 labels: 14 app: demo 15 spec: 16 containers: 17 - name: demo 18 image: docker.io/library/debian:stable-slim 19 command: 20 - sleep 21 - "3600" 22 resources: 23 requests: 24 memory: "100Mi" 25 cpu: "100m" 26 limits: 27 memory: "200Mi" 28 cpu: "200m"
创建
1[root@k8s-test-master01 demo]# kubectl create -f demo.yaml 2deployment.apps/demo created 3[root@k8s-test-master01 demo]# kubectl get pods -l app=demo 4NAME READY STATUS RESTARTS AGE 5demo-575ff999f8-dfs5s 1/1 Running 0 94s
创建 hpa
1[root@k8s-test-master01 demo]# cat demo-cpu-hpa.yaml 2apiVersion: autoscaling/v1 3kind: HorizontalPodAutoscaler 4metadata: 5 name: demo-cpu 6spec: 7 scaleTargetRef: 8 apiVersion: apps/v1 9 kind: Deployment 10 name: demo 11 minReplicas: 1 12 maxReplicas: 6 13 targetCPUUtilizationPercentage: 100 14[root@k8s-test-master01 demo]# kubectl create -f demo-cpu-hpa.yaml 15horizontalpodautoscaler.autoscaling/demo-cpu created 16[root@k8s-test-master01 demo]# kubectl get hpa 17NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE 18demo-cpu Deployment/demo 0%/100% 1 6 1 12s 19 20# 或使用 kubectl 命令创建 21kubectl autoscale deployment demo --cpu-percent=100 --min=1 --max=6
使用一个 for 循环消耗 CPU
1[root@k8s-test-master01 ~]# kubectl top pod 2[root@k8s-test-master01 demo]# kubectl exec -it demo-575ff999f8-ckpbg -- bash 3root@demo-575ff999f8-ckpbg:/# x=0 4root@demo-575ff999f8-ckpbg:/# while [ True ];do x=$x+1;done;
查看 hpa
1[root@k8s-test-master01 ~]# kubectl get hpa 2NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE 3demo-cpu Deployment/demo 201%/100% 1 6 2 2m53s 4[root@k8s-test-master01 ~]# kubectl get hpa 5NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE 6demo-cpu Deployment/demo 100%/100% 1 6 2 3m8s 7[root@k8s-test-master01 ~]#
可以看到已经扩容了 2 个 Pod ，我们定义了最大值是 6 ，那为什么只扩容了 2 个 pod 呢？ 这个就涉及到了 hpa 的算法细节了，官方给出的公式是
期望副本数 = ceil[当前副本数 * (当前指标 / 期望指标)]
因为当前度量值为 200，目标设定值为 100，那么 200/100 == 2，副本数量将会翻倍。当 pod 的数量为 2 时，平均的值为 100
1[root@k8s-test-master01 ~]# kubectl top po 2NAME CPU(cores) MEMORY(bytes) 3demo-575ff999f8-ckpbg 201m 2Mi 4demo-575ff999f8-gvq5l 0m 0Mi
在另外一个 pod 也执行下 for 循环后，查看 hpa
1[root@k8s-test-master01 ~]# kubectl get hpa 2NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE 3demo-cpu Deployment/demo 100%/100% 1 6 4 9m54s 4[root@k8s-test-master01 ~]# kubectl top pod 5NAME CPU(cores) MEMORY(bytes) 6demo-575ff999f8-ckpbg 200m 3Mi 7demo-575ff999f8-gvq5l 201m 2Mi 8demo-575ff999f8-q4zbj 0m 0Mi 9demo-575ff999f8-xblz6 0m 0Mi
换一句话说，如果 pod 当前度量的平均值大于 hpa 设定值，则自动扩容，直到平均值小于设定值。实际上不会出现这种极端的情况，kubernetes 有负载均衡。感兴趣的读者可以去试试。
1[root@k8s-test-master01 ~]# kubectl describe hpa demo-cpu 2Name: demo-cpu 3Namespace: default 4Labels: <none> 5Annotations: <none> 6CreationTimestamp: Sat, 27 Feb 2021 18:18:44 +0800 7Reference: Deployment/demo 8Metrics: ( current / target ) 9 resource cpu on pods (as a percentage of request): 0% (0) / 100% 10Min replicas: 1 11Max replicas: 6 12Deployment pods: 1 current / 1 desired 13Conditions: 14 Type Status Reason Message 15 ---- ------ ------ ------- 16 AbleToScale True ReadyForNewScale recommended size matches current size 17 ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request) 18 ScalingLimited True TooFewReplicas the desired replica count is less than the minimum replica count 19Events: 20 Type Reason Age From Message 21 ---- ------ ---- ---- ------- 22 Warning FailedGetResourceMetric 20m (x5 over 20m) horizontal-pod-autoscaler failed to get cpu utilization: did not receive metrics for any ready pods 23 Warning FailedComputeMetricsReplicas 20m (x5 over 20m) horizontal-pod-autoscaler invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: did not receive metrics for any ready pods 24 Normal SuccessfulRescale 18m horizontal-pod-autoscaler New size: 2; reason: cpu resource utilization (percentage of request) above target 25 Normal SuccessfulRescale 12m horizontal-pod-autoscaler New size: 3; reason: cpu resource utilization (percentage of request) above target 26 Normal SuccessfulRescale 12m horizontal-pod-autoscaler New size: 4; reason: cpu resource utilization (percentage of request) above target 27 Normal SuccessfulRescale 108s horizontal-pod-autoscaler New size: 2; reason: All metrics below target 28 Normal SuccessfulRescale 98s horizontal-pod-autoscaler New size: 1; reason: All metrics below target
基于内存
使用 beta API 创建 hpa
1[root@k8s-test-master01 demo]# cat demo-men-hpa.yaml 2apiVersion: autoscaling/v2beta2 3kind: HorizontalPodAutoscaler 4metadata: 5 name: demo-mem 6 namespace: default 7spec: 8 scaleTargetRef: 9 apiVersion: apps/v1 10 kind: Deployment 11 name: demo 12 minReplicas: 1 13 maxReplicas: 6 14 metrics: 15 - type: Resource 16 resource: 17 name: memory 18 target: 19 type: AverageValue 20 averageValue: 10Mi 21[root@k8s-test-master01 demo]# kubectl get hpa 22NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE 23demo-mem Deployment/demo 823296/100Mi 1 6 1 27s 24[root@k8s-test-master01 demo]#
与 CPU 差不多，这里就不展开了。
1[root@k8s-test-master01 demo]# kubectl get hpa 2NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE 3demo-mem Deployment/demo 14737408/10Mi 1 6 2 6m10s
基于自定义指标
除了基于 CPU 和内存来进行自动扩缩容之外，还可以使用 Prometheus Adapter 获取普罗米修斯收集的指标并使用来设置扩展策略。
写一个程序，获取 http 请求连接数等指标。
完整代码：
https://github.com/prodanlabs/kubernetes-hpa-examples
1package prometheus 2 3import ( 4 "strconv" 5 "time" 6 7 "github.com/kataras/iris/v12" 8 "github.com/prometheus/client_golang/prometheus" 9) 10 11const ( 12 reqsName = "http_requests_total" 13 latencyName = "http_request_duration_seconds" 14 connectionsName = "tcp_connections_total" 15) 16 17type Prometheus struct { 18 reqs *prometheus.CounterVec 19 latency *prometheus.HistogramVec 20 connections *prometheus.GaugeVec 21}
示例程序 yaml
1[root@k8s-test-master01 demo]# cat hpa-examples.yaml 2--- 3apiVersion: apps/v1 4kind: Deployment 5metadata: 6 name: hpa-examples 7spec: 8 selector: 9 matchLabels: 10 app: hpa-examples 11 replicas: 2 12 template: 13 metadata: 14 labels: 15 app: hpa-examples 16 annotations: 17 prometheus.io/port: "8080" 18 prometheus.io/scrape: "true" 19 spec: 20 containers: 21 - name: hpa-examples 22 image: prodan/kubernetes-hpa-examples:latest 23 env: 24 - name: POD_NAME 25 valueFrom: 26 fieldRef: 27 fieldPath: metadata.name 28 - name: POD_NAMESPACE 29 valueFrom: 30 fieldRef: 31 fieldPath: metadata.namespace 32 ports: 33 - containerPort: 8080 34 protocol: TCP 35 resources: 36 requests: 37 memory: "50Mi" 38 cpu: "100m" 39 limits: 40 memory: "256Mi" 41 cpu: "500m" 42--- 43apiVersion: v1 44kind: Service 45metadata: 46 name: hpa-examples 47 labels: 48 app: hpa-examples 49spec: 50 ports: 51 - port: 8080 52 targetPort: 8080 53 protocol: TCP 54 selector: 55 app: hpa-examples
确认是否接入普罗米修斯
prometheus-adapter 的 configmap 加入下列配置
1 - seriesQuery: '{__name__=~"^http_requests_.*",kubernetes_pod_name!="",kubernetes_namespace!=""}' 2 seriesFilters: [] 3 resources: 4 overrides: 5 kubernetes_namespace: 6 resource: namespace 7 kubernetes_pod_name: 8 resource: pod 9 name: 10 matches: ^(.*)_(total)$ 11 as: "${1}" 12 metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>)
查看是否能获取到自定义指标
1[root@k8s-test-master01 demo]# kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/http_requests" | jq . 2{ 3 "kind": "MetricValueList", 4 "apiVersion": "custom.metrics.k8s.io/v1beta1", 5 "metadata": { 6 "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/http_requests" 7 }, 8 "items": [ 9 { 10 "describedObject": { 11 "kind": "Pod", 12 "namespace": "default", 13 "name": "hpa-examples-954c4fb6c-h7j2g", 14 "apiVersion": "/v1" 15 }, 16 "metricName": "http_requests", 17 "timestamp": "2021-02-27T12:03:56Z", 18 "value": "100m", 19 "selector": null 20 }, 21 { 22 "describedObject": { 23 "kind": "Pod", 24 "namespace": "default", 25 "name": "hpa-examples-954c4fb6c-vxnrb", 26 "apiVersion": "/v1" 27 }, 28 "metricName": "http_requests", 29 "timestamp": "2021-02-27T12:03:56Z", 30 "value": "100m", 31 "selector": null 32 } 33 ] 34}
创建hpa
1--- 2apiVersion: autoscaling/v2beta2 3kind: HorizontalPodAutoscaler 4metadata: 5 name: hpa-examples-requests 6spec: 7 scaleTargetRef: 8 apiVersion: apps/v1 9 kind: Deployment 10 name: hpa-examples 11 minReplicas: 2 12 maxReplicas: 6 13 metrics: 14 - type: Pods 15 pods: 16 metric: 17 name: http_requests 18 target: 19 type: AverageValue 20 averageValue: 5 21[root@k8s-test-master01 demo]# kubectl get hpa 22NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE 23hpa-examples-requests Deployment/hpa-examples 100m/5 2 6 2 25s
测试
1kubectl run -i --tty load-generator --rm --image=busybox --restart=Never -- /bin/sh -c "while sleep 0.01; do wget -q -O- http://hpa-examples:8080/api/v1/hostname; done"
查看 hpa 扩容过程
1[root@k8s-test-master01 demo]# kubectl get hpa 2NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE 3hpa-examples-requests Deployment/hpa-examples 14982m/5 2 6 6 3m18s 4[root@k8s-test-master01 demo]# kubectl describe hpa hpa-examples-requests 5Name: hpa-examples-requests 6Namespace: default 7Labels: <none> 8Annotations: <none> 9CreationTimestamp: Sat, 27 Feb 2021 20:29:27 +0800 10Reference: Deployment/hpa-examples 11Metrics: ( current / target ) 12 "http_requests" on pods: 15280m / 5 13Min replicas: 2 14Max replicas: 6 15Deployment pods: 6 current / 6 desired 16Conditions: 17 Type Status Reason Message 18 ---- ------ ------ ------- 19 AbleToScale True ReadyForNewScale recommended size matches current size 20 ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from pods metric http_requests 21 ScalingLimited True TooManyReplicas the desired replica count is more than the maximum replica count 22Events: 23 Type Reason Age From Message 24 ---- ------ ---- ---- ------- 25 Normal SuccessfulRescale 82s horizontal-pod-autoscaler New size: 4; reason: pods metric http_requests above target 26 Normal SuccessfulRescale 72s horizontal-pod-autoscaler New size: 6; reason: pods metric http_requests above target 27[root@k8s-test-master01 demo]# kubectl get po -l app=hpa-examples 28NAME READY STATUS RESTARTS AGE 29hpa-examples-954c4fb6c-4rlvx 1/1 Running 0 74s 30hpa-examples-954c4fb6c-9fz2f 1/1 Running 0 74s 31hpa-examples-954c4fb6c-h7j2g 1/1 Running 0 35m 32hpa-examples-954c4fb6c-s95lf 1/1 Running 0 84s 33hpa-examples-954c4fb6c-vxnrb 1/1 Running 0 35m 34hpa-examples-954c4fb6c-zlp5b 1/1 Running 0 84s
参考文档：
https://github.com/kubernetes-sigs/prometheus-adapter https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/
感兴趣的读者可以关注下微信号
