prometheus监控k8s集群
- 具体版本
Prometheus:v2.2.1
kubernetes:v1.18.9
Grafana:latest
alertmanager:v0.14.0
metrics:v1.3.0
Prometheus是继Kubernetes项目以后CNCF基金会第二个托管项目，最初建于SoundClound，是一个开源的系统监控和警报工具包，独立于任何公司的一个独立的开源项目
详细介绍请查看官方文档：https://prometheus.io/docs/introduction/overview/
实现思路：
各node_exporter收集的信息Push到Prometheus，集群部署prometheus进行服务发现和采集信息，然后经过Prometheus进行处理后，Grafana进行效果展示，Altermanager进行报警管理
prometheus监控k8s架构
prometheus采用的是StatefulSet有状态部署，我们需要提前搭建好NFS存储，采用动态PV的存储给prometheus；
官方文档地址：https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config
官方部署文档：
https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/prometheus
https://grafana.com/grafana/download
每个node节点都执行
yum install nfs-common nfs-utils -y
promethues-rbac.yaml
1[root@master prometheus-k8s]# cat prometheus-rbac.yaml 2apiVersion: v1 3kind: ServiceAccount 4metadata: 5 name: prometheus 6 namespace: kube-system 7 labels: 8 kubernetes.io/cluster-service: "true" 9 addonmanager.kubernetes.io/mode: Reconcile 10--- 11apiVersion: rbac.authorization.k8s.io/v1beta1 12kind: ClusterRole 13metadata: 14 name: prometheus 15 labels: 16 kubernetes.io/cluster-service: "true" 17 addonmanager.kubernetes.io/mode: Reconcile 18rules: 19 - apiGroups: 20 - "" 21 resources: 22 - nodes 23 - nodes/metrics 24 - services 25 - endpoints 26 - pods 27 verbs: 28 - get 29 - list 30 - watch 31 - apiGroups: 32 - "" 33 resources: 34 - configmaps 35 verbs: 36 - get 37 - nonResourceURLs: 38 - "/metrics" 39 verbs: 40 - get 41--- 42apiVersion: rbac.authorization.k8s.io/v1beta1 43kind: ClusterRoleBinding 44metadata: 45 name: prometheus 46 labels: 47 kubernetes.io/cluster-service: "true" 48 addonmanager.kubernetes.io/mode: Reconcile 49roleRef: 50 apiGroup: rbac.authorization.k8s.io 51 kind: ClusterRole 52 name: prometheus 53subjects: 54- kind: ServiceAccount 55 name: prometheus 56 namespace: kube-system 57 58[root@master1 prometheus]# kubectl apply -f prometheus-rbac.yaml
prometheus-configmap.yaml
1root@master prometheus-k8s]# cat prometheus-configmap.yaml 2# Prometheus configuration format https://prometheus.io/docs/prometheus/latest/configuration/configuration/ 3apiVersion: v1 4kind: ConfigMap 5metadata: 6 name: prometheus-config 7 namespace: kube-system 8 labels: 9 kubernetes.io/cluster-service: "true" 10 addonmanager.kubernetes.io/mode: EnsureExists 11data: 12 prometheus.yml: | 13 rule_files: 14 - /etc/config/rules/*.rules 15 16 scrape_configs: 17 - job_name: prometheus 18 static_configs: 19 - targets: 20 - localhost:9090 21 22 - job_name: kubernetes-nodes 23 scrape_interval: 30s 24 static_configs: 25 - targets: 26 - 172.16.75.2:9100 27 - 172.16.75.3:9100 28 29 - job_name: kubernetes-apiservers 30 kubernetes_sd_configs: 31 - role: endpoints 32 relabel_configs: 33 - action: keep 34 regex: default;kubernetes;https 35 source_labels: 36 - __meta_kubernetes_namespace 37 - __meta_kubernetes_service_name 38 - __meta_kubernetes_endpoint_port_name 39 scheme: https 40 tls_config: 41 ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt 42 insecure_skip_verify: true 43 bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token 44 45 - job_name: kubernetes-nodes-kubelet 46 kubernetes_sd_configs: 47 - role: node 48 relabel_configs: 49 - action: labelmap 50 regex: __meta_kubernetes_node_label_(.+) 51 scheme: https 52 tls_config: 53 ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt 54 insecure_skip_verify: true 55 bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token 56 57 - job_name: kubernetes-nodes-cadvisor 58 kubernetes_sd_configs: 59 - role: node 60 relabel_configs: 61 - action: labelmap 62 regex: __meta_kubernetes_node_label_(.+) 63 - target_label: __metrics_path__ 64 replacement: /metrics/cadvisor 65 scheme: https 66 tls_config: 67 ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt 68 insecure_skip_verify: true 69 bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token 70 71 - job_name: kubernetes-service-endpoints 72 kubernetes_sd_configs: 73 - role: endpoints 74 relabel_configs: 75 - action: keep 76 regex: true 77 source_labels: 78 - __meta_kubernetes_service_annotation_prometheus_io_scrape 79 - action: replace 80 regex: (https?) 81 source_labels: 82 - __meta_kubernetes_service_annotation_prometheus_io_scheme 83 target_label: __scheme__ 84 - action: replace 85 regex: (.+) 86 source_labels: 87 - __meta_kubernetes_service_annotation_prometheus_io_path 88 target_label: __metrics_path__ 89 - action: replace 90 regex: ([^:]+)(?::\d+)?;(\d+) 91 replacement: $1:$2 92 source_labels: 93 - __address__ 94 - __meta_kubernetes_service_annotation_prometheus_io_port 95 target_label: __address__ 96 - action: labelmap 97 regex: __meta_kubernetes_service_label_(.+) 98 - action: replace 99 source_labels: 100 - __meta_kubernetes_namespace 101 target_label: kubernetes_namespace 102 - action: replace 103 source_labels: 104 - __meta_kubernetes_service_name 105 target_label: kubernetes_name 106 107 - job_name: kubernetes-services 108 kubernetes_sd_configs: 109 - role: service 110 metrics_path: /probe 111 params: 112 module: 113 - http_2xx 114 relabel_configs: 115 - action: keep 116 regex: true 117 source_labels: 118 - __meta_kubernetes_service_annotation_prometheus_io_probe 119 - source_labels: 120 - __address__ 121 target_label: __param_target 122 - replacement: blackbox 123 target_label: __address__ 124 - source_labels: 125 - __param_target 126 target_label: instance 127 - action: labelmap 128 regex: __meta_kubernetes_service_label_(.+) 129 - source_labels: 130 - __meta_kubernetes_namespace 131 target_label: kubernetes_namespace 132 - source_labels: 133 - __meta_kubernetes_service_name 134 target_label: kubernetes_name 135 136 - job_name: kubernetes-pods 137 kubernetes_sd_configs: 138 - role: pod 139 relabel_configs: 140 - action: keep 141 regex: true 142 source_labels: 143 - __meta_kubernetes_pod_annotation_prometheus_io_scrape 144 - action: replace 145 regex: (.+) 146 source_labels: 147 - __meta_kubernetes_pod_annotation_prometheus_io_path 148 target_label: __metrics_path__ 149 - action: replace 150 regex: ([^:]+)(?::\d+)?;(\d+) 151 replacement: $1:$2 152 source_labels: 153 - __address__ 154 - __meta_kubernetes_pod_annotation_prometheus_io_port 155 target_label: __address__ 156 - action: labelmap 157 regex: __meta_kubernetes_pod_label_(.+) 158 - action: replace 159 source_labels: 160 - __meta_kubernetes_namespace 161 target_label: kubernetes_namespace 162 - action: replace 163 source_labels: 164 - __meta_kubernetes_pod_name 165 target_label: kubernetes_pod_name 166 alerting: 167 alertmanagers: 168 - static_configs: 169 - targets: ["alertmanager:80"] 170[root@master1 prometheus]# kubectl apply -f prometheus-configmap.yaml
prometheus-rules.yaml
1[root@master prometheus-k8s]# cat prometheus-rules.yaml 2apiVersion: v1 3kind: ConfigMap 4metadata: 5 name: prometheus-rules 6 namespace: kube-system 7data: 8 general.rules: | 9 groups: 10 - name: general.rules 11 rules: 12 - alert: InstanceDown 13 expr: up == 0 14 for: 1m 15 labels: 16 severity: error 17 annotations: 18 summary: "Instance {{ $labels.instance }} 停止工作" 19 description: "{{ $labels.instance }} job {{ $labels.job }} 已经停止5分钟以上." 20 node.rules: | 21 groups: 22 - name: node.rules 23 rules: 24 - alert: NodeFilesystemUsage 25 expr: 100 - (node_filesystem_free_bytes{fstype=~"ext4|xfs"} / node_filesystem_size_bytes{fstype=~"ext4|xfs"} * 100) > 80 26 for: 1m 27 labels: 28 severity: warning 29 annotations: 30 summary: "Instance {{ $labels.instance }} : {{ $labels.mountpoint }} 分区使用率过高" 31 description: "{{ $labels.instance }}: {{ $labels.mountpoint }} 分区使用大于80% (当前值: {{ $value }})" 32 33 - alert: NodeMemoryUsage 34 expr: 100 - (node_memory_MemFree_bytes+node_memory_Cached_bytes+node_memory_Buffers_bytes) / node_memory_MemTotal_bytes * 100 > 80 35 for: 1m 36 labels: 37 severity: warning 38 annotations: 39 summary: "Instance {{ $labels.instance }} 内存使用率过高" 40 description: "{{ $labels.instance }}内存使用大于80% (当前值: {{ $value }})" 41 42 - alert: NodeCPUUsage 43 expr: 100 - (avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) by (instance) * 100) > 60 44 for: 1m 45 labels: 46 severity: warning 47 annotations: 48 summary: "Instance {{ $labels.instance }} CPU使用率过高" 49 description: "{{ $labels.instance }}CPU使用大于60% (当前值: {{ $value }})" 50[root@master1 prometheus]# kubectl apply -f prometheus-rules.yaml 51[root@master1 opt]# kubectl get sc -A -o wide 52NAME PROVISIONER AGE 53managed-nfs-storage fuseim.pri/ifs 6d
prometheus-statefulset.yaml
1[root@master prometheus-k8s]# cat prometheus-statefulset.yaml 2apiVersion: apps/v1 3kind: StatefulSet 4metadata: 5 name: prometheus 6 namespace: kube-system 7 labels: 8 k8s-app: prometheus 9 kubernetes.io/cluster-service: "true" 10 addonmanager.kubernetes.io/mode: Reconcile 11 version: v2.2.1 12spec: 13 serviceName: "prometheus" 14 replicas: 1 15 podManagementPolicy: "Parallel" 16 updateStrategy: 17 type: "RollingUpdate" 18 selector: 19 matchLabels: 20 k8s-app: prometheus 21 template: 22 metadata: 23 labels: 24 k8s-app: prometheus 25 annotations: 26 scheduler.alpha.kubernetes.io/critical-pod: '' 27 spec: 28 priorityClassName: system-cluster-critical 29 serviceAccountName: prometheus 30 initContainers: 31 - name: "init-chown-data" 32 image: "busybox:latest" 33 imagePullPolicy: "IfNotPresent" 34 command: ["chown", "-R", "65534:65534", "/data"] 35 volumeMounts: 36 - name: prometheus-data 37 mountPath: /data 38 subPath: "" 39 containers: 40 - name: prometheus-server-configmap-reload 41 image: "jimmidyson/configmap-reload:v0.1" 42 imagePullPolicy: "IfNotPresent" 43 args: 44 - --volume-dir=/etc/config 45 - --webhook-url=http://localhost:9090/-/reload 46 volumeMounts: 47 - name: config-volume 48 mountPath: /etc/config 49 readOnly: true 50 resources: 51 limits: 52 cpu: 10m 53 memory: 10Mi 54 requests: 55 cpu: 10m 56 memory: 10Mi 57 58 - name: prometheus-server 59 image: "prom/prometheus:v2.2.1" 60 imagePullPolicy: "IfNotPresent" 61 args: 62 - --config.file=/etc/config/prometheus.yml 63 - --storage.tsdb.path=/data 64 - --web.console.libraries=/etc/prometheus/console_libraries 65 - --web.console.templates=/etc/prometheus/consoles 66 - --web.enable-lifecycle 67 ports: 68 - containerPort: 9090 69 readinessProbe: 70 httpGet: 71 path: /-/ready 72 port: 9090 73 initialDelaySeconds: 30 74 timeoutSeconds: 30 75 livenessProbe: 76 httpGet: 77 path: /-/healthy 78 port: 9090 79 initialDelaySeconds: 30 80 timeoutSeconds: 30 81 # based on 10 running nodes with 30 pods each 82 resources: 83 limits: 84 cpu: 200m 85 memory: 1000Mi 86 requests: 87 cpu: 200m 88 memory: 1000Mi 89 90 volumeMounts: 91 - name: config-volume 92 mountPath: /etc/config 93 - name: prometheus-data 94 mountPath: /data 95 subPath: "" 96 - name: prometheus-rules 97 mountPath: /etc/config/rules 98 99 terminationGracePeriodSeconds: 300 100 volumes: 101 - name: config-volume 102 configMap: 103 name: prometheus-config 104 - name: prometheus-rules 105 configMap: 106 name: prometheus-rules 107 108 volumeClaimTemplates: 109 - metadata: 110 name: prometheus-data 111 spec: 112 storageClassName: managed-nfs-storage 113 accessModes: 114 - ReadWriteOnce 115 resources: 116 requests: 117 storage: "16Gi" 118 119[root@master1 prometheus]# kubectl apply -f prometheus-statefulset.yaml 120[root@master1 prometheus]# kubectl get sts -A 121NAMESPACE NAME READY AGE 122kube-system prometheus 0/1 14s 123[root@master1 prometheus]# kubectl get sts -A -o wide 124NAMESPACE NAME READY AGE CONTAINERS IMAGES 125kube-system prometheus 0/1 25s prometheus-server-configmap-reload,prometheus-server jimmidyson/configmap-reload:v0.1,prom/prometheus:v2.2.1
prometheus-service.yaml
1[root@master prometheus-k8s]# cat prometheus-service.yaml 2kind: Service 3apiVersion: v1 4metadata: 5 name: prometheus 6 namespace: kube-system 7 labels: 8 kubernetes.io/name: "Prometheus" 9 kubernetes.io/cluster-service: "true" 10 addonmanager.kubernetes.io/mode: Reconcile 11spec: 12 type: NodePort 13 ports: 14 - name: http 15 port: 9090 16 protocol: TCP 17 targetPort: 9090 18 nodePort: 30090 19 selector: 20 k8s-app: prometheus 21[root@master1 prometheus]# kubectl apply -f prometheus-service.yaml 22service/prometheus created
-
监控Pod性能，通过cADvisor监控，暴露的metrics的10250端口
[root@master prometheus-k8s]# for i in
ls kube-state-metrics-*; do ls $i && cat $i ; done kube-state-metrics-deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: kube-state-metrics namespace: kube-system labels: k8s-app: kube-state-metrics kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile version: v1.3.0 spec: selector: matchLabels: k8s-app: kube-state-metrics version: v1.3.0 replicas: 1 template: metadata: labels: k8s-app: kube-state-metrics version: v1.3.0 annotations: scheduler.alpha.kubernetes.io/critical-pod: '' spec: priorityClassName: system-cluster-critical serviceAccountName: kube-state-metrics containers: - name: kube-state-metrics image: lizhenliang/kube-state-metrics:v1.3.0 ports: - name: http-metrics containerPort: 8080 - name: telemetry containerPort: 8081 readinessProbe: httpGet: path: /healthz port: 8080 initialDelaySeconds: 5 timeoutSeconds: 5 - name: addon-resizer image: lizhenliang/addon-resizer:1.8.3 resources: limits: cpu: 100m memory: 30Mi requests: cpu: 100m memory: 30Mi env: - name: MY_POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: MY_POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace volumeMounts: - name: config-volume mountPath: /etc/config command: - /pod_nanny - --config-dir=/etc/config - --container=kube-state-metrics - --cpu=100m - --extra-cpu=1m - --memory=100Mi - --extra-memory=2Mi - --threshold=5 - --deployment=kube-state-metrics volumes: - name: config-volume configMap: name: kube-state-metrics-configConfig map for resource configuration.
apiVersion: v1 kind: ConfigMap metadata: name: kube-state-metrics-config namespace: kube-system labels: k8s-app: kube-state-metrics kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile data: NannyConfiguration: |- apiVersion: nannyconfig/v1alpha1 kind: NannyConfiguration
kube-state-metrics-rbac.yaml apiVersion: v1 kind: ServiceAccount metadata: name: kube-state-metrics namespace: kube-system labels: kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: kube-state-metrics labels: kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile rules:
- apiGroups: [""]
resources:
- configmaps
- secrets
- nodes
- pods
- services
- resourcequotas
- replicationcontrollers
- limitranges
- persistentvolumeclaims
- persistentvolumes
- namespaces
- endpoints verbs: ["list", "watch"]
- apiGroups: ["extensions"]
resources:
- daemonsets
- deployments
- replicasets verbs: ["list", "watch"]
- apiGroups: ["apps"]
resources:
- statefulsets verbs: ["list", "watch"]
- apiGroups: ["batch"]
resources:
- cronjobs
- jobs verbs: ["list", "watch"]
- apiGroups: ["autoscaling"]
resources:
- horizontalpodautoscalers verbs: ["list", "watch"]
apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: kube-state-metrics-resizer namespace: kube-system labels: kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile rules:
- apiGroups: [""]
resources:
- pods verbs: ["get"]
- apiGroups: ["extensions"]
resources:
- deployments resourceNames: ["kube-state-metrics"] verbs: ["get", "update"]
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: kube-state-metrics labels: kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: kube-state-metrics subjects:
- kind: ServiceAccount name: kube-state-metrics namespace: kube-system
apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: kube-state-metrics namespace: kube-system labels: kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: kube-state-metrics-resizer subjects:
- kind: ServiceAccount
name: kube-state-metrics
namespace: kube-system
kube-state-metrics-service.yaml
apiVersion: v1
kind: Service
metadata:
name: kube-state-metrics
namespace: kube-system
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
kubernetes.io/name: "kube-state-metrics"
annotations:
prometheus.io/scrape: 'true'
spec:
ports:
- name: http-metrics port: 8080 targetPort: http-metrics protocol: TCP
- name: telemetry port: 8081 targetPort: telemetry protocol: TCP selector: k8s-app: kube-state-metrics
- apiGroups: [""]
resources:
grafana.yaml
1[root@master prometheus-k8s]# cat grafana.yaml 2apiVersion: apps/v1 3kind: StatefulSet 4metadata: 5 name: grafana 6 namespace: kube-system 7spec: 8 serviceName: "grafana" 9 replicas: 1 10 selector: 11 matchLabels: 12 app: grafana 13 template: 14 metadata: 15 labels: 16 app: grafana 17 spec: 18 containers: 19 - name: grafana 20 image: grafana/grafana 21 ports: 22 - containerPort: 3000 23 protocol: TCP 24 resources: 25 limits: 26 cpu: 100m 27 memory: 256Mi 28 requests: 29 cpu: 100m 30 memory: 256Mi 31 volumeMounts: 32 - name: grafana-data 33 mountPath: /var/lib/grafana 34 subPath: grafana 35 securityContext: 36 fsGroup: 472 37 runAsUser: 472 38 volumeClaimTemplates: 39 - metadata: 40 name: grafana-data 41 spec: 42 storageClassName: managed-nfs-storage 43 accessModes: 44 - ReadWriteOnce 45 resources: 46 requests: 47 storage: "1Gi" 48 49--- 50 51apiVersion: v1 52kind: Service 53metadata: 54 name: grafana 55 namespace: kube-system 56spec: 57 type: NodePort 58 ports: 59 - port : 80 60 targetPort: 3000 61 nodePort: 30007 62 selector: 63 app: grafana 64[root@master1 prometheus]# kubectl apply -f grafana.yaml 65statefulset.apps/grafana created 66service/grafana created 67[root@master1 prometheus]# kubectl get svc -n kube-system 68NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE 69grafana NodePort 10.0.0.120 <none> 80:30007/TCP 4s 70kube-dns ClusterIP 10.0.0.2 <none> 53/UDP,53/TCP 15d 71kuboard NodePort 10.0.0.174 <none> 80:32567/TCP 15d 72metrics-server ClusterIP 10.0.0.52 <none> 443/TCP 15d 73prometheus NodePort 10.0.0.86 <none> 9090:30090/TCP 22m
AlterManager报警
1[root@master prometheus-k8s]# for i in `ls alertmanager-*`; do ls $i && cat $i ; done 2alertmanager-configmap.yaml 3apiVersion: v1 4kind: ConfigMap 5metadata: 6 name: alertmanager-config 7 namespace: kube-system 8 labels: 9 kubernetes.io/cluster-service: "true" 10 addonmanager.kubernetes.io/mode: EnsureExists 11data: 12 alertmanager.yml: | 13 global: 14 resolve_timeout: 5m 15 smtp_smarthost: 'smtp.163.com:25' 16 smtp_from: 'ljy_153@163.com' 17 smtp_auth_username: 'ljy_153@163.com' 18 smtp_auth_password: '************' 19 20 receivers: 21 - name: default-receiver 22 email_configs: 23 - to: "ljy_153@163.com" 24 25 route: 26 group_interval: 1m 27 group_wait: 10s 28 receiver: default-receiver 29 repeat_interval: 1m 30alertmanager-deployment.yaml 31apiVersion: apps/v1 32kind: Deployment 33metadata: 34 name: alertmanager 35 namespace: kube-system 36 labels: 37 k8s-app: alertmanager 38 kubernetes.io/cluster-service: "true" 39 addonmanager.kubernetes.io/mode: Reconcile 40 version: v0.14.0 41spec: 42 replicas: 1 43 selector: 44 matchLabels: 45 k8s-app: alertmanager 46 version: v0.14.0 47 template: 48 metadata: 49 labels: 50 k8s-app: alertmanager 51 version: v0.14.0 52 annotations: 53 scheduler.alpha.kubernetes.io/critical-pod: '' 54 spec: 55 priorityClassName: system-cluster-critical 56 containers: 57 - name: prometheus-alertmanager 58 image: "prom/alertmanager:v0.14.0" 59 imagePullPolicy: "IfNotPresent" 60 args: 61 - --config.file=/etc/config/alertmanager.yml 62 - --storage.path=/data 63 - --web.external-url=/ 64 ports: 65 - containerPort: 9093 66 readinessProbe: 67 httpGet: 68 path: /#/status 69 port: 9093 70 initialDelaySeconds: 30 71 timeoutSeconds: 30 72 volumeMounts: 73 - name: config-volume 74 mountPath: /etc/config 75 - name: storage-volume 76 mountPath: "/data" 77 subPath: "" 78 resources: 79 limits: 80 cpu: 10m 81 memory: 50Mi 82 requests: 83 cpu: 10m 84 memory: 50Mi 85 - name: prometheus-alertmanager-configmap-reload 86 image: "jimmidyson/configmap-reload:v0.1" 87 imagePullPolicy: "IfNotPresent" 88 args: 89 - --volume-dir=/etc/config 90 - --webhook-url=http://localhost:9093/-/reload 91 volumeMounts: 92 - name: config-volume 93 mountPath: /etc/config 94 readOnly: true 95 resources: 96 limits: 97 cpu: 10m 98 memory: 10Mi 99 requests: 100 cpu: 10m 101 memory: 10Mi 102 volumes: 103 - name: config-volume 104 configMap: 105 name: alertmanager-config 106 - name: storage-volume 107 persistentVolumeClaim: 108 claimName: alertmanager 109alertmanager-pvc.yaml 110apiVersion: v1 111kind: PersistentVolumeClaim 112metadata: 113 name: alertmanager 114 namespace: kube-system 115 labels: 116 kubernetes.io/cluster-service: "true" 117 addonmanager.kubernetes.io/mode: EnsureExists 118spec: 119 storageClassName: managed-nfs-storage 120 accessModes: 121 - ReadWriteOnce 122 resources: 123 requests: 124 storage: "2Gi" 125alertmanager-service.yaml 126apiVersion: v1 127kind: Service 128metadata: 129 name: alertmanager 130 namespace: kube-system 131 labels: 132 kubernetes.io/cluster-service: "true" 133 addonmanager.kubernetes.io/mode: Reconcile 134 kubernetes.io/name: "Alertmanager" 135spec: 136 ports: 137 - name: http 138 port: 80 139 protocol: TCP 140 targetPort: 9093 141 selector: 142 k8s-app: alertmanager 143 type: "ClusterIP" 144 145[root@master1 prometheus]# kubectl apply -f alertmanager-deployment.yaml 146deployment.apps/alertmanager created 147[root@master1 prometheus]# kubectl apply -f alertmanager-configmap.yaml 148configmap/alertmanager-config created 149[root@master1 prometheus]# kubectl apply -f alertmanager-pvc.yaml 150persistentvolumeclaim/alertmanager created 151[root@master1 prometheus]# kubectl apply -f alertmanager-service.yaml 152service/alertmanager created
Grafana展示：
-
连接prometheus
-
Dashboard
报警展示：
-
配置报警Rules，修改rule以后apply一下
-
邮箱展示
