Prometheus部署+邮箱报警+企业微信报警+钉钉报警

1 部署Prometheus server

1.1 下载二进制包

$ wget https://github.com/prometheus/prometheus/releases/download/v2.12.0/prometheus-2.12.0.linux-amd64.tar.gz

1.2 解压并move至/work/admin目录下

1$ tar zcvf prometheus-2.12.0.linux-amd64.tar.gz
2
3$ mv prometheus-2.12.0.linux-amd64 /work/admin/prometheus

1.3 配置并启动

1$ cat prometheus.yml
2# my global config
3global:
4  scrape_interval:     15s # 默认抓取间隔, 15秒向目标抓取一次数据。
5  scrape_timeout: 15s
6  evaluation_interval: 20s # Evaluate rules every 15 seconds. The default is every 1 minute.
7  # scrape_timeout is set to the global default (10s).
8
9# Alertmanager configuration
10alerting:
11  alertmanagers:
12  - static_configs:
13    - targets: ['localhost:9093']
14       #- alertmanager: ['localhost:9093']
15
16# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
17rule_files:
18  - "/work/admin/prometheus/alerts/*.rules"
19  # - "first_rules.yml"
20  # - "second_rules.yml"
21
22# A scrape configuration containing exactly one endpoint to scrape:
23# Here it's Prometheus itself.
24scrape_configs:
25  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
26  - job_name: 'prometheus_product'
27
28    # metrics_path defaults to '/metrics'
29    # scheme defaults to 'http'.
30    #metrics_path: /metrics
31    #scheme: http
32
33    static_configs:
34    - targets: ['localhost:9090','localhost:9100']
35      labels: {cluster: 'product',type: 'basic',env: 'prometheus',job: 'prometheus',export: 'prometheus'}
36
37$ /work/admin/prometheus/prometheus --config.file=/work/admin/prometheus/prometheus.yml --storage.tsdb.path=/work/admin/prometheus/data

2 部署node_exporter

2.1 下载二进制包

$ wget https://github.com/prometheus/node_exporter/releases/download/v0.17.0/node_exporter-0.17.0.linux-amd64.tar.gz

2.2 解压并move至/work/admin目录下

1$ tar zcvf node_exporter-0.17.0.linux-amd64.tar.gz
2
3$ mv node_exporter-0.17.0.linux-amd64 /work/admin/node_exporter

2.3 启动

$ /work/admin/node_exporter/node_exporter --web.listen-address=:9100

3 部署alertmanager

3.1 下载二进制包

$ wget https://github.com/prometheus/alertmanager/releases/download/v0.18.0/alertmanager-0.18.0.linux-amd64.tar.gz

3.2 解压并move至/work/admin目录下

1$ tar zcvf alertmanager-0.18.0.linux-amd64.tar.gz
2
3$ mv alertmanager-0.18.0.linux-amd64 /work/admin/alertmanager

3.3 修改配置文件并启动

1$ cat alertmanager.yml
2
3global:
4
5  resolve_timeout: 5m
6
7  smtp_smarthost: 'smtp.163.com:25' # 邮箱smtp服务器代理
8
9  smtp_from: 'XXXXXX@163.com' # 发送邮箱名称
10
11  smtp_auth_username: 'XXXXX@163.com' # 邮箱名称
12
13  smtp_auth_password: 'XXXXXXXX' # 邮箱密码或授权码
14
15templates:
16
17  - 'template/*.tmpl'
18
19route:
20
21  group_by: ['alertname']
22
23  group_wait: 10s
24
25  group_interval: 10s
26
27  repeat_interval: 24h
28
29  receiver: 'ops_dingding'
30
31receivers:
32
33  - name: 'email'
34
35    email_configs:
36
37    - to: 'XXXXX@163.com'  # 接收警报的email配置
38
39      html: '{{ template "test.html" . }}' # 设定邮箱的内容模板
40
41      headers: { Subject: "[WARN] 报警邮件"} # 接收邮件的标题
42
43  - name: 'wechat'
44
45    wechat_configs:
46
47    - corp_id: 'XXXXX'
48
49      to_party: '1'
50
51      agent_id: '1000002'
52
53      api_secret: 'XXXXX'
54
55  - name: 'ops_dingding'
56
57    webhook_configs:
58
59    - url: 'http://localhost:8060/dingtalk/ops_dingding/send'
60
61inhibit_rules:
62
63  - source_match:
64
65      severity: 'critical'
66
67    target_match:
68
69      severity: 'warning'
70
71    equal: ['alertname', 'dev', 'instance']
72
73$ /work/admin/alertmanager/alertmanager --config.file=/work/admin/alertmanager/alertmanager.yml

4 prometheus通过webhook推送告警至钉钉

4.1 添加钉钉机器人，获取webhook

参考 https://open-doc.dingtalk.com/docs/doc.htm?treeId=257&articleId=105735&docType=1

4.2 下载插件（二进制文件）

$ wget https://github.com/timonwong/prometheus-webhook-dingtalk/releases/download/v0.3.0/prometheus-webhook-dingtalk-0.3.0.linux-amd64.tar.gz

4.3 解压并move至/usr/local/prometheus目录下

1$ tar zxvf prometheus-webhook-dingtalk-0.3.0.linux-amd64.tar.gz
2
3$ mv prometheus-webhook-dingtalk-0.3.0.linux-amd64/prometheus-webhook-dingtalk /work/admin/alertmanager

4.4 编辑启动脚本（请替换为自己的webhook URL 及 ding.profile）

1$ cat dingding_start.sh
2
3nohup /work/admin/alertmanager/prometheus-webhook-dingtalk --ding.profile="ops_dingding=https://oapi.dingtalk.com/robot/send?access_token=XXXXXXX"  2>&1 1>/work/admin/alertmanager/dingding.log &
4
5$ sh dingding_start.sh

4.5 编辑alertmanager.yml，增加web_hook配置并重启alertmanager

1  - name: 'ops_dingding'
2
3    webhook_configs:
4
5    - url: 'http://localhost:8060/dingtalk/ops_dingding/send'