监控告诉我们"发生了什么"，告警告诉我们"需要做什么"。一个好的告警体系不是让你手机震个不停，而是在正确的时间把正确的问题推给正确的人。
一、告警体系概述
一个完整的告警体系由以下五层组成：
数据采集 → 指标计算 → 规则评估 → 告警路由 → 响应处置
Prometheus PromQL AlertManager 通知渠道 Runbook/SOP
Telegraf recording 评估引擎 飞书/钉钉 自动化修复
Exporters rules Silences 邮件/SMS Postmortem核心目标：
告警必须可行动：每条告警都应关联一个明确的响应动作
告警必须分级：不同严重程度走不同通道、不同响应时效
告警必须降噪：噪声是告警体系的第一杀手
告警必须闭环：从触发到恢复，全链路可追溯
二、告警分级设计
2.1 P0-P4 五级告警体系
2.2 SLA 与告警关联
# Prometheus recording rules - SLA 计算
groups:
- name: sla_rules
interval: 1m
rules:
# 服务可用性 SLA（按月滚动）
- record: service:sla:availability_30d
expr: |
1 - (
sum_over_time(http_requests_total{status=~"5.."}[30d])
/
sum_over_time(http_requests_total[30d])
)
# 错误预算消耗速率
- record: service:error_budget_burn_rate
expr: |
(
1 - (
sum(rate(http_requests_total{status!~"5.."}[1h]))
/
sum(rate(http_requests_total[1h]))
)
) / (1 - 0.999) # 假设 SLA 目标 99.9%2.3 升级机制
┌─────────────┐ 15min未响应 ┌─────────────┐ 30min未响应 ┌─────────────┐
│ 初级值班 │ ──────────────→ │ 高级值班 │ ──────────────→ │ 值班经理 │
│ (一线) │ │ (二线) │ │ (三线) │
└─────────────┘ └─────────────┘ └─────────────┘
↑ ↑
│ P0直接通知 │ P0超30min直接
└─────────────────────────────────┘# AlertManager 升级配置示例
route:
receiver: 'oncall-primary'
group_wait: 30s
routes:
- match:
severity: critical
receiver: 'oncall-primary'
group_wait: 10s
routes:
- receiver: 'oncall-secondary'
matchers:
- ack = ""
repeat_interval: 15m
- receiver: 'oncall-manager'
matchers:
- ack = ""
repeat_interval: 30m三、告警规则设计
3.1 阈值告警（静态阈值）
最基础的告警方式，适合指标有明确安全边界的场景。
groups:
- name: host_alerts
rules:
# CPU 使用率告警
- alert: HighCpuUsage
expr: |
100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
for: 5m
labels:
severity: warning
team: infra
annotations:
summary: "CPU 使用率过高: {{ $labels.instance }}"
description: "CPU 使用率 {{ $value | printf \"%.1f\" }}%，已持续5分钟"
runbook: "https://wiki.internal/runbook/high-cpu"
# 磁盘空间告警
- alert: DiskSpaceRunningLow
expr: |
(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes) * 100 < 15
for: 10m
labels:
severity: warning
annotations:
summary: "磁盘空间不足: {{ $labels.instance }}:{{ $labels.mountpoint }}"
description: "剩余空间 {{ $value | printf \"%.1f\" }}%"
# OOM Kill 检测
- alert: OOMKillDetected
expr: increase(node_vmstat_oom_kill[5m]) > 0
labels:
severity: critical
annotations:
summary: "OOM Kill 发生: {{ $labels.instance }}"
description: "5分钟内发生 {{ $value }} 次 OOM Kill"3.2 趋势告警（基于变化率）
当指标本身不触发阈值，但恶化速度异常时，趋势告警就能派上用场。
# 内存泄漏检测：内存使用率持续上升
- alert: MemoryLeakSuspected
expr: |
deriv(node_memory_MemAvailable_bytes[30m]) < -104857600 # 30分钟下降>100MB
and
node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 0.3
for: 1h
labels:
severity: warning
annotations:
summary: "疑似内存泄漏: {{ $labels.instance }}"
description: "可用内存30分钟内下降超过100MB，当前剩余 {{ $value }}"
# 磁盘写入速率异常
- alert: DiskWriteRateSpike
expr: |
rate(node_disk_written_bytes_total[5m]) > 3 * rate(node_disk_written_bytes_total[1h] offset 1d)
for: 10m
labels:
severity: warning
annotations:
summary: "磁盘写入速率异常飙升: {{ $labels.device }}"3.3 同比/环比告警
适合有明显周期性规律的业务指标（如工作日流量、电商大促）。
# 环比：当前小时 vs 上一小时
- alert: TrafficDropHourOverHour
expr: |
sum(rate(http_requests_total[5m]))
<
0.5 * sum(rate(http_requests_total[5m] offset 1h))
for: 10m
labels:
severity: critical
annotations:
summary: "流量环比骤降50%"
description: "当前QPS {{ $value }}，1小时前为对照值的2倍以上"
# 同比：今天 vs 上周同一天
- alert: TrafficDropWeekOverWeek
expr: |
sum(rate(http_requests_total[5m]))
<
0.3 * sum(rate(http_requests_total[5m] offset 7d))
for: 15m
labels:
severity: warning
annotations:
summary: "流量同比大幅下降"3.4 异常检测告警（基于统计学）
用标准差/百分位数做动态阈值，比静态阈值更聪明。
groups:
- name: anomaly_detection
rules:
# 基于3-sigma的动态阈值告警
- alert: ResponseTimeAnomaly
expr: |
# 当前5分钟的p99延迟 > 过去24小时平均值 + 3倍标准差
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))
>
(
avg_over_time(
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))[24h:5m]
)
+
3 * stddev_over_time(
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))[24h:5m]
)
)
for: 10m
labels:
severity: warning
annotations:
summary: "P99延迟异常偏离: {{ $labels.service }}"
description: "当前P99 {{ $value }}s，超出历史3σ范围"
# 错误率异常（基于历史分位数）
- alert: ErrorRateAnomaly
expr: |
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))
>
3 * quantile_over_time(0.99,
(sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])))[7d:5m]
)
for: 5m
labels:
severity: critical
annotations:
summary: "错误率异常飙升，超过历史P99的3倍"四、告警降噪
告警降噪是告警体系能否长期存活的关键。没有降噪的告警系统，最终命运就是被告警疲劳淹没。
4.1 去重（Dedup）
相同告警在同一 group 中只发送一次，直到 resolve。
# AlertManager 默认就会对相同 labelset 去重
# 关键是控制 group_by 的粒度
route:
group_by: ['alertname', 'cluster', 'service']
group_wait: 30s
group_interval: 5m
repeat_interval: 4h4.2 聚合（Grouping）
把相关的多条告警打包成一条通知，减少通知数量。
route:
# 按服务维度聚合，同一服务的多条告警合并为一条通知
- match:
team: platform
group_by: ['cluster', 'service']
group_wait: 60s # 等60秒让告警聚拢
group_interval: 10m # 同组告警至少间隔10分钟
receiver: 'platform-oncall'4.3 抑制（Inhibition）
高优先级告警触发时，自动抑制低优先级的相关告警。
inhibit_rules:
# 当集群级别的告警触发时，抑制该集群下节点级别的告警
- source_matchers:
- severity = "critical"
- alertname = "ClusterDown"
target_matchers:
- severity =~ "warning|info"
equal: ['cluster']
# 当核心服务不可用时，抑制其性能告警
- source_matchers:
- alertname = "ServiceDown"
target_matchers:
- alertname =~ "High.*Latency|HighErrorRate"
equal: ['service']4.4 静默（Silences）
维护窗口、已知问题期间的临时告警屏蔽。
# 通过 amtool 创建静默（2小时维护窗口）
amtool silence add \
--author="ops-team" \
--comment="数据库维护窗口 2026-06-10 02:00-04:00" \
--duration=2h \
alertname=~"Database.*" \
cluster="prod-cn-east"
# 查看当前所有静默
amtool silence query
# 删除指定静默
amtool silence expire <silence-id>
# API 方式创建静默
curl -X POST http://alertmanager:9093/api/v2/silences \
-H "Content-Type: application/json" \
-d '{
"matchers": [
{"name": "service", "value": "payment", "isRegex": false}
],
"startsAt": "2026-06-10T02:00:00Z",
"endsAt": "2026-06-10T04:00:00Z",
"createdBy": "ops-bot",
"comment": "支付服务计划维护"
}'4.5 降噪进阶：标签规范化
# Prometheus relabel_configs - 统一标签，避免同类告警分裂
alerting:
alert_relabel_configs:
# 归一化 instance 标签
- source_labels: [instance]
regex: '(.+):\d+'
target_label: instance
# 移除高基数标签，避免告警分裂
- action: labeldrop
regex: 'request_id|trace_id'五、AlertManager 进阶配置
5.1 路由树设计
路由树是 AlertManager 的核心，决定了"什么样的告警发给谁"。
global:
resolve_timeout: 5m
smtp_from: 'alertmanager@company.com'
smtp_smarthost: 'smtp.company.com:587'
smtp_auth_username: 'alertmanager@company.com'
smtp_auth_password: '***'
slack_api_url: 'https://hooks.slack.com/services/xxx'
route:
# 默认路由
receiver: 'default-webhook'
group_by: ['alertname', 'cluster', 'namespace']
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
routes:
# P0 致命告警 - 立即通知所有渠道
- matchers:
- severity = "critical"
receiver: 'p0-pager'
group_wait: 10s
group_interval: 1m
repeat_interval: 5m
routes:
# 数据库告警 - 发给 DBA
- matchers:
- team = "dba"
receiver: 'dba-oncall'
# 网络告警 - 发给网络组
- matchers:
- team = "network"
receiver: 'network-oncall'
# P1 严重告警
- matchers:
- severity = "warning"
receiver: 'p1-webhook'
group_wait: 1m
repeat_interval: 1h
# P2-P4 低优先级 - 只发邮件和工单
- matchers:
- severity =~ "info|none"
receiver: 'low-priority-email'
repeat_interval: 12h
receivers:
- name: 'default-webhook'
webhook_configs:
- url: 'http://alert-gateway:8080/webhook'
send_resolved: true
- name: 'p0-pager'
webhook_configs:
- url: 'http://alert-gateway:8080/webhook?level=p0'
slack_configs:
- channel: '#alert-critical'
title: '🚨 P0 告警'
text: '{{ range .Alerts }}{{ .Annotations.summary }}\n{{ end }}'
email_configs:
- to: 'oncall@company.com'
headers:
Subject: '[P0] {{ .GroupLabels.alertname }}'
- name: 'p1-webhook'
webhook_configs:
- url: 'http://alert-gateway:8080/webhook?level=p1'
- name: 'low-priority-email'
email_configs:
- to: 'ops@company.com'
- name: 'dba-oncall'
webhook_configs:
- url: 'http://alert-gateway:8080/webhook?team=dba'
- name: 'network-oncall'
webhook_configs:
- url: 'http://alert-gateway:8080/webhook?team=network'5.2 告警分组策略
# 按不同维度分组，实现精细化路由
route:
routes:
# 基础设施告警 - 按集群+环境分组
- matchers:
- category = "infrastructure"
group_by: ['cluster', 'env']
receiver: 'infra-oncall'
# 应用告警 - 按服务+命名空间分组
- matchers:
- category = "application"
group_by: ['service', 'namespace']
receiver: 'app-oncall'
# 业务告警 - 按业务线分组
- matchers:
- category = "business"
group_by: ['business_line']
receiver: 'biz-oncall'5.3 告警模板
让告警通知更有信息量、更易读。
# /etc/alertmanager/templates/custom.tmpl
{{ define "custom.title" }}
[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}]
{{ .GroupLabels.SortedPairs.Values | join " " }}
{{ end }}
{{ define "custom.text" }}
{{ range .Alerts }}
━━━━━━━━━━━━━━━━━━━━
**告警名称**: {{ .Labels.alertname }}
**严重程度**: {{ .Labels.severity }}
**实例**: {{ .Labels.instance }}
**集群**: {{ .Labels.cluster }}
**状态**: {{ .Status }}
**开始时间**: {{ .StartsAt.Format "2006-01-02 15:04:05" }}
{{ if .EndsAt }}**恢复时间**: {{ .EndsAt.Format "2006-01-02 15:04:05" }}{{ end }}
**描述**: {{ .Annotations.description }}
{{ if .Annotations.runbook }}**Runbook**: {{ .Annotations.runbook }}{{ end }}
{{ end }}
━━━━━━━━━━━━━━━━━━━━
**聚合标签**: {{ .GroupLabels.SortedPairs.Values | join " / " }}
{{ end }}# 在 AlertManager 中引用模板
templates:
- '/etc/alertmanager/templates/*.tmpl'
receivers:
- name: 'slack-critical'
slack_configs:
- channel: '#alert-critical'
title: '{{ template "custom.title" . }}'
text: '{{ template "custom.text" . }}'
color: '{{ if eq .Status "firing" }}danger{{ else }}good{{ end }}'六、通知渠道集成
6.1 飞书（Feishu/Lark）
# 飞书自定义机器人 Webhook
cat > feishu-webhook.py << 'EOF'
import json, requests, sys
def send_feishu_alert(webhook_url, alert_data):
"""将 AlertManager webhook 转为飞书消息"""
for alert in alert_data.get('alerts', []):
status = alert.get('status', 'firing')
severity = alert.get('labels', {}).get('severity', 'unknown')
# 构造飞书卡片消息
card = {
"msg_type": "interactive",
"card": {
"header": {
"title": {
"tag": "plain_text",
"content": f"{'🔴' if status=='firing' else '🟢'} [{severity.upper()}] {alert['labels'].get('alertname', 'Unknown')}"
},
"template": "red" if severity == "critical" else "orange" if severity == "warning" else "blue"
},
"elements": [
{
"tag": "div",
"fields": [
{"is_short": True, "text": {"tag": "lark_md", "content": f"**实例**\n{alert['labels'].get('instance', 'N/A')}"}},
{"is_short": True, "text": {"tag": "lark_md", "content": f"**集群**\n{alert['labels'].get('cluster', 'N/A')}"}},
{"is_short": True, "text": {"tag": "lark_md", "content": f"**状态**\n{status}"}},
{"is_short": True, "text": {"tag": "lark_md", "content": f"**时间**\n{alert.get('startsAt', '')[:19]}"}}
]
},
{
"tag": "div",
"text": {"tag": "lark_md", "content": f"**描述**: {alert.get('annotations', {}).get('description', 'N/A')}"}
},
{
"tag": "action",
"actions": [
{
"tag": "button",
"text": {"tag": "plain_text", "content": "📋 Runbook"},
"url": alert.get('annotations', {}).get('runbook', '#'),
"type": "primary"
}
]
}
]
}
}
requests.post(webhook_url, json=card, timeout=5)
print(f"Sent alert: {alert['labels'].get('alertname')}")
if __name__ == '__main__':
webhook_url = sys.argv[1]
data = json.load(sys.stdin)
send_feishu_alert(webhook_url, data)
EOF
# AlertManager webhook receiver 配置
# 使用中间件转发
receivers:
- name: 'feishu-alerts'
webhook_configs:
- url: 'http://localhost:9095/alertmanager/feishu'
send_resolved: true6.2 钉钉（DingTalk）
# 钉钉机器人 Webhook 配置
# 由于钉钉不直接支持 AlertManager 格式，需要中间件转换
# 使用 prometheus-webhook-dingtalk
docker run -d --name dingtalk-hook \
-p 8060:8060 \
timonwong/prometheus-webhook-dingtalk \
--ding.profile="ops=https://oapi.dingtalk.com/robot/send?access_token=YOUR_TOKEN"
# dingtalk 配置文件 config.yml
cat > dingtalk-config.yml << 'EOF'
targets:
ops:
url: https://oapi.dingtalk.com/robot/send?access_token=xxxx
# 签名验证
secret: SECxxxxxxxx
message:
title: '{{ template "ding.link.title" . }}'
text: '{{ template "ding.link.content" . }}'
critical:
url: https://oapi.dingtalk.com/robot/send?access_token=yyyy
message:
title: '🚨 {{ template "ding.link.title" . }}'
EOF
# AlertManager 中使用
receivers:
- name: 'dingtalk-ops'
webhook_configs:
- url: 'http://dingtalk-hook:8060/dingtalk/ops/send'6.3 企业微信（WeCom）
# wecom-webhook.py - 企业微信告警转发
import json, requests, sys
WEBHOOK_BASE = "https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key={key}"
def send_wecom(key, alert_data):
webhook_url = WEBHOOK_BASE.format(key=key)
for alert in alert_data.get('alerts', []):
status = alert['status']
labels = alert.get('labels', {})
annotations = alert.get('annotations', {})
content = f"""## {'🔴 告警触发' if status == 'firing' else '🟢 告警恢复'}
> **告警名称**: {labels.get('alertname', 'N/A')}
> **严重程度**: {labels.get('severity', 'N/A')}
> **实例**: {labels.get('instance', 'N/A')}
> **集群**: {labels.get('cluster', 'N/A')}
> **时间**: {alert.get('startsAt', '')[:19]}
> **描述**: {annotations.get('description', 'N/A')}
---
[📋 查看Runbook]({annotations.get('runbook', '#')})"""
payload = {
"msgtype": "markdown",
"markdown": {"content": content}
}
resp = requests.post(webhook_url, json=payload, timeout=5)
print(f"WeCom response: {resp.json()}")
if __name__ == '__main__':
send_wecom(sys.argv[1], json.load(sys.stdin))6.4 Slack
receivers:
- name: 'slack-ops'
slack_configs:
- api_url: 'https://hooks.slack.com/services/T00/B00/xxx'
channel: '#ops-alerts'
title: '{{ template "slack.title" . }}'
text: '{{ template "slack.text" . }}'
color: '{{ if eq .Status "firing" }}danger{{ else }}good{{ end }}'
send_resolved: true
actions:
- type: button
text: 'Silence :no_bell:'
url: '{{ template "slack.silenceURL" . }}'
- type: button
text: 'Runbook :book:'
url: '{{ (index .Alerts 0).Annotations.runbook }}'6.5 通用 Webhook（万能适配）
receivers:
- name: 'universal-webhook'
webhook_configs:
- url: 'http://alert-gateway:8080/api/v1/alerts'
send_resolved: true
http_config:
authorization:
type: Bearer
credentials: 'your-api-token'
follow_redirects: true
max_alerts: 50# 通用 Webhook 接收端示例（Go）
# 可以对接任意自定义系统
cat > webhook-gateway.go << 'GOEOF'
package main
import (
"encoding/json"
"log"
"net/http"
"os"
"time"
)
type AlertGroup struct {
Status string `json:"status"`
Alerts []Alert `json:"alerts"`
GroupLabels map[string]string `json:"groupLabels"`
CommonLabels map[string]string `json:"commonLabels"`
}
type Alert struct {
Status string `json:"status"`
Labels map[string]string `json:"labels"`
Annotations map[string]string `json:"annotations"`
StartsAt time.Time `json:"startsAt"`
EndsAt time.Time `json:"endsAt"`
}
func alertHandler(w http.ResponseWriter, r *http.Request) {
var group AlertGroup
if err := json.NewDecoder(r.Body).Decode(&group); err != nil {
http.Error(w, err.Error(), 400)
return
}
// 分发到各渠道：飞书/钉钉/企微/Slack
for _, alert := range group.Alerts {
log.Printf("[%s] %s - %s", alert.Status,
alert.Labels["alertname"],
alert.Annotations["summary"])
}
w.WriteHeader(200)
}
func main() {
http.HandleFunc("/api/v1/alerts", alertHandler)
log.Fatal(http.ListenAndServe(":8080", nil))
}
GOEOF七、On-Call 轮值
7.1 轮值排班模型
# 建议使用 PagerDuty / OpsGenie / 自建排班系统
# 这里给出一个基于 cron 的轻量方案
# 值班表配置 (schedule.yml)
schedules:
- name: primary-oncall
timezone: Asia/Shanghai
rotations:
- name: weekly-rotation
start: "2026-01-05T00:00:00+08:00" # 从周一开始
period: weekly
participants:
- user: zhangsan
- user: lisi
- user: wangwu
- user: zhaoliu
handoff_time: "10:00" # 每周一10:00交接
- name: secondary-oncall
timezone: Asia/Shanghai
rotations:
- name: daily-rotation
start: "2026-01-05T00:00:00+08:00"
period: daily
participants:
- user: senior-a
- user: senior-b7.2 轮值自动化脚本
#!/bin/bash
# oncall-scheduler.sh - 每日值班通知脚本
# 放入 crontab: 0 9 * * * /opt/scripts/oncall-scheduler.sh
DATE=$(date +%Y-%m-%d)
DAY_OF_WEEK=$(date +%u)
# 值班人员池
PRIMARY_ONCALL=("张三" "李四" "王五" "赵六")
SECONDARY_ONCALL=("高级A" "高级B" "高级C")
# 按周轮转
PRIMARY_IDX=$(( ($(date +%s) / 86400 / 7) % ${#PRIMARY_ONCALL[@]} ))
SECONDARY_IDX=$(( ($(date +%s) / 86400) % ${#SECONDARY_ONCALL[@]} ))
PRIMARY=${PRIMARY_ONCALL[$PRIMARY_IDX]}
SECONDARY=${SECONDARY_ONCALL[$SECONDARY_IDX]}
# 发送飞书通知
curl -s -X POST "$FEISHU_WEBHOOK" \
-H "Content-Type: application/json" \
-d "{
\"msg_type\": \"interactive\",
\"card\": {
\"header\": {
\"title\": {\"tag\": \"plain_text\", \"content\": \"📅 今日值班通知 - $DATE\"},
\"template\": \"green\"
},
\"elements\": [
{\"tag\": \"div\", \"text\": {\"tag\": \"lark_md\", \"content\": \"**一线值班**: $PRIMARY\n**二线值班**: $SECONDARY\n\n📞 P0告警请电话联系一线值班人\"}}
]
}
}"
echo "[$(date)] On-call notification sent: Primary=$PRIMARY, Secondary=$SECONDARY"7.3 值班最佳实践
✅ 做到：
- 每班次至少2人（主+备）
- 交接时同步未关闭告警和进行中的事件
- 值班期间确保手机畅通、VPN可用
- 记录所有告警处理过程
- 轮值周期不超过1周，避免疲劳
❌ 避免：
- 同一个人连续值班超过7天
- P0告警只通知一个人
- 没有备份值班人员
- 值班期间做高风险变更八、告警响应 SOP
8.1 Runbook 模板
每个告警规则都应关联一个 Runbook，以下是一个标准模板：
# Runbook: HighCpuUsage（CPU使用率过高）
## 告警描述
实例 {{ instance }} 的CPU使用率超过85%，持续5分钟以上。
## 影响范围
- 可能导致请求延迟增加
- 严重时触发服务降级
## 排查步骤
### Step 1: 确认告警
```bashSSH 到目标机器
ssh {{ instance }}
查看整体负载
uptime top -bn1 | head -20
查看CPU使用详情
mpstat -P ALL 1 5
### Step 2: 定位高CPU进程
```bash找到CPU消耗最高的进程
ps aux --sort=-%cpu | head -10
查看进程的线程级CPU消耗
top -H -p
生成火焰图（持续30秒）
perf record -F 99 -p -g -- sleep 30 perf script | stackcollapse-perf.pl | flamegraph.pl > cpu-flamegraph.svg
### Step 3: 常见原因及处理
| 原因 | 判断依据 | 处理方式 |
|------|---------|---------|
| 死循环 | 单线程100% CPU | 重启服务或修复bug |
| GC频繁 | Java进程+高CPU | 检查堆内存、调优GC |
| 流量突增 | QPS明显上升 | 扩容或限流 |
| 慢查询 | DB CPU高 | 优化SQL或加索引 |
| 挖矿木马 | 异常进程名 | 立即隔离机器、排查入侵 |
### Step 4: 临时缓解
```bash限制进程CPU使用（Linux cgroups）
systemctl set-property CPUQuota=80%
或使用 cpulimit
cpulimit -p -l 80
### Step 5: 长期解决
- [ ] 根因分析完成
- [ ] 代码/配置修复已上线
- [ ] 相关监控规则已优化
- [ ] Runbook 已更新
## 升级条件
- 排查30分钟未定位原因
- 影响核心业务链路
- 同时多台机器触发8.2 Postmortem（事后复盘）模板
# 事件复盘报告
## 基本信息
- **事件标题**: 生产数据库主库CPU 100%导致支付链路中断
- **事件等级**: P0
- **影响时间**: 2026-06-10 02:15 - 02:47 (32分钟)
- **影响范围**: 支付成功率从99.9%降至45%
- **直接损失**: 约￥50,000（估算）
- **复盘人员**: 张三(DBA)、李四(后端)、王五(运维)
## 时间线
| 时间 | 事件 |
|------|------|
| 02:15 | 告警触发：db-master CPU > 95% |
| 02:17 | 值班人张三收到通知，开始排查 |
| 02:22 | 定位到慢查询：全表扫描订单表 |
| 02:30 | kill 慢查询，CPU未恢复 |
| 02:35 | 发现是定时任务触发大量全表扫描 |
| 02:40 | 停止定时任务，CPU恢复正常 |
| 02:47 | 支付成功率恢复到99.9% |
## 根因分析（5-Whys）
1. **Why**: 支付链路中断
→ 数据库主库CPU 100%
2. **Why**: 数据库CPU满载
→ 大量全表扫描查询
3. **Why**: 出现大量全表扫描
→ 定时任务查询缺少索引
4. **Why**: 定时任务缺少索引
→ 新上线的对账脚本未经过DBA审核
5. **Why**: 未经过DBA审核
→ 变更流程未覆盖定时任务的SQL审核
## 改进措施
| 措施 | 负责人 | 截止日期 | 状态 |
|------|--------|---------|------|
| 对账脚本SQL优化并加索引 | 李四 | 2026-06-12 | 进行中 |
| 定时任务上线流程增加DBA审核环节 | 王五 | 2026-06-17 | 待开始 |
| 添加慢查询自动告警（>3s） | 张三 | 2026-06-14 | 待开始 |
| 定时任务执行增加限流保护 | 李四 | 2026-06-20 | 待开始 |
## 经验教训
1. 任何SQL上线前必须经过DBA审核（包括脚本和定时任务）
2. 大数据量操作需要在低峰期执行并限制并发
3. 告警响应Runbook中应包含"如何快速kill异常查询"的步骤九、告警治理
9.1 告警质量度量
#!/bin/bash
# alert-audit.sh - 告警质量审计脚本
PROMETHEUS_URL="http://prometheus:9090"
echo "=== 告警质量月度审计 ==="
echo ""
# 1. 告警规则总数
echo "--- 告警规则统计 ---"
total_rules=$(curl -s "$PROMETHEUS_URL/api/v1/rules" | jq '.data.groups[].rules | length' | awk '{sum+=$1} END{print sum}')
echo "总规则数: $total_rules"
# 2. 过去30天触发次数 Top 20
echo ""
echo "--- 高频告警 Top 20（过去30天）---"
curl -s "$PROMETHEUS_URL/api/v1/query" \
--data-urlencode 'query=topk(20, changes(ALERTS_FOR_STATE[30d]))' \
| jq -r '.data.result[] | "\(.metric.alertname) | \(.value[1]) 次 | severity=\(.metric.severity // "N/A")"'
# 3. 过去30天从未触发的告警
echo ""
echo "--- 冷却告警（30天未触发，考虑清理）---"
curl -s "$PROMETHEUS_URL/api/v1/rules" \
| jq -r '.data.groups[].rules[] | select(.type=="alerting") | .name' \
| sort > /tmp/all_alerts.txt
# 对比实际触发过的历史告警...
# 4. 告警响应率
echo ""
echo "--- 告警响应率（近30天）---"
# 需要从告警管理系统获取 ACK 数据
echo "请从告警管理平台导出数据计算"9.2 告警生命周期管理
┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
│ 创建 │ → │ 审核 │ → │ 上线 │ → │ 运营 │ → │ 退役 │
│ (Draft) │ │ (Review) │ │ (Active) │ │ (Monitor)│ │ (Archive)│
└──────────┘ └──────────┘ └──────────┘ └──────────┘ └──────────┘
│ │
↓ ↓
同行评审 季度审计
Runbook必备 30天未触发→清理
分级标注 误报率>50%→优化9.3 告警治理 Checklist
## 季度告警审计清单
### 1. 规则质量
- [ ] 每条告警都有明确的 severity 标签
- [ ] 每条告警都有 annotations（summary/description）
- [ ] 每条告警都关联了 Runbook URL
- [ ] 没有超过30天未触发的"僵尸告警"
- [ ] 没有误报率超过50%的"噪声告警"
### 2. 路由配置
- [ ] 路由树覆盖所有 team 标签
- [ ] 告警升级机制正常工作
- [ ] 抑制规则覆盖相关场景
- [ ] 默认 receiver 已配置（兜底）
### 3. 通知渠道
- [ ] 所有 Webhook 端点健康检查正常
- [ ] 值班人员名单已更新
- [ ] 通知模板格式正确
### 4. 响应流程
- [ ] Runbook 已更新并经过验证
- [ ] 值班交接流程正常执行
- [ ] 上月 Postmortem 的改进措施已落地十、完整告警体系配置模板
10.1 Prometheus 告警规则文件结构
# 推荐的规则文件组织结构
/etc/prometheus/rules/
├── 00-global.rules.yml # 全局基础告警
├── 10-node.rules.yml # 节点级告警
├── 20-container.rules.yml # 容器/K8s 告警
├── 30-middleware.rules.yml # 中间件告警（DB/Redis/MQ）
├── 40-application.rules.yml # 应用级告警
├── 50-business.rules.yml # 业务指标告警
├── 60-slo.rules.yml # SLA/SLO 相关告警
├── 90-recording.rules.yml # Recording rules
└── 99-test.rules.yml # 测试规则（不带入生产）10.2 完整 AlertManager 配置
# /etc/alertmanager/alertmanager.yml - 生产级完整配置
global:
resolve_timeout: 5m
smtp_from: 'alertmanager@company.com'
smtp_smarthost: 'smtp.company.com:587'
smtp_auth_username: 'alertmanager@company.com'
smtp_auth_password: '${SMTP_PASSWORD}'
slack_api_url: '${SLACK_WEBHOOK_URL}'
templates:
- '/etc/alertmanager/templates/*.tmpl'
route:
receiver: 'default-webhook'
group_by: ['alertname', 'cluster', 'namespace', 'service']
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
routes:
# === P0 致命告警 ===
- matchers:
- severity = "critical"
receiver: 'p0-escalation'
group_wait: 10s
group_interval: 1m
repeat_interval: 5m
continue: true
routes:
- matchers:
- team = "dba"
receiver: 'dba-oncall'
continue: true
- matchers:
- team = "network"
receiver: 'network-oncall'
continue: true
- matchers:
- team = "security"
receiver: 'security-oncall'
# === P1 严重告警 ===
- matchers:
- severity = "warning"
receiver: 'p1-webhook'
group_wait: 1m
group_interval: 5m
repeat_interval: 1h
# === P2-P4 低优先级 ===
- matchers:
- severity =~ "info|none"
receiver: 'low-priority'
group_wait: 5m
group_interval: 30m
repeat_interval: 12h
inhibit_rules:
# 集群级告警抑制节点级
- source_matchers:
- severity = "critical"
- alertname = "ClusterDown"
target_matchers:
- severity =~ "warning|info"
equal: ['cluster']
# 服务不可用抑制性能告警
- source_matchers:
- alertname =~ ".*Down$|.*Unavailable$"
target_matchers:
- alertname =~ "High.*|Slow.*"
equal: ['service', 'namespace']
# 依赖服务故障抑制上游告警
- source_matchers:
- alertname = "DependencyDown"
target_matchers:
- alertname =~ "HighErrorRate|HighLatency"
equal: ['service']
receivers:
- name: 'default-webhook'
webhook_configs:
- url: 'http://alert-gateway:8080/webhook'
send_resolved: true
- name: 'p0-escalation'
webhook_configs:
- url: 'http://alert-gateway:8080/webhook?level=p0'
send_resolved: true
slack_configs:
- channel: '#alert-critical'
title: '🚨 P0: {{ .GroupLabels.alertname }}'
text: '{{ range .Alerts }}{{ .Annotations.summary }}\n{{ end }}'
email_configs:
- to: 'oncall@company.com'
headers:
Subject: '[P0-CRITICAL] {{ .GroupLabels.alertname }}'
- name: 'p1-webhook'
webhook_configs:
- url: 'http://alert-gateway:8080/webhook?level=p1'
send_resolved: true
- name: 'low-priority'
email_configs:
- to: 'ops-batch@company.com'
send_resolved: true
- name: 'dba-oncall'
webhook_configs:
- url: 'http://alert-gateway:8080/webhook?team=dba'
send_resolved: true
- name: 'network-oncall'
webhook_configs:
- url: 'http://alert-gateway:8080/webhook?team=network'
send_resolved: true
- name: 'security-oncall'
webhook_configs:
- url: 'http://alert-gateway:8080/webhook?team=security'
send_resolved: true10.3 Docker Compose 一键部署
# docker-compose.alerting.yml
version: '3.8'
services:
prometheus:
image: prom/prometheus:latest
ports:
- "9090:9090"
volumes:
- ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
- ./prometheus/rules:/etc/prometheus/rules
- prometheus_data:/prometheus
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.retention.time=30d'
- '--web.enable-lifecycle'
restart: unless-stopped
alertmanager:
image: prom/alertmanager:latest
ports:
- "9093:9093"
volumes:
- ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml
- ./alertmanager/templates:/etc/alertmanager/templates
- alertmanager_data:/alertmanager
command:
- '--config.file=/etc/alertmanager/alertmanager.yml'
- '--storage.path=/alertmanager'
- '--cluster.listen-address=0.0.0.0:9094'
restart: unless-stopped
alert-gateway:
image: your-registry/alert-gateway:latest
ports:
- "9095:8080"
environment:
- FEISHU_WEBHOOK=${FEISHU_WEBHOOK}
- DINGTALK_TOKEN=${DINGTALK_TOKEN}
- WECOM_KEY=${WECOM_KEY}
- SLACK_WEBHOOK=${SLACK_WEBHOOK}
restart: unless-stopped
# 钉钉告警适配器
dingtalk-adapter:
image: timonwong/prometheus-webhook-dingtalk:latest
ports:
- "8060:8060"
volumes:
- ./dingtalk/config.yml:/etc/prometheus-webhook-dingtalk/config.yml
restart: unless-stopped
volumes:
prometheus_data:
alertmanager_data:10.4 Prometheus 告警规则配置
# prometheus.yml 完整配置
global:
scrape_interval: 15s
evaluation_interval: 15s
rule_files:
- '/etc/prometheus/rules/*.rules.yml'
alerting:
alertmanagers:
- static_configs:
- targets:
- 'alertmanager:9093'
timeout: 10s
api_version: v2
scrape_configs:
- job_name: 'node-exporter'
static_configs:
- targets: ['node1:9100', 'node2:9100', 'node3:9100']
- job_name: 'cadvisor'
static_configs:
- targets: ['cadvisor:8080']
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']写在最后
一个好的告警体系，不在于规则有多多，而在于每条告警都能驱动行动。记住三个原则：
宁缺毋滥：一条精准的告警胜过一百条噪声
闭环管理：告警 → 响应 → 复盘 → 优化，缺一不可
持续治理：告警体系不是一劳永逸的，需要定期审计和迭代
告警体系 = 好的规则 × 好的降噪 × 好的路由 × 好的 SOP如果告警太多让人麻木，那就砍掉一半；如果告警太少让人安心，那就加一些。找到那个"恰到好处"的平衡点，是每个运维团队的必修课。