运维

混沌工程实战：用Chaos Mesh构建生产级故障注入与韧性测试体系

北盛科技 阅读 580

为什么需要混沌工程

"假设它会出故障"是SRE（站点可靠性工程师）的基本信条。混沌工程的核心原则：与其等待真实故障发生，不如主动制造故障、验证系统的应对能力。

Netflix的Chaos Monkey项目证明：经过混沌工程磨炼的系统，在真实故障发生时的MTTR（平均恢复时间）降低了67%。

一、Chaos Mesh安装与配置

1.1 安装

# 使用Helm安装Chaos Mesh v2.7helm repo add chaos-mesh https://charts.chaos-mesh.orghelm repo updatekubectl create ns chaos-meshhelm install chaos-mesh chaos-mesh/chaos-mesh \ --namespace=chaos-mesh \ --version 2.7.0 \ --set chaosDaemon.runtime=containerd \ --set chaosDaemon.socketPath=/run/containerd/containerd.sock \ --set dashboard.create=true \ --set dashboard.serviceType=NodePort \ --set dnsServer.create=true # 支持DNS故障注入# 验证安装kubectl get pods -n chaos-mesh

1.2 RBAC权限配置

# 为混沌工程团队创建专用ServiceAccountapiVersion: v1kind: ServiceAccountmetadata: name: chaos-operator namespace: staging # 限定在staging命名空间---apiVersion: rbac.authorization.k8s.io/v1kind: Rolemetadata: name: chaos-operator-role namespace: stagingrules:# 允许管理混沌实验- apiGroups: ["chaos-mesh.org"] resources: ["*"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]# 禁止在production命名空间操作- apiGroups: [""] resources: ["pods"] verbs: ["get", "list", "watch"] # 只读！

二、常用故障场景配置

2.1 Pod故障

# 随机删除Pod（模拟节点故障/OOM Kill）apiVersion: chaos-mesh.org/v1alpha1kind: PodChaosmetadata: name: pod-kill-test namespace: stagingspec: action: pod-kill mode: random-max-percent # 随机比例 value: "30" # 最多30%的Pod selector: namespaces: - staging labelSelectors: app: api-service # 安全守卫：高级别保护 gracePeriod: 10 # 优雅停机10秒 # 定时触发（可选） scheduler: cron: "@hourly" # 每小时随机干掉30%的Pod

2.2 网络故障

# 网络延迟注入（模拟跨区域调用）apiVersion: chaos-mesh.org/v1alpha1kind: NetworkChaosmetadata: name: network-delay-testspec: action: delay mode: all selector: namespaces: [staging] labelSelectors: app: frontend delay: latency: "200ms" correlation: "50" # 50%的相关性（更真实的网络抖动模拟） jitter: "50ms" # ±50ms抖动 direction: to # 只影响出方向流量 target: selector: namespaces: [staging] labelSelectors: app: api-service mode: all duration: "5m" # 持续5分钟---# 网络丢包（模拟不稳定网络）apiVersion: chaos-mesh.org/v1alpha1kind: NetworkChaosmetadata: name: packet-loss-testspec: action: loss mode: all selector: namespaces: [staging] labelSelectors: app: payment-service loss: loss: "10" # 10%丢包率 correlation: "25" duration: "3m"

2.3 磁盘IO故障（模拟磁盘性能下降）

apiVersion: chaos-mesh.org/v1alpha1kind: IOChaosmetadata: name: io-stress-testspec: action: latency # 增加IO延迟 mode: one # 只影响一个Pod selector: namespaces: [staging] labelSelectors: app: mysql volumePath: /var/lib/mysql delay: "10ms" # 每次IO增加10ms延迟 percent: 50 # 50%的IO请求受影响 duration: "10m"

2.4 内存压力（模拟内存泄漏）

apiVersion: chaos-mesh.org/v1alpha1kind: StressChaosmetadata: name: memory-stress-testspec: mode: one selector: namespaces: [staging] labelSelectors: app: java-service stressors: memory: workers: 4 size: "512MB" # 消耗512MB内存 duration: "5m"

三、混沌工作流编排

# Workflow：模拟完整的"数据库故障恢复"场景apiVersion: chaos-mesh.org/v1alpha1kind: Workflowmetadata: name: db-failover-scenariospec: entry: entry templates: # 入口节点：顺序执行 - name: entry templateType: Serial children: - pre-check - inject-db-failure - wait-for-recovery - verify-data-integrity - cleanup # 前置检查 - name: pre-check templateType: Task task: container: name: check image: bitnami/kubectl:latest command: - sh - -c - | # 确认备库已就绪 kubectl exec mysql-replica-0 -- mysql -e "SHOW SLAVE STATUS\G" | \ grep -q "Seconds_Behind_Master: 0" echo "前置检查通过：备库延迟为0" # 注入主库故障 - name: inject-db-failure templateType: Suspend deadline: 10m children: - db-pod-kill - monitor-failover - name: db-pod-kill templateType: PodChaos podChaos: action: pod-kill mode: one selector: namespaces: [production] labelSelectors: app: mysql role: primary # 等待自动故障转移 - name: monitor-failover templateType: Task deadline: 5m task: container: image: bitnami/kubectl:latest command: - sh - -c - | # 等待直到新主库就绪 until kubectl exec mysql-replica-0 -- mysql -e \ "SELECT 1" > /dev/null 2>&1; do sleep 5 echo "等待故障转移..." done echo "故障转移完成！" # 数据完整性验证 - name: verify-data-integrity templateType: Task task: container: image: python:3.11 command: - python - /scripts/verify_data.py # 自定义数据一致性检查脚本

四、集成到CI/CD

# .github/workflows/chaos-test.ymlname: Chaos Engineering Testson: schedule: - cron: '0 2 * * 1' # 每周一凌晨2点 workflow_dispatch: # 支持手动触发jobs: chaos-test: runs-on: ubuntu-latest environment: staging steps: - uses: actions/checkout@v4 - name: Setup kubectl uses: azure/setup-kubectl@v4 - name: Run network chaos test run: | # 注入网络故障 kubectl apply -f chaos/network-delay-test.yaml # 同时运行负载测试 kubectl apply -f chaos/load-test-job.yaml # 等待5分钟 sleep 300 # 检查服务健康状态 SUCCESS_RATE=$(curl -s http://prometheus/api/v1/query \ --data 'query=rate(http_requests_total{status="200"}[1m]) / rate(http_requests_total[1m])' | \ jq -r '.data.result[0].value[1]') echo "当前成功率：${SUCCESS_RATE}" if (( $(echo "$SUCCESS_RATE < 0.95" | bc -l) )); then echo "❌ 成功率低于95%，混沌测试失败" exit 1 fi echo "✅ 混沌测试通过，服务在网络故障下保持>95%成功率" - name: Cleanup if: always() run: | kubectl delete -f chaos/ --ignore-not-found

五、混沌工程安全守则

  1. 从小范围开始：先在staging，再在生产的1%流量，循序渐进
  2. 明确爆炸半径：每次实验前清晰定义"最坏情况"并确认可接受
  3. 准备紧急停止：Chaos Mesh提供一键暂停所有实验的功能
  4. 自动化监控：实验期间自动监控SLO，违反则立即中止
  5. 完整记录：每次实验记录假设、结果和改进措施

混沌工程不是破坏系统，而是在受控条件下提前发现系统的薄弱点，让真实故障来临时胸有成竹。