Session 04 - Incident Management & Postmortem

Incident Severity Levels

Incident Response Workflow

On-Call Rotation Best Practices

Rotation Structure

Primary on-call:   1 tuần rotation
Secondary on-call:  Backup khi primary không available
Escalation:         Team lead → Engineering Manager → VP Engineering

Guidelines

Giảm On-Call Burden

csharp
// Automate common responses
// Ví dụ: Auto-restart service khi health check fail
public class SelfHealingHealthCheck : IHealthCheck
{
    private readonly IHostApplicationLifetime _lifetime;
    private static int _consecutiveFailures = 0;

    public SelfHealingHealthCheck(IHostApplicationLifetime lifetime)
        => _lifetime = lifetime;

    public async Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context,
        CancellationToken ct = default)
    {
        var isHealthy = await CheckDependencies(ct);

        if (!isHealthy)
        {
            var failures = Interlocked.Increment(ref _consecutiveFailures);
            if (failures >= 3)
            {
                // Kubernetes sẽ restart pod khi liveness fail
                return HealthCheckResult.Unhealthy(
                    $"Failed {failures} consecutive health checks");
            }
            return HealthCheckResult.Degraded("Temporary issue detected");
        }

        Interlocked.Exchange(ref _consecutiveFailures, 0);
        return HealthCheckResult.Healthy();
    }

    private Task<bool> CheckDependencies(CancellationToken ct) => /* ... */ Task.FromResult(true);
}

Communication During Incidents

Stakeholder Update Template

markdown
## Incident Update - [Incident Title]
**Time**: 2024-01-15 14:30 UTC
**Severity**: SEV1
**Status**: Investigating / Mitigating / Resolved
**Impact**: [Mô tả cụ thể ảnh hưởng tới users]

### Current Situation
[Chúng ta biết gì hiện tại]

### Actions Being Taken
[Đang làm gì để fix]

### Next Update
[Thời gian update tiếp theo - thường 30 phút cho SEV1]

Communication Timeline

Blameless Postmortem Template

Mẫu Postmortem Document

markdown
# Postmortem: [Incident Title]

## Metadata
- **Date**: 2024-01-15
- **Severity**: SEV1
- **Duration**: 2 hours 15 minutes (14:00 - 16:15 UTC)
- **Authors**: [Tên người viết]
- **Reviewers**: [Tên người review]

## Summary
[1-2 câu tóm tắt incident và impact]

## Impact
- **Users affected**: ~15,000 users (30% total)
- **Revenue impact**: ~$5,000 estimated lost orders
- **Duration**: 2h15m total, 1h45m user-facing impact

## Timeline (UTC)
| Time | Event |
|------|-------|
| 13:45 | Deploy v2.3.1 to production |
| 14:00 | Error rate alert fires |
| 14:05 | On-call acknowledges, begins investigation |
| 14:15 | Identified: new code path causes N+1 query |
| 14:20 | Decision: rollback to v2.3.0 |
| 14:35 | Rollback complete |
| 14:45 | Error rate returns to normal |
| 16:15 | Confirmed all queued orders processed |

## Root Cause
[Mô tả chi tiết nguyên nhân gốc - technical, không blame]

## Detection
- **How was it detected?** Automated alert on error rate > 5%
- **Time to detect**: 15 phút
- **Could we have detected sooner?** Yes - canary deployment sẽ catch

## Mitigation
[Các bước đã thực hiện để khôi phục]

## Lessons Learned

### What went well
- Alert fired nhanh (15 phút)
- Rollback process smooth (15 phút)
- Clear communication với stakeholders

### What went wrong
- Không có canary deployment
- Load test không cover new code path
- Staging environment thiếu realistic data

### Where we got lucky
- Incident xảy ra vào off-peak hours

## Action Items
| Action | Owner | Priority | Due Date | Ticket |
|--------|-------|----------|----------|--------|
| Implement canary deployment | DevOps team | P1 | 2024-02-01 | INFRA-456 |
| Add load test cho order flow | QA team | P2 | 2024-02-15 | QA-789 |
| Seed staging with production-like data | Backend team | P2 | 2024-02-15 | BE-123 |

Action Item Tracking

Nguyên tắc Action Items

1. Specific & Measurable: "Add circuit breaker to PaymentClient" - không phải "improve resilience"
2. Có owner rõ ràng: Một người chịu trách nhiệm, không phải "the team"
3. Có deadline: Dựa trên priority, không phải "when we have time"
4. Tracked trong system: JIRA/Linear ticket, review hàng tuần

Priority Framework

Learning From Incidents Culture

Blameless ≠ Accountable-less

• Blameless: Không tìm ai để phạt - tìm systemic failures
• Accountable: Action items có owner và deadline

Practices để xây dựng culture

Metrics to Track

MTTD (Mean Time to Detect)   = Thời gian từ lúc issue xảy ra → alert
MTTR (Mean Time to Resolve)  = Thời gian từ lúc alert → resolved
MTBF (Mean Time Between Failures) = Thời gian giữa 2 incidents
Recurrence Rate = % incidents có cùng root cause lặp lại

Trade-offs