Session 03 - SLA, SLO & SLI

Definitions & Relationships

Quy tắc vàng

• SLO luôn strict hơn SLA: Nếu SLA = 99.5%, SLO nên = 99.9%
• SLI phải đo từ góc nhìn user: Đo ở load balancer/edge, không phải server-side
• Error budget là công cụ ra quyết định, không chỉ là con số

Cách Định Nghĩa SLIs cho .NET API

SLI Types

Ví dụ: SLI cho Order API

csharp
// SLI 1: Availability - requests trả về status < 500
// Đo bằng middleware
public class SliMiddleware
{
    private static readonly Counter<long> _totalRequests;
    private static readonly Counter<long> _successfulRequests;

    static SliMiddleware()
    {
        var meter = new Meter("SLI.OrderService");
        _totalRequests = meter.CreateCounter<long>("sli.requests.total");
        _successfulRequests = meter.CreateCounter<long>("sli.requests.successful");
    }

    private readonly RequestDelegate _next;

    public SliMiddleware(RequestDelegate next) => _next = next;

    public async Task InvokeAsync(HttpContext context)
    {
        await _next(context);

        // Chỉ đếm user-facing endpoints
        if (context.Request.Path.StartsWithSegments("/api"))
        {
            _totalRequests.Add(1);
            if (context.Response.StatusCode < 500)
            {
                _successfulRequests.Add(1);
            }
        }
    }
}

// SLI 2: Latency - P99 < 500ms
// Đo bằng histogram
public class LatencySliMiddleware
{
    private static readonly Histogram<double> _requestDuration;
    private readonly RequestDelegate _next;

    static LatencySliMiddleware()
    {
        var meter = new Meter("SLI.OrderService");
        _requestDuration = meter.CreateHistogram<double>(
            "sli.request.duration", unit: "ms");
    }

    public LatencySliMiddleware(RequestDelegate next) => _next = next;

    public async Task InvokeAsync(HttpContext context)
    {
        var sw = Stopwatch.GetTimestamp();
        await _next(context);
        var elapsed = Stopwatch.GetElapsedTime(sw);

        if (context.Request.Path.StartsWithSegments("/api"))
        {
            _requestDuration.Record(elapsed.TotalMilliseconds,
                new KeyValuePair<string, object?>("endpoint",
                    context.GetEndpoint()?.DisplayName ?? "unknown"));
        }
    }
}

Error Budgets & Burn Rate

Error Budget Calculation

SLO = 99.9% trong 30 ngày
Error Budget = 0.1% = 100% - 99.9%
Thời gian downtime cho phép = 30 * 24 * 60 * 0.001 = 43.2 phút/tháng

Burn Rate

Burn rate đo tốc độ tiêu hao error budget:

Multi-window Burn Rate Alerts

yaml
# Fast burn - phát hiện nhanh, short window
- alert: SLO_HighBurnRate_Fast
  expr: |
    (
      sum(rate(sli_requests_successful_total[1h]))
      / sum(rate(sli_requests_total_total[1h]))
    ) < (1 - 14.4 * 0.001)  # 14.4x burn rate, 1h window
  for: 2m
  labels:
    severity: P1
  annotations:
    summary: "Error budget burning 14.4x nhanh hơn bình thường"

# Slow burn - phát hiện trend, long window
- alert: SLO_HighBurnRate_Slow
  expr: |
    (
      sum(rate(sli_requests_successful_total[3d]))
      / sum(rate(sli_requests_total_total[3d]))
    ) < (1 - 1.0 * 0.001)  # 1x burn rate, 3d window
  for: 1h
  labels:
    severity: P3
  annotations:
    summary: "Error budget đang tiêu hao đều đặn trong 3 ngày"

SLO Examples

Web API SLOs

Background Service SLOs

SLO-Based Decision Making

Error Budget Policy (mẫu)

Khi error budget cạn:

1. Feature freeze - chỉ ship reliability improvements
2. Mandatory rollback - deploy gây tăng error → rollback ngay
3. Post-mortem required - mọi incident phải có postmortem
4. Extra testing - tăng coverage, thêm load tests trước deploy

Trade-offs