Session 02 - Metrics, Dashboards & Alerting

Four Golden Signals

Google SRE Book định nghĩa 4 golden signals - đây là foundation cho mọi monitoring strategy:

RED & USE Methods

RED Method (Request-driven services)

• Rate: Số requests/second
• Errors: Số requests bị lỗi/second
• Duration: Distribution of request latency

USE Method (Infrastructure resources)

• Utilization: % thời gian resource bận
• Saturation: Lượng work đang queue
• Errors: Số error events

Custom Metrics trong .NET

Sử dụng System.Diagnostics.Metrics

csharp
using System.Diagnostics.Metrics;

public class OrderMetrics
{
    private readonly Counter<long> _ordersCreated;
    private readonly Counter<long> _ordersFailed;
    private readonly Histogram<double> _orderProcessingDuration;
    private readonly UpDownCounter<long> _activeOrders;
    private readonly ObservableGauge<int> _queueDepth;

    public OrderMetrics(IMeterFactory meterFactory)
    {
        var meter = meterFactory.Create("OrderService.Metrics");

        _ordersCreated = meter.CreateCounter<long>(
            "orders.created",
            unit: "{order}",
            description: "Total orders created");

        _ordersFailed = meter.CreateCounter<long>(
            "orders.failed",
            unit: "{order}",
            description: "Total orders failed");

        _orderProcessingDuration = meter.CreateHistogram<double>(
            "orders.processing.duration",
            unit: "ms",
            description: "Order processing duration in milliseconds");

        _activeOrders = meter.CreateUpDownCounter<long>(
            "orders.active",
            unit: "{order}",
            description: "Currently processing orders");

        _queueDepth = meter.CreateObservableGauge(
            "orders.queue.depth",
            () => GetCurrentQueueDepth(),
            unit: "{order}",
            description: "Orders waiting in queue");
    }

    public void RecordOrderCreated(string region, string tier)
    {
        _ordersCreated.Add(1,
            new KeyValuePair<string, object?>("region", region),
            new KeyValuePair<string, object?>("tier", tier));
    }

    public void RecordOrderFailed(string reason)
    {
        _ordersFailed.Add(1,
            new KeyValuePair<string, object?>("failure.reason", reason));
    }

    public TrackedOrder TrackProcessing()
    {
        _activeOrders.Add(1);
        return new TrackedOrder(this);
    }

    public class TrackedOrder : IDisposable
    {
        private readonly OrderMetrics _metrics;
        private readonly long _startTimestamp = Stopwatch.GetTimestamp();

        internal TrackedOrder(OrderMetrics metrics) => _metrics = metrics;

        public void Dispose()
        {
            var elapsed = Stopwatch.GetElapsedTime(_startTimestamp);
            _metrics._orderProcessingDuration.Record(elapsed.TotalMilliseconds);
            _metrics._activeOrders.Add(-1);
        }
    }

    private int GetCurrentQueueDepth() => /* lấy từ message queue */ 0;
}

Đăng ký trong DI

csharp
builder.Services.AddSingleton<OrderMetrics>();

// OpenTelemetry sẽ tự collect từ Meter name đã đăng ký
builder.Services.AddOpenTelemetry()
    .WithMetrics(m => m.AddMeter("OrderService.Metrics"));

Grafana vs Azure Dashboard

Dashboard Design Principles

Level 1: Business KPIs     → Revenue, Active Users, Conversion Rate
Level 2: Service Health    → Latency P99, Error Rate, Throughput
Level 3: Infrastructure    → CPU, Memory, Disk, Network
Level 4: Debug             → Individual traces, logs, spans

Alert Design - Tránh Alert Fatigue

Phân loại Alerts

Nguyên tắc Alert Hiệu quả

yaml
# Mỗi alert PHẢI có:
alert_name: "OrderService High Error Rate"
description: "Error rate vượt 5% trong 5 phút liên tục"
severity: P2
runbook_url: "https://wiki.internal/runbooks/order-high-error-rate"
dashboard_url: "https://grafana.internal/d/order-service"
escalation_policy: "backend-oncall"

Alerting Rules ví dụ (Prometheus format)

yaml
groups:
  - name: order-service
    rules:
      # Error rate > 5% trong 5 phút
      - alert: HighErrorRate
        expr: |
          sum(rate(http_server_request_duration_seconds_count{status_code=~"5.."}[5m]))
          /
          sum(rate(http_server_request_duration_seconds_count[5m]))
          > 0.05
        for: 5m
        labels:
          severity: P2
          team: backend
        annotations:
          summary: "Error rate {{ $value | humanizePercentage }}"
          runbook: "https://wiki.internal/runbooks/high-error-rate"

      # P99 latency > 2s trong 10 phút
      - alert: HighLatencyP99
        expr: |
          histogram_quantile(0.99,
            sum(rate(http_server_request_duration_seconds_bucket[5m])) by (le)
          ) > 2
        for: 10m
        labels:
          severity: P2

Azure Monitor Alert (Bicep)

bicep
resource metricAlert 'Microsoft.Insights/metricAlerts@2018-03-01' = {
  name: 'order-service-high-error-rate'
  location: 'global'
  properties: {
    severity: 2
    evaluationFrequency: 'PT1M'
    windowSize: 'PT5M'
    criteria: {
      'odata.type': 'Microsoft.Azure.Monitor.SingleResourceMultipleMetricCriteria'
      allOf: [
        {
          name: 'HighErrorRate'
          metricName: 'Http5xx'
          operator: 'GreaterThan'
          threshold: 50
          timeAggregation: 'Total'
        }
      ]
    }
    actions: [
      { actionGroupId: actionGroup.id }
    ]
  }
}

Runbook-Linked Alerts

Mỗi alert bắt buộc phải link tới runbook chứa:

1. Mô tả: Alert này nghĩa là gì?
2. Impact: Ảnh hưởng tới user/business như thế nào?
3. Diagnosis steps: Kiểm tra gì đầu tiên?
4. Mitigation: Các bước xử lý tạm thời
5. Resolution: Cách fix triệt để
6. Escalation: Khi nào cần escalate và contact ai?

Trade-offs