Session 06 - Azure Monitoring & Alerting

Tổng quan

Observability là nền tảng của Operational Excellence. Không thể fix những gì không nhìn thấy. Azure Monitor ecosystem gồm Application Insights, Log Analytics, Metrics, và Alerts - tạo thành full-stack monitoring cho .NET applications.

Application Insights Setup cho .NET

Step 1: Cài đặt NuGet packages

bash
dotnet add package Microsoft.ApplicationInsights.AspNetCore
dotnet add package Microsoft.ApplicationInsights.Kubernetes  # cho AKS

Step 2: Cấu hình trong Program.cs

csharp
var builder = WebApplication.CreateBuilder(args);

// Thêm Application Insights telemetry
builder.Services.AddApplicationInsightsTelemetry(options =>
{
    options.ConnectionString = builder.Configuration["ApplicationInsights:ConnectionString"];
    options.EnableAdaptiveSampling = true;
});

// Custom telemetry initializer - thêm metadata
builder.Services.AddSingleton<ITelemetryInitializer, CustomTelemetryInitializer>();

var app = builder.Build();

csharp
// Custom TelemetryInitializer - gắn context vào mọi telemetry
public class CustomTelemetryInitializer : ITelemetryInitializer
{
    public void Initialize(ITelemetry telemetry)
    {
        telemetry.Context.Cloud.RoleName = "OrderApi";
        telemetry.Context.GlobalProperties["deployment-slot"] = "production";
    }
}

Step 3: appsettings.json

json
{
  "ApplicationInsights": {
    "ConnectionString": "InstrumentationKey=xxx;IngestionEndpoint=https://southeastasia-1.in.applicationinsights.azure.com/"
  },
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    },
    "ApplicationInsights": {
      "LogLevel": {
        "Default": "Information"
      }
    }
  }
}

Custom Events & Metrics

csharp
public class OrderService
{
    private readonly TelemetryClient _telemetry;

    public OrderService(TelemetryClient telemetry)
    {
        _telemetry = telemetry;
    }

    public async Task<Order> CreateOrder(CreateOrderRequest request)
    {
        var stopwatch = Stopwatch.StartNew();
        try
        {
            var order = await ProcessOrder(request);

            // Track custom event
            _telemetry.TrackEvent("OrderCreated", new Dictionary<string, string>
            {
                ["OrderId"] = order.Id.ToString(),
                ["PaymentMethod"] = request.PaymentMethod,
                ["ItemCount"] = request.Items.Count.ToString()
            }, new Dictionary<string, double>
            {
                ["OrderTotal"] = (double)order.Total,
                ["ProcessingTimeMs"] = stopwatch.ElapsedMilliseconds
            });

            // Track custom metric
            _telemetry.GetMetric("OrderProcessingTime").TrackValue(stopwatch.ElapsedMilliseconds);

            return order;
        }
        catch (Exception ex)
        {
            _telemetry.TrackException(ex, new Dictionary<string, string>
            {
                ["Operation"] = "CreateOrder",
                ["CustomerId"] = request.CustomerId.ToString()
            });
            throw;
        }
    }
}

Log Analytics Workspace Design

bash
# Tạo Log Analytics Workspace
az monitor log-analytics workspace create \
  --resource-group rg-monitoring \
  --workspace-name law-central-prod \
  --location southeastasia \
  --retention-time 90 \
  --sku PerGB2018

# Link Application Insights vào workspace
az monitor app-insights component create \
  --app appi-orderapi-prod \
  --location southeastasia \
  --resource-group rg-monitoring \
  --workspace law-central-prod

KQL Queries - Common Scenarios

1. Slow Requests (P95 > 2s)

kql
requests
| where timestamp > ago(24h)
| where success == true
| summarize
    P50 = percentile(duration, 50),
    P95 = percentile(duration, 95),
    P99 = percentile(duration, 99),
    Count = count()
  by bin(timestamp, 1h), name
| where P95 > 2000
| order by P95 desc

2. Error Rate by Endpoint

kql
requests
| where timestamp > ago(1h)
| summarize
    TotalRequests = count(),
    FailedRequests = countif(success == false),
    ErrorRate = round(100.0 * countif(success == false) / count(), 2)
  by name
| where ErrorRate > 1
| order by ErrorRate desc

3. Exception Hotspots

kql
exceptions
| where timestamp > ago(24h)
| summarize Count = count() by type, outerMessage, problemId
| order by Count desc
| take 20

4. Dependency Failures (SQL, Redis, HTTP calls)

kql
dependencies
| where timestamp > ago(1h)
| where success == false
| summarize
    FailCount = count(),
    AvgDuration = avg(duration)
  by target, type, resultCode
| order by FailCount desc

5. Custom Business Metrics

kql
customEvents
| where name == "OrderCreated"
| where timestamp > ago(24h)
| extend
    OrderTotal = todouble(customMeasurements.OrderTotal),
    ProcessingTime = todouble(customMeasurements.ProcessingTimeMs),
    PaymentMethod = tostring(customDimensions.PaymentMethod)
| summarize
    OrderCount = count(),
    TotalRevenue = sum(OrderTotal),
    AvgProcessingTime = avg(ProcessingTime)
  by bin(timestamp, 1h), PaymentMethod

Metric Alerts vs Log Alerts

Azure Monitor Alert Rules - Playbook

Metric Alert - High Response Time

bash
# Alert khi response time P95 > 3 giây
az monitor metrics alert create \
  --resource-group rg-monitoring \
  --name "alert-high-response-time" \
  --scopes /subscriptions/{sub}/resourceGroups/rg-app/providers/Microsoft.Web/sites/app-orderapi \
  --condition "avg HttpResponseTime > 3" \
  --window-size 5m \
  --evaluation-frequency 1m \
  --severity 2 \
  --action-group ag-platform-team \
  --description "Response time exceeds 3s threshold"

Log Alert - Error Spike

bash
# Alert khi error rate > 5% trong 5 phút
az monitor scheduled-query create \
  --resource-group rg-monitoring \
  --name "alert-error-spike" \
  --scopes /subscriptions/{sub}/resourceGroups/rg-monitoring/providers/Microsoft.OperationalInsights/workspaces/law-central-prod \
  --condition "count 'ErrorRate' > 5" \
  --condition-query "requests | where timestamp > ago(5m) | summarize ErrorRate = round(100.0 * countif(success == false) / count(), 2) | where ErrorRate > 5" \
  --severity 1 \
  --evaluation-frequency 5m \
  --window-size 5m \
  --action-groups ag-platform-team

Action Groups & Notification Channels

bash
# Tạo Action Group
az monitor action-group create \
  --resource-group rg-monitoring \
  --name ag-platform-team \
  --short-name PlatformTeam \
  --email-receiver name=TeamLead email=lead@company.com \
  --email-receiver name=OnCall email=oncall@company.com \
  --webhook-receiver name=Slack uri="https://hooks.slack.com/services/xxx" \
  --webhook-receiver name=PagerDuty uri="https://events.pagerduty.com/integration/xxx/enqueue"

Alert Severity Guide

Distributed Tracing với Correlation IDs

csharp
// Middleware để ensure correlation ID propagation
public class CorrelationIdMiddleware
{
    private readonly RequestDelegate _next;

    public CorrelationIdMiddleware(RequestDelegate next) => _next = next;

    public async Task InvokeAsync(HttpContext context)
    {
        // Lấy correlation ID từ header hoặc tạo mới
        var correlationId = context.Request.Headers["X-Correlation-ID"]
            .FirstOrDefault() ?? Activity.Current?.Id ?? Guid.NewGuid().ToString();

        context.Items["CorrelationId"] = correlationId;
        context.Response.Headers["X-Correlation-ID"] = correlationId;

        // Set correlation vào Application Insights
        var telemetry = context.Features.Get<RequestTelemetry>();
        if (telemetry != null)
        {
            telemetry.Properties["CorrelationId"] = correlationId;
        }

        using (LogContext.PushProperty("CorrelationId", correlationId))
        {
            await _next(context);
        }
    }
}

Tracing across services - KQL query

kql
// Trace toàn bộ request flow bằng Operation ID
union requests, dependencies, exceptions
| where operation_Id == "abc123xyz"
| project
    timestamp,
    itemType,
    name,
    duration,
    success,
    resultCode,
    target,
    type
| order by timestamp asc

Dashboard Creation - Bicep

bicep
resource dashboard 'Microsoft.Portal/dashboards@2022-12-01-preview' = {
  name: 'dash-app-overview'
  location: location
  tags: {
    'hidden-title': 'Application Overview Dashboard'
  }
  properties: {
    lenses: [
      {
        order: 0
        parts: [
          {
            position: { x: 0, y: 0, colSpan: 6, rowSpan: 4 }
            metadata: {
              type: 'Extension/Microsoft_OperationsManagementSuite_Workspace/PartType/LogsDashboardPart'
              inputs: [
                { name: 'query', value: 'requests | where timestamp > ago(24h) | summarize count() by bin(timestamp, 1h), success' }
              ]
            }
          }
        ]
      }
    ]
  }
}

Monitoring Checklist cho Production .NET App

Trade-off Analysis

Nguyên tắc: Bắt đầu với Application Insights + 4 basic alerts (error rate, response time, availability, CPU). Mở rộng monitoring khi hiểu rõ hơn patterns của application.