Session 01 - OpenTelemetry & Distributed Tracing

Tổng quan OpenTelemetry

OpenTelemetry (OTel) là vendor-neutral standard cho observability, cung cấp 3 pillars:

Tại sao chọn OpenTelemetry?

• Vendor lock-in elimination: Đổi backend (Jaeger → Zipkin → Azure Monitor) mà không sửa code
• W3C Trace Context: Standard propagation header (traceparent, tracestate)
• Auto-instrumentation: HTTP, gRPC, EF Core, HttpClient tự động tạo spans

.NET Integration - Complete Setup

1. NuGet Packages

xml
<ItemGroup>
  <PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.9.*" />
  <PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.9.*" />
  <PackageReference Include="OpenTelemetry.Instrumentation.Http" Version="1.9.*" />
  <PackageReference Include="OpenTelemetry.Instrumentation.SqlClient" Version="1.9.*" />
  <PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.9.*" />
  <PackageReference Include="Azure.Monitor.OpenTelemetry.Exporter" Version="1.3.*" />
</ItemGroup>

2. Program.cs Configuration

csharp
using OpenTelemetry;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
using OpenTelemetry.Metrics;
using OpenTelemetry.Logs;

var builder = WebApplication.CreateBuilder(args);

var serviceName = "OrderService";
var serviceVersion = "1.0.0";

// Resource cho tất cả signals
var resource = ResourceBuilder.CreateDefault()
    .AddService(serviceName, serviceVersion: serviceVersion)
    .AddAttributes(new Dictionary<string, object>
    {
        ["deployment.environment"] = builder.Environment.EnvironmentName,
        ["host.name"] = Environment.MachineName
    });

// === TRACING ===
builder.Services.AddOpenTelemetry()
    .WithTracing(tracing =>
    {
        tracing
            .SetResourceBuilder(resource)
            .AddAspNetCoreInstrumentation(opts =>
            {
                opts.RecordException = true;
                opts.Filter = ctx => !ctx.Request.Path.StartsWithSegments("/health");
            })
            .AddHttpClientInstrumentation()
            .AddSqlClientInstrumentation(opts =>
            {
                opts.SetDbStatementForText = true; // Caution: có thể log sensitive data
                opts.RecordException = true;
            })
            .AddSource("OrderService.Activities") // Custom ActivitySource
            .AddOtlpExporter(opts =>
            {
                opts.Endpoint = new Uri("http://otel-collector:4317");
            });
    })
    // === METRICS ===
    .WithMetrics(metrics =>
    {
        metrics
            .SetResourceBuilder(resource)
            .AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation()
            .AddMeter("OrderService.Metrics") // Custom Meter
            .AddOtlpExporter();
    });

// === LOGGING ===
builder.Logging.AddOpenTelemetry(logging =>
{
    logging.SetResourceBuilder(resource);
    logging.IncludeScopes = true;
    logging.IncludeFormattedMessage = true;
    logging.AddOtlpExporter();
});

3. Custom Spans & Attributes

csharp
using System.Diagnostics;

public class OrderService
{
    private static readonly ActivitySource _activitySource = new("OrderService.Activities");

    public async Task<Order> CreateOrderAsync(CreateOrderRequest request)
    {
        using var activity = _activitySource.StartActivity(
            "CreateOrder",
            ActivityKind.Internal);

        // Thêm attributes để debug và search
        activity?.SetTag("order.customer_id", request.CustomerId);
        activity?.SetTag("order.item_count", request.Items.Count);
        activity?.SetTag("order.total_amount", request.TotalAmount);

        try
        {
            var order = await ProcessOrder(request);

            activity?.SetTag("order.id", order.Id);
            activity?.SetStatus(ActivityStatusCode.Ok);

            return order;
        }
        catch (Exception ex)
        {
            activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
            activity?.RecordException(ex);
            throw;
        }
    }
}

Trace Propagation

Correlation ID Middleware

csharp
public class CorrelationIdMiddleware
{
    private readonly RequestDelegate _next;

    public CorrelationIdMiddleware(RequestDelegate next) => _next = next;

    public async Task InvokeAsync(HttpContext context)
    {
        var correlationId = context.Request.Headers["X-Correlation-Id"].FirstOrDefault()
            ?? Activity.Current?.TraceId.ToString()
            ?? Guid.NewGuid().ToString();

        context.Items["CorrelationId"] = correlationId;
        context.Response.Headers["X-Correlation-Id"] = correlationId;

        using (var scope = context.RequestServices
            .GetRequiredService<ILogger<CorrelationIdMiddleware>>()
            .BeginScope(new Dictionary<string, object> { ["CorrelationId"] = correlationId }))
        {
            await _next(context);
        }
    }
}

Sampling Strategies

csharp
// Production: Sample 10% + always sample errors
tracing.SetSampler(new ParentBasedSampler(
    new TraceIdRatioBasedSampler(0.1)));

Tail-based Sampling (Collector-side)

Thay vì quyết định lúc tạo trace, tail-based sampling chờ trace hoàn thành rồi mới quyết định:

• Giữ lại 100% traces có error
• Giữ lại traces có latency > P99
• Sample random cho traces bình thường

Exporters Comparison

Trade-offs & Best Practices

Nên

• Dùng OTel Collector làm trung gian thay vì export trực tiếp từ app
• Đặt semantic conventions cho span names (http.method, db.system)
• Luôn set RecordException = true cho production

Tránh

• Over-instrumentation: Không tạo span cho mọi method call - chỉ tạo cho boundary operations
• Sensitive data trong tags: Không log PII trong span attributes
• Quá nhiều cardinality: Tránh dùng userId làm metric label - sẽ explode time series