Monitoring & Observability

Comprehensive monitoring, observability, and alerting strategies for production systems.

Three Pillars of Observability

Metrics

• •Quantitative measurements (counters, gauges, histograms)
• •Time-series data (Prometheus, InfluxDB, Datadog)
• •Examples: request latency, error rate, CPU usage

Logs

• •Structured event records
• •Searchable and filterable
• •Examples: application logs, access logs, error logs

Traces

• •Request flow through system
• •Distributed tracing (Jaeger, Zipkin)
• •Shows dependencies and bottlenecks

Implementation Approaches

Metrics Collection

from prometheus_client import Counter, Histogram

request_count = Counter('http_requests_total', 'Total requests')
latency = Histogram('http_request_duration_seconds', 'Request latency')

@app.route('/api/users')
def get_users():
    request_count.inc()
    with latency.time():
        return fetch_users()

Structured Logging

{
  "timestamp": "2025-02-07T10:30:00Z",
  "level": "ERROR",
  "service": "user-service",
  "request_id": "req_12345",
  "user_id": "user_789",
  "error_code": "DB_CONNECTION_FAILED",
  "message": "Failed to connect to database",
  "duration_ms": 1500
}

Distributed Tracing

• •Instrument application code
• •Propagate trace IDs across services
• •Collect traces centrally (Jaeger, Zipkin)
• •Visualize service dependencies

Popular Tools

Best Practices

1. •Structured Logging - JSON format with context
2. •Contextual Data - Request IDs, user IDs, service names
3. •Sampling - Don't log everything to save costs
4. •Retention Policy - Balance cost and retention needs
5. •Alerts - On error rates, latency, resource usage
6. •Dashboards - Visualize key metrics
7. •Runbooks - Document how to respond to alerts

Key Metrics to Monitor

• •Request rate and latency (p50, p95, p99)
• •Error rate and error types
• •Resource usage (CPU, memory, disk)
• •Database query performance
• •Cache hit rates
• •Queue depths
• •User session counts

References

• •Prometheus Monitoring Best Practices
• •Observability Engineering (O'Reilly)
• •Google SRE Book
• •ELK Stack Documentation
• •OpenTelemetry Project