Monitoring & Observability Skill

Purpose

Standards for monitoring, metrics, alerting, and observability.

Auto-Invoke Triggers

• •Setting up monitoring infrastructure
• •Defining metrics and KPIs
• •Configuring alerts
• •Implementing distributed tracing

Three Pillars of Observability

Key Metrics (Golden Signals)

The Four Golden Signals

RED Method (Request-focused)

• •Rate - Requests per second
• •Errors - Failed requests per second
• •Duration - Request latency

USE Method (Resource-focused)

• •Utilization - Resource % used
• •Saturation - Queue depth
• •Errors - Error count

Application Metrics

HTTP Endpoints

Database

Business Metrics

Metric Types

Naming Convention

{namespace}_{subsystem}_{name}_{unit}

http_server_request_duration_seconds
db_pool_connections_active
app_users_registered_total

Alerting Strategy

Alert Severity

Alert Rules

• •Alert on symptoms, not causes
• •Include runbook links
• •Set appropriate thresholds
• •Avoid alert fatigue
• •Group related alerts

What to Alert On

SLOs and SLIs

Service Level Indicators (SLIs)

• •Availability: Successful requests / Total requests
• •Latency: % requests < threshold
• •Throughput: Requests per second
• •Error rate: Failed requests / Total requests

Service Level Objectives (SLOs)

Error Budget

• •Monthly allowed downtime
• •Spend on risky deployments
• •Freeze deploys when exhausted

Distributed Tracing

Concepts

What to Trace

• •Cross-service calls
• •Database queries
• •External API calls
• •Message queue operations

Trace Propagation

• •Pass trace context in headers
• •Standard: W3C Trace Context
• •Headers: traceparent, tracestate

Health Checks

Endpoint Types

Readiness Check Components

• •Database connection
• •Cache connection
• •External service connectivity
• •Required configuration present

Health Response Format

{
  "status": "healthy",
  "checks": {
    "database": "healthy",
    "cache": "healthy",
    "external-api": "degraded"
  },
  "version": "1.2.3"
}

Dashboard Design

Layout Principles

• •Most important metrics at top
• •Group related metrics
• •Use consistent time ranges
• •Include context (deploy markers)

Standard Panels

1. •Overview - Traffic, errors, latency
2. •Resources - CPU, memory, disk
3. •Dependencies - DB, cache, external APIs
4. •Business - Domain-specific metrics

Best Practices

• •Link to runbooks
• •Add annotations for deploys
• •Use consistent colors
• •Set reasonable refresh rates

Runbooks

Structure

1. •Alert description - What triggered
2. •Impact - User/business effect
3. •Diagnosis steps - How to investigate
4. •Resolution steps - How to fix
5. •Escalation - Who to contact

Required Runbooks

• •Service restart procedure
• •Database failover
• •Rollback deployment
• •Scale up/down
• •Incident communication

Best Practices

DO

• •Monitor the four golden signals
• •Set up alerts before incidents
• •Create runbooks for alerts
• •Use distributed tracing
• •Track SLOs and error budgets
• •Review dashboards regularly

DON'T

• •Alert on every metric
• •Ignore alert fatigue
• •Skip health checks
• •Forget to trace async operations
• •Set unrealistic SLOs
• •Neglect runbook maintenance

Observability Checklist

Metrics

• • Golden signals tracked
• • Custom business metrics
• • Resource utilization metrics
• • Dependency health metrics

Alerting

• • Critical alerts defined
• • Severity levels set
• • Runbooks linked
• • On-call rotation configured

Tracing

• • Trace context propagated
• • Key operations traced
• • Trace sampling configured

Dashboards

• • Service overview dashboard
• • Dependency dashboard
• • Business metrics dashboard
• • Deploy annotations enabled