Observability Design

Purpose

Design a comprehensive observability strategy covering metrics, logging, tracing, alerting, and SLI/SLO definitions. Produces a monitoring architecture that enables rapid incident detection, diagnosis, and resolution.

Inputs

• •System architecture (services, databases, APIs, third-party dependencies)
• •Current monitoring setup (existing tools, dashboards, alerts)
• •Reliability requirements (SLA commitments, uptime targets)
• •Team structure (on-call rotation, escalation paths)

Process

Step 1: Define Observability Pillars

Establish the three pillars for this system:

• •Metrics: What to measure — request rate, error rate, latency, saturation, business KPIs
• •Logs: What to record — request lifecycle, state changes, errors, audit events
• •Traces: What to follow — cross-service request flows, database queries, external API calls
• •Map each pillar to specific use cases: debugging, alerting, capacity planning, business intelligence

Step 2: Design Metric Collection

Define the metric taxonomy:

• •Application metrics: Request count, error count, latency histograms, queue depth, cache hit rate
• •Infrastructure metrics: CPU, memory, disk I/O, network throughput, connection pool utilization
• •Business metrics: Sign-ups, conversions, revenue events, feature adoption rates
• •Custom instrumentation: Counters (events), gauges (current values), histograms (distributions)
• •Specify metric naming conventions, label/tag strategy, and cardinality limits

Step 3: Define Alert Thresholds and Escalation

Design the alerting strategy:

• •Warning alerts: Early indicators — elevated error rate, latency creep, resource approaching limits
• •Critical alerts: Immediate action required — service down, error rate spike, SLO burn rate exceeded
• •Escalation paths: Primary on-call → secondary → engineering lead → incident commander
• •Runbook links: Every alert includes a link to its diagnosis and remediation runbook
• •Alert fatigue prevention: Grouping, deduplication, silence windows, alert quality reviews

Step 4: Plan Structured Logging

Design the logging architecture:

• •Log levels: DEBUG (development only), INFO (normal operations), WARN (unexpected but handled), ERROR (requires attention)
• •Structured fields: timestamp, service, request_id, user_id, action, duration_ms, status
• •Correlation IDs: Request ID propagation across services for distributed request tracing
• •PII redaction: Identify sensitive fields, implement automatic redaction/masking
• •Log aggregation: Collection, indexing, retention periods, search capabilities

Step 5: Design Distributed Tracing

Plan request flow visibility:

• •Span naming conventions: service.operation format, consistent across services
• •Context propagation: How trace context passes between services (headers, message metadata)
• •Sampling strategy: Head-based vs tail-based sampling, sampling rate by endpoint or error status
• •Trace enrichment: Adding business context (user tier, feature flag state) to spans
• •Critical paths: Which request flows must always be traced (payments, auth, data mutations)

Step 6: Specify Dashboard Requirements

Define dashboard hierarchy:

• •Operational dashboards: Service health overview, real-time traffic, error rates, latency percentiles
• •Business dashboards: User activity, feature adoption, conversion funnels, revenue metrics
• •SLO dashboards: Error budget remaining, burn rate, SLO compliance history
• •Incident dashboards: Pre-built investigation views for common failure modes
• •Specify dashboard layout, refresh intervals, time range defaults, and access controls

Step 7: Define SLIs/SLOs

Establish reliability targets:

• •Availability SLI: Successful requests / total requests (define "successful")
• •Latency SLI: Proportion of requests faster than threshold (p50, p95, p99 targets)
• •Error rate SLI: Proportion of requests without errors (define "error")
• •SLO targets: e.g., 99.9% availability, p95 latency < 200ms, error rate < 0.1%
• •Error budgets: Calculate error budget from SLO, define burn rate alerts (fast burn, slow burn)
• •SLO review cadence: Weekly error budget check, monthly SLO review, quarterly target adjustment

Output Format

# Observability Design: [Service/Feature Name]

## Observability Architecture

[Application] → [Metrics Agent] → [Metrics Store] → [Dashboards] ↓ ↓ [Structured Logs] → [Log Aggregator] → [Log Search] [Alerts] → [On-call] ↓ [Trace SDK] → [Trace Collector] → [Trace UI]

## Metric Catalog

| Metric Name | Type | Labels | Description | Alert Threshold |
|-------------|------|--------|-------------|-----------------|
| http_requests_total | counter | method, path, status | Request count | N/A |
| http_request_duration_ms | histogram | method, path | Request latency | p95 > 500ms |
| ... | ... | ... | ... | ... |

## Alert Catalog

| Alert Name | Severity | Condition | Duration | Runbook |
|------------|----------|-----------|----------|---------|
| HighErrorRate | critical | error_rate > 5% | 5m | [link] |
| LatencyDegraded | warning | p95 > 500ms | 10m | [link] |
| ... | ... | ... | ... | ... |

## Logging Schema

```json
{
  "timestamp": "ISO8601",
  "level": "INFO",
  "service": "api",
  "request_id": "uuid",
  "user_id": "string (optional)",
  "action": "string",
  "duration_ms": "number",
  "status": "number",
  "message": "string"
}

SLI/SLO Definitions

Dashboard Specifications

## Quality Checks

- [ ] All three observability pillars (metrics, logs, traces) are covered
- [ ] Every alert has a defined severity, threshold, and linked runbook
- [ ] Structured logging schema includes correlation IDs for distributed tracing
- [ ] PII fields are identified with redaction strategy
- [ ] SLIs are measurable and SLO targets are realistic for the service tier
- [ ] Error budgets are calculated with burn rate alert thresholds
- [ ] Dashboard hierarchy covers operational, business, and SLO views
- [ ] Sampling strategy balances trace coverage with storage costs

## Evolution Notes
<!-- Observations appended after each use -->