Observability Standards

Guidance for implementing observability requirements including logging, metrics, tracing, and monitoring configuration.

Tooling

Available Tools: If using Claude Code, the agents:sre-engineer agent specializes in observability setup and SLO management. The agents:devops-engineer agent can help configure monitoring infrastructure.

Logging Requirements

Structured Logging (MUST)

All logging MUST use structured format (JSON preferred):

{
  "timestamp": "2024-01-15T10:30:00.000Z",
  "level": "info",
  "message": "Request processed",
  "service": "api-gateway",
  "trace_id": "abc123",
  "span_id": "def456",
  "duration_ms": 45,
  "status_code": 200
}

Log Levels (MUST)

Use consistent log levels with defined semantics:

Required Log Fields (MUST)

All log entries MUST include:

Recommended Log Fields (SHOULD)

Log entries SHOULD include when applicable:

Sensitive Data (MUST NOT)

Logs MUST NOT contain:

• •Passwords or secrets
• •API keys or tokens
• •Personal identifiable information (PII)
• •Credit card numbers
• •Session tokens

Metrics Requirements

Metric Types (MUST)

Implement appropriate metric types:

Required Metrics (MUST)

Services MUST expose:

Metric Naming (MUST)

Follow naming conventions:

# Format: <namespace>_<name>_<unit>
http_requests_total
http_request_duration_seconds
db_connections_active
cache_hits_total

Metric Labels (SHOULD)

Use consistent label naming:

Distributed Tracing

Tracing Implementation (SHOULD)

Implement distributed tracing using OpenTelemetry:

# OpenTelemetry configuration
exporters:
  otlp:
    endpoint: "collector:4317"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]

Trace Context (MUST)

When tracing is implemented, propagate context:

Span Requirements (SHOULD)

Spans SHOULD include:

• •Operation name
• •Start/end timestamps
• •Status (OK, ERROR)
• •Relevant attributes
• •Error details (if applicable)

Health Checks

Health Endpoints (MUST)

Services MUST expose health endpoints:

Health Response Format (MUST)

{
  "status": "healthy",
  "checks": {
    "database": {
      "status": "healthy",
      "latency_ms": 5
    },
    "cache": {
      "status": "healthy",
      "latency_ms": 1
    }
  },
  "version": "1.2.3",
  "uptime_seconds": 3600
}

Dependency Checks (SHOULD)

Health checks SHOULD verify:

• •Database connectivity
• •Cache availability
• •External service reachability
• •Disk space adequacy
• •Memory availability

Alerting

Alert Configuration (MUST)

Define alerts for critical conditions:

Alert Requirements (MUST)

Alerts MUST include:

• •Clear description of condition
• •Severity level
• •Runbook link
• •Affected service/component

Implementation Checklist

• • Configure structured logging
• • Define log level policies
• • Implement required metrics
• • Set up health endpoints
• • Configure distributed tracing (if applicable)
• • Define alerting rules
• • Create runbooks for alerts
• • Verify sensitive data exclusion

Compliance Verification

# Verify structured log output
app_command 2>&1 | jq .

# Check health endpoint
curl -s http://localhost:8080/health | jq .

# Verify metrics endpoint
curl -s http://localhost:8080/metrics | grep -E "^(http_|app_)"

# Check for sensitive data in logs
grep -r -i "password\|secret\|token" logs/ | wc -l
# Should be 0

Language-Specific Logging

Rust (tracing)

use tracing::{info, instrument};

#[instrument]
fn process_request(id: &str) {
    info!(request_id = %id, "Processing request");
}

TypeScript (pino)

import pino from "pino";

const logger = pino({
  level: "info",
  formatters: {
    level: (label) => ({ level: label }),
  },
});

logger.info({ requestId, duration }, "Request processed");

Python (structlog)

import structlog

logger = structlog.get_logger()
logger.info("request_processed", request_id=request_id, duration=duration)

Additional Resources

Reference Files

• •references/logging-config.md - Logging configuration guide
• •references/metrics-guide.md - Metrics implementation patterns

Examples

• •examples/otel-config.yaml - OpenTelemetry configuration
• •examples/alerting-rules.yaml - Prometheus alerting rules