Implementing Observability

Goal

Make the system's internal state inferable from its external outputs. Answer "Why is it slow?" and "Why did it fail?" without SSH-ing into a server.

When to Use

• •Before launching to production.
• •When debugging a performance bottleneck.
• •When integrating a new microservice or external API.

Instructions

1. Structured Logging

Text logs are hard to query. Use JSON.

• •Context: Every log must have trace_id, request_id, user_id.
• •Levels: INFO for normal ops, WARN for handled issues, ERROR for unhandled crashes.

{"level": "info", "msg": "User logged in", "user_id": 123, "trace_id": "abc-123"}

2. Distributed Tracing (OpenTelemetry)

Trace a request across boundaries (Frontend -> API -> DB).

• •Instrument HTTP clients and server frameworks.
• •Visualize the "waterfall" to find the slow span.

3. Golden Signals (Metrics)

Track the four key metrics for every service:

• •Latency: Time to serve a request.
• •Traffic: Request rate (RPS).
• •Errors: Rate of 5xx responses.
• •Saturation: CPU/Memory/Disk usage.

4. Alerting

Alert on symptoms (High Error Rate), not causes (High CPU).

• •Page: If Error Rate > 1% for 5 minutes.
• •Ticket: If Disk Usage > 80%.

Constraints

✅ Do

• •DO: Use OpenTelemetry standards for portability.
• •DO: Correlate logs and traces (inject trace ID into logs).
• •DO: Sample high-volume traces (10%) to save costs, but keep 100% of errors.

❌ Don't

• •DON'T: Log PII (Emails, Passwords, Credit Cards).
• •DON'T: Create alerts that auto-resolve in seconds (flapping).
• •DON'T: Rely solely on "system up" checks; check "business logic working".

Output Format

• •docker-compose.yml with Prometheus/Grafana/Jaeger (for dev).
• •Code instrumentation (e.g., tracing.py).

Dependencies

• •backend/managing-flask-middleware/SKILL.md (where instrumentation lives)
• •shared/debugging/SKILL.md