This skill is adpated from "Logging sucks. And here's how to make it better. by Boris Tane.
When helping with logging, observability, or debugging strategies, follow these principles:
Core Philosophy
- •Logs are optimized for querying, not writing — always design with debugging in mind
- •Context is everything — a log without correlation IDs is useless in distributed systems
- •Logs are for humans during incidents, not just for compliance or "just in case"
- •If you can't filter and search your logs effectively, they provide zero value
Structured Logging Requirements
- •Always use key-value pairs (JSON) instead of string interpolation
- •Bad: "Payment failed for user 123"
- •Good: {"event": "payment_failed", "user_id": "123", "reason": "insufficient_funds", "amount": 99.99}
- •Structured logs are machine-parseable, enabling aggregation, alerting, and dashboards
Required Fields for Every Log Event
- •timestamp — ISO 8601 with timezone (e.g., 2025-01-24T20:00:00Z)
- •level — debug, info, warn, error (be consistent, don't invent new levels)
- •event — machine-readable event name, snake_case (e.g., user_login_success)
- •request_id or trace_id — for correlating logs across a single request
- •service — which service/application emitted this log
- •environment — prod, staging, dev
Examples of High-Cardinality Fields (always include when available):
- •user_id, org_id, account_id — who is affected
- •request_id, trace_id, span_id — for distributed tracing
- •order_id, transaction_id, job_id — domain-specific identifiers
These fields are what make logs actually queryable during incidents. Without them, you're grepping through millions of lines blindly.
Look for opportunities for high-cardinality fields that can help you identify the root cause of an issue quickly.
Context Propagation
- •Pass trace/request IDs through all service boundaries (HTTP headers, message queues, etc.)
- •Downstream services must inherit correlation IDs from upstream
- •Use middleware or interceptors to automatically inject context into every log
- •For async jobs, store and restore the original request context
Log Levels — Use Them Correctly:
- •debug — Verbose details for local development, usually disabled in production
- •info — Normal operations worth recording (user actions, job completions, deploys)
- •warn — Something unexpected happened but the system handled it (retries, fallbacks)
- •error — Something failed and likely needs human attention (exceptions, failed requests)
Don't log errors for expected conditions (e.g., user enters wrong password)
What to Log:
- •Request entry and exit points (with duration)
- •State transitions (order created → paid → shipped)
- •External service calls (with latency and response codes)
- •Authentication and authorization events
- •Background job starts, completions, and failures
- •Retry attempts and circuit breaker state changes
What NOT to Log:
- •Sensitive data (passwords, tokens, PII, credit card numbers)
- •Logs inside tight loops (will generate millions of useless entries)
- •Success cases that provide no debugging value
- •Redundant information already captured by infrastructure (load balancer logs, etc.)
Naming Conventions:
- •Be consistent across all services — agree on field names as a team
- •Use snake_case for field names: user_id, not userId or user-id
- •Use past-tense verbs for events: payment_completed, not complete_payment
- •Prefix events by domain when helpful: auth.login_failed, billing.invoice_created
Performance Considerations:
- •Use sampling for high-volume debug logs in production
- •Avoid logging inside hot paths unless absolutely necessary
- •Buffer and batch log writes to reduce I/O overhead
- •Consider log levels that can be changed at runtime without redeploying
During Incidents:
- •Your logs should answer: Who was affected? What failed? When? Why?
- •If you can't answer these within 5 minutes of querying, your logging strategy needs work
- •Post-incident: add the logs you wished you had