Log Analysis Methodology

Core Philosophy: Partition-First

NEVER start by reading raw log samples.

Logs can be overwhelming. The partition-first approach prevents:

• •Missing the forest for the trees
• •Wasting time on irrelevant data
• •Overwhelming context with noise

The 4-Step Process

Step 1: Get Statistics

Before ANY log search, understand the landscape:

CloudWatch Insights:

# How many errors?
filter @message like /ERROR/
| stats count(*) as total

# Error rate over time
filter @message like /ERROR/
| stats count(*) by bin(5m)

# What types of errors?
filter @message like /ERROR/
| parse @message /(?<error_type>[\w.]+Exception)/
| stats count(*) by error_type
| sort count desc

Datadog:

# Error distribution by service
service:* status:error | stats count by service

# Error types
service:myapp status:error | stats count by @error.kind

Questions to answer:

• •What's the total error volume?
• •Is it increasing, stable, or decreasing?
• •What are the unique error types?
• •Which services/hosts are affected?

Step 2: Identify Patterns

Look for correlations:

Temporal patterns:

• •Did errors start at a specific time?
• •Is there periodicity (every hour, every day)?
• •Correlation with deployments or traffic spikes?

Service patterns:

• •Is one service the source?
• •Is the error propagating across services?

Error patterns:

• •What's the most frequent error?
• •Are errors clustered or distributed?

Step 3: Sample Strategically

Only NOW read actual log samples:

Sample from anomalies:

• •Get logs from the peak error time
• •Get logs from normal time for comparison

Sample by error type:

• •Get examples of each distinct error type
• •Limit to 5-10 per type

Sample around events:

• •Logs before/after a deployment
• •Logs around a specific incident timestamp

Step 4: Correlate with Events

Connect logs to system changes:

# Use git_log to find recent deployments
git_log --since="2 hours ago"

# Use get_deployment_history for K8s
get_deployment_history deployment=api-server

# Compare log patterns before/after changes

Platform-Specific Tips

CloudWatch Insights

Best practices:

# Always include time filter
filter @timestamp > ago(1h)

# Use parse for structured extraction
parse @message /status=(?<status>\d+)/

# Aggregate before displaying
stats count(*) by status | sort count desc | limit 10

Common queries:

# Latency distribution
filter @type = "REPORT"
| stats avg(@duration) as avg,
        pct(@duration, 95) as p95,
        pct(@duration, 99) as p99

# Error messages with context
filter @message like /ERROR/
| fields @timestamp, @message
| sort @timestamp desc
| limit 20

Datadog Logs

Query syntax:

# Filter by service and status
service:api-gateway status:error

# Field queries
@http.status_code:>=500

# Wildcard
@error.message:*timeout*

# Time comparison
service:api (now-1h TO now) vs (now-25h TO now-24h)

Kubernetes Logs

Use get_pod_logs wisely:

• •Always specify tail_lines (default: 100)
• •Filter to specific containers in multi-container pods
• •Use get_pod_events first for crashes/restarts

Anti-Patterns to Avoid

1. •Dumping all logs - Never request unbounded log queries
2. •Starting with samples - Always get statistics first
3. •Ignoring time windows - Narrow to incident window
4. •Missing correlation - Always connect to deployments/changes
5. •Single-service focus - Check upstream/downstream services

Investigation Template

## Log Analysis Report

### Statistics
- Time window: [start] to [end]
- Total log volume: X events
- Error count: Y events (Z%)
- Error rate trend: [increasing/stable/decreasing]

### Top Error Types
1. [ErrorType1]: N occurrences - [description]
2. [ErrorType2]: M occurrences - [description]

### Temporal Pattern
- Errors started at: [timestamp]
- Correlation: [deployment X / traffic spike / external event]

### Sample Errors
[Quote 2-3 representative error messages]

### Root Cause Hypothesis
[Based on patterns, what's the likely cause?]