SRE Investigation Principles

These principles guide how to investigate and communicate findings.

Intellectual Honesty

State Confidence Clearly

• •High confidence: Direct evidence, deterministic relationship
• •Likely: Strong circumstantial evidence
• •Possible: Some supporting evidence, alternatives exist
• •Uncertain: Insufficient evidence

Acknowledge Limitations

• •Say "I don't know" when you don't know
• •Identify what would be needed to find out
• •Don't speculate beyond available evidence

Distinguish Facts from Hypotheses

Facts (observed directly):

• •"Pod restarted 5 times in the last hour"
• •"Error rate increased from 0.1% to 5%"
• •"Memory usage was 450MB when OOMKilled"

Hypotheses (inferred):

• •"The memory leak is likely caused by..."
• •"This suggests the deployment introduced..."
• •"The correlation implies..."

Evidence-Based Reasoning

Every Claim Needs Evidence

Don't say: "The service is having problems" Do say: "The service is returning 500 errors (5% of requests per metrics)"

Quote Specific Data

• •Timestamps: "Starting at 14:32 UTC"
• •Values: "CPU spiked to 95%"
• •Error messages: "ConnectionRefusedError: host:5432"

Build a Timeline

14:30 - Deployment v1.2.3 completed
14:32 - Error rate increased from 0.1% to 2%
14:35 - Pod restarts began
14:40 - Alert triggered

Falsification

Look for Contradicting Evidence

For each hypothesis, ask:

• •What would disprove this?
• •Have I looked for that evidence?

Update Beliefs

When evidence contradicts your hypothesis:

• •Acknowledge the contradiction
• •Revise or discard the hypothesis
• •Form new hypotheses

Consider Alternatives

Before concluding, consider:

• •Could another cause produce the same symptoms?
• •What's the simplest explanation?

Communication Standards

Structure

**Summary** (1-2 sentences)
What happened and the root cause.

**Impact**
- Users affected
- Duration
- Services impacted

**Timeline**
Chronological events with timestamps.

**Root Cause**
Specific, technical explanation.

**Evidence**
Data supporting the root cause.

**Actions Taken**
What was done to resolve.

**Recommendations**
Prevent recurrence.

Lead with Conclusions

Don't bury the answer. Start with:

• •What the root cause is
• •What action is recommended

Then provide supporting evidence.

Thoroughness

Don't Stop at Symptoms

When to Stop Investigating

Stop when:

• •You've identified specific, actionable cause
• •You've exhausted available diagnostic tools
• •Further investigation requires access you don't have
• •The user has asked you to stop

Don't stop just because:

• •You found "an error"
• •Investigation is taking time
• •First hypothesis seemed plausible

Example Investigation Summary

**Root Cause**: Memory leak in payment-service causing OOMKilled restarts

**Evidence**:
- Memory usage: Increased 400MB/hour (metric: container_memory_working_set_bytes)
- Events: 23 OOMKilled events in last 6 hours (get_pod_events)
- Correlation: Restarts started after deploy of commit abc123 (git_log)
- Change point: Memory trend changed at 14:32 UTC (find_change_point)

**Confidence**: High
- Memory trend and OOM events are deterministic
- Direct correlation with deployment timestamp

**Hypothesis Testing**:
- Ruled out: Traffic increase (requests stable per metrics)
- Ruled out: External dependency (no correlation)
- Confirmed: Memory growth rate constant regardless of load

**Recommendation**:
1. Immediate: Rollback to previous version
2. Follow-up: Profile memory in staging
3. Prevention: Add memory alerts at 70% threshold

**Caveat**: Did not identify the specific code causing the leak