Incident Log Analyzer

Analyze logs to identify patterns, extract errors, and generate actionable insights for incident response.

Instructions

1. •

   Discovery Phase

   • •Locate log files using Glob tool
   • •Identify log format (JSON, plain text, structured)
   • •Determine time range of incident
2. •

   Analysis Phase

   • •Run scripts/parse_logs.py to extract structured data
   • •Use scripts/pattern_detector.py to find error patterns
   • •Identify:
     • •Error frequency and distribution
     • •Affected components/services
     • •Timeline of events
     • •Potential root causes
3. •

   Report Phase

   • •Generate incident summary with key metrics
   • •Provide timeline visualization
   • •List top errors with context
   • •Suggest next investigation steps

Usage Examples

Example 1: Analyze application logs

User: "Analyze the logs in /var/logs/app/ for errors in the last hour"

Claude executes:
1. python scripts/parse_logs.py /var/logs/app/ --since "1 hour ago"
2. python scripts/pattern_detector.py --input parsed_logs.json
3. Generates report with findings

Example 2: Root cause analysis

User: "Why did the API start failing at 3 PM?"

Claude executes:
1. Filters logs around 3 PM
2. Identifies spike in 500 errors
3. Traces error source to database connection pool exhaustion
4. Provides evidence and recommendations

Example 3: Multi-service correlation

User: "Check if the frontend errors are related to backend issues"

Claude:
1. Analyzes frontend logs
2. Analyzes backend logs
3. Correlates timestamps and error patterns
4. Maps frontend errors to backend failures

Scripts

parse_logs.py

Parses log files and extracts structured data.

Usage:

python scripts/parse_logs.py <log_directory> [options]

Options:
  --format json|text|auto     Log format (default: auto)
  --since "time"              Start time (e.g., "1 hour ago", "2024-01-01")
  --until "time"              End time
  --level error|warn|info     Filter by log level
  --output FILE               Output file (default: parsed_logs.json)

Output: JSON file with structured log entries

pattern_detector.py

Detects patterns, clusters similar errors, generates statistics.

Usage:

python scripts/pattern_detector.py [options]

Options:
  --input FILE                Input JSON from parse_logs.py
  --threshold N               Minimum occurrences to report (default: 5)
  --output FILE               Output report file

Output: JSON report with error patterns and statistics

timeline_visualizer.py

Generates ASCII timeline visualization of incidents.

Usage:

python scripts/timeline_visualizer.py --input parsed_logs.json

Output: ASCII chart showing error frequency over time

Report Format

# Incident Log Analysis Report
**Analysis Period**: 2024-01-01 14:00 - 15:00
**Logs Analyzed**: 45,234 entries
**Errors Found**: 1,247

## Summary
Critical errors detected in payment service causing cascade failures
across dependent services.

## Timeline

14:05 ████░░░░░░░░ First errors appear (DB connection) 14:15 ████████████ Error spike (payment service) 14:30 ████████░░░░ Partial recovery 14:45 ██░░░░░░░░░░ Normal operation resumed

## Top Errors
1. **DatabaseConnectionError** (423 occurrences)
   - First seen: 14:05:23
   - Last seen: 14:32:15
   - Affected: payment-service, order-service
   - Pattern: Connection pool exhausted

2. **PaymentTimeoutException** (312 occurrences)
   - First seen: 14:08:45
   - Last seen: 14:28:33
   - Affected: payment-service
   - Pattern: Downstream service timeout

## Root Cause Analysis
**Primary**: Database connection pool exhausted
**Contributing factors**:
- Sudden traffic spike (3x normal)
- Connection timeout too high (30s)
- No connection pooling limits

## Recommendations
1. Increase connection pool size
2. Reduce connection timeout to 5s
3. Implement circuit breaker
4. Add connection pool monitoring

## Evidence

[14:05:23] ERROR [payment-service] DatabaseConnectionError: Cannot get connection from pool (exhausted) at ConnectionPool.getConnection()

[14:08:45] ERROR [payment-service] PaymentTimeoutException: Timeout waiting for payment processor response at PaymentGateway.processPayment()

Advanced Features

Correlation Analysis

The analyzer can correlate errors across multiple services by:

• •Matching request IDs across logs
• •Analyzing temporal proximity
• •Identifying cascade failures

Anomaly Detection

Detects unusual patterns:

• •Sudden error rate changes
• •New error types
• •Missing expected log entries
• •Irregular timing patterns

Metrics Extraction

Automatically extracts:

• •Error rate (errors/minute)
• •Mean time between failures (MTBF)
• •Error distribution by severity
• •Service availability percentage

Integration with Subagents

This Skill can delegate to:

• •log-slicer subagent: For detailed log segmentation
• •sre-incident-scribe style: For incident documentation

Best Practices

1. •Always specify time range: Reduces noise and speeds analysis
2. •Check multiple log sources: Single source may not show full picture
3. •Look for patterns, not just errors: Warnings often precede failures
4. •Correlate with deployments: Check if errors started after deploy
5. •Preserve evidence: Copy relevant log sections for postmortem

Dependencies

Scripts require Python 3.8+ with:

pip install python-dateutil pandas numpy

For JSON logs: jq command-line tool (optional, improves performance)