Kubernetes Debugging Expertise

Golden Rule: Events Before Logs

When debugging Kubernetes issues, ALWAYS check events first:

1. •get_pod_events - Shows scheduling, pulling, starting, probes, OOM
2. •THEN get_pod_logs - Application-level errors

Events explain most crash/scheduling issues faster than logs.

Typical Investigation Flow

1. list_pods        → Get overview of pod health in namespace
2. get_pod_events   → Understand WHY pods are in their state
3. get_pod_logs     → Only if events don't explain the issue
4. get_pod_resources → For performance/resource issues
5. describe_deployment → Check deployment status and conditions

Common Issue Patterns

CrashLoopBackOff

First check: get_pod_events

Checklist:

• • Memory limits vs actual usage
• • Recent deployment changes (get_deployment_history)
• • Missing config/secrets
• • Dependency failures (database, external services)

OOMKilled

First check: get_pod_events (confirms OOMKilled) Then: get_pod_resources (compare usage to limits)

Common causes:

• •Memory limit set too low for workload
• •Memory leak (usage increases over time)
• •Sudden traffic spike causing memory pressure
• •Large request payloads cached in memory

ImagePullBackOff

First check: get_pod_events

Common causes:

• •Wrong image name or tag
• •Private registry without imagePullSecrets
• •Rate limiting from registry
• •Network issues reaching registry

Pending Pods

First check: get_pod_events

Look for:

• •FailedScheduling - Insufficient resources
• •Unschedulable - Node affinity/taints
• •No matching nodes for nodeSelector

Readiness/Liveness Probe Failures

First check: describe_pod (shows probe config) Then: get_pod_events (probe failure events) Then: get_pod_logs (why endpoint isn't responding)

Evicted Pods

First check: get_pod_events

Causes:

• •Node resource pressure (disk, memory)
• •Priority preemption
• •Taint-based eviction

Deployment Issues

Stuck Rollout

describe_deployment  → Check replicas (desired vs ready vs available)
get_deployment_history → Compare current vs previous revision
get_pod_events → For pods in new ReplicaSet

Common causes:

• •New pods failing (CrashLoopBackOff)
• •Readiness probes failing
• •Resource constraints preventing scheduling

Rollback Decision

Use get_deployment_history to see previous working versions.

Error Classification

Non-Retryable (Stop Immediately)

• •401 Unauthorized - Invalid credentials
• •403 Forbidden - No permission
• •404 Not Found - Resource doesn't exist
• •"config_required": true - Integration not configured

Retryable (May retry once)

• •429 Too Many Requests
• •500/502/503/504 Server errors
• •Timeout
• •Connection refused

Resource Investigation Pattern

For memory/CPU issues:

1. get_pod_resources → See allocation vs usage
2. describe_pod → See full container spec
3. get_cloudwatch_metrics/query_datadog_metrics → Historical usage
4. detect_anomalies on historical data → Find when issue started