☸️ Kubernetes SRE Incident Drill
kubectl Crash Simulator
Production is failing. Alerts are firing. Diagnose and fix CrashLoopBackOff, Exit Code 137 OOMKills, Pending PVCs, and broken Services under a 60-second timer.
Step 1: Choose Diagnostic Command
What is your first triage command?
$
Quick Diagnostic Commands:
kubectl get pods -n prod -o wide
kubectl describe pod
kubectl logs --previous
kubectl get endpoints
kubectl get events
⭐ Staff SRE (Level 4/5)
Cluster Outage Triage Complete
You diagnosed and resolved 8 of 10 production cluster incidents under live timer conditions.
Final Score
3,850
Incidents Fixed
8 / 10
Mean Time to Detect (MTTD)
24.2s
Max Streak
5x 🔥
The SRE 4-Step Algorithmic Kubernetes Triage Framework
When an outage alerts you at 2:00 AM, guessing and deleting pods wastes critical time. Follow this deterministic triage sequence engineered for production reliability.
1
Scope the Blast Radius
Determine whether the incident is isolated to a single pod, spreading across a namespace, or concentrated on a physical worker node.
kubectl get pods -n <namespace> -o wide
Single Pod Crashing
➔ Investigate Application Config, missing
Secret / ConfigMap, database credentials, or code panic.
All Pods on 1 Node Failing
➔ Inspect Worker Node Health (
kubectl get nodes), Kubelet service daemon, disk pressure, or CNI pod network.
2
Inspect Container Lifecycle & Termination States
Read the exact kernel exit signal, verify termination reasons, and inspect recent Kubelet lifecycle events.
kubectl describe pod <pod-name> -n <namespace>
Last State: Terminated
➔ Check Exit Code & Reason:
137 (OOMKilled) vs 1 (Runtime Crash) vs 143 (SIGTERM / Eviction).
Events Section
➔ Review chronological warnings:
FailedScheduling, FailedMount, BackOff, or Unhealthy probes.
Limits & Requests
➔ Verify if memory usage hit cgroup ceiling defined in
resources.limits.memory.
3
Capture Post-Mortem Crash Logs
Extract stdout/stderr from the terminated container instance right before the exit signal was dispatched.
kubectl logs <pod-name> -n <namespace> --previous --tail=100
⚡ Essential SRE Rule
Running standard
kubectl logs during CrashLoopBackOff often returns empty because the container is in exponential restart sleep! Always append --previous to capture pre-crash logs.
4
Verify Dependencies, Endpoints & Event Chronology
Ensure network endpoints are registered and correlate cluster events across the namespace.
kubectl get endpoints <service-name> -n <namespace>
kubectl get events -n <namespace> --sort-by='.metadata.creationTimestamp'
Service Routing
➔ Confirm ready IP addresses exist in EndpointSlice. Empty endpoints cause upstream
502 / 503 Bad Gateway errors.
Event Chronology
➔ Correlate timestamps between pod termination, PVC detach/attach operations, and node condition changes.
Exit Code Rosetta Stone: Decoding Kernel & Kubelet Signals
Before inspecting application logs, the container exit code provides immediate insight into the root failure vector.
| Exit Code | Signal Name | Primary Root Cause | First Diagnostic Command |
|---|---|---|---|
137 |
SIGKILL (128 + 9) | OOMKilled: Container memory exceeded cgroup limits or node ran out of RAM. | kubectl describe pod <pod> | grep -i oom |
143 |
SIGTERM (128 + 15) | Graceful Kill Interrupted: Failed liveness probe or pod eviction timeout. | Check liveness probe initialDelaySeconds and grace periods. |
1 |
Application Error | Runtime Panic: Missing required env var, DB handshake error, or syntax bug. | kubectl logs <pod> --previous |
127 |
Command Not Found | Missing Binary: Entrypoint or command executable not found in image $PATH. | Review Dockerfile base image and script shebang. |
139 |
SIGSEGV (128 + 11) | Segmentation Fault: Memory corruption or C-library mismatch (Alpine vs glibc). | Inspect native compiled dependencies and glibc versions. |
0 |
Success / Clean Exit | Premature Completion: Daemon launched in background; main process exited. | Ensure foreground execution flag (e.g. nginx -g 'daemon off;'). |
📚 Frequently Asked Kubernetes Triage Questions
What is the difference between CrashLoopBackOff and an application error?
+
CrashLoopBackOff is not an error code; it is a Kubernetes state condition. When a container process terminates unexpectedly with a non-zero exit code, the kubelet attempts to restart it. If it crashes repeatedly, the kubelet applies an exponential back-off delay (10s, 20s, 40s up to 5 minutes) before the next restart to avoid overloading node CPU and disk. The application error is the underlying cause, while CrashLoopBackOff is the orchestrator's pacing response.
How do you diagnose an Exit Code 137 in Kubernetes?
+
Exit Code 137 indicates that the Linux kernel sent a SIGKILL signal (128 + 9 = 137) to terminate the container. The primary cause is OOMKilled (Out Of Memory), which happens when a container exceeds the memory limit defined in
resources.limits.memory. Run kubectl describe pod <pod-name> and inspect the Last State: Terminated block. If Reason: OOMKilled is present, the process breached its cgroup limit. Resolution requires profiling memory leaks or increasing the container memory limit.
Why does kubectl logs show no output when a pod is in CrashLoopBackOff?
+
Running
kubectl logs <pod-name> queries the currently running container instance. If the container is currently waiting in its back-off period between restarts, the active container does not exist or has not yet written logs. To inspect the stdout and stderr logs from the failed container instance right before it crashed, pass the --previous flag: kubectl logs <pod-name> --previous.
How do you triage a Kubernetes Service returning HTTP 503 errors when pods are Running?
+
If pods show 1/1 Running but the Service returns HTTP 503, the issue is almost always a label selector mismatch. Run
kubectl get endpoints <service-name> or kubectl get endpointslices. If the ENDPOINTS column shows <none>, the Service's spec.selector labels do not match the Pod's metadata.labels. Align the labels so kube-proxy can register the pod IPs as healthy backend endpoints.
What causes a Kubernetes Pod to stay in Pending state indefinitely?
+
A Pod stays in Pending when the Kubernetes scheduler cannot place it on any worker node. Run
kubectl describe pod <pod-name> and inspect the Events section at the bottom. Common causes include: 1) Insufficient CPU or Memory requests exceeding available allocatable capacity on all nodes; 2) PersistentVolumeClaim (PVC) unbound due to storageClass mismatch or volume binding mode; 3) Unmatched nodeSelector, nodeAffinity, or taints without matching tolerations.
What is the difference between Exit Code 143 and Exit Code 137?
+
Exit Code 143 corresponds to SIGTERM (128 + 15 = 143), which is a graceful termination signal sent by the kubelet (e.g. during pod eviction, rolling update, or when a liveness probe fails). The application is given a grace period (default 30s) to shut down cleanly. Exit Code 137 corresponds to SIGKILL (128 + 9 = 137), an immediate, ungraceful kill triggered by the Linux OOM killer when cgroup memory limits are exceeded.