kubectl Crash Simulator
Production is failing. Alerts are firing. Play with kubernetes failure scenarios and make kubectl practice a habit by diagnosing CrashLoopBackOff, Exit Code 137 OOMKills, Pending PVCs, and broken Services under a 60-second timer.
Cluster Outage Triage Complete
You diagnosed and resolved production cluster incidents under live timer conditions. Practice systematic Kubernetes triage workflows to identify pod restart loops, inspect container exit codes, verify service endpoints, and restore cluster service availability rapidly without guessing or wasting critical response time.
The SRE 4-Step Algorithmic Kubernetes Triage Framework
When an outage alerts you at 2:00 AM, guessing and deleting pods wastes critical response time. Instead, run kubectl diagnostic commands systematically. This deterministic four-step triage sequence isolates the blast radius with kubectl get pods, inspects container lifecycle termination states with describe, captures previous crash logs, and validates service endpoint routing to resolve cluster failures rapidly.
Step 1: Scope the Blast Radius and Identify Failing Pods
During kubectl deployment troubleshooting, determine whether the outage is isolated to a single pod, spreading across an entire namespace, or concentrated on a physical worker node. Rather than immediately executing kubectl delete pod or triggering an abrupt kubectl restart pod, measure the scope of disruption first to prevent cluster-wide thrashing.
kubectl get pods -n <namespace> -o wide
Secret / ConfigMap, database credentials, or code panic.
kubectl get nodes), Kubelet service daemon, disk pressure, or CNI pod network.
For architectural lifecycle mechanics, explore Kubernetes: Describe the pod lifecycle from kubectl apply.
Step 2: Inspect Container Lifecycle, Events, and Termination States
Running kubectl describe pod reveals exact container state transitions, volume mount statuses, and chronological probe checks. SRE practitioners know kubectl describe pod tips: inspect kubectl describe events first then conditions at the base of the output to spot preemption or failed image pulls before diving into application code.
kubectl describe pod <pod-name> -n <namespace>
137 (OOMKilled) vs 1 (Runtime Crash) vs 143 (SIGTERM / Eviction).
FailedScheduling, FailedMount, BackOff, or Unhealthy probes.
resources.limits.memory (configured under container resources limits memory specifications).
Step 3: Capture Post-Mortem Crash Logs with Previous Flag
Standard log streaming queries only active processes. When an instance terminates abruptly, execute the kubectl logs command with --previous to retrieve stdout and stderr from the kubectl logs previous container run. The basic command syntax kubectl logs pod (often written with placeholders like kubectl logs lt pod gt or shorthand kubectl logs lt pod) targets an active process, so you must pass --previous to kubectl get logs of a pod before it restarts.
kubectl logs <pod-name> -n <namespace> --previous --tail=100
CrashLoopBackOff returns empty because the container is in exponential restart sleep. Always append --previous to capture pre-crash logs.
Step 4: Verify Dependencies, Endpoints, and Event Chronology
Ensure network endpoints are registered and correlate cluster events across the namespace. Correlate timestamps between pod termination, volume detach operations, and node condition changes. For deep kubectl deployment troubleshooting and pending pods, check whether PersistentVolumeClaims are bound and whether service endpoints match selector labels.
kubectl get endpoints <service-name> -n <namespace>
kubectl get events -n <namespace> --sort-by='.metadata.creationTimestamp'
502 / 503 Bad Gateway errors.
Exit Code Rosetta Stone: Decoding Kernel & Kubelet Signals
Kubernetes container exit codes reveal whether a failure originated from kernel enforcement, orchestrator lifecycle eviction, or internal application panics. Inspecting termination states with kubectl describe pod highlights Exit Code 137 for OOMKills, Exit Code 143 for graceful shutdown timeouts, and Exit Code 1 for unhandled runtime exceptions before reviewing lengthy log streams.
To inspect exit code via kubectl describe, look under the Last State: Terminated block (referencing last state terminated conditions) for numeric codes and kernel signals before analyzing application stacks.
| Exit Code | Signal Name | Primary Root Cause | First Diagnostic Command |
|---|---|---|---|
137 |
SIGKILL (128 + 9) | OOMKilled: Container memory exceeded cgroup limits or node ran out of RAM. | kubectl describe pod <pod> | grep -i oom |
143 |
SIGTERM (128 + 15) | Graceful Kill Interrupted: Failed liveness probe or pod eviction timeout. | Check liveness probe initialDelaySeconds and grace periods. |
1 |
Application Error | Runtime Panic: Missing required env var, DB handshake error, or syntax bug. | kubectl logs <pod> --previous |
127 |
Command Not Found | Missing Binary: Entrypoint or command executable not found in image $PATH. | Review Dockerfile base image and script shebang. |
139 |
SIGSEGV (128 + 11) | Segmentation Fault: Memory corruption or C-library mismatch (Alpine vs glibc). | Inspect native compiled dependencies and glibc versions. |
0 |
Success / Clean Exit | Premature Completion: Daemon launched in background; main process exited. | Ensure foreground execution flag (e.g. nginx -g 'daemon off;'). |
How to Troubleshoot Kubernetes Deployments and Pods with kubectl?
Troubleshooting Kubernetes workloads systematically requires validating pod scheduling phases, container restart loops, and network endpoint registrations. Engineers use targeted kubectl commands to isolate crashloopbackoff failures, inspect pod event conditions, stream previous container logs, preview manifest diffs, and verify service connectivity before applying configuration patches to production clusters.
Kubernetes Pod Failure States and Diagnostic Decision Logic
Systematic cluster triage demands understanding how the kubelet and control plane controllers transition workload lifecycle phases. When an outage occurs, isolating the operational phase reveals the root cause before you execute any remediation command:
- Pending Phase: The workload specification has been accepted by the API server, but the Kubernetes scheduler cannot bind it to a worker node. Failures typically stem from insufficient allocatable compute capacity, unfulfilled nodeAffinity rules, unresolved taints, or PersistentVolumeClaims awaiting volume binding mode placement.
- ContainerCreating Phase: The pod is assigned to a node, but the local container runtime cannot initialize the sandboxed network namespace or mount volumes. Common culprits include slow cloud storage attachment, missing ConfigMaps or Secrets, and CNI IP address allocation pool exhaustion.
- CrashLoopBackOff Condition: The container process starts, fails, and terminates with a non-zero exit code repeatedly. The kubelet enforces an exponential back-off restart delay starting at 10 seconds and doubling up to 300 seconds to protect node CPU cycles, disk I/O, and container runtime sockets.
- OOMKilled State (Exit Code 137): The container process breached its allocated memory ceiling. The Linux kernel memory cgroup controller triggers the Out-Of-Memory killer to protect host operating system stability, dispatching an immediate SIGKILL signal.
- Terminating Stuck State: A pod remains stuck during deletion when persistent finalizers wait for cleanup confirmation or when an unmounted storage volume hangs during detachment from the underlying storage driver.
How to Debug CrashLoopBackOff and Application Failures?
To debug CrashLoopBackOff efficiently, check the previous container logs and verify runtime dependencies. Using kubectl debug crashloopbackoff scenarios allows engineers to attach interactive ephemeral debug containers or clone pods with modified entrypoints, isolating missing configuration keys and memory breaches without mutating live cluster production workloads.
When a container instance crashes repeatedly upon startup, running standard interactive shells via kubectl exec -it <pod> -- /bin/sh fails because the target container process is not running. In modern clusters, attach an ephemeral troubleshooting container to inspect the root filesystem, environment variables, and local socket bindings directly:
kubectl debug -it <pod-name> --image=busybox:1.36 --target=<container-name>
If you need to extract heap dumps, core dumps, or local runtime artifacts for offline post-mortem analysis, run the cp utility to copy diagnostic files between the container and your local workstation:
kubectl cp <namespace>/<pod-name>:/var/log/app.log ./local-app.log
Kubernetes Service Networking and DNS Resolution Runbook
When microservices communicate across clusters, service discovery depends on CoreDNS and kube-proxy synchronization. Correlating network failures requires inspecting both pod endpoints and node packet filtering rules:
- Endpoint Population Mechanics: The Kubernetes endpoint controller watches pod lifecycle states. If all container readiness probes fail, the controller immediately evicts pod IP addresses from the corresponding EndpointSlice object. Upstream reverse proxies receive empty backend lists and return HTTP 503 Service Unavailable errors.
- Kube-Proxy Mode Trade-offs: In iptables mode, every service registration appends chains and rules sequentially, causing performance degradation in clusters with tens of thousands of services. In IPVS mode, packet forwarding relies on hashed ipset data structures, maintaining sub-millisecond connection routing under massive traffic volume.
- CoreDNS Resolution Latencies: When internal domain names fail to resolve, check CoreDNS pod logs in the kube-system namespace. High query concurrency or upstream DNS timeouts cause 5-second connection delays due to glibc single-request-reopen resolver behavior.
Production CLI Commands Cheat Sheet: Triage, Diff & Resource Inspection
A reliable kubectl diagnostic production command cheat sheet provides instant access to high-impact operational commands during incidents. Bookmark this reference to query namespaces, inspect admission controllers, perform client-side dry runs, preview manifest changes, and port-forward endpoints during high-pressure cluster troubleshooting sessions.
When triage demands quick context scoping, pass the target pod name along with name n namespace scoping flags (or shorthand n namespace parameters) to query resources without affecting adjacent environments. When validating network routing, query the target endpoints service name to verify that backend pods register valid IP addresses.
| Diagnostic Category | Operational Target | Production Command Syntax | SRE Verification Focus |
|---|---|---|---|
| Pod Filtering | Find non-running pods | get pods -A --field-selector=status.phase!=Running |
Leverage kubectl get lists filtering and scripting to instantly surface CrashLoopBackOff, Pending, and Evicted pods across all namespaces. |
| Namespace Audit | Audit namespace resources | get all -n <namespace> |
Execute kubectl get all resources in a namespace to audit pods, deployments, replicaSets, daemonSets, and services in a single call. |
| Port Forwarding | Local port redirection | port-forward -n monitoring svc/prometheus-k8s 9090:9090 |
Establish a secure kubectl port forward session to test endpoints across kubectl n monitoring and kubectl n monitoring port targets without public ingress exposure. |
| Manifest Diff | Preview change before apply | diff -f deployment.yaml |
Execute kubectl diff preview changes before apply to compare live API server state against proposed YAML manifests before applying updates. |
| Dry-Run Generation | Generate clean YAML specs | run debug-pod --image=busybox:1.36 --dry-run=client -o yaml |
Explore kubectl dry run examples to construct pod manifests and test syntax using kubectl create pod and kubectl apply command flags safely. |
| Declarative Deploy | Apply production manifest | apply -f deployment.yaml |
Execute kubectl apply deployment yaml to roll out declarative updates with zero downtime and automatic rolling revision tracking. |
| Cluster Context | Switch clusters & namespaces | config get-contexts && config set-context --current --namespace=<ns> |
Review configurations with kubectl get contexts and update target namespaces using kubectl set context to prevent accidental production commands. |
| Admission Control | Inspect active webhooks | get mutatingwebhookconfigurations,validatingwebhookconfigurations |
Audit webhooks via kubectl get admission controllers and kubectl list admission controllers when pod creation is silently rejected. |
How to Install and Set Up kubectl on Windows and Linux?
Before debugging production clusters, engineers must properly configure their local workstation CLI environment. Setting up kubectl install windows via winget or Chocolatey and kubectl install ubuntu through official apt repositories ensures you have the latest stable binary, context switching utilities, and shell auto-completion for daily cluster operations.
For Windows systems, complete your local environment setup using winget or PowerShell package managers for automated path registration. Engineers managing hybrid clusters or running kubectl windows environments can verify installation via PowerShell:
winget install -e --id Kubernetes.kubectl
On Debian and Ubuntu workstations, configure the official Kubernetes package repository to complete your package installation:
sudo apt-get update && sudo apt-get install -y kubectl
Verify your local workstation installation with kubectl version --client. For standalone binary setups on Windows or Linux, execute direct kubectl download workflows from official releases on dl.k8s.io.
📚 Frequently Asked Kubernetes Triage Questions
Looking for more interactive drills? Explore our companion guide and kubectl practice game for in-depth scenario walkthroughs.
CrashLoopBackOff is not an error code; it is a Kubernetes state condition. When a container process terminates unexpectedly with a non-zero exit code, the kubelet attempts to restart it. If it crashes repeatedly, the kubelet applies an exponential back-off delay (10s, 20s, 40s up to 5 minutes) before the next restart to avoid overloading node CPU and disk. The application error is the underlying cause, while CrashLoopBackOff is the orchestrator's pacing response.
resources.limits.memory (configured under container resources limits memory specifications). Run kubectl describe pod <pod-name> and inspect the Last State: Terminated block. If Reason: OOMKilled is present, the process breached its cgroup limit. Resolution requires profiling memory leaks or increasing the container memory limit.
kubectl logs <pod-name> queries the currently running container instance. If the container is currently waiting in its back-off period between restarts, the active container does not exist or has not yet written logs. To inspect the stdout and stderr logs from the failed container instance right before it crashed, pass the --previous flag: kubectl logs <pod-name> --previous.
kubectl get endpoints <service-name> or kubectl get endpointslices. If the ENDPOINTS column shows <none>, the Service's spec.selector labels do not match the Pod's metadata.labels. Align the labels so kube-proxy can register the pod IPs as healthy backend endpoints.
kubectl describe pod <pod-name> and inspect the Events section at the bottom. Common causes include: 1) Insufficient CPU or Memory requests exceeding available allocatable capacity on all nodes; 2) PersistentVolumeClaim (PVC) unbound due to storageClass mismatch or volume binding mode; 3) Unmatched nodeSelector, nodeAffinity, or taints without matching tolerations.