⚡ ~/naveed DevOps Arcade
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
⚡ DevOps Arcade
☸️ Kubernetes SRE Incident Drill

kubectl Crash Simulator

Production is failing. Alerts are firing. Play with kubernetes failure scenarios and make kubectl practice a habit by diagnosing CrashLoopBackOff, Exit Code 137 OOMKills, Pending PVCs, and broken Services under a 60-second timer.

Incident #1 of 10
Time Remaining 60s
SRE Score 0 PTS
Streak 0x 🔥
Triage Rank L1 Operator
bash: naveed@k8s-prod-cluster-01 (112x32)
ns: prod
Step 1: Choose Diagnostic Command What is your first triage command?
$
Quick Diagnostic Commands:
get pods -n prod -o wide describe pod logs --previous get endpoints get events
⭐ Staff SRE (Level 4/5)

Cluster Outage Triage Complete

You diagnosed and resolved production cluster incidents under live timer conditions. Practice systematic Kubernetes triage workflows to identify pod restart loops, inspect container exit codes, verify service endpoints, and restore cluster service availability rapidly without guessing or wasting critical response time.

Final Score
3,850
Incidents Fixed
8 / 10
Mean Time to Detect (MTTD)
24.2s
Max Streak
5x 🔥

The SRE 4-Step Algorithmic Kubernetes Triage Framework

When an outage alerts you at 2:00 AM, guessing and deleting pods wastes critical response time. Instead, run kubectl diagnostic commands systematically. This deterministic four-step triage sequence isolates the blast radius with kubectl get pods, inspects container lifecycle termination states with describe, captures previous crash logs, and validates service endpoint routing to resolve cluster failures rapidly.

🚨
INCIDENT TRIGGER AlertManager / PagerDuty Alert: Pod entering NotReady or recurring CrashLoopBackOff in Production
1

Step 1: Scope the Blast Radius and Identify Failing Pods

During kubectl deployment troubleshooting, determine whether the outage is isolated to a single pod, spreading across an entire namespace, or concentrated on a physical worker node. Rather than immediately executing kubectl delete pod or triggering an abrupt kubectl restart pod, measure the scope of disruption first to prevent cluster-wide thrashing.

kubectl get pods -n <namespace> -o wide
Single Pod Crashing → Investigate Application Config, missing Secret / ConfigMap, database credentials, or code panic.
All Pods on 1 Node Failing → Inspect Worker Node Health (kubectl get nodes), Kubelet service daemon, disk pressure, or CNI pod network.

For architectural lifecycle mechanics, explore Kubernetes: Describe the pod lifecycle from kubectl apply.

2

Step 2: Inspect Container Lifecycle, Events, and Termination States

Running kubectl describe pod reveals exact container state transitions, volume mount statuses, and chronological probe checks. SRE practitioners know kubectl describe pod tips: inspect kubectl describe events first then conditions at the base of the output to spot preemption or failed image pulls before diving into application code.

kubectl describe pod <pod-name> -n <namespace>
Last State: Terminated → Check Exit Code & Reason: 137 (OOMKilled) vs 1 (Runtime Crash) vs 143 (SIGTERM / Eviction).
Events Section → Review chronological warnings: FailedScheduling, FailedMount, BackOff, or Unhealthy probes.
Limits & Requests → Verify if memory usage hit cgroup ceiling defined in resources.limits.memory (configured under container resources limits memory specifications).
3

Step 3: Capture Post-Mortem Crash Logs with Previous Flag

Standard log streaming queries only active processes. When an instance terminates abruptly, execute the kubectl logs command with --previous to retrieve stdout and stderr from the kubectl logs previous container run. The basic command syntax kubectl logs pod (often written with placeholders like kubectl logs lt pod gt or shorthand kubectl logs lt pod) targets an active process, so you must pass --previous to kubectl get logs of a pod before it restarts.

kubectl logs <pod-name> -n <namespace> --previous --tail=100
⚡ Essential SRE Rule Running standard log streams during CrashLoopBackOff returns empty because the container is in exponential restart sleep. Always append --previous to capture pre-crash logs.
4

Step 4: Verify Dependencies, Endpoints, and Event Chronology

Ensure network endpoints are registered and correlate cluster events across the namespace. Correlate timestamps between pod termination, volume detach operations, and node condition changes. For deep kubectl deployment troubleshooting and pending pods, check whether PersistentVolumeClaims are bound and whether service endpoints match selector labels.

kubectl get endpoints <service-name> -n <namespace>
kubectl get events -n <namespace> --sort-by='.metadata.creationTimestamp'
Service Routing → Confirm ready IP addresses exist in EndpointSlice. Empty endpoints cause upstream 502 / 503 Bad Gateway errors.
Event Chronology → Correlate timestamps between pod termination, PVC detach/attach operations, and node condition changes.

Exit Code Rosetta Stone: Decoding Kernel & Kubelet Signals

Kubernetes container exit codes reveal whether a failure originated from kernel enforcement, orchestrator lifecycle eviction, or internal application panics. Inspecting termination states with kubectl describe pod highlights Exit Code 137 for OOMKills, Exit Code 143 for graceful shutdown timeouts, and Exit Code 1 for unhandled runtime exceptions before reviewing lengthy log streams.

To inspect exit code via kubectl describe, look under the Last State: Terminated block (referencing last state terminated conditions) for numeric codes and kernel signals before analyzing application stacks.

Exit Code Signal Name Primary Root Cause First Diagnostic Command
137 SIGKILL (128 + 9) OOMKilled: Container memory exceeded cgroup limits or node ran out of RAM. kubectl describe pod <pod> | grep -i oom
143 SIGTERM (128 + 15) Graceful Kill Interrupted: Failed liveness probe or pod eviction timeout. Check liveness probe initialDelaySeconds and grace periods.
1 Application Error Runtime Panic: Missing required env var, DB handshake error, or syntax bug. kubectl logs <pod> --previous
127 Command Not Found Missing Binary: Entrypoint or command executable not found in image $PATH. Review Dockerfile base image and script shebang.
139 SIGSEGV (128 + 11) Segmentation Fault: Memory corruption or C-library mismatch (Alpine vs glibc). Inspect native compiled dependencies and glibc versions.
0 Success / Clean Exit Premature Completion: Daemon launched in background; main process exited. Ensure foreground execution flag (e.g. nginx -g 'daemon off;').

How to Troubleshoot Kubernetes Deployments and Pods with kubectl?

Troubleshooting Kubernetes workloads systematically requires validating pod scheduling phases, container restart loops, and network endpoint registrations. Engineers use targeted kubectl commands to isolate crashloopbackoff failures, inspect pod event conditions, stream previous container logs, preview manifest diffs, and verify service connectivity before applying configuration patches to production clusters.

Kubernetes Pod Failure States and Diagnostic Decision Logic

Systematic cluster triage demands understanding how the kubelet and control plane controllers transition workload lifecycle phases. When an outage occurs, isolating the operational phase reveals the root cause before you execute any remediation command:

How to Debug CrashLoopBackOff and Application Failures?

To debug CrashLoopBackOff efficiently, check the previous container logs and verify runtime dependencies. Using kubectl debug crashloopbackoff scenarios allows engineers to attach interactive ephemeral debug containers or clone pods with modified entrypoints, isolating missing configuration keys and memory breaches without mutating live cluster production workloads.

When a container instance crashes repeatedly upon startup, running standard interactive shells via kubectl exec -it <pod> -- /bin/sh fails because the target container process is not running. In modern clusters, attach an ephemeral troubleshooting container to inspect the root filesystem, environment variables, and local socket bindings directly:

kubectl debug -it <pod-name> --image=busybox:1.36 --target=<container-name>

If you need to extract heap dumps, core dumps, or local runtime artifacts for offline post-mortem analysis, run the cp utility to copy diagnostic files between the container and your local workstation:

kubectl cp <namespace>/<pod-name>:/var/log/app.log ./local-app.log

Kubernetes Service Networking and DNS Resolution Runbook

When microservices communicate across clusters, service discovery depends on CoreDNS and kube-proxy synchronization. Correlating network failures requires inspecting both pod endpoints and node packet filtering rules:

Production CLI Commands Cheat Sheet: Triage, Diff & Resource Inspection

A reliable kubectl diagnostic production command cheat sheet provides instant access to high-impact operational commands during incidents. Bookmark this reference to query namespaces, inspect admission controllers, perform client-side dry runs, preview manifest changes, and port-forward endpoints during high-pressure cluster troubleshooting sessions.

When triage demands quick context scoping, pass the target pod name along with name n namespace scoping flags (or shorthand n namespace parameters) to query resources without affecting adjacent environments. When validating network routing, query the target endpoints service name to verify that backend pods register valid IP addresses.

Diagnostic Category Operational Target Production Command Syntax SRE Verification Focus
Pod Filtering Find non-running pods get pods -A --field-selector=status.phase!=Running Leverage kubectl get lists filtering and scripting to instantly surface CrashLoopBackOff, Pending, and Evicted pods across all namespaces.
Namespace Audit Audit namespace resources get all -n <namespace> Execute kubectl get all resources in a namespace to audit pods, deployments, replicaSets, daemonSets, and services in a single call.
Port Forwarding Local port redirection port-forward -n monitoring svc/prometheus-k8s 9090:9090 Establish a secure kubectl port forward session to test endpoints across kubectl n monitoring and kubectl n monitoring port targets without public ingress exposure.
Manifest Diff Preview change before apply diff -f deployment.yaml Execute kubectl diff preview changes before apply to compare live API server state against proposed YAML manifests before applying updates.
Dry-Run Generation Generate clean YAML specs run debug-pod --image=busybox:1.36 --dry-run=client -o yaml Explore kubectl dry run examples to construct pod manifests and test syntax using kubectl create pod and kubectl apply command flags safely.
Declarative Deploy Apply production manifest apply -f deployment.yaml Execute kubectl apply deployment yaml to roll out declarative updates with zero downtime and automatic rolling revision tracking.
Cluster Context Switch clusters & namespaces config get-contexts && config set-context --current --namespace=<ns> Review configurations with kubectl get contexts and update target namespaces using kubectl set context to prevent accidental production commands.
Admission Control Inspect active webhooks get mutatingwebhookconfigurations,validatingwebhookconfigurations Audit webhooks via kubectl get admission controllers and kubectl list admission controllers when pod creation is silently rejected.

How to Install and Set Up kubectl on Windows and Linux?

Before debugging production clusters, engineers must properly configure their local workstation CLI environment. Setting up kubectl install windows via winget or Chocolatey and kubectl install ubuntu through official apt repositories ensures you have the latest stable binary, context switching utilities, and shell auto-completion for daily cluster operations.

For Windows systems, complete your local environment setup using winget or PowerShell package managers for automated path registration. Engineers managing hybrid clusters or running kubectl windows environments can verify installation via PowerShell:

winget install -e --id Kubernetes.kubectl

On Debian and Ubuntu workstations, configure the official Kubernetes package repository to complete your package installation:

sudo apt-get update && sudo apt-get install -y kubectl

Verify your local workstation installation with kubectl version --client. For standalone binary setups on Windows or Linux, execute direct kubectl download workflows from official releases on dl.k8s.io.

📚 Frequently Asked Kubernetes Triage Questions

Looking for more interactive drills? Explore our companion guide and kubectl practice game for in-depth scenario walkthroughs.

What is the difference between CrashLoopBackOff and an application error? +
CrashLoopBackOff is not an error code; it is a Kubernetes state condition. When a container process terminates unexpectedly with a non-zero exit code, the kubelet attempts to restart it. If it crashes repeatedly, the kubelet applies an exponential back-off delay (10s, 20s, 40s up to 5 minutes) before the next restart to avoid overloading node CPU and disk. The application error is the underlying cause, while CrashLoopBackOff is the orchestrator's pacing response.
How do you diagnose an Exit Code 137 in Kubernetes? +
Exit Code 137 indicates that the Linux kernel sent a SIGKILL signal (128 + 9 = 137) to terminate the container. The primary cause is OOMKilled (Out Of Memory), which happens when a container exceeds the memory limit defined in resources.limits.memory (configured under container resources limits memory specifications). Run kubectl describe pod <pod-name> and inspect the Last State: Terminated block. If Reason: OOMKilled is present, the process breached its cgroup limit. Resolution requires profiling memory leaks or increasing the container memory limit.
Why does kubectl logs show no output when a pod is in CrashLoopBackOff? +
Running kubectl logs <pod-name> queries the currently running container instance. If the container is currently waiting in its back-off period between restarts, the active container does not exist or has not yet written logs. To inspect the stdout and stderr logs from the failed container instance right before it crashed, pass the --previous flag: kubectl logs <pod-name> --previous.
How do you triage a Kubernetes Service returning HTTP 503 errors when pods are Running? +
If pods show 1/1 Running but the Service returns HTTP 503, the issue is almost always a label selector mismatch. Run kubectl get endpoints <service-name> or kubectl get endpointslices. If the ENDPOINTS column shows <none>, the Service's spec.selector labels do not match the Pod's metadata.labels. Align the labels so kube-proxy can register the pod IPs as healthy backend endpoints.
What causes a Kubernetes Pod to stay in Pending state indefinitely? +
A Pod stays in Pending when the Kubernetes scheduler cannot place it on any worker node. Run kubectl describe pod <pod-name> and inspect the Events section at the bottom. Common causes include: 1) Insufficient CPU or Memory requests exceeding available allocatable capacity on all nodes; 2) PersistentVolumeClaim (PVC) unbound due to storageClass mismatch or volume binding mode; 3) Unmatched nodeSelector, nodeAffinity, or taints without matching tolerations.
What is the difference between Exit Code 143 and Exit Code 137? +
Exit Code 143 corresponds to SIGTERM (128 + 15 = 143), which is a graceful termination signal sent by the kubelet (e.g. during pod eviction, rolling update, or when a liveness probe fails). The application is given a grace period (default 30s) to shut down cleanly. Exit Code 137 corresponds to SIGKILL (128 + 9 = 137), an immediate, ungraceful kill triggered by the Linux OOM killer when cgroup memory limits are exceeded.