> ## Documentation Index > Fetch the complete documentation index at: https://mcp-server-langgraph.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Pod Failure Troubleshooting > Comprehensive runbook for diagnosing and resolving Kubernetes pod failures # Pod Failure Troubleshooting This runbook provides comprehensive guidance for diagnosing and resolving Kubernetes pod failures. ## Quick Diagnosis ### Check Pod Status ```bash theme={null} # Get all pods in namespace kubectl get pods -n # Get pods with issues kubectl get pods -n | grep -E "Error|CrashLoop|Pending|CreateContainer|ImagePull|Evicted" # Get detailed pod information kubectl describe pod -n ``` ### Check Recent Events ```bash theme={null} # Get recent events sorted by timestamp kubectl get events -n --sort-by='.lastTimestamp' | tail -50 # Get events for specific pod kubectl get events -n --field-selector involvedObject.name= ``` ### Check Pod Logs ```bash theme={null} # Get current container logs kubectl logs -c -n # Get previous container logs (if pod restarted) kubectl logs -c -n --previous # Follow logs in real-time kubectl logs -f -c -n ``` *** ## CrashLoopBackOff **Symptoms**: * Pod status: `CrashLoopBackOff` * Restart count increasing * Pod alternates between Running and Terminating ### Application Errors ```bash theme={null} # Check logs for application errors kubectl logs -n --previous | tail -100 ``` **Solutions**: * Fix application code errors * Update configuration * Check environment variables ### Filesystem Permissions (readOnlyRootFilesystem) **Error**: `java.nio.file.ReadOnlyFileSystemException` or `Permission denied` **Root Cause**: `readOnlyRootFilesystem: true` without proper volume mounts **Solution**: ```yaml theme={null} # Add writable volume mounts volumeMounts: - name: tmp mountPath: /tmp - name: cache mountPath: /var/tmp - name: app-data mountPath: /app/data # Application-specific writable path volumes: - name: tmp emptyDir: {} - name: cache emptyDir: {} - name: app-data emptyDir: {} ``` **Prevention**: 1. Test `readOnlyRootFilesystem` in development first 2. Run validation script: `python3 scripts/validate_gke_autopilot_compliance.py` 3. Monitor for filesystem errors in logs ### Missing Dependencies **Error**: `ModuleNotFoundError`, `ImportError`, or similar **Solutions**: * Verify Docker image contains all dependencies * Check requirements.txt or package.json is complete * Rebuild and push updated image ### Configuration Errors **Error**: Configuration file not found or invalid **Solutions**: * Verify ConfigMaps are created and mounted correctly * Check Secret references are correct * Validate configuration syntax *** ## CreateContainerConfigError **Symptoms**: * Pod status: `CreateContainerConfigError` * Container never starts * Pod stuck in this state ### Missing Secrets/ConfigMaps **Error**: `secret "example-secret" not found` or `configmap "example-config" not found` **Diagnosis**: ```bash theme={null} kubectl describe pod -n | grep -A 5 "Events:" ``` **Solutions**: ```bash theme={null} # List secrets kubectl get secrets -n # List configmaps kubectl get configmaps -n # Create missing secret kubectl create secret generic \ --from-literal=key=value \ -n ``` ### Invalid Volume Mounts **Error**: Volume mount path conflicts or invalid **Solutions**: * Check volume mount paths don't overlap * Verify volume names match between `volumes` and `volumeMounts` * Ensure mount paths are absolute *** ## ImagePullBackOff / ErrImagePull **Symptoms**: * Pod status: `ImagePullBackOff` or `ErrImagePull` * Image cannot be pulled from registry ### Image Does Not Exist **Solutions**: * Verify image tag is correct * Check image exists in registry: ```bash theme={null} docker pull : ``` * Update deployment with correct image tag ### Authentication Failure **Solutions**: ```bash theme={null} # Create image pull secret kubectl create secret docker-registry \ --docker-server= \ --docker-username= \ --docker-password= \ --docker-email= \ -n # Add to ServiceAccount kubectl patch serviceaccount \ -p '{"imagePullSecrets": [{"name": ""}]}' \ -n ``` *** ## Pending (Insufficient Resources) **Symptoms**: * Pod status: `Pending` * Events show: `Insufficient cpu` or `Insufficient memory` **Diagnosis**: ```bash theme={null} # Check node resources kubectl top nodes # Check pod resource requests kubectl describe pod -n | grep -A 10 "Requests:" # Check resource quotas kubectl get resourcequota -n ``` **Solutions**: ### Reduce Resource Requests ```yaml theme={null} resources: requests: cpu: 100m # Reduce from 500m memory: 128Mi # Reduce from 512Mi ``` ### Scale Up Cluster ```bash theme={null} # GKE Autopilot auto-scales, but you can trigger it gcloud container clusters resize \ --num-nodes= \ --zone= ``` *** ## GKE Autopilot Specific Errors ### CPU Limit/Request Ratio Violation **Error**: `cpu max limit to request ratio per Container is 4, but provided ratio is X` **Root Cause**: GKE Autopilot enforces max ratio of 4.0 **Diagnosis**: ```bash theme={null} # Calculate ratio # ratio = cpu_limit / cpu_request # Example: 1000m / 200m = 5.0 (FAILS) ``` **Solution**: ```yaml theme={null} resources: requests: cpu: 250m # Increased from 200m limits: cpu: 1000m # New ratio: 1000/250 = 4.0 ✅ ``` **Prevention**: * Run validation: `python3 scripts/validate_gke_autopilot_compliance.py` * Pre-commit hook automatically checks ### Memory Limit/Request Ratio Violation **Error**: `memory max limit to request ratio per Container is 4, but provided ratio is X` **Solution**: Same as CPU - ensure ratio ≤ 4.0 *** ## Liveness/Readiness Probe Failures **Symptoms**: * Pod restarts frequently * Events show: `Liveness probe failed` or `Readiness probe failed` **Diagnosis**: ```bash theme={null} # Check probe configuration kubectl get deployment -n -o yaml | grep -A 10 "livenessProbe:" ``` ### Probe Endpoint Not Available **Error**: `connection refused` or `404 Not Found` **Solutions**: * Verify health endpoint exists and is accessible * Check port numbers match * Ensure health endpoint doesn't require authentication ### Probe Timeout Too Short **Solutions**: ```yaml theme={null} livenessProbe: httpGet: path: /health port: 8000 initialDelaySeconds: 60 # Increase if app starts slowly periodSeconds: 30 timeoutSeconds: 10 # Increase if endpoint is slow failureThreshold: 3 ``` ### Missing Health Check Extension **Error** (OTEL Collector): `connection refused on port 13133` **Root Cause**: health\_check extension not enabled **Solution**: ```yaml theme={null} # In OTEL Collector config extensions: health_check: endpoint: 0.0.0.0:13133 service: extensions: [health_check] # Must be enabled! ``` *** ## Step-by-Step Troubleshooting ### Step 1: Identify the Problem ```bash theme={null} # Get overview of all pods kubectl get pods -n # Identify problematic pods kubectl get pods -n | grep -vE "Running|Completed" ``` ### Step 2: Gather Information ```bash theme={null} # Describe pod for detailed info kubectl describe pod -n # Check logs kubectl logs -n kubectl logs -n --previous # Check events kubectl get events -n --sort-by='.lastTimestamp' | tail -50 ``` ### Step 3: Analyze Root Cause Use the patterns above to match symptoms to common causes. ### Step 4: Apply Fix Follow solution steps for the identified problem. ### Step 5: Verify Fix ```bash theme={null} # Watch pod status kubectl get pods -n -w # Check if pod is running and ready kubectl get pod -n # Verify no restarts kubectl describe pod -n | grep "Restart Count:" ``` ### Step 6: Clean Up ```bash theme={null} # Delete old ReplicaSets (keep last 2-3 for rollback) kubectl delete replicaset -n # Scale down deployments if needed kubectl scale deployment --replicas= -n ``` *** ## Prevention Checklist ### Before Deployment * [ ] Run validation script: ```bash theme={null} python3 scripts/validate_gke_autopilot_compliance.py deployments/overlays/ ``` * [ ] Run regression tests: ```bash theme={null} pytest tests/regression/test_pod_deployment_regression.py -v ``` * [ ] Verify kustomize builds: ```bash theme={null} kubectl kustomize deployments/overlays/ | kubectl apply --dry-run=client -f - ``` * [ ] Check resource ratios (GKE Autopilot): * CPU ratio ≤ 4.0 * Memory ratio ≤ 4.0 * [ ] Verify no environment variable conflicts: * No env vars with both `value` and `valueFrom` * [ ] Test in development/staging first ### After Deployment * [ ] Monitor pod status for 10 minutes * [ ] Check logs for errors * [ ] Verify metrics are being collected * [ ] Test application functionality * [ ] Set up alerts for pod failures *** ## Quick Reference Commands ### Essential Commands ```bash theme={null} # Get pod status kubectl get pods -n # Describe pod kubectl describe pod -n # Get logs kubectl logs -c -n # Get previous logs (after restart) kubectl logs -c -n --previous # Get events kubectl get events -n --sort-by='.lastTimestamp' # Delete pod (forces recreation) kubectl delete pod -n # Rollout restart deployment kubectl rollout restart deployment/ -n # Rollback deployment kubectl rollout undo deployment/ -n # Scale deployment kubectl scale deployment/ --replicas= -n ``` ### Resource Commands ```bash theme={null} # Check node resources kubectl top nodes # Check pod resources kubectl top pods -n # Get resource quotas kubectl get resourcequota -n # Get limit ranges kubectl get limitrange -n ``` ### Debugging Commands ```bash theme={null} # Execute command in pod kubectl exec -it -c -n -- /bin/sh # Port forward to pod kubectl port-forward : -n # Copy files from pod kubectl cp /: ``` *** ## Related Documentation * [Deployment Checklist](/deployment/deployment-checklist) * [GKE Runbooks](/deployment/operations/gke-runbooks) * [GKE Autopilot Resource Constraints](/deployment/gke-autopilot-resource-constraints) * [ADR-0054: Pod Failure Prevention Framework](/architecture/adr-0054-pod-failure-prevention-framework)