Deployment Problems

Troubleshooting guide for Kubernetes, Docker, and Cloud Run deployment issues

Deployment Problems

This guide covers common deployment issues across Kubernetes, Docker, and Cloud Run environments.

Kubernetes Pod CrashLoopBackOff

Symptom: Pods continuously restart with CrashLoopBackOff status

Diagnosis:

# Check pod logs
kubectl logs -n mcp-server <pod-name> --previous

# Describe pod for events
kubectl describe pod -n mcp-server <pod-name>

# Check resource constraints
kubectl top pod -n mcp-server <pod-name>

Common Causes & Solutions:

1. Missing Environment Variables

# Check ConfigMap
kubectl get configmap -n mcp-server mcp-server-config -o yaml

# Check Secrets
kubectl get secret -n mcp-server mcp-server-secrets -o yaml

# Verify required secrets exist
kubectl get secret -n mcp-server cloud-sql-credentials
kubectl get secret -n mcp-server keycloak-admin

Fix: Add missing secrets to deployments/base/secrets.yaml or External Secrets configuration.

2. Cloud SQL Proxy Connection Failure

# Check Cloud SQL Proxy sidecar logs
kubectl logs -n mcp-server <pod-name> -c cloud-sql-proxy

# Common errors:
# - "could not refresh token": IAM permissions issue
# - "connection refused": Wrong connection name
# - "invalid instance": Instance doesn't exist

Fix: See our GKE Runbooks.

3. Resource Limits Too Low

# Check if OOMKilled
kubectl get events -n mcp-server | grep OOMKilled

# Increase memory limits in deployment
resources:
  requests:
    memory: "512Mi"  # Increase from 256Mi
  limits:
    memory: "2Gi"    # Increase from 1Gi

---

Docker Container Won’t Start

Symptom: docker-compose up fails with container exit code

Diagnosis:

# View container logs
docker-compose logs mcp-server

# Check specific service
docker-compose logs keycloak
docker-compose logs redis

# Inspect container
docker inspect mcp-server_mcp-server_1

Common Causes:

1. Port Already in Use

# Find process using port
lsof -i :8000  # For MCP server
lsof -i :8080  # For Keycloak

# Kill the process or change port in docker-compose.yml

2. Health Check Failing

# Fix Qdrant health check (grpc_health_probe required)
qdrant:
  healthcheck:
    test: ["CMD", "grpc_health_probe", "-addr=:6334"]
    # NOT: test: ["CMD", "curl", "-f", "http://localhost:6333"]

3. Volume Mount Permissions

# Fix permissions for PostgreSQL data volume
sudo chown -R 999:999 ./data/postgres

# Fix permissions for Redis data
sudo chown -R 999:999 ./data/redis

---

Cloud Run Deployment Fails

Symptom: Cloud Run deployment fails with ERROR: (gcloud.run.deploy) INVALID_ARGUMENT

Common Errors:

1. Container Image Not Found

# Verify image exists in Artifact Registry
gcloud artifacts docker images list \
  us-central1-docker.pkg.dev/YOUR_PROJECT/mcp-server

# Ensure image is pushed
docker tag mcp-server:latest \
  us-central1-docker.pkg.dev/YOUR_PROJECT/mcp-server/mcp-server:latest
docker push us-central1-docker.pkg.dev/YOUR_PROJECT/mcp-server/mcp-server:latest

2. Service Account Permissions

# Grant required permissions
gcloud projects add-iam-policy-binding YOUR_PROJECT \
  --member="serviceAccount:mcp-server@YOUR_PROJECT.iam.gserviceaccount.com" \
  --role="roles/cloudsql.client"

# Check current permissions
gcloud projects get-iam-policy YOUR_PROJECT \
  --flatten="bindings[].members" \
  --filter="bindings.members:mcp-server@"

3. Environment Variables Too Large

# Error: "The total size of environment variables exceeds 32KB"
# Solution: Use Secret Manager instead
gcloud run services update mcp-server \
  --update-secrets=OPENAI_API_KEY=openai-api-key:latest \
  --region=us-central1

---

Kustomize Build Errors

Symptom: Error: accumulating resources: accumulation err='merging resources...'

Common Issues:

1. ConfigMap Generator with behavior: replace

# Don't use behavior: replace with generators
configMapGenerator:
  - name: app-config
    # behavior: replace  # REMOVE THIS
    literals:
      - KEY=value

2. Missing Base Resources

# Ensure base exists
bases:
  - ../../base  # Path must exist

3. Duplicate Resource Names

# Each resource must have unique metadata.name
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config  # Must be unique

---

Still Having Issues?

For more detailed deployment troubleshooting:

• Kubernetes: See Pod Failures Troubleshooting
• GKE Specific: See GKE Runbooks
• EKS Specific: See EKS Runbooks
• Report Bug: Create an issue