> ## Documentation Index
> Fetch the complete documentation index at: https://mcp-server-langgraph.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Infrastructure Deployment Checklist

> Reproducible and idempotent infrastructure deployment guide with standardized naming conventions

# Infrastructure Deployment Checklist

```mermaid theme={null}
flowchart TD
    subgraph PreDeploy["Pre-Deployment"]
        NC["Naming Convention Check"]
        GHV["GitHub Variables Check"]
        TF["Terraform Validate"]
    end

    subgraph Deploy["Deployment"]
        APPLY["terraform apply"]
        K8S["kubectl apply"]
        HELM["helm upgrade"]
    end

    subgraph PostDeploy["Post-Deployment"]
        HEALTH["Health Checks"]
        SMOKE["Smoke Tests"]
        MONITOR["Monitoring Verification"]
    end

    NC --> GHV --> TF
    TF --> APPLY --> K8S --> HELM
    HELM --> HEALTH --> SMOKE --> MONITOR

    %% ColorBrewer2 Set3 palette
    classDef preStyle fill:#8dd3c7,stroke:#2a9d8f,stroke-width:2px,color:#333
    classDef deployStyle fill:#bebada,stroke:#7e5eb0,stroke-width:2px,color:#333
    classDef postStyle fill:#b3de69,stroke:#7cb342,stroke-width:2px,color:#333

    class NC,GHV,TF preStyle
    class APPLY,K8S,HELM deployStyle
    class HEALTH,SMOKE,MONITOR postStyle
```

## Overview

This checklist ensures reproducible and idempotent infrastructure deployments following the standardized naming convention: `{environment}-mcp-server-langgraph-{resource-type}`.

**Last Updated:** 2025-11-05
**Status:** ✅ Validated with staging rebuild

***

## Pre-Deployment Validation

### 1. Naming Convention Compliance

Run this validation before any infrastructure changes:

```bash theme={null}
# Check for legacy naming patterns
! grep -r "mcp-staging-cluster\|mcp-prod-gke\|mcp-dev-cluster" \
  terraform/environments/ .github/workflows/deploy-*.yaml deployments/overlays/ || \
  echo "❌ FAIL: Legacy naming found"

# Verify new naming in active configs
grep -r "staging-mcp-server-langgraph-gke\|production-mcp-server-langgraph-gke" \
  .github/workflows/deploy-*.yaml terraform/environments/gcp-*/main.tf && \
  echo "✅ PASS: New naming present"
```

### 2. GitHub Variables Check

```bash theme={null}
gh variable list | grep -E "GKE_.*_CLUSTER|NAMESPACE"
# Should show:
# - GKE_STAGING_CLUSTER: staging-mcp-server-langgraph-gke
# - GKE_PROD_CLUSTER: production-mcp-server-langgraph-gke
# - STAGING_NAMESPACE: staging-mcp-server-langgraph
# - PRODUCTION_NAMESPACE: production-mcp-server-langgraph
```

***

### 2. GKE Autopilot Compliance (for GKE deployments)

* [ ] **Run automated validation**
  ```bash theme={null}
  python3 scripts/validate_gke_autopilot_compliance.py deployments/overlays/<overlay-name>
  ```

* [ ] **CPU limit/request ratio ≤ 4.0**
  * Formula: `cpu_limit / cpu_request`
  * Example valid: 1000m / 250m = 4.0 ✅
  * Example invalid: 1000m / 200m = 5.0 ❌

* [ ] **Memory limit/request ratio ≤ 4.0**
  * Formula: `memory_limit / memory_request`

* [ ] **CPU/Memory within allowed ranges**
  * CPU: min 50m, max 4 cores
  * Memory: min 64Mi, max 8Gi

### 3. Security Configuration

* [ ] **If readOnlyRootFilesystem = true**
  * [ ] `/tmp` is mounted as writable volume
  * [ ] All application writable paths have volume mounts
  * [ ] Tested in development environment first

* [ ] **No privileged containers** (unless required)

* [ ] **Non-root user specified**
  ```yaml theme={null}
  securityContext:
    runAsNonRoot: true
    runAsUser: <uid>
  ```

* [ ] **Capabilities dropped**
  ```yaml theme={null}
  securityContext:
    capabilities:
      drop:
      - ALL
  ```

### 4. Resource Specifications

* [ ] **All containers have resource requests and limits**
  ```yaml theme={null}
  resources:
    requests:
      cpu: <value>
      memory: <value>
    limits:
      cpu: <value>
      memory: <value>
  ```

* [ ] **Resource requests are realistic**
  * Based on actual usage patterns
  * Not over-provisioned
  * Not under-provisioned

* [ ] **Ephemeral storage specified if needed**
  ```yaml theme={null}
  resources:
    requests:
      ephemeral-storage: 1Gi
    limits:
      ephemeral-storage: 2Gi
  ```

### 5. Environment Variables

* [ ] **No env vars with both `value` and `valueFrom`**
  ```yaml theme={null}
  # INVALID:
  env:
  - name: MY_VAR
    value: "something"
    valueFrom:  # Cannot have both!
      secretKeyRef:
        name: my-secret
        key: my-key

  # VALID:
  env:
  - name: MY_VAR
    valueFrom:
      secretKeyRef:
        name: my-secret
        key: my-key
  ```

* [ ] **All valueFrom sources have exactly one key**
  * `configMapKeyRef` OR `secretKeyRef` OR `fieldRef` (not multiple)

* [ ] **All referenced secrets exist**
  ```bash theme={null}
  kubectl get secret <secret-name> -n <namespace>
  ```

* [ ] **All referenced configmaps exist**
  ```bash theme={null}
  kubectl get configmap <configmap-name> -n <namespace>
  ```

### 6. Health Probes

* [ ] **Liveness probe configured**
  ```yaml theme={null}
  livenessProbe:
    httpGet:
      path: /health/live
      port: 8000
    initialDelaySeconds: 30
    periodSeconds: 10
  ```

* [ ] **Readiness probe configured**
  ```yaml theme={null}
  readinessProbe:
    httpGet:
      path: /health/ready
      port: 8000
    initialDelaySeconds: 10
    periodSeconds: 5
  ```

* [ ] **Probe endpoints exist and are accessible**
  * Test endpoint returns 200 OK when healthy
  * Endpoint doesn't require authentication

* [ ] **initialDelaySeconds is sufficient for app startup**

* [ ] **Timeout and period values are appropriate**

### 7. Volumes and Storage

* [ ] **All required volumes are defined**
  ```yaml theme={null}
  volumes:
  - name: tmp
    emptyDir: {}
  ```

* [ ] **All volume mounts reference existing volumes**

* [ ] **PersistentVolumeClaims exist and are bound**
  ```bash theme={null}
  kubectl get pvc -n <namespace>
  ```

* [ ] **Storage class is available**
  ```bash theme={null}
  kubectl get storageclass
  ```

### 8. Networking

* [ ] **Service exists for deployment (if needed)**
  ```bash theme={null}
  kubectl get service -n <namespace>
  ```

* [ ] **Service ports match container ports**

* [ ] **Network policies allow required traffic**
  ```bash theme={null}
  kubectl get networkpolicy -n <namespace>
  ```

* [ ] **Ingress configured correctly (if needed)**

### 9. IAM and Permissions

* [ ] **ServiceAccount exists**
  ```bash theme={null}
  kubectl get serviceaccount -n <namespace>
  ```

* [ ] **Workload Identity configured (for GCP)**
  ```yaml theme={null}
  metadata:
    annotations:
      iam.gke.io/gcp-service-account: <gcp-sa>@<project>.iam.gserviceaccount.com
  ```

* [ ] **GCP service account has required IAM roles**
  ```bash theme={null}
  gcloud iam service-accounts get-iam-policy <sa-email>
  ```

* [ ] **Workload Identity binding configured**
  ```bash theme={null}
  gcloud iam service-accounts add-iam-policy-binding <sa-email> \
    --role roles/iam.workloadIdentityUser \
    --member "serviceAccount:<project>.svc.id.goog[<namespace>/<ksa>]"
  ```

### 10. Image Configuration

* [ ] **Image tag is immutable (not `latest`)**
  * Use specific version tags or commit SHAs

* [ ] **Image exists in registry**
  ```bash theme={null}
  docker pull <image>:<tag>
  ```

* [ ] **Image pull secrets configured (if private registry)**

* [ ] **imagePullPolicy set appropriately**
  * `Always` for staging/production
  * `IfNotPresent` for development

***

## Post-Deployment Validation

### 1. Immediate Checks (0-5 minutes)

* [ ] **All pods are Running**
  ```bash theme={null}
  kubectl get pods -n <namespace>
  ```

* [ ] **No restarts occurring**
  ```bash theme={null}
  watch kubectl get pods -n <namespace>
  ```

* [ ] **Check logs for startup errors**
  ```bash theme={null}
  kubectl logs <pod-name> -n <namespace>
  ```

* [ ] **Health checks passing**
  ```bash theme={null}
  kubectl describe pod <pod-name> -n <namespace> | grep -A 5 "Readiness:"
  ```

### 2. Short-term Monitoring (5-30 minutes)

* [ ] **Pods remain stable (no CrashLoopBackOff)**

* [ ] **Memory/CPU usage is reasonable**
  ```bash theme={null}
  kubectl top pods -n <namespace>
  ```

* [ ] **No errors in application logs**

* [ ] **Metrics are being collected**

* [ ] **External dependencies are reachable**
  * Database connections working
  * External API calls succeeding

### 3. Functional Testing

* [ ] **Application endpoints are accessible**
  ```bash theme={null}
  kubectl port-forward <pod> 8080:8080 -n <namespace>
  curl http://localhost:8080/health
  ```

* [ ] **Core functionality works**
  * Test critical user flows
  * Verify data persistence
  * Check integrations

* [ ] **Performance is acceptable**
  * Response times normal
  * No unusual latency

### 4. Long-term Monitoring (24-48 hours)

* [ ] **Set up alerts for**:
  * Pod restarts
  * OOM kills
  * Probe failures
  * Error rate increases

* [ ] **Monitor resource usage trends**

* [ ] **Check for memory leaks**

* [ ] **Verify log aggregation working**

***

## Rollback Procedure

If issues are detected post-deployment:

### 1. Quick Rollback

```bash theme={null}
# Rollback to previous revision
kubectl rollout undo deployment/<deployment> -n <namespace>

# Verify rollback succeeded
kubectl rollout status deployment/<deployment> -n <namespace>
```

### 2. Rollback to Specific Revision

```bash theme={null}
# List revisions
kubectl rollout history deployment/<deployment> -n <namespace>

# Rollback to specific revision
kubectl rollout undo deployment/<deployment> --to-revision=<revision> -n <namespace>
```

### 3. Rollback with Kustomize

```bash theme={null}
# Revert Git changes
git revert <commit-hash>

# Reapply previous configuration
kubectl apply -k deployments/overlays/<overlay> --namespace=<namespace>
```

***

## Tools and Scripts

### Available Tools:

1. `scripts/validate_gke_autopilot_compliance.py` - Validation script
2. `tests/regression/test_pod_deployment_regression.py` - Regression tests
3. `.githooks/pre-commit` - Pre-commit validation hook
4. `.github/workflows/validate-k8s-configs.yml` - CI/CD pipeline

### Usage Examples:

```bash theme={null}
# Validate before deploying
python3 scripts/validate_gke_autopilot_compliance.py deployments/overlays/production-gke

# Run all regression tests
pytest tests/regression/ -v

# Install pre-commit hook
git config core.hooksPath .githooks
```

***

## Common Mistakes to Avoid

1. ❌ **Deploying without validation**
   * Always run validation script first

2. ❌ **Enabling readOnlyRootFilesystem without testing**
   * Test in development first
   * Ensure all writable paths are mounted

3. ❌ **Not checking CPU/memory ratios on GKE Autopilot**
   * Ratio must be ≤ 4.0

4. ❌ **Using `latest` image tag**
   * Use specific versions or commit SHAs

5. ❌ **Not testing rollback procedure**
   * Verify you can rollback before deploying

6. ❌ **Insufficient health probe delays**
   * Account for slow application startup

7. ❌ **Missing IAM permissions**
   * Verify service account permissions before deploying

8. ❌ **Not monitoring after deployment**
   * Watch pods for at least 10 minutes

9. ❌ **Forgetting to delete old ReplicaSets**
   * Clean up after successful deployment

10. ❌ **Not documenting changes**
    * Always document what was changed and why

***

## Emergency Contacts

* **Platform Team**: @platform-team
* **On-Call Engineer**: Check PagerDuty
* **Documentation**: See [Troubleshooting Guide](/troubleshooting/overview#deployment-problems)

***

## Related Documentation

* [Troubleshooting Guide - Deployment Problems](/troubleshooting/overview#deployment-problems)
* [GKE Autopilot Best Practices](https://cloud.google.com/kubernetes-engine/docs/concepts/autopilot-overview)
* [ADR-0040: GCP GKE Autopilot Deployment](/architecture/adr-0040-gcp-gke-autopilot-deployment)
* [ADR-0054: Pod Failure Prevention Framework](/architecture/adr-0054-pod-failure-prevention-framework)

***

**Last Validated:** 2025-11-12 (Staging rebuild from scratch succeeded)