> ## Documentation Index
> Fetch the complete documentation index at: https://mcp-server-langgraph.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# GKE Operational Runbooks

> Day-2 operations, incident response, and maintenance procedures for MCP Server on GKE Autopilot

### Overview

This guide provides operational procedures for managing MCP Server LangGraph on GKE Autopilot in production, including incident response, maintenance tasks, and troubleshooting.

<Note>
  This guide provides essential operational procedures for managing GKE Autopilot deployments in production.
</Note>

***

### Daily Health Check (5 minutes)

Run this every morning to ensure system health:

<Steps>
  <Step title="Check Cluster Status">
    ```bash theme={null}
    gcloud container clusters describe production-mcp-server-langgraph-gke \
      --region=us-central1 \
      --format="value(status)"
    ```

    <Check>Should return: `RUNNING`</Check>
  </Step>

  <Step title="Check Pod Health">
    ```bash theme={null}
    kubectl get pods -n production-mcp-server-langgraph

    # Check for failed pods
    kubectl get pods -n production-mcp-server-langgraph \
      --field-selector=status.phase!=Running,status.phase!=Succeeded
    ```

    <Check>All pods should show `Running` status with 0-1 restarts</Check>
  </Step>

  <Step title="Review Recent Events">
    ```bash theme={null}
    kubectl get events -n production-mcp-server-langgraph \
      --sort-by='.lastTimestamp' \
      --field-selector=type!=Normal \
      | tail -10
    ```

    <Check>No ERROR or WARNING events in last hour</Check>
  </Step>

  <Step title="Check Database Status">
    ```bash theme={null}
    gcloud sql instances describe mcp-prod-postgres \
      --format="table(name,state,ipAddresses[0].ipAddress)"
    ```

    <Check>State should be: `RUNNABLE`</Check>
  </Step>

  <Step title="View Recent Errors">
    ```bash theme={null}
    gcloud logging read \
      'resource.type="k8s_container" AND resource.labels.namespace_name="mcp-production" AND severity>=ERROR' \
      --limit=10 \
      --format="table(timestamp,jsonPayload.message)"
    ```

    <Check>No critical errors in last hour</Check>
  </Step>
</Steps>

***

### Incident Response

#### P0: Service Down (Complete Outage)

<Warning>
  **Symptoms**: All pods crashing, health checks failing, users cannot access service
</Warning>

**Response Time**: 5-10 minutes

<Steps>
  <Step title="Immediate Assessment">
    ```bash theme={null}
    # Check pod status
    kubectl get pods -n production-mcp-server-langgraph

    # Get pod logs
    kubectl logs -n production-mcp-server-langgraph -l app=mcp-server-langgraph --tail=100

    # Describe failing pods
    kubectl describe pod POD_NAME -n production-mcp-server-langgraph
    ```
  </Step>

  <Step title="Quick Fixes">
    <Tabs>
      <Tab title="Restart Deployment">
        ```bash theme={null}
        kubectl rollout restart deployment/production-mcp-server-langgraph \
          -n production-mcp-server-langgraph
        ```
      </Tab>

      <Tab title="Rollback">
        ```bash theme={null}
        kubectl rollout undo deployment/production-mcp-server-langgraph \
          -n production-mcp-server-langgraph

        kubectl rollout status deployment/production-mcp-server-langgraph \
          -n production-mcp-server-langgraph
        ```
      </Tab>

      <Tab title="Scale Up">
        ```bash theme={null}
        kubectl scale deployment production-mcp-server-langgraph \
          -n production-mcp-server-langgraph \
          --replicas=6
        ```
      </Tab>
    </Tabs>
  </Step>

  <Step title="Verify Recovery">
    ```bash theme={null}
    kubectl get pods -n production-mcp-server-langgraph
    kubectl exec -it -n production-mcp-server-langgraph POD_NAME -- curl http://localhost:8000/health/live
    ```
  </Step>

  <Step title="Post-Incident Analysis">
    ```bash theme={null}
    # Export logs for analysis
    gcloud logging read \
      'resource.type="k8s_container" AND severity>=ERROR' \
      --limit=50 \
      --format=json \
      > incident-logs.json
    ```
  </Step>
</Steps>

**Escalation**: If not resolved in 10 minutes, escalate to on-call architect

***

#### P1: Performance Degradation

<Warning>
  **Symptoms**: Slow response times, high CPU/memory, increased error rates
</Warning>

<Steps>
  <Step title="Check Resource Usage">
    ```bash theme={null}
    kubectl top pods -n production-mcp-server-langgraph
    kubectl top nodes
    ```
  </Step>

  <Step title="Check HPA Status">
    ```bash theme={null}
    kubectl get hpa -n production-mcp-server-langgraph -o yaml
    ```
  </Step>

  <Step title="Review Database Performance">
    ```bash theme={null}
    gcloud sql operations list \
      --instance=mcp-prod-postgres \
      --filter="startTime>=$(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%S).000000Z"
    ```
  </Step>

  <Step title="Temporary Scale Up">
    ```bash theme={null}
    kubectl scale deployment production-mcp-server-langgraph \
      -n production-mcp-server-langgraph \
      --replicas=10

    # Monitor improvement
    watch -n 5 'kubectl top pods -n production-mcp-server-langgraph'
    ```
  </Step>
</Steps>

**Permanent Fix**: Adjust resource requests/limits based on actual usage

***

#### P2: Database Connection Issues

<AccordionGroup>
  <Accordion title="Connection Refused">
    **Check Cloud SQL Proxy**:

    ```bash theme={null}
    kubectl logs -n production-mcp-server-langgraph POD_NAME -c cloud-sql-proxy --tail=50
    ```

    **Verify Instance Running**:

    ```bash theme={null}
    gcloud sql instances describe mcp-prod-postgres --format="value(state)"
    ```

    **Restart Proxy**:

    ```bash theme={null}
    kubectl rollout restart deployment/production-mcp-server-langgraph -n production-mcp-server-langgraph
    ```
  </Accordion>

  <Accordion title="Too Many Connections">
    **Check Connection Count**:

    ```bash theme={null}
    kubectl port-forward -n production-mcp-server-langgraph svc/production-mcp-server-langgraph 5432:5432 &
    psql "host=localhost user=postgres" -c "SELECT count(*) FROM pg_stat_activity;"
    ```

    **Kill Idle Connections**:

    ```bash theme={null}
    psql -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE state = 'idle' AND state_change < NOW() - INTERVAL '10 minutes';"
    ```

    **Increase Max Connections** (via Terraform):

    ```hcl theme={null}
    database_flags = {
      "max_connections" = "200"
    }
    ```
  </Accordion>
</AccordionGroup>

***

### Deployment Operations

#### Standard Deployment (via CI/CD)

<Info>
  **Recommended**: Use the automated GitHub Actions workflow for production deployments.
</Info>

<Steps>
  <Step title="Create Release">
    ```bash theme={null}
    gh release create v1.1.0 \
      --title "Release 1.1.0" \
      --notes "Release notes here"
    ```
  </Step>

  <Step title="Monitor CI/CD">
    The workflow automatically:

    * Builds and scans image
    * Requests manual approval
    * Deploys to production
    * Runs validation tests
    * Rolls back on failure

    Monitor at: `https://github.com/USER/REPO/actions`
  </Step>

  <Step title="Verify Deployment">
    ```bash theme={null}
    kubectl rollout status deployment/production-mcp-server-langgraph -n production-mcp-server-langgraph
    kubectl get pods -n production-mcp-server-langgraph
    ```
  </Step>
</Steps>

#### Manual Deployment (Emergency)

<Warning>
  **Use only for emergencies**. Bypasses automated testing and approval gates.
</Warning>

```bash theme={null}
cd deployments/overlays/production-gke

## Update image tag
kustomize edit set image mcp-server-langgraph=REGISTRY/IMAGE:NEW_TAG

## Dry run
kubectl apply -k . --dry-run=server

## Apply
kubectl apply -k .

## Monitor
kubectl rollout status deployment/production-mcp-server-langgraph -n production-mcp-server-langgraph
```

#### Rollback Deployment

<Tabs>
  <Tab title="Quick Rollback">
    ```bash theme={null}
    # Rollback to previous revision
    kubectl rollout undo deployment/production-mcp-server-langgraph \
      -n production-mcp-server-langgraph

    # Monitor
    kubectl rollout status deployment/production-mcp-server-langgraph \
      -n production-mcp-server-langgraph
    ```
  </Tab>

  <Tab title="Specific Revision">
    ```bash theme={null}
    # List revisions
    kubectl rollout history deployment/production-mcp-server-langgraph \
      -n production-mcp-server-langgraph

    # Rollback to specific revision
    kubectl rollout undo deployment/production-mcp-server-langgraph \
      -n production-mcp-server-langgraph \
      --to-revision=5
    ```
  </Tab>
</Tabs>

***

### Database Operations

#### Manual Backup

```bash theme={null}
gcloud sql backups create \
  --instance=mcp-prod-postgres \
  --description="Manual backup before major change" \
  --project=PROJECT_ID

## List backups
gcloud sql backups list --instance=mcp-prod-postgres
```

#### Restore from Backup

<Warning>
  **Caution**: Restoring to the same instance is destructive. Restore to a new instance first for safety.
</Warning>

<Tabs>
  <Tab title="Restore to New Instance">
    ```bash theme={null}
    # Clone to new instance
    gcloud sql instances clone mcp-prod-postgres mcp-prod-postgres-restored \
      --backup-id=BACKUP_ID \
      --project=PROJECT_ID
    ```

    **Recommended**: Test on new instance, then switch connection if successful
  </Tab>

  <Tab title="Point-in-Time Recovery">
    ```bash theme={null}
    # Restore to specific timestamp
    gcloud sql instances clone mcp-prod-postgres mcp-prod-postgres-pitr \
      --point-in-time='2025-11-01T12:00:00.000Z' \
      --project=PROJECT_ID
    ```

    **PITR Window**: 7 days (configurable)
  </Tab>
</Tabs>

#### Database Maintenance

**Check Maintenance Window**:

```bash theme={null}
gcloud sql instances describe mcp-prod-postgres \
  --format="value(settings.maintenanceWindow)"
```

**Reschedule Maintenance**:

```bash theme={null}
gcloud sql instances patch mcp-prod-postgres \
  --maintenance-window-day=1 \
  --maintenance-window-hour=3 \
  --project=PROJECT_ID
```

***

### Scaling Operations

#### Manual Scaling

<CodeGroup>
  ````bash Scale Deployment theme={null}
  kubectl scale deployment production-mcp-server-langgraph \
    -n production-mcp-server-langgraph \
    --replicas=10
  ```bash
  ```bash Update HPA
  kubectl patch hpa production-mcp-server-langgraph \
    -n production-mcp-server-langgraph \
    --patch '{"spec":{"minReplicas":5,"maxReplicas":30}}'
  ````

  ```bash Update Cluster Limits theme={null}
  ## Via Terraform
  cd terraform/environments/gcp-prod

  ## Edit terraform.tfvars:
  ## max_cluster_cpu    = 2000
  ## max_cluster_memory = 20000

  terraform apply
  ```
</CodeGroup>

#### Cluster Resource Monitoring

```bash theme={null}
## View resource usage
kubectl top pods -n production-mcp-server-langgraph
kubectl top nodes

## Cloud Monitoring query
gcloud monitoring time-series list \
  --filter='metric.type="kubernetes.io/container/cpu/core_usage_time"' \
  --project=PROJECT_ID
```

***

### Security Operations

#### Rotate Secrets

<Steps>
  <Step title="Generate New Secrets">
    ```properties theme={null}
    NEW_JWT_SECRET=$(openssl rand -base64 32)
    NEW_API_KEY=$(openssl rand -base64 32)
    ```
  </Step>

  <Step title="Update Secret Manager">
    ```bash theme={null}
    # Update secret version
    gcloud secrets versions add mcp-production-secrets \
      --data-file=<(jq --arg jwt "$NEW_JWT_SECRET" '.jwt_secret = $jwt' secrets.json) \
      --project=PROJECT_ID
    ```
  </Step>

  <Step title="Pods Auto-Restart">
    External Secrets Operator automatically syncs secrets and Reloader restarts pods.

    Monitor:

    ```bash theme={null}
    kubectl get pods -n production-mcp-server-langgraph -w
    ```
  </Step>
</Steps>

#### Audit Access Logs

```bash theme={null}
## View cluster access
gcloud logging read \
  'protoPayload.serviceName="container.googleapis.com" AND protoPayload.methodName="io.k8s.core.v1.pods.exec"' \
  --limit=50

## View kubectl commands
gcloud logging read \
  'protoPayload.serviceName="container.googleapis.com"' \
  --limit=20
```

#### Review Binary Authorization Denials

```bash theme={null}
gcloud logging read \
  'protoPayload.serviceName="binaryauthorization.googleapis.com" AND protoPayload.response.allow=false' \
  --limit=20
```

***

### Common Tasks

<AccordionGroup>
  <Accordion title="View Application Logs">
    <CodeGroup>
      ```bash Real-time theme={null}
      kubectl logs -f -n production-mcp-server-langgraph \
        -l app=mcp-server-langgraph \
        --max-log-requests=10
      ```

      ```bash Cloud Logging theme={null}
      gcloud logging read \
        'resource.type="k8s_container" AND resource.labels.namespace_name="mcp-production"' \
        --limit=100 \
        --format=json
      ```

      ```bash Specific Container theme={null}
      kubectl logs POD_NAME -n production-mcp-server-langgraph -c mcp-server-langgraph
      kubectl logs POD_NAME -n production-mcp-server-langgraph -c cloud-sql-proxy
      ```
    </CodeGroup>
  </Accordion>

  <Accordion title="Update Configuration">
    ```bash theme={null}
    # Edit ConfigMap
    kubectl edit configmap production-mcp-server-langgraph-config \
      -n production-mcp-server-langgraph

    # Restart to pick up changes (Reloader does this automatically)
    kubectl rollout restart deployment/production-mcp-server-langgraph \
      -n production-mcp-server-langgraph
    ```
  </Accordion>

  <Accordion title="Connect to Database">
    <Tabs>
      <Tab title="Cloud SQL Proxy">
        ```bash theme={null}
        cloud-sql-proxy PROJECT_ID:us-central1:mcp-prod-postgres &
        psql "host=localhost port=5432 user=postgres dbname=mcp_langgraph"
        ```
      </Tab>

      <Tab title="kubectl Port Forward">
        ```bash theme={null}
        kubectl port-forward -n production-mcp-server-langgraph \
          svc/production-mcp-server-langgraph 5432:5432 &

        psql "host=localhost port=5432 user=postgres dbname=mcp_langgraph"
        ```
      </Tab>
    </Tabs>
  </Accordion>

  <Accordion title="Export Metrics">
    ```bash theme={null}
    # Export to JSON for analysis
    gcloud monitoring time-series list \
      --filter='resource.type="k8s_container"' \
      --format=json \
      > metrics-$(date +%Y%m%d).json
    ```
  </Accordion>
</AccordionGroup>

***

### Maintenance Windows

#### Cluster Upgrades

GKE Autopilot upgrades automatically based on release channel (STABLE for production).

<Info>
  **Release Channels**:

  * **RAPID**: Weekly (for testing)
  * **REGULAR**: Monthly (for general use)
  * **STABLE**: Quarterly (for production) ✅
</Info>

**Check upgrade status**:

```bash theme={null}
gcloud container clusters describe production-mcp-server-langgraph-gke \
  --region=us-central1 \
  --format="yaml(currentMasterVersion,releaseChannel)"
```

**Manual upgrade** (if needed):

```bash theme={null}
## Check available versions
gcloud container get-server-config --region=us-central1

## Upgrade
gcloud container clusters upgrade production-mcp-server-langgraph-gke \
  --region=us-central1 \
  --cluster-version=VERSION
```

#### Database Maintenance

Scheduled maintenance: **Sunday 3 AM UTC** (configured in Terraform)

**Reschedule**:

```bash theme={null}
gcloud sql instances patch mcp-prod-postgres \
  --maintenance-window-day=1 \  # Monday
  --maintenance-window-hour=3 \  # 3 AM UTC
  --project=PROJECT_ID
```

**Defer one-time**:

```bash theme={null}
gcloud sql instances reschedule-maintenance mcp-prod-postgres \
  --reschedule-type=NEXT_AVAILABLE_WINDOW
```

***

### Monitoring & Alerting

#### Active Alerts

```bash theme={null}
## List firing alerts
gcloud alpha monitoring policies list \
  --filter="enabled=true" \
  --project=PROJECT_ID

## Get alert details
gcloud alpha monitoring policies describe POLICY_ID
```

#### Create Custom Alert

```bash theme={null}
gcloud alpha monitoring policies create \
  --notification-channels=CHANNEL_ID \
  --display-name="High Error Rate" \
  --condition-threshold-value=0.05 \
  --condition-threshold-duration=300s
```

#### View Metrics

Access Cloud Monitoring dashboards:

<CardGroup cols={2}>
  <Card title="GKE Dashboard" icon="chart-line" href="https://console.cloud.google.com/kubernetes">
    Pod metrics, cluster health, node status
  </Card>

  <Card title="Custom Dashboard" icon="chart-mixed" href="https://console.cloud.google.com/monitoring/dashboards">
    Custom metrics created by setup script
  </Card>
</CardGroup>

***

### Disaster Recovery

#### Cloud SQL Failover

**Manual Failover** (for testing):

```bash theme={null}
gcloud sql instances failover mcp-prod-postgres --project=PROJECT_ID

## Monitor
watch -n 5 'gcloud sql instances describe mcp-prod-postgres --format="value(state)"'
```

**Recovery Time**: 2-3 minutes for automatic failover

#### Full DR Procedure

For complete disaster recovery automation:

```bash theme={null}
./deployments/disaster-recovery/gcp-dr-automation.sh PROJECT_ID us-east1 us-central1 full
```

<Card title="Complete DR Guide" icon="life-ring" href="/deployment/disaster-recovery">
  Multi-region failover, backup restoration, RTO/RPO targets
</Card>

***

### Emergency Procedures

#### Emergency Stop

<Warning>
  **This stops all traffic**. Use only in critical situations (security breach, data corruption).
</Warning>

```bash theme={null}
## Scale to zero
kubectl scale deployment production-mcp-server-langgraph \
  -n production-mcp-server-langgraph \
  --replicas=0

## Resume
kubectl scale deployment production-mcp-server-langgraph \
  -n production-mcp-server-langgraph \
  --replicas=3
```

#### Emergency Maintenance Mode

```yaml theme={null}
## Apply maintenance page
apiVersion: v1
kind: Service
metadata:
  name: maintenance-page
spec:
  selector:
    app: maintenance
  ports:
  - port: 80
    targetPort: 8080
```

***

### Contacts & Escalation

<Card title="On-Call Rotation" icon="phone">
  Configure in PagerDuty/Opsgenie

  **Escalation Path**:

  1. P0/P1: On-call engineer (immediate)
  2. P2: Engineering team lead (within 4 hours)
  3. P3: Ticket for next sprint

  **Communication**: #production-incidents (Slack)
</Card>

***

### Related Documentation

<CardGroup cols={2}>
  <Card title="Troubleshooting Guide" icon="screwdriver-wrench" href="/advanced/troubleshooting">
    Detailed debugging procedures
  </Card>

  <Card title="Security Hardening" icon="shield" href="/security/gcp-security-hardening">
    Security operations and auditing
  </Card>

  <Card title="Cost Optimization" icon="money-bill-trend-up" href="/deployment/cost-optimization">
    Resource rightsizing and cost controls
  </Card>

  <Card title="Deployment Guide" icon="rocket" href="/deployment/kubernetes/gke-production">
    Production deployment procedures
  </Card>
</CardGroup>