> ## Documentation Index
> Fetch the complete documentation index at: https://mcp-server-langgraph.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Disaster Recovery

> Backup, restore, and failover strategies for production resilience

### Overview

Implement comprehensive disaster recovery (DR) to protect against data loss, service disruption, and regional failures. This guide covers backup strategies, restore procedures, and multi-region failover.

<Info>
  **RTO vs RPO**: Recovery Time Objective (RTO) is how quickly you can restore service. Recovery Point Objective (RPO) is how much data you can afford to lose. Balance both based on business requirements.
</Info>

### Disaster Recovery Flow

```mermaid theme={null}
%% ColorBrewer2 Set3 palette - each component type uniquely colored
flowchart TB
    subgraph "Production Environment (Primary)"
        App[MCP Server<br/>GKE Cluster]
        PG[(PostgreSQL<br/>Keycloak + OpenFGA)]
        Redis[(Redis<br/>Sessions + Cache)]
        K8s[Kubernetes<br/>Manifests]
    end

    subgraph "Backup Process (Automated)"
        Cron[Cron Jobs<br/>Scheduled Backups]
        PGBackup[PostgreSQL Backup<br/>pg_dump / pg_basebackup]
        RedisBackup[Redis Backup<br/>RDB snapshots]
        K8sBackup[Kubernetes Backup<br/>Velero]
        SecretBackup[Secret Manager<br/>Backup]
    end

    subgraph "Cloud Storage (Multi-Region)"
        GCS[Google Cloud Storage<br/>Versioned Buckets]
        S3[AWS S3<br/>Cross-Region Replication]
        Vault[Secret Vault<br/>Encrypted Backups]
    end

    subgraph "Disaster Scenarios"
        Regional[Regional Failure<br/>GCP us-central1 down]
        Data[Data Corruption<br/>Accidental deletion]
        Security[Security Breach<br/>Ransomware attack]
    end

    subgraph "Recovery Environment (Secondary)"
        AppDR[MCP Server<br/>Standby Cluster]
        PGDR[(PostgreSQL<br/>Restored from backup)]
        RedisDR[(Redis<br/>Rebuilt from snapshot)]
        K8sDR[Kubernetes<br/>Restored manifests]
    end

    %% Backup flows
    App --> Cron
    PG --> PGBackup
    Redis --> RedisBackup
    K8s --> K8sBackup

    Cron --> PGBackup
    Cron --> RedisBackup
    Cron --> K8sBackup
    Cron --> SecretBackup

    PGBackup --> GCS
    RedisBackup --> GCS
    K8sBackup --> GCS
    SecretBackup --> Vault

    GCS --> S3
    Vault --> S3

    %% Disaster triggers
    Regional -.->|Triggers| Restore[Restore Process]
    Data -.->|Triggers| Restore
    Security -.->|Triggers| Restore

    %% Recovery flows
    Restore --> GCS
    GCS --> PGDR
    GCS --> RedisDR
    GCS --> K8sDR
    Vault --> SecretBackup2[Restore Secrets]
    SecretBackup2 --> AppDR

    PGDR --> AppDR
    RedisDR --> AppDR
    K8sDR --> AppDR

    AppDR --> Verify[Verification<br/>Health checks + Smoke tests]
    Verify --> Switch[DNS Failover<br/>Traffic to DR cluster]

    %% ColorBrewer2 + semantic color palette styling
    classDef failureStyle fill:#f8d7da,stroke:#dc3545,stroke-width:2px,color:#333
    classDef activeStyle fill:#b3de69,stroke:#7cb342,stroke-width:2px,color:#333
    classDef dataStyle fill:#80b1d3,stroke:#3498db,stroke-width:2px,color:#333
    classDef cacheStyle fill:#fb8072,stroke:#c0392b,stroke-width:2px,color:#333
    classDef infraStyle fill:#bebada,stroke:#8e44ad,stroke-width:2px,color:#333
    classDef storageStyle fill:#ffffb3,stroke:#f39c12,stroke-width:2px,color:#333
    classDef secretStyle fill:#bc80bd,stroke:#8e44ad,stroke-width:2px,color:#333
    classDef recoveryStyle fill:#d4edda,stroke:#28a745,stroke-width:2px,color:#333
    classDef verifyStyle fill:#fff3cd,stroke:#ffc107,stroke-width:2px,color:#333

    class Regional,Data,Security failureStyle
    class App activeStyle
    class PG dataStyle
    class Redis cacheStyle
    class K8s infraStyle
    class GCS,S3 storageStyle
    class Vault secretStyle
    class AppDR,PGDR,RedisDR recoveryStyle
    class Verify,Switch verifyStyle
```

**Backup and Restore Timeline**:

```mermaid theme={null}
sequenceDiagram
    %%{init: {'theme':'base', 'themeVariables': {'primaryColor':'#8dd3c7','primaryTextColor':'#1a202c','primaryBorderColor':'#2a9d8f','lineColor':'#fb8072','secondaryColor':'#fdb462','tertiaryColor':'#b3de69','actorBkg':'#8dd3c7','actorBorder':'#2a9d8f','actorTextColor':'#1a202c','actorLineColor':'#2a9d8f','signalColor':'#7cb342','signalTextColor':'#1a202c','labelBoxBkgColor':'#fdb462','labelBoxBorderColor':'#e67e22','labelTextColor':'#1a202c','noteBorderColor':'#e67e22','noteBkgColor':'#fdb462','noteTextColor':'#1a202c','activationBorderColor':'#7cb342','activationBkgColor':'#b3de69','sequenceNumberColor':'#4a5568'}}}%%
    participant Prod as Production<br/>Environment
    participant Cron as Backup Cron
    participant Storage as Cloud Storage<br/>(GCS/S3)
    participant Monitor as Monitoring<br/>Alerting
    participant Ops as Ops Team
    participant DR as DR Environment

    Note over Prod,Cron: Daily Automated Backups
    loop Every 24 hours
        Cron->>Prod: Trigger backup<br/>PostgreSQL full dump
        Prod->>Prod: Create snapshot<br/>pg_basebackup
        Prod->>Storage: Upload backup<br/>keycloak_20250129.dump
        Storage->>Storage: Replicate to S3<br/>Cross-region
        Cron->>Monitor: Report success<br/>Backup completed
    end

    loop Every 1 hour
        Cron->>Prod: Incremental backup<br/>WAL archives
        Prod->>Storage: Upload WAL files
    end

    Note over Prod,Monitor: Disaster Occurs
    Prod->>Prod: Regional failure<br/>GCP us-central1 unavailable
    Monitor->>Monitor: Detect failure<br/>Health checks fail
    Monitor->>Ops: Alert: Production DOWN<br/>PagerDuty escalation

    Note over Ops,DR: Disaster Recovery Initiated
    Ops->>DR: Provision DR cluster<br/>GKE in us-east1
    DR->>DR: Create infrastructure<br/>Terraform apply

    Ops->>Storage: Fetch latest backups<br/>keycloak_20250129_2300.dump
    Storage->>DR: Download to DR cluster

    Note over DR: Restore Database
    DR->>DR: Create PostgreSQL<br/>Fresh instance
    DR->>DR: Restore from backup<br/>pg_restore keycloak.dump
    DR->>DR: Apply WAL files<br/>Point-in-time recovery

    Note over DR: Restore Redis
    DR->>Storage: Download Redis RDB
    Storage->>DR: redis_snapshot.rdb
    DR->>DR: redis-server --appendonly yes<br/>Load RDB + AOF

    Note over DR: Restore Kubernetes
    DR->>Storage: Download K8s backups<br/>Velero restore
    Storage->>DR: Manifests + PVCs
    DR->>DR: kubectl apply<br/>Restore all resources

    Note over DR: Restore Secrets
    DR->>Storage: Download encrypted secrets
    Storage->>DR: Vault backup
    DR->>DR: Decrypt and apply<br/>to Secret Manager

    Note over DR: Verification
    DR->>DR: Run health checks<br/>/health, /ready
    DR->>DR: Run smoke tests<br/>Authentication, LLM calls
    DR->>DR: Verify data integrity<br/>User count, permissions

    DR->>Ops: DR Ready<br/>All checks passed ✓

    Note over Ops: Failover Decision
    Ops->>Ops: Update DNS<br/>api.example.com → DR IP
    Ops->>Ops: Update load balancer<br/>Point to DR cluster

    Note over DR: DR Now Active
    DR->>Monitor: Send health metrics<br/>Normal operation
    Monitor->>Ops: Alert: DR ACTIVE<br/>RTO: 45 minutes ✓

```

### Backup Strategy

#### What to Backup

<CardGroup cols={2}>
  <Card title="PostgreSQL Databases" icon="database">
    * Keycloak user data
    * OpenFGA authorization tuples
    * Application metadata
  </Card>

  <Card title="Redis Data" icon="memory">
    * Active sessions
    * Cache data
    * Rate limit counters
  </Card>

  <Card title="Kubernetes Resources" icon="dharmachakra">
    * Deployments
    * ConfigMaps/Secrets
    * Ingress rules
    * PVCs
  </Card>

  <Card title="Configuration" icon="gear">
    * Environment variables
    * Helm values
    * Infrastructure as Code
  </Card>
</CardGroup>

#### Backup Schedule

| Component                | Frequency | Retention | RPO | RTO |
| ------------------------ | --------- | --------- | --- | --- |
| PostgreSQL (full)        | Daily     | 30 days   | 24h | 1h  |
| PostgreSQL (incremental) | Hourly    | 7 days    | 1h  | 30m |
| Redis                    | Hourly    | 24 hours  | 1h  | 15m |
| Kubernetes               | Daily     | 7 days    | 24h | 2h  |
| Secrets                  | Weekly    | 90 days   | 7d  | 4h  |

### PostgreSQL Backup

#### Automated Backups

<Tabs>
  <Tab title="pg_dump">
    ```bash theme={null}
    #!/bin/bash
    # backup-postgres.sh

    TIMESTAMP=$(date +%Y%m%d_%H%M%S)
    BACKUP_DIR="/backups/postgres"

    # Keycloak database
    pg_dump -U keycloak -h postgres-keycloak \
      -d keycloak \
      -F c -b -v \
      -f "${BACKUP_DIR}/keycloak_${TIMESTAMP}.dump"

    # OpenFGA database
    pg_dump -U openfga -h postgres-openfga \
      -d openfga \
      -F c -b -v \
      -f "${BACKUP_DIR}/openfga_${TIMESTAMP}.dump"

    # Compress
    gzip "${BACKUP_DIR}"/*.dump

    # Upload to S3
    aws s3 sync ${BACKUP_DIR} s3://my-backups/postgres/ \
      --storage-class STANDARD_IA

    # Clean up local files older than 7 days
    find ${BACKUP_DIR} -name "*.dump.gz" -mtime +7 -delete
    ```

    **Schedule with cron**:

    ```bash theme={null}
    # Daily full backup at 2 AM
    0 2 * * * /scripts/backup-postgres.sh
    ```
  </Tab>

  <Tab title="CronJob (Kubernetes)">
    ```yaml theme={null}
    apiVersion: batch/v1
    kind: CronJob
    metadata:
      name: postgres-backup
      namespace: mcp-server-langgraph
    spec:
      schedule: "0 2 * * *"  # Daily at 2 AM
      successfulJobsHistoryLimit: 3
      failedJobsHistoryLimit: 1
      jobTemplate:
        spec:
          template:
            spec:
              restartPolicy: OnFailure
              containers:
              - name: backup
                image: postgres:15-alpine
                env:
                - name: PGPASSWORD
                  valueFrom:
                    secretKeyRef:
                      name: postgres-credentials
                      key: password
                command:
                - /bin/sh
                - -c
                - |
                  TIMESTAMP=$(date +%Y%m%d_%H%M%S)

                  # Backup Keycloak
                  pg_dump -U keycloak -h postgres-keycloak -d keycloak \
                    -F c -b -v -f /backups/keycloak_${TIMESTAMP}.dump

                  # Backup OpenFGA
                  pg_dump -U openfga -h postgres-openfga -d openfga \
                    -F c -b -v -f /backups/openfga_${TIMESTAMP}.dump

                  # Upload to S3
                  aws s3 sync /backups s3://my-backups/postgres/

                volumeMounts:
                - name: backup-storage
                  mountPath: /backups
              volumes:
              - name: backup-storage
                persistentVolumeClaim:
                  claimName: backup-pvc
    ```
  </Tab>

  <Tab title="Cloud-Managed">
    **GCP Cloud SQL**:

    ```bash theme={null}
    # Enable automated backups
    gcloud sql instances patch keycloak-db \
      --backup-start-time=02:00 \
      --enable-bin-log \
      --retained-backups-count=30

    # Point-in-time recovery
    gcloud sql instances patch keycloak-db \
      --enable-point-in-time-recovery \
      --retained-transaction-log-days=7
    ```

    **AWS RDS**:

    ```bash theme={null}
    # Enable automated backups
    aws rds modify-db-instance \
      --db-instance-identifier keycloak-db \
      --backup-retention-period 30 \
      --preferred-backup-window 02:00-03:00 \
      --enable-iam-database-authentication
    ```

    **Azure Database**:

    ```bash theme={null}
    az postgres server update \
      --resource-group langgraph-rg \
      --name keycloak-db \
      --backup-retention 30 \
      --geo-redundant-backup Enabled
    ```
  </Tab>
</Tabs>

#### Point-in-Time Recovery

```bash theme={null}
## Restore to specific timestamp
pg_restore -U keycloak -h postgres-keycloak \
  -d keycloak_recovery \
  -c -v \
  /backups/keycloak_20251012_020000.dump.gz

## Verify data
psql -U keycloak -h postgres-keycloak -d keycloak_recovery -c "SELECT COUNT(*) FROM users;"

## Switch to recovered database
kubectl exec -it postgres-keycloak-0 -- psql -U postgres -c "ALTER DATABASE keycloak RENAME TO keycloak_old;"
kubectl exec -it postgres-keycloak-0 -- psql -U postgres -c "ALTER DATABASE keycloak_recovery RENAME TO keycloak;"
```

#### Continuous Archiving (WAL)

```
## postgresql.conf
wal_level = replica
archive_mode = on
archive_command = 'aws s3 cp %p s3://my-backups/postgres-wal/%f'
archive_timeout = 300  # 5 minutes

## Restore from WAL
restore_command = 'aws s3 cp s3://my-backups/postgres-wal/%f %p'
recovery_target_time = '2025-10-12 10:00:00'
```

### Redis Backup

#### RDB Snapshots

```conf theme={null}
## redis.conf
save 900 1      # Save if 1 key changed in 15 min
save 300 10     # Save if 10 keys changed in 5 min
save 60 10000   # Save if 10000 keys changed in 1 min

dir /data
dbfilename dump.rdb
```

**Backup Script**:

```bash theme={null}
#!/bin/bash
## backup-redis.sh

TIMESTAMP=$(date +%Y%m%d_%H%M%S)
BACKUP_DIR="/backups/redis"

## Trigger save
redis-cli -h redis-master BGSAVE

## Wait for save to complete
while [ $(redis-cli -h redis-master LASTSAVE) -eq $LAST_SAVE ]; do
  sleep 1
done

## Copy RDB file
kubectl cp redis-master-0:/data/dump.rdb \
  "${BACKUP_DIR}/dump_${TIMESTAMP}.rdb"

## Upload to S3
aws s3 cp "${BACKUP_DIR}/dump_${TIMESTAMP}.rdb" \
  s3://my-backups/redis/
```

#### AOF (Append-Only File)

```conf theme={null}
## redis.conf
appendonly yes
appendfilename "appendonly.aof"
appendfsync everysec  # Balance durability/performance

## Rewrite AOF when it grows 100% and is at least 64mb
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
```

#### Redis Replication

```yaml theme={null}
## Redis Sentinel for HA
apiVersion: v1
kind: ConfigMap
metadata:
  name: redis-sentinel
data:
  sentinel.conf: |
    sentinel monitor mymaster redis-master 6379 2
    sentinel down-after-milliseconds mymaster 5000
    sentinel parallel-syncs mymaster 1
    sentinel failover-timeout mymaster 10000
```

### Kubernetes Resource Backup

#### Velero

**Install Velero**:

```bash theme={null}
## Install CLI
brew install velero

## Install in cluster (AWS)
velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.9.0 \
  --bucket my-velero-backups \
  --backup-location-config region=us-east-1 \
  --snapshot-location-config region=us-east-1 \
  --secret-file ./credentials-velero

## Verify installation
velero version
```

**Create Backup Schedule**:

```yaml theme={null}
apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: langgraph-daily-backup
  namespace: velero
spec:
  schedule: "0 2 * * *"  # Daily at 2 AM
  template:
    includedNamespaces:
    - mcp-server-langgraph
    excludedResources:
    - events
    - events.events.k8s.io
    storageLocation: default
    ttl: 720h0m0s  # 30 days
```

**Manual Backup**:

```bash theme={null}
## Backup entire namespace
velero backup create langgraph-backup \
  --include-namespaces mcp-server-langgraph

## Backup specific resources
velero backup create langgraph-deployments \
  --include-namespaces mcp-server-langgraph \
  --include-resources deployments,services,configmaps,secrets

## Check backup status
velero backup get
velero backup describe langgraph-backup

## Download backup
velero backup download langgraph-backup
```

#### kubectl Export

```bash theme={null}
#!/bin/bash
## export-k8s-resources.sh

NAMESPACE="mcp-server-langgraph"
BACKUP_DIR="/backups/k8s/$(date +%Y%m%d)"
mkdir -p ${BACKUP_DIR}

## Export all resource types
for resource in deployment service configmap secret ingress pvc; do
  kubectl get ${resource} -n ${NAMESPACE} -o yaml > \
    "${BACKUP_DIR}/${resource}.yaml"
done

## Create archive
tar -czf "${BACKUP_DIR}.tar.gz" -C /backups/k8s "$(date +%Y%m%d)"

## Upload to S3
aws s3 cp "${BACKUP_DIR}.tar.gz" s3://my-backups/k8s/
```

### Secrets Backup

#### Infisical Backup

```python theme={null}
## backup-secrets.py
from infisical import InfisicalClient
import json
from datetime import datetime

client = InfisicalClient(
    client_id=os.getenv("INFISICAL_CLIENT_ID"),
    client_secret=os.getenv("INFISICAL_CLIENT_SECRET")
)

## Export all secrets
secrets = client.get_all_secrets(
    project_id=os.getenv("INFISICAL_PROJECT_ID"),
    environment="production"
)

## Save to file
backup_file = f"secrets-backup-{datetime.now().strftime('%Y%m%d_%H%M%S')}.json"
with open(backup_file, 'w') as f:
    json.dump([{
        'name': s.secret_name,
        'value': s.secret_value
    } for s in secrets], f, indent=2)

## Encrypt backup
os.system(f"gpg --encrypt --recipient admin@company.com {backup_file}")

## Upload encrypted backup
os.system(f"aws s3 cp {backup_file}.gpg s3://my-backups/secrets/")
```

#### Kubernetes Secrets

```bash theme={null}
## Export all secrets (base64 encoded)
kubectl get secrets -n mcp-server-langgraph -o yaml > secrets-backup.yaml

## Encrypt with SOPS
sops --encrypt --kms "arn:aws:kms:us-east-1:123456789:key/abc-123" \
  secrets-backup.yaml > secrets-backup.enc.yaml

## Store encrypted file in Git
git add secrets-backup.enc.yaml
git commit -m "Backup secrets $(date +%Y-%m-%d)"
```

### Restore Procedures

#### Full System Restore

<Steps>
  <Step title="Restore Infrastructure">
    ```bash theme={null}
    # Deploy Kubernetes cluster (if needed)
    eksctl create cluster -f cluster-config.yaml

    # Install Helm charts
    helm install openfga ./charts/openfga
    helm install redis ./charts/redis
    helm install keycloak ./charts/keycloak
    ```
  </Step>

  <Step title="Restore Databases">
    ```bash theme={null}
    # Restore PostgreSQL
    pg_restore -U keycloak -h postgres-keycloak \
      -d keycloak -c -v \
      /backups/keycloak_20251012_020000.dump

    pg_restore -U openfga -h postgres-openfga \
      -d openfga -c -v \
      /backups/openfga_20251012_020000.dump
    ```
  </Step>

  <Step title="Restore Redis">
    ```bash theme={null}
    # Stop Redis
    kubectl scale statefulset redis-master --replicas=0

    # Copy RDB file
    kubectl cp /backups/dump_20251012_020000.rdb \
      redis-master-0:/data/dump.rdb

    # Start Redis
    kubectl scale statefulset redis-master --replicas=1
    ```
  </Step>

  <Step title="Restore Kubernetes Resources">
    ```bash theme={null}
    # Using Velero
    velero restore create langgraph-restore \
      --from-backup langgraph-backup-20251012

    # Monitor restore
    velero restore get
    velero restore describe langgraph-restore --details
    ```
  </Step>

  <Step title="Restore Secrets">
    ```bash theme={null}
    # Decrypt and apply
    sops --decrypt secrets-backup.enc.yaml | kubectl apply -f -

    # Or restore to Infisical
    python restore-secrets.py secrets-backup-20251012.json
    ```
  </Step>

  <Step title="Verify System">
    ```bash theme={null}
    # Check all pods running
    kubectl get pods -n mcp-server-langgraph

    # Test authentication
    curl -X POST http://api.yourdomain.com/auth/login \
      -d '{"username":"admin","password":"***"}'

    # Test API
    curl http://api.yourdomain.com/health

    # Verify data integrity
    psql -U keycloak -d keycloak -c "SELECT COUNT(*) FROM users;"
    ```
  </Step>
</Steps>

#### Partial Restore

**Restore Single Database**:

```bash theme={null}
## Create new database
createdb -U postgres keycloak_restored

## Restore from backup
pg_restore -U keycloak \
  -d keycloak_restored \
  -v /backups/keycloak_20251012.dump

## Verify
psql -U keycloak -d keycloak_restored -c "\dt"
```

**Restore Specific Kubernetes Resource**:

```bash theme={null}
## Restore from backup file
kubectl apply -f /backups/k8s/20251012/deployment.yaml

## Or using Velero
velero restore create deployment-restore \
  --from-backup langgraph-backup \
  --include-resources deployments \
  --selector app=mcp-server-langgraph
```

### Disaster Recovery Testing

#### DR Drill Procedure

**Monthly DR Test**:

```bash theme={null}
#!/bin/bash
## dr-drill.sh

set -e

echo "=== Starting DR Drill ==="

## 1. Create test namespace
kubectl create namespace langgraph-dr-test

## 2. Restore latest backup
LATEST_BACKUP=$(velero backup get --output json | jq -r '.items[0].metadata.name')
velero restore create dr-test-restore \
  --from-backup ${LATEST_BACKUP} \
  --namespace-mappings mcp-server-langgraph:langgraph-dr-test

## 3. Wait for restore
velero restore wait dr-test-restore

## 4. Run smoke tests
pytest tests/smoke/ --namespace=langgraph-dr-test

## 5. Measure RTO
START_TIME=$(date -d "$(velero backup get ${LATEST_BACKUP} -o json | jq -r '.status.completionTimestamp')" +%s)
END_TIME=$(date +%s)
RTO=$((END_TIME - START_TIME))
echo "RTO: ${RTO} seconds"

## 6. Cleanup
kubectl delete namespace langgraph-dr-test
velero restore delete dr-test-restore

echo "=== DR Drill Complete ==="
echo "RTO: ${RTO}s (Target: 3600s)"
```

#### Automated Testing

```yaml theme={null}
## .github/workflows/dr-test.yml
name: DR Testing
on:
  schedule:
    - cron: '0 0 1 * *'  # Monthly

jobs:
  dr-test:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout
      uses: actions/checkout@v3

    - name: Setup kubectl
      uses: azure/setup-kubectl@v3

    - name: Run DR drill
      run: ./scripts/dr-drill.sh

    - name: Report results
      if: failure()
      uses: 8398a7/action-slack@v3
      with:
        status: ${{ job.status }}
        text: 'DR test failed!'
        webhook_url: ${{ secrets.SLACK_WEBHOOK }}
```

### Multi-Region Failover

#### Active-Passive Setup

```yaml theme={null}
## Primary region (us-east-1)
primary:
  region: us-east-1
  postgres:
    replication: async
    replica_regions:
      - us-west-2
  redis:
    replication: async

## Secondary region (us-west-2)
secondary:
  region: us-west-2
  postgres:
    mode: read-replica
  redis:
    mode: standby
```

**Failover Procedure**:

```
## 1. Promote secondary database
aws rds promote-read-replica \
  --db-instance-identifier keycloak-db-west

## 2. Update DNS
aws route53 change-resource-record-sets \
  --hosted-zone-id Z123 \
  --change-batch file://failover-dns.json

## 3. Scale up secondary region
kubectl config use-context us-west-2
kubectl scale deployment mcp-server-langgraph --replicas=10

## 4. Verify traffic shift
watch -n 5 'kubectl top pods'
```

#### Active-Active Setup

```yaml theme={null}
## Global load balancer
apiVersion: networking.gke.io/v1
kind: MultiClusterIngress
metadata:
  name: langgraph-global
spec:
  template:
    spec:
      backend:
        serviceName: mcp-server-langgraph
        servicePort: 80
      rules:
      - http:
          paths:
          - backend:
              serviceName: mcp-server-langgraph
              servicePort: 80
```

**Database Synchronization**:

```yaml theme={null}
## CockroachDB for multi-region
cockroachdb:
  replicas: 9
  regions:
    - us-east-1 (3 nodes)
    - us-west-2 (3 nodes)
    - eu-west-1 (3 nodes)
  survivability: region
```

### Monitoring & Alerting

#### Backup Monitoring

```promql theme={null}
## Backup age
time() - backup_last_success_timestamp_seconds > 86400

## Backup failures
rate(backup_failures_total[1h]) > 0

## Backup size
backup_size_bytes{backup_type="postgres"} < 1e9
```

**Alerts**:

```yaml theme={null}
groups:
- name: backup_alerts
  rules:
  - alert: BackupFailed
    expr: backup_last_success_timestamp_seconds < time() - 86400
    for: 1h
    annotations:
      summary: "Backup has not succeeded in 24h"

  - alert: BackupMissing
    expr: absent(backup_last_success_timestamp_seconds)
    for: 30m
    annotations:
      summary: "Backup metrics missing"
```

#### Recovery Testing Dashboard

```json theme={null}
{
  "dashboard": {
    "title": "DR Metrics",
    "panels": [
      {
        "title": "Last Successful Backup",
        "targets": [{
          "expr": "time() - backup_last_success_timestamp_seconds"
        }]
      },
      {
        "title": "RTO Trend",
        "targets": [{
          "expr": "dr_test_rto_seconds"
        }]
      },
      {
        "title": "RPO Actual",
        "targets": [{
          "expr": "time() - backup_last_success_timestamp_seconds"
        }]
      }
    ]
  }
}
```

### Best Practices

<AccordionGroup>
  <Accordion title="Test Restores Regularly" icon="vial">
    **Never trust untested backups!**

    * Monthly DR drills
    * Quarterly full restore tests
    * Document restore procedures
    * Measure actual RTO/RPO
    * Automate testing where possible
  </Accordion>

  <Accordion title="3-2-1 Backup Rule" icon="copy">
    Maintain:

    * **3** copies of data
    * **2** different storage types
    * **1** off-site backup

    ```bash theme={null}
    # Example implementation
    Primary: Live database
    Copy 1: Daily snapshots (same region)
    Copy 2: S3 backups (different region)
    Copy 3: Glacier archive (off-site)
    ```
  </Accordion>

  <Accordion title="Encrypt Backups" icon="lock">
    Always encrypt sensitive data:

    ```bash theme={null}
    # Encrypt with GPG
    gpg --encrypt --recipient backup@company.com backup.sql

    # Or use AWS KMS
    aws s3 cp backup.sql s3://backups/ \
      --sse aws:kms \
      --sse-kms-key-id arn:aws:kms:...
    ```
  </Accordion>

  <Accordion title="Document Everything" icon="book">
    Maintain runbooks for:

    * Backup procedures
    * Restore procedures
    * Failover procedures
    * DR contacts and escalation
    * Test results and improvements
  </Accordion>

  <Accordion title="Automate Recovery" icon="robot">
    Infrastructure as Code for DR:

    ```yaml theme={null}
    # Terraform/Pulumi
    - Database clusters
    - Kubernetes clusters
    - Load balancers
    - DNS configuration

    # Ansible/Helm
    - Application deployment
    - Configuration management
    - Secret restoration
    ```
  </Accordion>
</AccordionGroup>

### Compliance & Audit

#### Backup Retention Policies

```yaml theme={null}
## retention-policy.yaml
policies:
  daily_backups:
    retention_days: 7
    backup_time: "02:00 UTC"

  weekly_backups:
    retention_days: 28
    backup_day: "Sunday"

  monthly_backups:
    retention_days: 365
    backup_day: "1st"

  yearly_backups:
    retention_days: 2555  # 7 years
    backup_day: "Jan 1"
```

#### Audit Trail

```bash theme={null}
## Log all backup/restore operations
logger -t backup "Started PostgreSQL backup"

## Send to centralized logging
echo "$(date): Backup completed" | \
  aws logs put-log-events \
    --log-group-name /aws/backup \
    --log-stream-name postgres
```

### Next Steps

<CardGroup cols={2}>
  <Card title="Kubernetes Deployment" icon="dharmachakra" href="/deployment/kubernetes">
    Deploy production infrastructure
  </Card>

  <Card title="Monitoring" icon="chart-line" href="/guides/observability">
    Set up observability
  </Card>

  <Card title="Security Best Practices" icon="shield" href="/security/best-practices">
    Secure your backups
  </Card>

  <Card title="Production Checklist" icon="clipboard-check" href="/deployment/production-checklist">
    DR requirements
  </Card>
</CardGroup>

***

<Check>
  **Resilient Infrastructure**: Comprehensive DR ensures business continuity and data protection!
</Check>