Secret Rotation - MCP Server with LangGraph

Overview

Regular secret rotation is a critical security practice that limits the window of exposure if credentials are compromised. This guide covers automated rotation strategies for API keys, passwords, tokens, and certificates.

Improper secret rotation can cause service disruptions. Always test rotation procedures in non-production environments first.

Why Rotate Secrets?

Security

Limit exposure window
Mitigate credential compromise
Comply with security policies
Reduce blast radius of breaches

Compliance

GDPR requirements
SOC 2 controls
PCI DSS standards
Industry best practices

Operational

Automated processes
Reduce manual errors
Audit trail
Policy enforcement

Risk Management

Defense in depth
Insider threat mitigation
Supply chain security
Zero trust principles

Rotation Schedule

Secret Type	Rotation Frequency	Automation Level
API Keys (LLM)	90 days	Automated
Database Passwords	60 days	Automated
JWT Signing Keys	30 days	Automated
Service Account Keys	90 days	Semi-automated
TLS Certificates	Before expiry	Automated
Encryption Keys	180 days	Manual + Automated
User Passwords	90 days	User-initiated

Infisical Secret Rotation

Automatic Rotation

Infisical supports automatic secret rotation:

from infisical import InfisicalClient
import secrets
import string
from datetime import datetime, timedelta

client = InfisicalClient(
    client_id=os.getenv("INFISICAL_CLIENT_ID"),
    client_secret=os.getenv("INFISICAL_CLIENT_SECRET")
)

def generate_secure_secret(length: int = 64) -> str:
    """Generate cryptographically secure secret"""
    alphabet = string.ascii_letters + string.digits + "!@#$%^&*"
    return ''.join(secrets.choice(alphabet) for _ in range(length))

async def rotate_api_key(secret_name: str, project_id: str, environment: str):
    """Rotate API key in Infisical"""

    # Generate new key
    new_key = generate_secure_secret(64)

    # Update in Infisical
    client.update_secret(
        secret_name=secret_name,
        secret_value=new_key,
        project_id=project_id,
        environment=environment
    )

    # Log rotation
    logger.info(
        "Secret rotated",
        secret_name=secret_name,
        rotated_at=datetime.utcnow().isoformat()
    )

    return new_key

## Schedule rotation
import schedule

schedule.every(90).days.do(
    rotate_api_key,
    secret_name="ANTHROPIC_API_KEY",
    project_id=os.getenv("INFISICAL_PROJECT_ID"),
    environment="production"
)

Zero-Downtime Rotation

Implement graceful rotation without service interruption:

async def zero_downtime_rotation(secret_name: str):
    """Rotate secret with zero downtime"""

    # Step 1: Create new secret with version suffix
    new_secret = generate_secure_secret()
    client.create_secret(
        secret_name=f"{secret_name}_V2",
        secret_value=new_secret,
        project_id=project_id,
        environment="production"
    )

    # Step 2: Deploy application update to use new secret
    # (Application should try V2 first, fallback to V1)
    logger.info("Deploying with new secret version")
    await deploy_application()

    # Step 3: Wait for rollout completion
    await asyncio.sleep(300)  # 5 minutes

    # Step 4: Verify new secret is working
    health_check = await verify_new_secret()
    if not health_check:
        logger.error("New secret verification failed, rolling back")
        await rollback_deployment()
        return False

    # Step 5: Delete old secret
    client.delete_secret(
        secret_name=f"{secret_name}_V1",
        project_id=project_id,
        environment="production"
    )

    # Step 6: Rename V2 to primary
    # (This happens in next deployment)

    logger.info("Zero-downtime rotation completed successfully")
    return True

Database Password Rotation

PostgreSQL Password Rotation

import asyncpg
from infisical import InfisicalClient

async def rotate_postgres_password(db_name: str):
    """Rotate PostgreSQL password"""

    # Generate new password
    new_password = generate_secure_secret(32)

    # Connect as superuser
    conn = await asyncpg.connect(
        host="postgres",
        user="postgres",
        password=os.getenv("POSTGRES_ADMIN_PASSWORD"),
        database="postgres"
    )

    try:
        # Update password
        await conn.execute(
            f"ALTER USER {db_name} WITH PASSWORD '{new_password}'"
        )

        # Update in Infisical
        client.update_secret(
            secret_name=f"{db_name.upper()}_PASSWORD",
            secret_value=new_password,
            project_id=project_id,
            environment="production"
        )

        # Restart application to pick up new password
        logger.info(f"Password rotated for user: {db_name}")
        await restart_application_pods()

        # Verify connectivity
        test_conn = await asyncpg.connect(
            host="postgres",
            user=db_name,
            password=new_password,
            database=db_name
        )
        await test_conn.close()

        logger.info("Password rotation verified successfully")

    finally:
        await conn.close()

## Schedule rotation
schedule.every(60).days.do(rotate_postgres_password, db_name="keycloak")
schedule.every(60).days.do(rotate_postgres_password, db_name="openfga")

Redis Password Rotation

import redis.asyncio as redis

async def rotate_redis_password():
    """Rotate Redis password"""

    # Generate new password
    new_password = generate_secure_secret(32)

    # Connect to Redis
    r = await redis.Redis(
        host="redis-master",
        port=6379,
        password=os.getenv("REDIS_PASSWORD")
    )

    try:
        # Set new password (keeps old one active)
        await r.config_set("requirepass", new_password)

        # Update in Infisical
        client.update_secret(
            secret_name="REDIS_PASSWORD",
            secret_value=new_password,
            project_id=project_id,
            environment="production"
        )

        # Restart application pods
        logger.info("Redis password rotated")
        await restart_application_pods()

        # Verify with new password
        test_r = await redis.Redis(
            host="redis-master",
            port=6379,
            password=new_password
        )
        await test_r.ping()
        await test_r.close()

        logger.info("Redis password rotation verified")

    finally:
        await r.close()

JWT Signing Key Rotation

RSA Key Pair Rotation

from cryptography.hazmat.primitives.asymmetric import rsa
from cryptography.hazmat.primitives import serialization
from datetime import datetime

async def rotate_jwt_signing_key():
    """Rotate JWT RSA signing key"""

    # Generate new RSA key pair
    private_key = rsa.generate_private_key(
        public_exponent=65537,
        key_size=4096
    )

    # Serialize private key
    private_pem = private_key.private_bytes(
        encoding=serialization.Encoding.PEM,
        format=serialization.PrivateFormat.PKCS8,
        encryption_algorithm=serialization.NoEncryption()
    ).decode('utf-8')

    # Serialize public key
    public_key = private_key.public_key()
    public_pem = public_key.public_bytes(
        encoding=serialization.Encoding.PEM,
        format=serialization.PublicFormat.SubjectPublicKeyInfo
    ).decode('utf-8')

    # Key ID (timestamp-based)
    key_id = f"key-{datetime.utcnow().strftime('%Y%m')}"

    # Store new keys in Infisical
    client.create_secret(
        secret_name=f"JWT_PRIVATE_KEY_{key_id}",
        secret_value=private_pem,
        project_id=project_id,
        environment="production"
    )

    client.create_secret(
        secret_name=f"JWT_PUBLIC_KEY_{key_id}",
        secret_value=public_pem,
        project_id=project_id,
        environment="production"
    )

    # Update current key reference
    client.update_secret(
        secret_name="JWT_CURRENT_KEY_ID",
        secret_value=key_id,
        project_id=project_id,
        environment="production"
    )

    # Keep old key for 24h for token validation
    logger.info(f"JWT signing key rotated: {key_id}")

    # Schedule old key deletion
    schedule_key_deletion(old_key_id, delay_hours=24)

## Monthly rotation
schedule.every(30).days.do(rotate_jwt_signing_key)

JWKS Endpoint Update

from fastapi import FastAPI

app = FastAPI()

@app.get("/.well-known/jwks.json")
async def get_jwks():
    """Serve JWKS with multiple keys during rotation"""

    # Get current and previous key
    current_key_id = client.get_secret(
        secret_name="JWT_CURRENT_KEY_ID",
        project_id=project_id,
        environment="production"
    ).secret_value

    current_public = client.get_secret(
        secret_name=f"JWT_PUBLIC_KEY_{current_key_id}",
        project_id=project_id,
        environment="production"
    ).secret_value

    # Build JWKS
    jwks = {
        "keys": [
            {
                "kty": "RSA",
                "use": "sig",
                "kid": current_key_id,
                "n": extract_n_from_public_key(current_public),
                "e": "AQAB"
            }
        ]
    }

    # Include previous key if still valid
    try:
        previous_key_id = get_previous_key_id(current_key_id)
        previous_public = client.get_secret(
            secret_name=f"JWT_PUBLIC_KEY_{previous_key_id}",
            project_id=project_id,
            environment="production"
        ).secret_value

        jwks["keys"].append({
            "kty": "RSA",
            "use": "sig",
            "kid": previous_key_id,
            "n": extract_n_from_public_key(previous_public),
            "e": "AQAB"
        })
    except:
        pass  # No previous key

    return jwks

API Key Rotation

LLM Provider Keys

async def rotate_llm_api_key(provider: str):
    """Rotate LLM provider API key"""

    # This requires manual regeneration from provider
    # 1. Generate new key from provider console
    # 2. Update in Infisical
    # 3. Test new key
    # 4. Deploy
    # 5. Revoke old key

    logger.warning(
        f"Manual intervention required for {provider} API key rotation",
        provider=provider,
        rotation_due=datetime.utcnow().isoformat()
    )

    # Send notification
    await send_notification(
        channel="security",
        message=f"Action required: Rotate {provider} API key",
        severity="warning"
    )

## Monitor key age
async def check_key_age():
    """Check if keys need rotation"""

    secrets_to_check = [
        ("ANTHROPIC_API_KEY", 90),
        ("OPENAI_API_KEY", 90),
        ("GOOGLE_API_KEY", 90)
    ]

    for secret_name, max_age_days in secrets_to_check:
        # Get secret metadata
        secret = client.get_secret(
            secret_name=secret_name,
            project_id=project_id,
            environment="production"
        )

        # Check age
        created_at = datetime.fromisoformat(secret.created_at)
        age_days = (datetime.utcnow() - created_at).days

        if age_days >= max_age_days:
            await rotate_llm_api_key(secret_name.split("_")[0].lower())

## Daily check
schedule.every().day.at("09:00").do(check_key_age)

TLS Certificate Rotation

Cert-Manager Auto-Renewal

## Certificate with auto-renewal
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: mcp-server-langgraph-tls
  namespace: mcp-server-langgraph
spec:
  secretName: mcp-server-langgraph-tls
  duration: 2160h  # 90 days
  renewBefore: 360h  # 15 days before expiry

  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer

  dnsNames:
  - api.yourdomain.com
  - "*.yourdomain.com"

Manual Certificate Rotation

from cryptography import x509
from cryptography.hazmat.primitives import hashes
import subprocess

async def rotate_tls_certificate():
    """Rotate TLS certificate"""

    # Generate new private key
    subprocess.run([
        "openssl", "genrsa",
        "-out", "/tmp/new-key.pem",
        "4096"
    ])

    # Generate CSR
    subprocess.run([
        "openssl", "req",
        "-new",
        "-key", "/tmp/new-key.pem",
        "-out", "/tmp/new-csr.pem",
        "-subj", "/CN=api.yourdomain.com"
    ])

    # Get certificate from CA (automated or manual)
    # ...

    # Update Kubernetes secret
    subprocess.run([
        "kubectl", "create", "secret", "tls",
        "mcp-server-langgraph-tls",
        "--cert=/tmp/new-cert.pem",
        "--key=/tmp/new-key.pem",
        "--dry-run=client",
        "-o", "yaml",
        "|",
        "kubectl", "apply", "-f", "-"
    ])

    # Reload ingress
    subprocess.run([
        "kubectl", "rollout", "restart",
        "deployment/nginx-ingress-controller"
    ])

    logger.info("TLS certificate rotated successfully")

Kubernetes Secret Rotation

Using External Secrets Operator

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: langgraph-secrets
  namespace: mcp-server-langgraph
spec:
  refreshInterval: 1h  # Check for updates hourly

  secretStoreRef:
    name: infisical
    kind: SecretStore

  target:
    name: mcp-server-langgraph-secrets
    creationPolicy: Owner

  data:
  - secretKey: ANTHROPIC_API_KEY
    remoteRef:
      key: ANTHROPIC_API_KEY

  - secretKey: REDIS_PASSWORD
    remoteRef:
      key: REDIS_PASSWORD

Restart Pods After Rotation

from kubernetes import client, config

async def restart_pods_after_rotation(namespace: str, deployment: str):
    """Restart pods to pick up new secrets"""

    # Load k8s config
    config.load_incluster_config()
    apps_v1 = client.AppsV1Api()

    # Get deployment
    deployment_obj = apps_v1.read_namespaced_deployment(
        name=deployment,
        namespace=namespace
    )

    # Add annotation to trigger rollout
    if deployment_obj.spec.template.metadata.annotations is None:
        deployment_obj.spec.template.metadata.annotations = {}

    deployment_obj.spec.template.metadata.annotations[
        "secret-rotation"
    ] = datetime.utcnow().isoformat()

    # Update deployment
    apps_v1.patch_namespaced_deployment(
        name=deployment,
        namespace=namespace,
        body=deployment_obj
    )

    logger.info(f"Triggered rollout for {deployment}")

## Use after secret rotation
await restart_pods_after_rotation(
    namespace="mcp-server-langgraph",
    deployment="mcp-server-langgraph"
)

Rotation Workflow Automation

Complete Rotation Pipeline

from dataclasses import dataclass
from enum import Enum
from typing import List, Callable

class RotationStatus(Enum):
    PENDING = "pending"
    IN_PROGRESS = "in_progress"
    COMPLETED = "completed"
    FAILED = "failed"
    ROLLED_BACK = "rolled_back"

@dataclass
class RotationJob:
    secret_name: str
    rotation_func: Callable
    verification_func: Callable
    rollback_func: Callable
    status: RotationStatus = RotationStatus.PENDING

async def execute_rotation_pipeline(jobs: List[RotationJob]):
    """Execute rotation pipeline with verification and rollback"""

    for job in jobs:
        try:
            logger.info(f"Starting rotation: {job.secret_name}")
            job.status = RotationStatus.IN_PROGRESS

            # Execute rotation
            await job.rotation_func()

            # Wait for propagation
            await asyncio.sleep(30)

            # Verify
            logger.info(f"Verifying rotation: {job.secret_name}")
            if not await job.verification_func():
                raise Exception("Verification failed")

            job.status = RotationStatus.COMPLETED
            logger.info(f"Rotation completed: {job.secret_name}")

        except Exception as e:
            logger.error(
                f"Rotation failed: {job.secret_name}",
                error=str(e)
            )
            job.status = RotationStatus.FAILED

            # Attempt rollback
            try:
                logger.info(f"Rolling back: {job.secret_name}")
                await job.rollback_func()
                job.status = RotationStatus.ROLLED_BACK
            except Exception as rollback_error:
                logger.error(
                    f"Rollback failed: {job.secret_name}",
                    error=str(rollback_error)
                )

            # Stop pipeline on failure
            break

## Define rotation jobs
jobs = [
    RotationJob(
        secret_name="REDIS_PASSWORD",
        rotation_func=rotate_redis_password,
        verification_func=verify_redis_connection,
        rollback_func=rollback_redis_password
    ),
    RotationJob(
        secret_name="JWT_SIGNING_KEY",
        rotation_func=rotate_jwt_signing_key,
        verification_func=verify_jwt_signing,
        rollback_func=rollback_jwt_key
    )
]

## Execute
await execute_rotation_pipeline(jobs)

Monitoring & Alerts

Track Rotation Status

from prometheus_client import Gauge, Counter

secret_age_days = Gauge(
    'secret_age_days',
    'Age of secret in days',
    ['secret_name']
)

rotation_success = Counter(
    'secret_rotation_success_total',
    'Successful secret rotations',
    ['secret_name']
)

rotation_failures = Counter(
    'secret_rotation_failures_total',
    'Failed secret rotations',
    ['secret_name']
)

## Update metrics
def update_secret_metrics():
    secrets = [
        "ANTHROPIC_API_KEY",
        "REDIS_PASSWORD",
        "JWT_SIGNING_KEY"
    ]

    for secret_name in secrets:
        secret = client.get_secret(
            secret_name=secret_name,
            project_id=project_id,
            environment="production"
        )

        created_at = datetime.fromisoformat(secret.created_at)
        age_days = (datetime.utcnow() - created_at).days

        secret_age_days.labels(secret_name=secret_name).set(age_days)

schedule.every().hour.do(update_secret_metrics)

Rotation Alerts

## prometheus-alerts.yaml
groups:
- name: secret_rotation
  rules:
  - alert: SecretRotationOverdue
    expr: secret_age_days > 100
    for: 24h
    annotations:
      summary: "Secret rotation overdue"
      description: "{{ $labels.secret_name }} is {{ $value }} days old"

  - alert: SecretRotationFailed
    expr: increase(secret_rotation_failures_total[1h]) > 0
    annotations:
      summary: "Secret rotation failed"
      description: "Failed to rotate {{ $labels.secret_name }}"

Best Practices

Test Rotation Procedures

Always test in non-production first:

# Test in staging
python rotate_secrets.py --environment=staging --dry-run

# Verify
python verify_secrets.py --environment=staging

# Apply to production
python rotate_secrets.py --environment=production

Maintain Grace Period

Keep old secrets valid during rotation:

Database passwords: Support both old and new for 5 minutes
JWT keys: Keep old key for token validation (24 hours)
API keys: Overlap period of 1 hour

Automate Where Possible

Automate rotation for:

Database passwords
JWT signing keys
TLS certificates
Internal service credentials

Require manual approval for:

External API keys
Root credentials
Encryption keys

Document Procedures

Maintain runbooks:

Rotation procedures
Rollback steps
Verification checks
Emergency contacts
Incident response

Next Steps

Infisical Setup

Secret management platform

Security Best Practices

Security hardening guide

Disaster Recovery

Backup and restore

Production Checklist

Pre-deployment security

Secret Rotation Ready: Automated, secure credential rotation for enhanced security!

Quick Start

LLM Providers

Production

Migration

Authorization

Enterprise Identity & Access

Secrets Management

Sessions & Storage

Observability

​Overview

​Why Rotate Secrets?

Security

Compliance

Operational

Risk Management

​Rotation Schedule

​Infisical Secret Rotation

​Automatic Rotation

​Zero-Downtime Rotation

​Database Password Rotation

​PostgreSQL Password Rotation

​Redis Password Rotation

​JWT Signing Key Rotation

​RSA Key Pair Rotation

​JWKS Endpoint Update

​API Key Rotation

​LLM Provider Keys

​TLS Certificate Rotation

​Cert-Manager Auto-Renewal

​Manual Certificate Rotation

​Kubernetes Secret Rotation

​Using External Secrets Operator

​Restart Pods After Rotation

​Rotation Workflow Automation

​Complete Rotation Pipeline

​Monitoring & Alerts

​Track Rotation Status

​Rotation Alerts

​Best Practices

​Next Steps

Infisical Setup

Security Best Practices

Disaster Recovery

Production Checklist

Overview

Why Rotate Secrets?

Rotation Schedule

Infisical Secret Rotation

Automatic Rotation

Zero-Downtime Rotation

Database Password Rotation

PostgreSQL Password Rotation

Redis Password Rotation

JWT Signing Key Rotation

RSA Key Pair Rotation

JWKS Endpoint Update

API Key Rotation

LLM Provider Keys

TLS Certificate Rotation

Cert-Manager Auto-Renewal

Manual Certificate Rotation

Kubernetes Secret Rotation

Using External Secrets Operator

Restart Pods After Rotation

Rotation Workflow Automation

Complete Rotation Pipeline

Monitoring & Alerts

Track Rotation Status

Rotation Alerts

Best Practices

Next Steps