> ## Documentation Index
> Fetch the complete documentation index at: https://mcp-server-langgraph.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Service Mesh (Anthos)

> Deploy Anthos Service Mesh on GKE for service-to-service mTLS, traffic management, and observability

## Overview

**Anthos Service Mesh** (managed Istio) provides secure service-to-service communication, advanced traffic management, and deep observability for microservices on GKE. Fully managed by Google with automatic upgrades.

<CardGroup cols={2}>
  <Card title="Mutual TLS" icon="lock">
    Automatic encryption between services
  </Card>

  <Card title="Traffic Control" icon="route">
    Canary deployments, A/B testing, circuit breaking
  </Card>

  <Card title="Observability" icon="chart-line">
    Service topology, latency, error rates
  </Card>

  <Card title="Policy Enforcement" icon="shield-check">
    Fine-grained authorization, rate limiting
  </Card>
</CardGroup>

***

## Why Service Mesh?

<AccordionGroup>
  <Accordion title="Zero-Trust Networking">
    **Challenge**: By default, pods can talk to any other pod

    **Solution**: Service mesh enforces mTLS + authorization policies

    **Implementation**:

    ```yaml theme={null}
    apiVersion: security.istio.io/v1beta1
    kind: PeerAuthentication
    metadata:
      name: default
      namespace: production-mcp-server-langgraph
    spec:
      mtls:
        mode: STRICT  # All traffic must be mTLS
    ```

    **Result**: Encrypted, authenticated communication
  </Accordion>

  <Accordion title="Advanced Deployments">
    **Use cases**:

    * Canary releases (10% traffic to v2)
    * A/B testing (iOS users → v2)
    * Blue-green deployments
    * Circuit breaking (prevent cascading failures)

    **Without mesh**: Complex custom code

    **With mesh**: Declarative traffic rules
  </Accordion>

  <Accordion title="Service-Level Observability">
    **Built-in metrics**:

    * Request rate (QPS per service)
    * P50/P95/P99 latency
    * Success rate (% 2xx responses)
    * Service dependency graph

    **Without mesh**: Instrumentation code in every service

    **With mesh**: Automatic sidecar collection
  </Accordion>

  <Accordion title="Multi-Cluster Mesh">
    **Scenario**: Services across dev, staging, prod clusters

    **Capability**: Single mesh spanning clusters

    **Benefit**: Consistent policies, cross-cluster service discovery
  </Accordion>
</AccordionGroup>

***

## Architecture

```mermaid theme={null}
flowchart TB
    subgraph "GKE Cluster (production-mcp-server-langgraph-gke)"
        subgraph "Namespace: istio-system"
            CP[Istiod Control Plane<br/>Managed by Google]
        end

        subgraph "Namespace: mcp-production"
            A[mcp-server Pod]
            A1[Envoy Sidecar]
            B[postgres-proxy Pod]
            B1[Envoy Sidecar]
            C[redis-proxy Pod]
            C1[Envoy Sidecar]
        end

        A --> A1
        B --> B1
        C --> C1

        A1 -.mTLS.-> B1
        A1 -.mTLS.-> C1

        CP -.Config.-> A1
        CP -.Config.-> B1
        CP -.Config.-> C1
    end

    subgraph "Cloud Monitoring"
        M[Service Metrics]
        T[Service Topology]
    end

    A1 -.Metrics.-> M
    B1 -.Metrics.-> M
    C1 -.Metrics.-> M

    A1 -.Traces.-> T

    %% ColorBrewer2 Set3 palette - each component type uniquely colored
    classDef controlPlaneStyle fill:#bc80bd,stroke:#8e44ad,stroke-width:2px,color:#fff
    classDef appPodStyle fill:#b3de69,stroke:#7cb342,stroke-width:2px,color:#333
    classDef sidecarStyle fill:#fdb462,stroke:#e67e22,stroke-width:2px,color:#333
    classDef monitoringStyle fill:#fccde5,stroke:#ec7ab8,stroke-width:2px,color:#333

    class CP controlPlaneStyle
    class A,B,C appPodStyle
    class A1,B1,C1 sidecarStyle
    class M,T monitoringStyle
```

**Components**:

* **Istiod**: Control plane (managed by Google, auto-upgraded)
* **Envoy sidecars**: Injected into each pod, handle traffic
* **Telemetry**: Metrics sent to Cloud Monitoring

***

## Quick Setup (30 minutes)

<Steps>
  <Step title="Enable APIs & Fleet Registration">
    ```bash theme={null}
    ./deployments/service-mesh/anthos/setup-anthos-service-mesh.sh \
      PROJECT_ID production-mcp-server-langgraph-gke us-central1
    ```

    **What it does**:

    * Enables Anthos Service Mesh APIs
    * Registers cluster with GKE Fleet
    * Enables managed service mesh
    * Waits for control plane (\~10-15 min)
  </Step>

  <Step title="Verify Installation">
    ```bash theme={null}
    # Check mesh status
    gcloud container fleet mesh describe --project=PROJECT_ID

    # Should show:
    # state: ACTIVE
    # controlPlaneManagement: AUTOMATIC

    # Verify Istiod running
    kubectl get pods -n istio-system
    ```

    <Check>istiod pod should be Running</Check>
  </Step>

  <Step title="Enable Sidecar Injection">
    ```bash theme={null}
    # Label namespace for automatic injection
    kubectl label namespace production-mcp-server-langgraph istio-injection=enabled

    # Restart deployments to inject sidecars
    kubectl rollout restart deployment/production-mcp-server-langgraph \
      -n production-mcp-server-langgraph
    ```
  </Step>

  <Step title="Verify Sidecars Injected">
    ```bash theme={null}
    # Check pods have 2 containers (app + envoy)
    kubectl get pods -n production-mcp-server-langgraph

    # Should show:
    # NAME                                   READY   STATUS
    # production-mcp-server-langgraph-...    2/2     Running
    #                                        ^^^
    #                                    app + sidecar

    # Describe pod to see istio-proxy container
    kubectl describe pod POD_NAME -n production-mcp-server-langgraph | grep istio-proxy
    ```
  </Step>

  <Step title="Enable Strict mTLS">
    ```bash theme={null}
    kubectl apply -f - <<EOF
    apiVersion: security.istio.io/v1beta1
    kind: PeerAuthentication
    metadata:
      name: default
      namespace: production-mcp-server-langgraph
    spec:
      mtls:
        mode: STRICT
    EOF
    ```

    All traffic now encrypted with mTLS!
  </Step>

  <Step title="Verify mTLS">
    ```bash theme={null}
    # Check Kiali dashboard or use istioctl
    istioctl proxy-config secret -n production-mcp-server-langgraph POD_NAME

    # Should show TLS certificates
    ```
  </Step>
</Steps>

***

## Traffic Management

### Canary Deployment

Deploy new version to 10% of traffic:

<CodeGroup>
  ```yaml VirtualService (Traffic Split) theme={null}
  apiVersion: networking.istio.io/v1beta1
  kind: VirtualService
  metadata:
    name: mcp-server
    namespace: production-mcp-server-langgraph
  spec:
    hosts:
    - mcp-server
    http:
    - match:
      - headers:
          x-canary:
            exact: "true"
      route:
      - destination:
          host: mcp-server
          subset: v2
        weight: 100
    - route:
      - destination:
          host: mcp-server
          subset: v1
        weight: 90
      - destination:
          host: mcp-server
          subset: v2
        weight: 10  # 10% to canary
  ```

  ```yaml DestinationRule (Define Subsets) theme={null}
  apiVersion: networking.istio.io/v1beta1
  kind: DestinationRule
  metadata:
    name: mcp-server
    namespace: production-mcp-server-langgraph
  spec:
    host: mcp-server
    trafficPolicy:
      tls:
        mode: ISTIO_MUTUAL  # mTLS
    subsets:
    - name: v1
      labels:
        version: v1
    - name: v2
      labels:
        version: v2
  ```
</CodeGroup>

**Workflow**:

1. Deploy v2 with label `version: v2`
2. Apply VirtualService (10% → v2)
3. Monitor metrics for 30 minutes
4. If healthy, increase to 50%, then 100%
5. If unhealthy, revert to 0%

### Circuit Breaking

Prevent cascading failures:

```yaml theme={null}
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: postgres-proxy
  namespace: production-mcp-server-langgraph
spec:
  host: postgres-proxy
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
      http:
        http1MaxPendingRequests: 50
        http2MaxRequests: 100
        maxRequestsPerConnection: 2
    outlierDetection:
      consecutiveErrors: 5
      interval: 30s
      baseEjectionTime: 30s
      maxEjectionPercent: 50
```

**Behavior**: After 5 consecutive errors, eject pod for 30 seconds

### Retry Policy

```yaml theme={null}
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: mcp-server
spec:
  http:
  - route:
    - destination:
        host: mcp-server
    retries:
      attempts: 3
      perTryTimeout: 2s
      retryOn: 5xx,reset,connect-failure
```

***

## Security

### Strict mTLS

<Tabs>
  <Tab title="Cluster-Wide">
    ```yaml theme={null}
    apiVersion: security.istio.io/v1beta1
    kind: PeerAuthentication
    metadata:
      name: default
      namespace: istio-system
    spec:
      mtls:
        mode: STRICT
    ```

    Applies to all namespaces.
  </Tab>

  <Tab title="Namespace-Specific">
    ```yaml theme={null}
    apiVersion: security.istio.io/v1beta1
    kind: PeerAuthentication
    metadata:
      name: default
      namespace: production-mcp-server-langgraph
    spec:
      mtls:
        mode: STRICT
    ```

    Only `mcp-production` namespace.
  </Tab>

  <Tab title="Permissive (Migration)">
    ```yaml theme={null}
    spec:
      mtls:
        mode: PERMISSIVE  # Allow both mTLS and plaintext
    ```

    Use during migration, then switch to STRICT.
  </Tab>
</Tabs>

### Authorization Policies

**Deny-all by default**:

```yaml theme={null}
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: deny-all
  namespace: production-mcp-server-langgraph
spec: {}  # Empty = deny all
```

**Allow specific service**:

```yaml theme={null}
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: allow-mcp-server
  namespace: production-mcp-server-langgraph
spec:
  selector:
    matchLabels:
      app: postgres-proxy
  action: ALLOW
  rules:
  - from:
    - source:
        principals: ["cluster.local/ns/mcp-production/sa/mcp-server"]
    to:
    - operation:
        methods: ["GET", "POST"]
        paths: ["/api/*"]
```

**Result**: Only `mcp-server` SA can call `postgres-proxy`

***

## Observability

### Service Topology

**View in Google Cloud Console**:

```text theme={null}
Navigation → Anthos → Service Mesh → Topology
```

Shows:

* Service dependency graph
* Traffic flow between services
* Error rates per edge

### Metrics

<Tabs>
  <Tab title="Request Rate">
    ```promql theme={null}
    rate(istio_requests_total{
      destination_service_name="mcp-server",
      destination_workload_namespace="mcp-production"
    }[1m])
    ```
  </Tab>

  <Tab title="Latency (P95)">
    ```promql theme={null}
    histogram_quantile(0.95,
      rate(istio_request_duration_milliseconds_bucket{
        destination_service_name="mcp-server"
      }[1m])
    )
    ```
  </Tab>

  <Tab title="Error Rate">
    ```promql theme={null}
    rate(istio_requests_total{
      destination_service_name="mcp-server",
      response_code=~"5.."
    }[1m])
    ```
  </Tab>
</Tabs>

### Dashboards

Import pre-built dashboards:

```bash theme={null}
# Install Kiali (service mesh dashboard)
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.20/samples/addons/kiali.yaml

# Port-forward
kubectl port-forward svc/kiali -n istio-system 20001:20001

# Open http://localhost:20001
```

**Features**:

* Service graph visualization
* Traffic animation
* Configuration validation
* Distributed tracing

***

## Multi-Cluster Mesh

<Steps>
  <Step title="Register All Clusters">
    ```bash theme={null}
    # Dev cluster
    gcloud container fleet memberships register mcp-dev-membership \
      --gke-cluster=us-central1/mcp-dev-gke \
      --project=PROJECT_ID

    # Staging cluster
    gcloud container fleet memberships register mcp-staging-membership \
      --gke-cluster=us-central1/mcp-preview-gke \
      --project=PROJECT_ID

    # Prod cluster (already registered)
    ```
  </Step>

  <Step title="Enable Mesh for All">
    ```bash theme={null}
    gcloud container fleet mesh update \
      --management automatic \
      --memberships=mcp-dev-membership,mcp-staging-membership,mcp-prod-membership \
      --project=PROJECT_ID
    ```
  </Step>

  <Step title="Configure Cross-Cluster Service Discovery">
    ```yaml theme={null}
    apiVersion: networking.istio.io/v1beta1
    kind: ServiceEntry
    metadata:
      name: external-staging-service
      namespace: production-mcp-server-langgraph
    spec:
      hosts:
      - mcp-server.mcp-staging.svc.cluster.local
      location: MESH_INTERNAL
      ports:
      - number: 8000
        name: http
        protocol: HTTP
      resolution: DNS
    ```
  </Step>
</Steps>

**Use case**: Production can call staging services for integration testing

***

## Troubleshooting

<AccordionGroup>
  <Accordion title="Sidecar not injected">
    **Symptom**: Pod has 1/1 containers (should be 2/2)

    **Checks**:

    ```bash theme={null}
    # Verify namespace labeled
    kubectl get namespace production-mcp-server-langgraph --show-labels

    # Should see: istio-injection=enabled

    # Check injection status
    kubectl get mutatingwebhookconfigurations
    ```

    **Solution**: Label namespace and restart pods
  </Accordion>

  <Accordion title="mTLS connection failure">
    **Symptom**: Service A can't connect to Service B

    **Checks**:

    ```bash theme={null}
    # Check PeerAuthentication
    kubectl get peerauthentication -n production-mcp-server-langgraph

    # Check DestinationRule
    kubectl get destinationrule -n production-mcp-server-langgraph

    # Verify certificates
    istioctl proxy-config secret POD_NAME -n production-mcp-server-langgraph
    ```

    **Common fix**: Ensure both sides have sidecars injected
  </Accordion>

  <Accordion title="Control plane not ready">
    **Symptom**: Mesh status shows PROVISIONING for >20 minutes

    **Solution**:

    ```bash theme={null}
    # Check fleet status
    gcloud container fleet mesh describe --project=PROJECT_ID

    # View logs
    kubectl logs -n istio-system deployment/istiod

    # If stuck, re-enable
    gcloud container fleet mesh update \
      --management automatic \
      --memberships=MEMBERSHIP_NAME \
      --project=PROJECT_ID
    ```
  </Accordion>
</AccordionGroup>

***

## Best Practices

<Check>**Start with PERMISSIVE mTLS**, then move to STRICT</Check>

```yaml theme={null}
# Week 1: Permissive (allow migration)
mtls:
  mode: PERMISSIVE

# Week 2: Strict (after all services have sidecars)
mtls:
  mode: STRICT
```

<Check>**Use namespace-scoped policies** for isolation</Check>

```yaml theme={null}
# Production has strict mTLS
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: production-mcp-server-langgraph
spec:
  mtls:
    mode: STRICT

# Dev can be permissive
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: mcp-dev
spec:
  mtls:
    mode: PERMISSIVE
```

<Check>**Enable resource limits** on sidecars</Check>

```yaml theme={null}
apiVersion: v1
kind: ConfigMap
metadata:
  name: istio-sidecar-injector
  namespace: istio-system
data:
  values: |
    global:
      proxy:
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 500m
            memory: 256Mi
```

<Check>**Monitor mesh health** with SLIs</Check>

```yaml theme={null}
# SLI: 99% of requests < 500ms
# SLI: 99.9% success rate (non-5xx)
# Alert if error budget depleted
```

***

## Related Documentation

<CardGroup cols={2}>
  <Card title="GKE Production" icon="kubernetes" href="/deployment/kubernetes/gke-production">
    Deploy with service mesh enabled
  </Card>

  <Card title="Security Hardening" icon="shield-halved" href="/security/gcp-security-hardening">
    mTLS as part of 67-control framework
  </Card>

  <Card title="Operations Runbooks" icon="book-medical" href="/deployment/operations/gke-runbooks">
    Service mesh troubleshooting
  </Card>

  <Card title="Monitoring" icon="chart-line" href="/deployment/monitoring">
    Service mesh metrics and alerting
  </Card>
</CardGroup>

***

## Next Steps

<Steps>
  <Step title="Install Anthos Service Mesh">
    ```bash theme={null}
    ./deployments/service-mesh/anthos/setup-anthos-service-mesh.sh PROJECT_ID
    ```
  </Step>

  <Step title="Enable Sidecar Injection">
    ```bash theme={null}
    kubectl label namespace production-mcp-server-langgraph istio-injection=enabled
    kubectl rollout restart deployment -n production-mcp-server-langgraph
    ```
  </Step>

  <Step title="Enable Strict mTLS">
    ```bash theme={null}
    kubectl apply -f deployments/service-mesh/anthos/peer-authentication.yaml
    ```
  </Step>

  <Step title="Configure Traffic Rules">
    Set up canary deployments, circuit breaking, retries
  </Step>

  <Step title="Monitor Service Topology">
    Console → Anthos → Service Mesh → Topology
  </Step>
</Steps>