Log Aggregation - MCP Server with LangGraph

Available in: v2.5.0+ View Release Notes →

Overview

Production-grade structured JSON logging with OpenTelemetry trace injection and support for 6 major log aggregation platforms.

Supported Platforms

AWS CloudWatch

CloudWatch Logs + Metrics (EMF) + X-Ray

GCP Cloud Logging

Cloud Logging + Monitoring + Trace

Azure Monitor

Application Insights (unified)

Elasticsearch

ELK Stack with daily indices

Datadog

Unified APM, Logs, Metrics

Splunk

Enterprise or Observability Cloud

Structured JSON Logging

Features

✅ Automatic trace injection - trace_id and span_id in every log
✅ ISO 8601 timestamps with millisecond precision
✅ Exception stack traces in structured format
✅ Custom fields via logging.extra parameter
✅ Backward compatible - Can use text format with LOG_FORMAT=text

Example Output

{
  "timestamp": "2025-10-15T14:23:45.123Z",
  "level": "INFO",
  "logger": "mcp-server-langgraph",
  "service": "mcp-server-langgraph",
  "hostname": "pod-abc123",
  "message": "User logged in successfully",
  "trace_id": "0af7651916cd43dd8448eb211c80319c",
  "span_id": "b7ad6b7169203331",
  "trace_flags": "01",
  "user_id": "alice",
  "ip_address": "192.168.1.100",
  "process": {"pid": 1234, "name": "MainProcess"},
  "thread": {"id": 5678, "name": "MainThread"},
  "location": {
    "file": "/app/auth.py",
    "line": 42,
    "function": "login"
  }
}

Configuration

## .env
LOG_FORMAT=json              # "json" or "text" (default: json)
LOG_JSON_INDENT=null         # null (compact) or 2 (pretty-print)

AWS CloudWatch

Overview

Export logs to CloudWatch Logs, metrics to CloudWatch Metrics (via EMF), and traces to X-Ray.

Prerequisites

AWS Account with CloudWatch and X-Ray enabled
IAM role with permissions (recommended) or access keys

Setup

1. Create IAM Policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:DescribeLogStreams"
      ],
      "Resource": "arn:aws:logs:*:*:log-group:/aws/mcp-server-langgraph/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "xray:PutTraceSegments",
        "xray:PutTelemetryRecords"
      ],
      "Resource": "*"
    }
  ]
}

2. Configure Environment

## .env
AWS_REGION=us-east-1
AWS_ACCESS_KEY_ID=your-access-key      # Or use IAM role
AWS_SECRET_ACCESS_KEY=your-secret-key  # Or use IAM role
ENVIRONMENT=production

3. Docker Compose

## Switch to AWS CloudWatch
./scripts/switch-log-exporter.sh aws

## Start services
docker compose restart otel-collector

4. Kubernetes (EKS)

## Create IAM role for service account (IRSA)
eksctl create iamserviceaccount \
  --name otel-collector \
  --namespace mcp-server-langgraph \
  --cluster my-cluster \
  --attach-policy-arn arn:aws:iam::ACCOUNT_ID:policy/OTELCollectorPolicy \
  --approve

## Deploy
kubectl apply -k deployments/kubernetes/overlays/aws

Configuration Details

Log Groups: /aws/mcp-server-langgraph/${ENVIRONMENT} Log Streams: {service.name}/{hostname} Metrics Namespace: MCPServer/${ENVIRONMENT} Retention: Configure via AWS Console (default: Never expire)

Verification

## View logs
aws logs tail /aws/mcp-server-langgraph/production --follow

## Query logs with CloudWatch Insights
aws logs start-query \
  --log-group-name /aws/mcp-server-langgraph/production \
  --start-time $(date -u -d '1 hour ago' +%s) \
  --end-time $(date -u +%s) \
  --query-string 'fields @timestamp, level, message, trace_id | filter level = "ERROR"'

GCP Cloud Logging

Overview

Export logs to Cloud Logging, metrics to Cloud Monitoring, and traces to Cloud Trace (unified exporter).

Prerequisites

GCP Project with Logging and Trace APIs enabled
Service Account with permissions or Workload Identity

Setup

1. Create Service Account

## Create service account
gcloud iam service-accounts create otel-collector \
  --display-name="OTEL Collector" \
  --project=YOUR_PROJECT_ID

## Grant permissions
gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
  --member="serviceAccount:otel-collector@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/logging.logWriter"

gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
  --member="serviceAccount:otel-collector@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/cloudtrace.agent"

gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
  --member="serviceAccount:otel-collector@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/monitoring.metricWriter"

## Create key (for Docker Compose)
gcloud iam service-accounts keys create sa-key.json \
  --iam-account=otel-collector@YOUR_PROJECT_ID.iam.gserviceaccount.com

2. Configure Environment

## .env
GCP_PROJECT_ID=your-project-id
GOOGLE_APPLICATION_CREDENTIALS=/path/to/sa-key.json
ENVIRONMENT=production

3. Docker Compose

## Switch to GCP
./scripts/switch-log-exporter.sh gcp

## Mount service account key
docker compose restart otel-collector

4. Kubernetes (GKE with Workload Identity)

## Bind Kubernetes SA to GCP SA
gcloud iam service-accounts add-iam-policy-binding \
  otel-collector@YOUR_PROJECT_ID.iam.gserviceaccount.com \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:YOUR_PROJECT_ID.svc.id.goog[mcp-server-langgraph/otel-collector]"

## Annotate Kubernetes service account
kubectl annotate serviceaccount otel-collector \
  --namespace mcp-server-langgraph \
  iam.gke.io/gcp-service-account=otel-collector@YOUR_PROJECT_ID.iam.gserviceaccount.com

## Deploy (create GCP overlay first)
kubectl apply -k deployments/kubernetes/overlays/gcp

Configuration Details

Log Name: mcp-server-langgraph Metric Prefix: custom.googleapis.com/mcp-server/ Resource Detection: Automatic (GKE, GCE, Cloud Run)

Verification

## View logs
gcloud logging read "resource.type=k8s_container AND logName:mcp-server-langgraph" \
  --limit 50 \
  --format json

## Query with Log Explorer (via Console)
## https://console.cloud.google.com/logs/query

Azure Monitor

Overview

Export logs, metrics, and traces to Application Insights (unified).

Prerequisites

Azure subscription
Application Insights resource created

Setup

1. Create Application Insights

## Create resource group
az group create --name mcp-server-rg --location eastus

## Create Application Insights
az monitor app-insights component create \
  --app mcp-server-insights \
  --location eastus \
  --resource-group mcp-server-rg \
  --application-type web

## Get connection string
az monitor app-insights component show \
  --app mcp-server-insights \
  --resource-group mcp-server-rg \
  --query connectionString -o tsv

2. Configure Environment

## .env
AZURE_MONITOR_CONNECTION_STRING="InstrumentationKey=...;IngestionEndpoint=https://..."
ENVIRONMENT=production

3. Docker Compose

## Switch to Azure
./scripts/switch-log-exporter.sh azure

## Start services
docker compose restart otel-collector

4. Kubernetes (AKS)

## Create secret with connection string
kubectl create secret generic otel-collector-secrets \
  --namespace mcp-server-langgraph \
  --from-literal=azure-connection-string="InstrumentationKey=..."

## Deploy (create Azure overlay first)
kubectl apply -k deployments/kubernetes/overlays/azure

Configuration Details

Application Map: Automatic service topology Live Metrics: Real-time monitoring Smart Detection: Anomaly detection enabled

Verification

Access Azure Portal:

Logs: https://portal.azure.com → Application Insights → Logs
Application Map: https://portal.azure.com → Application Insights → Application Map
Live Metrics: https://portal.azure.com → Application Insights → Live Metrics

Elasticsearch

Overview

Export logs and traces to Elasticsearch with daily index rotation and Kibana visualization.

Prerequisites

Elasticsearch cluster (self-hosted or Elastic Cloud)
Kibana for visualization

Setup

1. Configure Environment

## .env
ELASTICSEARCH_ENDPOINT=https://elasticsearch:9200
ELASTICSEARCH_USERNAME=elastic
ELASTICSEARCH_PASSWORD=changeme
ENVIRONMENT=production

## Or use API Key
ELASTICSEARCH_API_KEY=your-api-key

## Or use Elastic Cloud ID
ELASTICSEARCH_CLOUD_ID=cluster-name:dXMtY2VudHJhbDE...

2. Docker Compose

## Switch to Elasticsearch
./scripts/switch-log-exporter.sh elasticsearch

## Start services
docker compose restart otel-collector

3. Index Patterns

Logs are stored in daily indices:

Logs: mcp-server-langgraph-logs-2025.10.15
Traces: mcp-server-langgraph-traces-2025.10.15

Configuration Details

Index Lifecycle Management: Configure retention policies Mapping: Elastic Common Schema (ECS) Compression: gzip enabled Flush Interval: 30s, 5MB

Verification

## Check indices
curl -u elastic:changeme http://localhost:9200/_cat/indices/mcp-server*

## Query logs
curl -u elastic:changeme -X GET "http://localhost:9200/mcp-server-langgraph-logs-*/_search" \
  -H 'Content-Type: application/json' -d'
{
  "query": {
    "match": {
      "level": "ERROR"
    }
  }
}'

Access Kibana: http://localhost:5601

Datadog

Overview

Unified observability with APM, Log Management, and Infrastructure monitoring.

Prerequisites

Datadog account
API key from https://app.datadoghq.com/organization-settings/api-keys

Setup

1. Configure Environment

## .env
DATADOG_API_KEY=your-api-key
DATADOG_SITE=datadoghq.com  # or datadoghq.eu, us3.datadoghq.com, us5.datadoghq.com
ENVIRONMENT=production

2. Docker Compose

## Switch to Datadog
./scripts/switch-log-exporter.sh datadog

## Start services
docker compose restart otel-collector

3. Kubernetes

## Create secret with API key
kubectl create secret generic otel-collector-secrets \
  --namespace mcp-server-langgraph \
  --from-literal=datadog-api-key=your-api-key \
  --from-literal=datadog-site=datadoghq.com

## Deploy
kubectl apply -k deployments/kubernetes/overlays/datadog

Configuration Details

Service Map: Automatic distributed tracing Watchdog: Anomaly detection enabled Host Metadata: Auto-tagged with environment, service, version

Verification

Access Datadog:

Logs: https://app.datadoghq.com/logs
APM: https://app.datadoghq.com/apm/home
Infrastructure: https://app.datadoghq.com/infrastructure

Splunk

Overview

Export logs to Splunk Enterprise (via HEC) or Splunk Observability Cloud (via SAPM/SignalFx).

Prerequisites

Splunk Enterprise or Splunk Observability Cloud account
HEC token created

Setup (Splunk Enterprise)

1. Create HEC Token

In Splunk Web:

Settings → Data Inputs → HTTP Event Collector
Click “New Token”
Name: mcp-server-langgraph
Source type: _json
Index: main
Save and copy token

2. Configure Environment

## .env
SPLUNK_HEC_TOKEN=your-hec-token
SPLUNK_HEC_ENDPOINT=https://splunk:8088
ENVIRONMENT=production

3. Docker Compose

## Switch to Splunk
./scripts/switch-log-exporter.sh splunk

## Start services
docker compose restart otel-collector

Setup (Splunk Observability Cloud)

## .env
SPLUNK_ACCESS_TOKEN=your-access-token
SPLUNK_REALM=us0  # or us1, eu0, jp0, au0
ENVIRONMENT=production

Configuration Details

HEC Endpoint: ${SPLUNK_HEC_ENDPOINT}/services/collector Source: mcp-server-langgraph Sourcetype: _json (logs), metric (metrics) Compression: gzip enabled

Verification

Access Splunk and run:

index=main source="mcp-server-langgraph" | head 100

Platform Comparison

Feature	AWS	GCP	Azure	Elasticsearch	Datadog	Splunk
Unified Exporter	❌ (3 separate)	✅	✅	❌ (2 separate)	✅	❌ (2 modes)
Auto Trace Correlation	✅	✅	✅	⚠️ (manual)	✅	⚠️ (manual)
Service Map	✅ (X-Ray)	✅ (Trace)	✅ (App Map)	❌	✅	✅ (APM)
Anomaly Detection	⚠️ (CloudWatch Anomaly)	✅ (ML)	✅ (Smart Detection)	✅ (ML)	✅ (Watchdog)	✅ (ITSI)
Retention	Configurable	30 days default	90 days default	Configurable	15 days (standard)	Configurable
Query Language	CloudWatch Insights	Log Explorer	KQL	Elasticsearch DSL	Datadog Query	SPL
Cost	Pay per GB	Pay per GB	Pay per GB	Self-hosted or Cloud	Per host	License or Cloud

Troubleshooting

Logs Not Appearing

Check OTLP collector status:

docker compose logs otel-collector
kubectl logs -n mcp-server-langgraph -l app=otel-collector

Verify configuration:

## Test OTLP endpoint
curl -v http://localhost:13133/

Check credentials:

## Verify environment variables are set
docker compose exec otel-collector env | grep -E '(AWS|GCP|AZURE|DATADOG|ELASTICSEARCH|SPLUNK)'

Authentication Failures

AWS: Verify IAM permissions GCP: Check service account roles Azure: Validate connection string Datadog: Confirm API key is valid Elasticsearch: Test basic auth credentials Splunk: Verify HEC token

High Cardinality

Reduce log volume:

## .env
LOG_LEVEL=WARNING  # Instead of INFO or DEBUG

Enable sampling (edit otel-collector config):

processors:
  probabilistic_sampler:
    sampling_percentage: 10  # Sample 10% of logs

Best Practices

1. Use Structured Fields

## Good
logger.info("User action", extra={
    "user_id": "alice",
    "action": "login",
    "ip": "192.168.1.100"
})

## Bad
logger.info(f"User alice performed login from 192.168.1.100")

2. Include Trace Context

Trace context is automatically injected. Ensure OpenTelemetry tracing is enabled:

## .env
ENABLE_TRACING=true

3. Set Appropriate Log Levels

DEBUG: Development only
INFO: General events (default)
WARNING: Potential issues
ERROR: Errors that need attention
CRITICAL: System failures

4. Configure Retention

Set retention policies based on compliance requirements:

Audit logs: 7 years
Application logs: 30-90 days
Debug logs: 7 days

5. Monitor Costs

AWS CloudWatch: ~$0.50/GB ingested
GCP Cloud Logging: ~$0.50/GB ingested
Azure Monitor: ~$2.76/GB ingested
Datadog: Per host pricing
Elasticsearch: Infrastructure costs
Splunk: Per GB ingested

Next Steps

Log Query Examples - Platform-specific query examples
Observability Guide - Complete observability setup
Kubernetes Deployment - Production Kubernetes deployment

Getting Started

Deployment Options

LangGraph Platform

Kubernetes - GKE

Kubernetes - EKS & AKS

Kubernetes - Best Practices

Infrastructure as Code

Monitoring & Observability

Advanced Deployment

Configuration

Operations

​Overview

​Supported Platforms

AWS CloudWatch

GCP Cloud Logging

Azure Monitor

Elasticsearch

Datadog

Splunk

​Structured JSON Logging

​Features

​Example Output

​Configuration

​AWS CloudWatch

​Overview

​Prerequisites

​Setup

1. Create IAM Policy

2. Configure Environment

3. Docker Compose

4. Kubernetes (EKS)

​Configuration Details

​Verification

​GCP Cloud Logging

​Overview

​Prerequisites

​Setup

1. Create Service Account

2. Configure Environment

3. Docker Compose

4. Kubernetes (GKE with Workload Identity)

​Configuration Details

​Verification

​Azure Monitor

​Overview

​Prerequisites

​Setup

1. Create Application Insights

2. Configure Environment

3. Docker Compose

4. Kubernetes (AKS)

​Configuration Details

​Verification

​Elasticsearch

​Overview

​Prerequisites

​Setup

1. Configure Environment

2. Docker Compose

3. Index Patterns

​Configuration Details

​Verification

​Datadog

​Overview

​Prerequisites

​Setup

1. Configure Environment

2. Docker Compose

3. Kubernetes

​Configuration Details

​Verification

​Splunk

​Overview

​Prerequisites

​Setup (Splunk Enterprise)

1. Create HEC Token

2. Configure Environment

3. Docker Compose

​Setup (Splunk Observability Cloud)

​Configuration Details

Overview

Supported Platforms

Structured JSON Logging

Features

Example Output

Configuration

AWS CloudWatch

Overview

Prerequisites

Setup

Configuration Details

Verification

GCP Cloud Logging

Overview

Prerequisites

Setup

Configuration Details

Verification

Azure Monitor

Overview

Prerequisites

Setup

Configuration Details

Verification

Elasticsearch

Overview

Prerequisites

Setup

Configuration Details

Verification

Datadog

Overview

Prerequisites

Setup

Configuration Details

Verification

Splunk

Overview

Prerequisites

Setup (Splunk Enterprise)

Setup (Splunk Observability Cloud)

Configuration Details

Verification

Platform Comparison

Troubleshooting

Logs Not Appearing

Authentication Failures

High Cardinality

Best Practices

1. Use Structured Fields

2. Include Trace Context

3. Set Appropriate Log Levels

4. Configure Retention

5. Monitor Costs

Next Steps