> ## Documentation Index
> Fetch the complete documentation index at: https://mcp-server-langgraph.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Log Aggregation

> Multi-platform log aggregation with structured JSON logging

<Note type="success">
  **Available in:** v2.5.0+
  [View Release Notes →](/releases/v2-5-0)
</Note>

### Overview

Production-grade **structured JSON logging** with **OpenTelemetry trace injection** and support for 6 major log aggregation platforms.

#### Supported Platforms

<CardGroup cols={3}>
  <Card title="AWS CloudWatch" icon="aws">
    CloudWatch Logs + Metrics (EMF) + X-Ray
  </Card>

  <Card title="GCP Cloud Logging" icon="google">
    Cloud Logging + Monitoring + Trace
  </Card>

  <Card title="Azure Monitor" icon="microsoft">
    Application Insights (unified)
  </Card>

  <Card title="Elasticsearch" icon="magnifying-glass">
    ELK Stack with daily indices
  </Card>

  <Card title="Datadog" icon="chart-line">
    Unified APM, Logs, Metrics
  </Card>

  <Card title="Splunk" icon="tower-broadcast">
    Enterprise or Observability Cloud
  </Card>
</CardGroup>

***

### Structured JSON Logging

#### Features

* ✅ **Automatic trace injection** - trace\_id and span\_id in every log
* ✅ **ISO 8601 timestamps** with millisecond precision
* ✅ **Exception stack traces** in structured format
* ✅ **Custom fields** via logging.extra parameter
* ✅ **Backward compatible** - Can use text format with `LOG_FORMAT=text`

#### Example Output

```json theme={null}
{
  "timestamp": "2025-10-15T14:23:45.123Z",
  "level": "INFO",
  "logger": "mcp-server-langgraph",
  "service": "mcp-server-langgraph",
  "hostname": "pod-abc123",
  "message": "User logged in successfully",
  "trace_id": "0af7651916cd43dd8448eb211c80319c",
  "span_id": "b7ad6b7169203331",
  "trace_flags": "01",
  "user_id": "alice",
  "ip_address": "192.168.1.100",
  "process": {"pid": 1234, "name": "MainProcess"},
  "thread": {"id": 5678, "name": "MainThread"},
  "location": {
    "file": "/app/auth.py",
    "line": 42,
    "function": "login"
  }
}
```

#### Configuration

```properties theme={null}
## .env
LOG_FORMAT=json              # "json" or "text" (default: json)
LOG_JSON_INDENT=null         # null (compact) or 2 (pretty-print)
```

***

### AWS CloudWatch

#### Overview

Export logs to **CloudWatch Logs**, metrics to **CloudWatch Metrics** (via EMF), and traces to **X-Ray**.

#### Prerequisites

* AWS Account with CloudWatch and X-Ray enabled
* IAM role with permissions (recommended) or access keys

#### Setup

##### 1. Create IAM Policy

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:DescribeLogStreams"
      ],
      "Resource": "arn:aws:logs:*:*:log-group:/aws/mcp-server-langgraph/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "xray:PutTraceSegments",
        "xray:PutTelemetryRecords"
      ],
      "Resource": "*"
    }
  ]
}
```

##### 2. Configure Environment

```properties theme={null}
## .env
AWS_REGION=us-east-1
AWS_ACCESS_KEY_ID=your-access-key      # Or use IAM role
AWS_SECRET_ACCESS_KEY=your-secret-key  # Or use IAM role
ENVIRONMENT=production
```

##### 3. Docker Compose

```bash theme={null}
## Switch to AWS CloudWatch
./scripts/switch-log-exporter.sh aws

## Start services
docker compose restart otel-collector
```

##### 4. Kubernetes (EKS)

```bash theme={null}
## Create IAM role for service account (IRSA)
eksctl create iamserviceaccount \
  --name otel-collector \
  --namespace mcp-server-langgraph \
  --cluster my-cluster \
  --attach-policy-arn arn:aws:iam::ACCOUNT_ID:policy/OTELCollectorPolicy \
  --approve

## Deploy
kubectl apply -k deployments/kubernetes/overlays/aws
```

#### Configuration Details

**Log Groups**: `/aws/mcp-server-langgraph/${ENVIRONMENT}`
**Log Streams**: `{service.name}/{hostname}`
**Metrics Namespace**: `MCPServer/${ENVIRONMENT}`
**Retention**: Configure via AWS Console (default: Never expire)

#### Verification

```bash theme={null}
## View logs
aws logs tail /aws/mcp-server-langgraph/production --follow

## Query logs with CloudWatch Insights
aws logs start-query \
  --log-group-name /aws/mcp-server-langgraph/production \
  --start-time $(date -u -d '1 hour ago' +%s) \
  --end-time $(date -u +%s) \
  --query-string 'fields @timestamp, level, message, trace_id | filter level = "ERROR"'
```

***

### GCP Cloud Logging

#### Overview

Export logs to **Cloud Logging**, metrics to **Cloud Monitoring**, and traces to **Cloud Trace** (unified exporter).

#### Prerequisites

* GCP Project with Logging and Trace APIs enabled
* Service Account with permissions or Workload Identity

#### Setup

##### 1. Create Service Account

```bash theme={null}
## Create service account
gcloud iam service-accounts create otel-collector \
  --display-name="OTEL Collector" \
  --project=YOUR_PROJECT_ID

## Grant permissions
gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
  --member="serviceAccount:otel-collector@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/logging.logWriter"

gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
  --member="serviceAccount:otel-collector@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/cloudtrace.agent"

gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
  --member="serviceAccount:otel-collector@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/monitoring.metricWriter"

## Create key (for Docker Compose)
gcloud iam service-accounts keys create sa-key.json \
  --iam-account=otel-collector@YOUR_PROJECT_ID.iam.gserviceaccount.com
```

##### 2. Configure Environment

```properties theme={null}
## .env
GCP_PROJECT_ID=your-project-id
GOOGLE_APPLICATION_CREDENTIALS=/path/to/sa-key.json
ENVIRONMENT=production
```

##### 3. Docker Compose

```bash theme={null}
## Switch to GCP
./scripts/switch-log-exporter.sh gcp

## Mount service account key
docker compose restart otel-collector
```

##### 4. Kubernetes (GKE with Workload Identity)

```bash theme={null}
## Bind Kubernetes SA to GCP SA
gcloud iam service-accounts add-iam-policy-binding \
  otel-collector@YOUR_PROJECT_ID.iam.gserviceaccount.com \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:YOUR_PROJECT_ID.svc.id.goog[mcp-server-langgraph/otel-collector]"

## Annotate Kubernetes service account
kubectl annotate serviceaccount otel-collector \
  --namespace mcp-server-langgraph \
  iam.gke.io/gcp-service-account=otel-collector@YOUR_PROJECT_ID.iam.gserviceaccount.com

## Deploy (create GCP overlay first)
kubectl apply -k deployments/kubernetes/overlays/gcp
```

#### Configuration Details

**Log Name**: `mcp-server-langgraph`
**Metric Prefix**: `custom.googleapis.com/mcp-server/`
**Resource Detection**: Automatic (GKE, GCE, Cloud Run)

#### Verification

```bash theme={null}
## View logs
gcloud logging read "resource.type=k8s_container AND logName:mcp-server-langgraph" \
  --limit 50 \
  --format json

## Query with Log Explorer (via Console)
## https://console.cloud.google.com/logs/query
```

***

### Azure Monitor

#### Overview

Export logs, metrics, and traces to **Application Insights** (unified).

#### Prerequisites

* Azure subscription
* Application Insights resource created

#### Setup

##### 1. Create Application Insights

```bash theme={null}
## Create resource group
az group create --name mcp-server-rg --location eastus

## Create Application Insights
az monitor app-insights component create \
  --app mcp-server-insights \
  --location eastus \
  --resource-group mcp-server-rg \
  --application-type web

## Get connection string
az monitor app-insights component show \
  --app mcp-server-insights \
  --resource-group mcp-server-rg \
  --query connectionString -o tsv
```

##### 2. Configure Environment

```toml theme={null}
## .env
AZURE_MONITOR_CONNECTION_STRING="InstrumentationKey=...;IngestionEndpoint=https://..."
ENVIRONMENT=production
```

##### 3. Docker Compose

```bash theme={null}
## Switch to Azure
./scripts/switch-log-exporter.sh azure

## Start services
docker compose restart otel-collector
```

##### 4. Kubernetes (AKS)

```bash theme={null}
## Create secret with connection string
kubectl create secret generic otel-collector-secrets \
  --namespace mcp-server-langgraph \
  --from-literal=azure-connection-string="InstrumentationKey=..."

## Deploy (create Azure overlay first)
kubectl apply -k deployments/kubernetes/overlays/azure
```

#### Configuration Details

**Application Map**: Automatic service topology
**Live Metrics**: Real-time monitoring
**Smart Detection**: Anomaly detection enabled

#### Verification

Access Azure Portal:

* **Logs**: [https://portal.azure.com](https://portal.azure.com) → Application Insights → Logs
* **Application Map**: [https://portal.azure.com](https://portal.azure.com) → Application Insights → Application Map
* **Live Metrics**: [https://portal.azure.com](https://portal.azure.com) → Application Insights → Live Metrics

***

### Elasticsearch

#### Overview

Export logs and traces to **Elasticsearch** with daily index rotation and **Kibana** visualization.

#### Prerequisites

* Elasticsearch cluster (self-hosted or Elastic Cloud)
* Kibana for visualization

#### Setup

##### 1. Configure Environment

```properties theme={null}
## .env
ELASTICSEARCH_ENDPOINT=https://elasticsearch:9200
ELASTICSEARCH_USERNAME=elastic
ELASTICSEARCH_PASSWORD=changeme
ENVIRONMENT=production

## Or use API Key
ELASTICSEARCH_API_KEY=your-api-key

## Or use Elastic Cloud ID
ELASTICSEARCH_CLOUD_ID=cluster-name:dXMtY2VudHJhbDE...
```

##### 2. Docker Compose

```bash theme={null}
## Switch to Elasticsearch
./scripts/switch-log-exporter.sh elasticsearch

## Start services
docker compose restart otel-collector
```

##### 3. Index Patterns

Logs are stored in daily indices:

* **Logs**: `mcp-server-langgraph-logs-2025.10.15`
* **Traces**: `mcp-server-langgraph-traces-2025.10.15`

#### Configuration Details

**Index Lifecycle Management**: Configure retention policies
**Mapping**: Elastic Common Schema (ECS)
**Compression**: gzip enabled
**Flush Interval**: 30s, 5MB

#### Verification

```bash theme={null}
## Check indices
curl -u elastic:changeme http://localhost:9200/_cat/indices/mcp-server*

## Query logs
curl -u elastic:changeme -X GET "http://localhost:9200/mcp-server-langgraph-logs-*/_search" \
  -H 'Content-Type: application/json' -d'
{
  "query": {
    "match": {
      "level": "ERROR"
    }
  }
}'
```

Access Kibana: [http://localhost:5601](http://localhost:5601)

***

### Datadog

#### Overview

Unified observability with **APM**, **Log Management**, and **Infrastructure** monitoring.

#### Prerequisites

* Datadog account
* API key from [https://app.datadoghq.com/organization-settings/api-keys](https://app.datadoghq.com/organization-settings/api-keys)

#### Setup

##### 1. Configure Environment

```properties theme={null}
## .env
DATADOG_API_KEY=your-api-key
DATADOG_SITE=datadoghq.com  # or datadoghq.eu, us3.datadoghq.com, us5.datadoghq.com
ENVIRONMENT=production
```

##### 2. Docker Compose

```bash theme={null}
## Switch to Datadog
./scripts/switch-log-exporter.sh datadog

## Start services
docker compose restart otel-collector
```

##### 3. Kubernetes

```bash theme={null}
## Create secret with API key
kubectl create secret generic otel-collector-secrets \
  --namespace mcp-server-langgraph \
  --from-literal=datadog-api-key=your-api-key \
  --from-literal=datadog-site=datadoghq.com

## Deploy
kubectl apply -k deployments/kubernetes/overlays/datadog
```

#### Configuration Details

**Service Map**: Automatic distributed tracing
**Watchdog**: Anomaly detection enabled
**Host Metadata**: Auto-tagged with environment, service, version

#### Verification

Access Datadog:

* **Logs**: [https://app.datadoghq.com/logs](https://app.datadoghq.com/logs)
* **APM**: [https://app.datadoghq.com/apm/home](https://app.datadoghq.com/apm/home)
* **Infrastructure**: [https://app.datadoghq.com/infrastructure](https://app.datadoghq.com/infrastructure)

***

### Splunk

#### Overview

Export logs to **Splunk Enterprise** (via HEC) or **Splunk Observability Cloud** (via SAPM/SignalFx).

#### Prerequisites

* Splunk Enterprise or Splunk Observability Cloud account
* HEC token created

#### Setup (Splunk Enterprise)

##### 1. Create HEC Token

In Splunk Web:

1. Settings → Data Inputs → HTTP Event Collector
2. Click "New Token"
3. Name: `mcp-server-langgraph`
4. Source type: `_json`
5. Index: `main`
6. Save and copy token

##### 2. Configure Environment

```properties theme={null}
## .env
SPLUNK_HEC_TOKEN=your-hec-token
SPLUNK_HEC_ENDPOINT=https://splunk:8088
ENVIRONMENT=production
```

##### 3. Docker Compose

```bash theme={null}
## Switch to Splunk
./scripts/switch-log-exporter.sh splunk

## Start services
docker compose restart otel-collector
```

#### Setup (Splunk Observability Cloud)

```properties theme={null}
## .env
SPLUNK_ACCESS_TOKEN=your-access-token
SPLUNK_REALM=us0  # or us1, eu0, jp0, au0
ENVIRONMENT=production
```

#### Configuration Details

**HEC Endpoint**: `${SPLUNK_HEC_ENDPOINT}/services/collector`
**Source**: `mcp-server-langgraph`
**Sourcetype**: `_json` (logs), `metric` (metrics)
**Compression**: gzip enabled

#### Verification

Access Splunk and run:

```spl theme={null}
index=main source="mcp-server-langgraph" | head 100
```

***

### Platform Comparison

| Feature                    | AWS                     | GCP             | Azure               | Elasticsearch        | Datadog            | Splunk           |
| -------------------------- | ----------------------- | --------------- | ------------------- | -------------------- | ------------------ | ---------------- |
| **Unified Exporter**       | ❌ (3 separate)          | ✅               | ✅                   | ❌ (2 separate)       | ✅                  | ❌ (2 modes)      |
| **Auto Trace Correlation** | ✅                       | ✅               | ✅                   | ⚠️ (manual)          | ✅                  | ⚠️ (manual)      |
| **Service Map**            | ✅ (X-Ray)               | ✅ (Trace)       | ✅ (App Map)         | ❌                    | ✅                  | ✅ (APM)          |
| **Anomaly Detection**      | ⚠️ (CloudWatch Anomaly) | ✅ (ML)          | ✅ (Smart Detection) | ✅ (ML)               | ✅ (Watchdog)       | ✅ (ITSI)         |
| **Retention**              | Configurable            | 30 days default | 90 days default     | Configurable         | 15 days (standard) | Configurable     |
| **Query Language**         | CloudWatch Insights     | Log Explorer    | KQL                 | Elasticsearch DSL    | Datadog Query      | SPL              |
| **Cost**                   | Pay per GB              | Pay per GB      | Pay per GB          | Self-hosted or Cloud | Per host           | License or Cloud |

***

### Troubleshooting

#### Logs Not Appearing

**Check OTLP collector status**:

```bash theme={null}
docker compose logs otel-collector
kubectl logs -n mcp-server-langgraph -l app=otel-collector
```

**Verify configuration**:

```bash theme={null}
## Test OTLP endpoint
curl -v http://localhost:13133/
```

**Check credentials**:

```bash theme={null}
## Verify environment variables are set
docker compose exec otel-collector env | grep -E '(AWS|GCP|AZURE|DATADOG|ELASTICSEARCH|SPLUNK)'
```

#### Authentication Failures

**AWS**: Verify IAM permissions
**GCP**: Check service account roles
**Azure**: Validate connection string
**Datadog**: Confirm API key is valid
**Elasticsearch**: Test basic auth credentials
**Splunk**: Verify HEC token

#### High Cardinality

**Reduce log volume**:

```properties theme={null}
## .env
LOG_LEVEL=WARNING  # Instead of INFO or DEBUG
```

**Enable sampling** (edit otel-collector config):

```yaml theme={null}
processors:
  probabilistic_sampler:
    sampling_percentage: 10  # Sample 10% of logs
```

***

### Best Practices

#### 1. Use Structured Fields

```json theme={null}
## Good
logger.info("User action", extra={
    "user_id": "alice",
    "action": "login",
    "ip": "192.168.1.100"
})

## Bad
logger.info(f"User alice performed login from 192.168.1.100")
```

#### 2. Include Trace Context

Trace context is automatically injected. Ensure OpenTelemetry tracing is enabled:

```properties theme={null}
## .env
ENABLE_TRACING=true
```

#### 3. Set Appropriate Log Levels

* **DEBUG**: Development only
* **INFO**: General events (default)
* **WARNING**: Potential issues
* **ERROR**: Errors that need attention
* **CRITICAL**: System failures

#### 4. Configure Retention

Set retention policies based on compliance requirements:

* **Audit logs**: 7 years
* **Application logs**: 30-90 days
* **Debug logs**: 7 days

#### 5. Monitor Costs

* **AWS CloudWatch**: \~\$0.50/GB ingested
* **GCP Cloud Logging**: \~\$0.50/GB ingested
* **Azure Monitor**: \~\$2.76/GB ingested
* **Datadog**: Per host pricing
* **Elasticsearch**: Infrastructure costs
* **Splunk**: Per GB ingested

***

### Next Steps

* [Log Query Examples](/guides/log-queries) - Platform-specific query examples
* [Observability Guide](/guides/observability) - Complete observability setup
* [Kubernetes Deployment](/deployment/kubernetes) - Production Kubernetes deployment