> ## Documentation Index
> Fetch the complete documentation index at: https://mcp-server-langgraph.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# 17. Error Handling Strategy

> Architecture Decision Record: 17. Error Handling Strategy

# 17. Error Handling Strategy

Date: 2025-10-13

## Status

Accepted

## Category

Core Architecture

## Context

The MCP Server with LangGraph needs a consistent, production-grade error handling strategy that:

* Provides clear error messages to users
* Enables debugging with sufficient context
* Maintains security by not leaking sensitive information
* Supports observability and monitoring
* Handles failures gracefully with appropriate fallback mechanisms
* Complies with enterprise SLAs and uptime requirements

## Decision

We will implement a **layered error handling strategy** with the following principles:

### 1. Error Categories

We categorize all errors into five types:

#### Client Errors (4xx)

* **Authentication Errors** (401): Invalid or expired credentials
* **Authorization Errors** (403): Insufficient permissions
* **Validation Errors** (400): Invalid input data
* **Not Found Errors** (404): Resource doesn't exist

#### Server Errors (5xx)

* **Internal Errors** (500): Unexpected server failures
* **Service Unavailable** (503): Temporary service disruption
* **Timeout Errors** (504): Operation exceeded time limit

### 2. Error Propagation Pattern

```
Layer 1: Infrastructure (Database, External APIs)
   ↓
Layer 2: Service Layer (Business Logic)
   ↓
Layer 3: API Layer (Controllers/Handlers)
   ↓
Layer 4: Client (User-facing error messages)
```

**Rules**:

* **Catch at appropriate layer**: Catch errors where you have context to handle them
* **Add context**: Each layer adds relevant context before re-raising
* **Transform sensitive errors**: Never expose infrastructure details to clients
* **Log once**: Log at the layer where you handle the error, not at every layer

### 3. Error Response Format

All API errors return a consistent JSON structure:

```json theme={null}
{
  "error": {
    "code": "AUTHENTICATION_FAILED",
    "message": "Invalid credentials",
    "details": {
      "reason": "token_expired",
      "timestamp": "2025-10-13T12:00:00Z"
    },
    "trace_id": "abc123",
    "request_id": "req_xyz"
  }
}
```

**Fields**:

* `code`: Machine-readable error code (uppercase snake\_case)
* `message`: Human-readable error message
* `details`: Additional context (optional, non-sensitive)
* `trace_id`: OpenTelemetry trace ID for debugging
* `request_id`: Request identifier

### 4. Retry Strategy

#### Automatic Retries

* **LLM API Calls**: 3 retries with exponential backoff (1s, 2s, 4s)
* **OpenFGA Checks**: 2 retries with 500ms backoff
* **Redis Operations**: 2 retries with 100ms backoff

#### No Retries

* Authentication failures (permanent failures)
* Validation errors (requires user correction)
* Resource not found (won't exist on retry)

### 5. Fallback Mechanisms

#### Error Handling & Recovery Flow

```mermaid theme={null}
flowchart TB
    subgraph Request["Incoming Request"]
        Req[API Request]
    end

    subgraph Validation["Layer 1: Validation"]
        Validate{Valid Input?}
        ValErr[400 Bad Request]
    end

    subgraph Auth["Layer 2: Authentication"]
        AuthCheck{Token Valid?}
        AuthErr[401 Unauthorized]
    end

    subgraph Authz["Layer 3: Authorization"]
        OpenFGA{OpenFGA Check}
        Fallback{Fallback Mode?}
        FallbackCheck[Role-Based Check]
        AuthzErr[403 Forbidden]
    end

    subgraph Service["Layer 4: Service Execution"]
        Primary[Primary Service Call]
        Retry{Retry Available?}
        Fallback2[Fallback Service]
        SvcErr[503 Service Unavailable]
    end

    subgraph Response["Response"]
        Success[200 Success]
        ErrorResp[Error Response + trace_id]
    end

    Req --> Validate
    Validate -->|No| ValErr
    Validate -->|Yes| AuthCheck
    AuthCheck -->|No| AuthErr
    AuthCheck -->|Yes| OpenFGA
    OpenFGA -->|Unavailable| Fallback
    Fallback -->|Yes| FallbackCheck
    Fallback -->|No| AuthzErr
    FallbackCheck -->|Denied| AuthzErr
    FallbackCheck -->|Allowed| Primary
    OpenFGA -->|Denied| AuthzErr
    OpenFGA -->|Allowed| Primary
    Primary -->|Error| Retry
    Retry -->|Yes| Fallback2
    Retry -->|No| SvcErr
    Fallback2 -->|Error| SvcErr
    Fallback2 -->|Success| Success
    Primary -->|Success| Success
    ValErr --> ErrorResp
    AuthErr --> ErrorResp
    AuthzErr --> ErrorResp
    SvcErr --> ErrorResp

    %% ColorBrewer2 Set3 palette - each component type uniquely colored
    classDef reqStyle fill:#8dd3c7,stroke:#2a9d8f,stroke-width:2px,color:#333
    classDef valStyle fill:#fdb462,stroke:#e67e22,stroke-width:2px,color:#333
    classDef authStyle fill:#bebada,stroke:#7e5eb0,stroke-width:2px,color:#333
    classDef authzStyle fill:#bc80bd,stroke:#8e44ad,stroke-width:2px,color:#333
    classDef svcStyle fill:#b3de69,stroke:#7cb342,stroke-width:2px,color:#333
    classDef errStyle fill:#fb8072,stroke:#e74c3c,stroke-width:2px,color:#333
    classDef successStyle fill:#ffffb3,stroke:#f1c40f,stroke-width:2px,color:#333

    class Req reqStyle
    class Validate,ValErr valStyle
    class AuthCheck,AuthErr authStyle
    class OpenFGA,Fallback,FallbackCheck,AuthzErr authzStyle
    class Primary,Retry,Fallback2,SvcErr svcStyle
    class Success,ErrorResp successStyle
```

#### Multi-Provider LLM Fallback Decision Tree

```mermaid theme={null}
flowchart TB
    subgraph Request["LLM Request"]
        Input[User Query]
    end

    subgraph Primary["Primary Provider"]
        P1[Claude Sonnet 4.5]
        P1Err{Error?}
        P1Rate{Rate Limited?}
    end

    subgraph Fallback1["Fallback #1"]
        F1[Gemini 2.5 Pro]
        F1Err{Error?}
    end

    subgraph Fallback2["Fallback #2"]
        F2[GPT-4o]
        F2Err{Error?}
    end

    subgraph Circuit["Circuit Breaker"]
        CB{Circuit Open?}
        CBOpen[Skip to Fallback]
        CBIncr[Increment Failures]
    end

    subgraph Result["Response"]
        Success[LLM Response]
        Error[All Providers Failed]
    end

    Input --> CB
    CB -->|Yes| CBOpen
    CB -->|No| P1
    CBOpen --> F1
    P1 --> P1Err
    P1Err -->|No| Success
    P1Err -->|Yes| P1Rate
    P1Rate -->|No| CBIncr
    P1Rate -->|Yes| F1
    CBIncr --> F1
    F1 --> F1Err
    F1Err -->|No| Success
    F1Err -->|Yes| F2
    F2 --> F2Err
    F2Err -->|No| Success
    F2Err -->|Yes| Error

    %% ColorBrewer2 Set3 palette - each component type uniquely colored
    classDef reqStyle fill:#8dd3c7,stroke:#2a9d8f,stroke-width:2px,color:#333
    classDef primaryStyle fill:#fb8072,stroke:#e74c3c,stroke-width:2px,color:#333
    classDef fallback1Style fill:#fdb462,stroke:#e67e22,stroke-width:2px,color:#333
    classDef fallback2Style fill:#80b1d3,stroke:#3498db,stroke-width:2px,color:#333
    classDef circuitStyle fill:#bebada,stroke:#7e5eb0,stroke-width:2px,color:#333
    classDef successStyle fill:#b3de69,stroke:#7cb342,stroke-width:2px,color:#333
    classDef errorStyle fill:#d9d9d9,stroke:#95a5a6,stroke-width:2px,color:#333

    class Input reqStyle
    class P1,P1Err,P1Rate primaryStyle
    class F1,F1Err fallback1Style
    class F2,F2Err fallback2Style
    class CB,CBOpen,CBIncr circuitStyle
    class Success successStyle
    class Error errorStyle
```

#### LLM Fallback

When primary model fails:

```
1. Try fallback model #1 (gemini-2.5-pro)
2. Try fallback model #2 (claude-sonnet-4-5)
3. Try fallback model #3 (gpt-5.1)
4. Return error if all fail
```

#### Authorization Fallback

When OpenFGA is unavailable:

```python theme={null}
if openfga_strict_mode:
    # Fail-closed: deny all requests
    return deny_access("Authorization service unavailable")
else:
    # Fail-open: use role-based fallback
    return check_fallback_authorization(user, resource)
```

#### Session Storage Fallback

When Redis is unavailable:

```
Redis (primary) → InMemory (fallback) → Error
```

### 6. Logging Strategy

#### Error Logging Levels

* **ERROR**: Unexpected failures requiring investigation
  * Database connection failures
  * External API errors (after retries)
  * Unhandled exceptions

* **WARNING**: Expected failures with fallback handling
  * OpenFGA timeout (using fallback)
  * LLM primary model failure (trying fallback)
  * Rate limit approaching threshold

* **INFO**: Normal operational events
  * Successful fallback execution
  * Retry attempts
  * Circuit breaker state changes

#### Log Structure

```python theme={null}
logger.error(
    "LLM API call failed after retries",
    extra={
        "error_code": "LLM_API_ERROR",
        "provider": "google",
        "model": "gemini-2.5-flash",
        "retry_count": 3,
        "trace_id": trace_id,
        "user_id": user_id
    },
    exc_info=True  # Include stack trace
)
```

### 7. OpenTelemetry Integration

All errors are captured in distributed traces:

```python theme={null}
with tracer.start_as_current_span("llm.call") as span:
    try:
        result = await llm.call(prompt)
        span.set_status(StatusCode.OK)
    except LLMAPIError as e:
        span.set_status(StatusCode.ERROR, str(e))
        span.record_exception(e)
        raise
```

**Span Attributes**:

* `error.type`: Exception class name
* `error.message`: Error message
* `error.stack`: Stack trace (truncated)
* `error.recoverable`: Whether error is recoverable

### 8. Error Codes

#### Authentication/Authorization (AUTH\_\*)

* `AUTH_INVALID_CREDENTIALS`: Invalid username/password
* `AUTH_TOKEN_EXPIRED`: JWT token expired
* `AUTH_TOKEN_INVALID`: Malformed or invalid token
* `AUTH_INSUFFICIENT_PERMISSIONS`: User lacks required permissions
* `AUTH_SESSION_EXPIRED`: Session no longer valid
* `AUTH_MFA_REQUIRED`: Multi-factor authentication required

#### Validation (VALIDATION\_\*)

* `VALIDATION_REQUIRED_FIELD`: Required field missing
* `VALIDATION_INVALID_FORMAT`: Field format incorrect
* `VALIDATION_OUT_OF_RANGE`: Value outside allowed range
* `VALIDATION_CONSTRAINT_VIOLATION`: Business rule violated

#### Resource (RESOURCE\_\*)

* `RESOURCE_NOT_FOUND`: Requested resource doesn't exist
* `RESOURCE_ALREADY_EXISTS`: Resource already exists (conflict)
* `RESOURCE_LOCKED`: Resource is locked by another process
* `RESOURCE_QUOTA_EXCEEDED`: Resource quota limit reached

#### Infrastructure (INFRA\_\*)

* `INFRA_DATABASE_ERROR`: Database connection/query failure
* `INFRA_REDIS_ERROR`: Redis connection/operation failure
* `INFRA_OPENFGA_ERROR`: OpenFGA service error
* `INFRA_NETWORK_ERROR`: Network connectivity issue

#### External Service (EXT\_\*)

* `EXT_LLM_API_ERROR`: LLM provider API error
* `EXT_LLM_RATE_LIMIT`: LLM rate limit exceeded
* `EXT_KEYCLOAK_ERROR`: Keycloak authentication error
* `EXT_INFISICAL_ERROR`: Infisical secrets retrieval error

#### Internal (INTERNAL\_\*)

* `INTERNAL_UNEXPECTED_ERROR`: Unhandled exception
* `INTERNAL_TIMEOUT`: Operation timeout
* `INTERNAL_CIRCUIT_BREAKER_OPEN`: Circuit breaker protecting service

### 9. Error Metrics

Track error rates with Prometheus:

```python theme={null}
error_counter = Counter(
    "mcp_errors_total",
    "Total errors by type and layer",
    ["error_code", "layer", "severity"]
)

error_duration = Histogram(
    "mcp_error_handling_duration_seconds",
    "Time spent handling errors",
    ["error_code"]
)
```

### 10. Security Considerations

**Never expose**:

* Database connection strings
* Internal IP addresses or service names
* Authentication secrets or tokens
* Full stack traces (only trace\_id for debugging)
* SQL queries or database schema details
* API keys or credentials

**Always sanitize**:

* User input in error messages
* File paths (show relative, not absolute)
* Email addresses (show partial: a\*\*\*@acme.com)

## Implementation Examples

### Example 1: Authentication Error

```python theme={null}
# auth/middleware.py
async def verify_token(self, token: str) -> TokenVerification:
    """Verify JWT token with proper error handling"""

    with tracer.start_as_current_span("auth.verify_token") as span:
        try:
            payload = jwt.decode(
                token,
                self.secret_key,
                algorithms=[self.algorithm]
            )
            span.set_attribute("user_id", payload["user_id"])
            return TokenVerification(valid=True, user_id=payload["user_id"])

        except jwt.ExpiredSignatureError:
            span.set_status(StatusCode.ERROR, "Token expired")
            logger.warning(
                "Token verification failed: expired",
                extra={
                    "error_code": "AUTH_TOKEN_EXPIRED",
                    "trace_id": span.get_span_context().trace_id
                }
            )
            return TokenVerification(
                valid=False,
                error_code="AUTH_TOKEN_EXPIRED",
                error_message="Authentication token has expired"
            )

        except jwt.InvalidTokenError as e:
            span.set_status(StatusCode.ERROR, str(e))
            logger.warning(
                "Token verification failed: invalid",
                extra={
                    "error_code": "AUTH_TOKEN_INVALID",
                    "trace_id": span.get_span_context().trace_id
                }
            )
            return TokenVerification(
                valid=False,
                error_code="AUTH_TOKEN_INVALID",
                error_message="Invalid authentication token"
            )
```

### Example 2: LLM Fallback

```python theme={null}
# llm/factory.py
async def create_llm_with_fallback(
    self,
    model: str,
    temperature: float = 0.7
) -> ChatModel:
    """Create LLM with automatic fallback"""

    models_to_try = [model] + self.fallback_models

    for attempt, model_name in enumerate(models_to_try):
        try:
            logger.info(
                f"Attempting LLM creation: {model_name}",
                extra={"model": model_name, "attempt": attempt + 1}
            )

            llm = await self._create_llm(model_name, temperature)

            # Test the LLM with a simple call
            await llm.ainvoke("test")

            if attempt > 0:
                # Successfully failed over
                logger.warning(
                    "LLM fallback successful",
                    extra={
                        "primary_model": model,
                        "fallback_model": model_name,
                        "attempt": attempt + 1
                    }
                )

            return llm

        except Exception as e:
            logger.error(
                f"LLM creation failed: {model_name}",
                extra={
                    "model": model_name,
                    "error": str(e),
                    "attempt": attempt + 1,
                    "is_last_attempt": attempt == len(models_to_try) - 1
                },
                exc_info=True if attempt == len(models_to_try) - 1 else False
            )

            if attempt == len(models_to_try) - 1:
                # All models failed
                raise LLMCreationError(
                    f"All LLM models failed after {len(models_to_try)} attempts",
                    error_code="EXT_LLM_API_ERROR",
                    models_tried=models_to_try
                )
```

### Example 3: Service Layer Error

```python theme={null}
# core/compliance/data_export.py
async def export_user_data(
    self,
    user_id: str,
    username: str,
    email: str
) -> UserDataExport:
    """Export user data with comprehensive error handling"""

    with tracer.start_as_current_span("data_export.export_user_data") as span:
        span.set_attribute("user_id", user_id)

        try:
            # Gather data from multiple sources
            sessions = await self._get_user_sessions(user_id)

            export = UserDataExport(
                user_id=user_id,
                username=username,
                email=email,
                sessions=sessions,
                # ... other fields
            )

            span.set_status(StatusCode.OK)
            return export

        except Exception as e:
            # Add context and re-raise
            span.set_status(StatusCode.ERROR, str(e))
            span.record_exception(e)

            logger.error(
                "User data export failed",
                extra={
                    "user_id": user_id,
                    "error_code": "INTERNAL_EXPORT_ERROR",
                    "error": str(e)
                },
                exc_info=True
            )

            raise DataExportError(
                f"Failed to export data for user {user_id}",
                error_code="INTERNAL_EXPORT_ERROR",
                user_id=user_id
            ) from e
```

## Consequences

### Positive

* **Consistency**: All errors follow the same structure and patterns
* **Debuggability**: Trace IDs link errors to distributed traces
* **Reliability**: Fallback mechanisms prevent single points of failure
* **Observability**: Comprehensive metrics and logging
* **Security**: No sensitive information leaked in error messages

### Negative

* **Complexity**: Developers must understand error categorization
* **Overhead**: Additional logging and tracing adds latency (minimal)
* **Maintenance**: Error codes must be documented and maintained

### Mitigation

* Provide error handling templates and examples
* Create linting rules to enforce patterns
* Document all error codes in centralized registry
* Monitor error handling overhead in production

## Related ADRs

* [0003: Dual Observability Strategy](https://github.com/vishnu2kmohan/mcp-server-langgraph/blob/main/adr/adr-0003-dual-observability.md)
* [0001: Multi-Provider LLM Support](https://github.com/vishnu2kmohan/mcp-server-langgraph/blob/main/adr/adr-0001-llm-multi-provider.md)

## References

* [Google SRE: Error Handling](https://sre.google/sre-book/handling-overload/)
* [12 Factor App: Logs](https://12factor.net/logs)
* [OpenTelemetry Error Handling](https://opentelemetry.io/docs/instrumentation/python/manual/#errors)
* [OWASP: Error Handling](https://owasp.org/www-community/Improper_Error_Handling)