> ## Documentation Index
> Fetch the complete documentation index at: https://mcp-server-langgraph.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Multi-LLM Setup

> Configure multiple LLM providers with automatic fallback

### Overview

Configure **100+ LLM providers** via LiteLLM with automatic fallback, load balancing, and cost optimization. Supports cloud providers (Anthropic, OpenAI, Google) and open-source models (Llama, Mistral, Qwen via Ollama).

<Info>
  **LiteLLM** provides a unified interface to all major LLM providers with automatic retries, fallback, and intelligent routing.
</Info>

### Supported Providers

<CardGroup cols={3}>
  <Card title="Anthropic" icon="robot">
    * Claude Sonnet 4.5
    * Claude Opus 4.1
    * Claude Haiku 4.5
  </Card>

  <Card title="OpenAI" icon="openai">
    * GPT-5
    * GPT-5 Pro
    * GPT-5 Mini
    * GPT-5 Nano
  </Card>

  <Card title="Google" icon="google">
    * Gemini 2.5 Flash
    * Gemini 2.0 Pro
    * Gemini 1.5 Pro
  </Card>

  <Card title="Azure OpenAI" icon="microsoft">
    * GPT-4 (Azure)
    * GPT-3.5 (Azure)
    * Custom deployments
  </Card>

  <Card title="AWS Bedrock" icon="aws">
    * Claude (Bedrock)
    * Llama (Bedrock)
    * Titan
  </Card>

  <Card title="Ollama" icon="computer">
    * Llama 3.1
    * Mistral
    * Qwen 2.5
    * DeepSeek
  </Card>
</CardGroup>

### Quick Start

#### Anthropic Claude

<Steps>
  <Step title="Get API Key">
    1. Sign up at [https://console.anthropic.com](https://console.anthropic.com)
    2. Generate API key
    3. Note your organization ID
  </Step>

  <Step title="Configure">
    ```bash theme={null}
    # .env
    LLM_PROVIDER=anthropic
    MODEL_NAME=claude-sonnet-4-5
    ANTHROPIC_API_KEY=sk-ant-api03-...
    MODEL_TEMPERATURE=0.7
    MODEL_MAX_TOKENS=4096
    ```
  </Step>

  <Step title="Test">
    ```python theme={null}
    from mcp_server_langgraph.llm.factory import LLMFactory

    llm = LLMFactory(
        provider="anthropic",
        model_name="claude-sonnet-4-5"
    )

    response = llm.invoke([
        {"role": "user", "content": "Hello!"}
    ])
    print(response.content)
    ```
  </Step>
</Steps>

***

#### OpenAI GPT

<Steps>
  <Step title="Get API Key">
    1. Sign up at [https://platform.openai.com](https://platform.openai.com)
    2. Create API key
    3. Add billing information
  </Step>

  <Step title="Configure">
    ```bash theme={null}
    # .env
    LLM_PROVIDER=openai
    MODEL_NAME=gpt-5.1
    OPENAI_API_KEY=sk-proj-...
    MODEL_TEMPERATURE=0.7
    MODEL_MAX_TOKENS=4096
    ```
  </Step>

  <Step title="Test">
    ```python theme={null}
    llm = LLMFactory(
        provider="openai",
        model_name="gpt-5.1"
    )

    response = llm.invoke([
        {"role": "user", "content": "What is AI?"}
    ])
    ```
  </Step>
</Steps>

***

#### Google Gemini

<Steps>
  <Step title="Get API Key">
    1. Go to [https://makersuite.google.com/app/apikey](https://makersuite.google.com/app/apikey)
    2. Create API key
    3. Enable Gemini API
  </Step>

  <Step title="Configure">
    ```bash theme={null}
    # .env
    LLM_PROVIDER=google
    MODEL_NAME=gemini-2.5-flash
    GOOGLE_API_KEY=AIza...
    MODEL_TEMPERATURE=0.7
    MODEL_MAX_TOKENS=8192
    ```
  </Step>

  <Step title="Test">
    ```python theme={null}
    llm = LLMFactory(
        provider="google",
        model_name="gemini-2.5-flash"
    )

    response = llm.invoke([
        {"role": "user", "content": "Explain quantum computing"}
    ])
    ```
  </Step>
</Steps>

***

#### Azure OpenAI

<Steps>
  <Step title="Setup Azure">
    1. Create Azure OpenAI resource
    2. Deploy a model (e.g., gpt-4)
    3. Get endpoint and API key
  </Step>

  <Step title="Configure">
    ```bash theme={null}
    # .env
    LLM_PROVIDER=azure
    MODEL_NAME=azure/gpt-4
    AZURE_API_KEY=your-azure-key
    AZURE_API_BASE=https://your-resource.openai.azure.com
    AZURE_API_VERSION=2024-02-15-preview
    AZURE_DEPLOYMENT_NAME=gpt-4
    ```
  </Step>

  <Step title="Test">
    ```python theme={null}
    llm = LLMFactory(
        provider="azure",
        model_name="azure/gpt-4",
        api_base=os.getenv("AZURE_API_BASE"),
        api_version=os.getenv("AZURE_API_VERSION")
    )
    ```
  </Step>
</Steps>

***

#### AWS Bedrock

<Steps>
  <Step title="Setup AWS">
    1. Enable Bedrock in AWS Console
    2. Request model access
    3. Configure IAM credentials
  </Step>

  <Step title="Configure">
    ```bash theme={null}
    # .env
    LLM_PROVIDER=bedrock
    MODEL_NAME=bedrock/anthropic.claude-sonnet-4-5-v2:0
    AWS_ACCESS_KEY_ID=AKIA...
    AWS_SECRET_ACCESS_KEY=...
    AWS_REGION_NAME=us-east-1
    ```
  </Step>

  <Step title="Test">
    ```python theme={null}
    llm = LLMFactory(
        provider="bedrock",
        model_name="bedrock/anthropic.claude-sonnet-4-5-v2:0"
    )
    ```
  </Step>
</Steps>

***

#### Ollama (Local Models)

<Steps>
  <Step title="Install Ollama">
    ```bash theme={null}
    # macOS / Linux
    curl -fsSL https://ollama.com/install.sh | sh

    # Or download from https://ollama.com/download
    ```
  </Step>

  <Step title="Pull Model">
    ```bash theme={null}
    # Pull Llama 3.1
    ollama pull llama3.1:8b

    # Pull Mistral
    ollama pull mistral:7b

    # Pull Qwen
    ollama pull qwen2.5:7b

    # List models
    ollama list
    ```
  </Step>

  <Step title="Configure">
    ```bash theme={null}
    # .env
    LLM_PROVIDER=ollama
    MODEL_NAME=ollama/llama3.1:8b
    OLLAMA_API_BASE=http://localhost:11434
    MODEL_TEMPERATURE=0.7
    MODEL_MAX_TOKENS=4096
    ```
  </Step>

  <Step title="Test">
    ```python theme={null}
    llm = LLMFactory(
        provider="ollama",
        model_name="ollama/llama3.1:8b",
        api_base="http://localhost:11434"
    )

    response = llm.invoke([
        {"role": "user", "content": "Hello!"}
    ])
    ```
  </Step>
</Steps>

***

### Automatic Fallback

Configure automatic fallback when primary model fails:

```properties theme={null}
## .env
LLM_PROVIDER=anthropic
MODEL_NAME=claude-sonnet-4-5
ENABLE_FALLBACK=true
FALLBACK_MODELS=gemini-2.5-flash,gpt-5.1,ollama/llama3.1:8b
```

**Fallback Flow**:

```mermaid theme={null}
flowchart TD
    Request[User Request] --> Primary[Primary Model<br/>Claude 3.5 Sonnet]

    Primary -->|✓ Success| Response[Return Response]
    Primary -->|✗ Error| Fallback1[Fallback 1<br/>Gemini 2.5 Flash]

    Fallback1 -->|✓ Success| Response
    Fallback1 -->|✗ Error| Fallback2[Fallback 2<br/>GPT-5]

    Fallback2 -->|✓ Success| Response
    Fallback2 -->|✗ Error| Fallback3[Fallback 3<br/>Llama 3.1 Local]

    Fallback3 -->|✓ Success| Response
    Fallback3 -->|✗ Final Error| Error[Error Response]

    %% ColorBrewer2 Set3 palette - each component type uniquely colored
    classDef startStyle fill:#8dd3c7,stroke:#2a9d8f,stroke-width:2px,color:#333
    classDef primaryStyle fill:#fdb462,stroke:#e67e22,stroke-width:2px,color:#333
    classDef fallbackStyle fill:#ffffb3,stroke:#f1c40f,stroke-width:2px,color:#333
    classDef successStyle fill:#b3de69,stroke:#7cb342,stroke-width:2px,color:#333
    classDef errorStyle fill:#fb8072,stroke:#e74c3c,stroke-width:2px,color:#333

    class Request startStyle
    class Primary primaryStyle
    class Fallback1,Fallback2,Fallback3 fallbackStyle
    class Response successStyle
    class Error errorStyle
```

**Configuration**:

```python theme={null}
from mcp_server_langgraph.llm.factory import LLMFactory

llm = LLMFactory(
    provider="anthropic",
    model_name="claude-sonnet-4-5",
    enable_fallback=True,
    fallback_models=[
        "gemini-2.5-flash",
        "gpt-5.1",
        "ollama/llama3.1:8b"
    ]
)

## Automatically falls back on error
response = llm.invoke(messages)
```

**Common Failure Scenarios**:

* API quota exceeded
* Rate limiting
* Model unavailable
* Timeout
* Invalid API key

***

### Model Comparison

<Tabs>
  <Tab title="Performance">
    | Model             | Speed | Quality | Context | Cost   |
    | ----------------- | ----- | ------- | ------- | ------ |
    | Claude Sonnet 4.5 | ⚡⚡⚡   | ⭐⭐⭐⭐⭐   | 200K    | \$\$   |
    | GPT-4o            | ⚡⚡⚡   | ⭐⭐⭐⭐⭐   | 128K    | \$\$\$ |
    | Gemini 2.5 Flash  | ⚡⚡⚡⚡⚡ | ⭐⭐⭐⭐    | 1M      | \$     |
    | Llama 3.1 8B      | ⚡⚡⚡⚡  | ⭐⭐⭐     | 128K    | Free   |
  </Tab>

  <Tab title="Use Cases">
    **Claude Sonnet 4.5**: Best for complex reasoning, code generation, analysis

    **GPT-4o**: Strong general purpose, multimodal, reliable

    **Gemini 2.5 Flash**: Fast, cost-effective, huge context window

    **Llama 3.1**: Local/offline, privacy-sensitive, cost-free
  </Tab>

  <Tab title="Pricing">
    **Input / Output per 1M tokens**:

    * **Claude Sonnet 4.5**: $3 / $15
    * **GPT-4o**: $2.50 / $10
    * **Gemini 2.5 Flash**: $0.075 / $0.30
    * **Llama 3.1 (Ollama)**: $0 / $0 (local)
  </Tab>
</Tabs>

***

### Advanced Configuration

#### Load Balancing

Distribute requests across multiple providers:

```python theme={null}
from litellm import Router

router = Router(
    model_list=[
        {
            "model_name": "claude-sonnet-4-5",
            "litellm_params": {
                "model": "claude-sonnet-4-5",
                "api_key": os.getenv("ANTHROPIC_API_KEY")
            }
        },
        {
            "model_name": "gpt-5.1",
            "litellm_params": {
                "model": "gpt-5.1",
                "api_key": os.getenv("OPENAI_API_KEY")
            }
        }
    ],
    routing_strategy="least-busy"  # or "simple-shuffle", "latency-based"
)

response = await router.acompletion(
    model="claude-sonnet-4-5",
    messages=[{"role": "user", "content": "Hello"}]
)
```

#### Rate Limiting

Prevent quota exhaustion:

```python theme={null}
from litellm import Router

router = Router(
    model_list=[...],
    redis_host="localhost",
    redis_port=6379,
    rpm=1000,  # Requests per minute
    tpm=100000  # Tokens per minute
)
```

#### Cost Tracking

Monitor LLM costs:

```python theme={null}
from litellm import completion_cost

response = llm.invoke(messages)

## Calculate cost
cost = completion_cost(
    model=model_name,
    prompt_tokens=response.usage.prompt_tokens,
    completion_tokens=response.usage.completion_tokens
)

print(f"Cost: ${cost:.4f}")
```

#### Caching

Cache responses to reduce costs:

```python theme={null}
from litellm import Cache

cache = Cache(
    type="redis",
    host="localhost",
    port=6379,
    ttl=3600  # 1 hour
)

## Cached completion
response = completion(
    model="claude-sonnet-4-5",
    messages=messages,
    cache=cache
)
```

***

### LLM Streaming Response Flow

The following diagram shows how streaming responses flow from the LLM through the server to the client:

```mermaid theme={null}
flowchart TB
    %% ColorBrewer2 Set3 palette - each component type uniquely colored

    subgraph Client["Client"]
        C1[Send Request]
        C5[Receive Streaming Response]
    end

    subgraph API["API Gateway"]
        A1[Route Request]
        A2[Establish SSE Connection]
    end

    subgraph LangGraph["LangGraph Agent"]
        L1[Process Request]
        L2[Stream Tokens Back]
    end

    subgraph LLM["LLM Provider"]
        LLM1[Generate Response]
        LLM2[Stream Token by Token]
    end

    subgraph SSE["Server-Sent Events"]
        S1[Event Stream]
        S2[Push Tokens to Client]
    end

    C1 --> A1
    A1 --> L1
    L1 --> LLM1
    LLM1 --> LLM2
    LLM2 --> L2
    L2 --> A2
    A2 --> S1
    S1 --> S2
    S2 --> C5

    classDef clientStyle fill:#8dd3c7,stroke:#2a9d8f,stroke-width:2px,color:#333
    classDef apiStyle fill:#fdb462,stroke:#e67e22,stroke-width:2px,color:#333
    classDef agentStyle fill:#bebada,stroke:#7e5eb0,stroke-width:2px,color:#333
    classDef llmStyle fill:#fb8072,stroke:#e74c3c,stroke-width:2px,color:#333
    classDef streamStyle fill:#b3de69,stroke:#7cb342,stroke-width:2px,color:#333

    class C1,C5 clientStyle
    class A1,A2 apiStyle
    class L1,L2 agentStyle
    class LLM1,LLM2 llmStyle
    class S1,S2 streamStyle
```

**Flow Description**:

1. **Client Request**: Client sends request to API endpoint
2. **API Routing**: API Gateway routes request to LangGraph agent
3. **Agent Processing**: LangGraph agent processes request and invokes LLM
4. **LLM Generation**: LLM provider generates response token by token
5. **Token Streaming**: Tokens stream back through agent to API
6. **SSE Delivery**: Server-Sent Events push tokens to client in real-time
7. **Client Reception**: Client receives and displays streaming response

<Info>
  **Streaming Benefits**: Real-time responses, lower perceived latency, better user experience, and efficient token-by-token delivery.
</Info>

***

### Configuration Reference

#### Environment Variables

```properties theme={null}
## Provider Selection
LLM_PROVIDER=anthropic|openai|google|azure|bedrock|ollama
MODEL_NAME=model-identifier

## Model Parameters
MODEL_TEMPERATURE=0.0-2.0  # Default: 0.7
MODEL_MAX_TOKENS=1-32000   # Default: 4096
MODEL_TIMEOUT=10-300       # Seconds, Default: 60
MODEL_TOP_P=0.0-1.0        # Default: 1.0

## Fallback
ENABLE_FALLBACK=true|false  # Default: false
FALLBACK_MODELS=model1,model2,model3

## Provider API Keys
ANTHROPIC_API_KEY=sk-ant-...
OPENAI_API_KEY=sk-proj-...
GOOGLE_API_KEY=AIza...
AZURE_API_KEY=...
AWS_ACCESS_KEY_ID=AKIA...
AWS_SECRET_ACCESS_KEY=...

## Provider-Specific
AZURE_API_BASE=https://your-resource.openai.azure.com
AZURE_API_VERSION=2024-02-15-preview
AZURE_DEPLOYMENT_NAME=gpt-4
AWS_REGION_NAME=us-east-1
OLLAMA_API_BASE=http://localhost:11434
```

#### Model IDs

<Tabs>
  <Tab title="Anthropic">
    ```bash theme={null}
    claude-sonnet-4-5             # Latest Sonnet
    claude-opus-4-1               # Opus (extended reasoning)
    claude-haiku-4-5              # Haiku (fast, cost-effective)
    ```
  </Tab>

  <Tab title="OpenAI">
    ```bash theme={null}
    gpt-5.1                         # GPT-5 (flagship)
    gpt-5.1-pro                     # GPT-5 Pro (most capable)
    gpt-5-mini                    # GPT-5 Mini (fast, cost-effective)
    gpt-5.1-nano                    # GPT-5 Nano (smallest, fastest)
    ```
  </Tab>

  <Tab title="Google">
    ```bash theme={null}
    gemini-2.5-flash              # Gemini 2.5 Flash
    gemini-2.5-pro                # Gemini 2.5 Pro
    ```
  </Tab>

  <Tab title="Azure">
    ```bash theme={null}
    azure/gpt-4                   # Requires deployment name
    azure/gpt-35-turbo
    ```
  </Tab>

  <Tab title="Bedrock">
    ```bash theme={null}
    bedrock/anthropic.claude-sonnet-4-5-v2:0
    bedrock/meta.llama3-1-70b-instruct-v1:0
    bedrock/amazon.titan-text-express-v1
    ```
  </Tab>

  <Tab title="Ollama">
    ```bash theme={null}
    ollama/llama3.1:8b            # Llama 3.1 8B
    ollama/llama3.1:70b           # Llama 3.1 70B
    ollama/mistral:7b             # Mistral 7B
    ollama/qwen2.5:7b             # Qwen 2.5 7B
    ollama/deepseek-coder:6.7b    # DeepSeek Coder
    ```
  </Tab>
</Tabs>

***

### Troubleshooting

<AccordionGroup>
  <Accordion title="API key not working">
    ```bash theme={null}
    # Test API key
    curl https://api.anthropic.com/v1/messages \
      -H "x-api-key: $ANTHROPIC_API_KEY" \
      -H "anthropic-version: 2023-06-01" \
      -H "content-type: application/json" \
      -d '{"model":"claude-sonnet-4-5","max_tokens":10,"messages":[{"role":"user","content":"Hi"}]}'

    # Check environment
    echo $ANTHROPIC_API_KEY
    env | grep API_KEY
    ```
  </Accordion>

  <Accordion title="Rate limit errors">
    **Solutions**:

    * Enable fallback models
    * Implement request queuing
    * Increase rate limits (paid plans)
    * Add retry with exponential backoff

    ```python theme={null}
    llm = LLMFactory(
        provider="anthropic",
        enable_fallback=True,
        fallback_models=["gemini-2.5-flash"],
        timeout=120  # Longer timeout
    )
    ```
  </Accordion>

  <Accordion title="Ollama connection failed">
    ```bash theme={null}
    # Check Ollama running
    ollama list

    # Start Ollama
    ollama serve

    # Test connection
    curl http://localhost:11434/api/tags

    # Set correct endpoint
    OLLAMA_API_BASE=http://localhost:11434
    ```
  </Accordion>

  <Accordion title="Slow responses">
    **Optimizations**:

    * Use faster models (Gemini Flash, Claude Haiku)
    * Reduce max\_tokens
    * Increase temperature for faster sampling
    * Enable streaming

    ```python theme={null}
    llm = LLMFactory(
        model_name="gemini-2.5-flash",  # Faster
        max_tokens=1024,  # Lower limit
        temperature=0.9  # Faster sampling
    )
    ```
  </Accordion>
</AccordionGroup>

***

### Next Steps

<CardGroup cols={2}>
  <Card title="Observability" icon="chart-line" href="/guides/observability">
    Track LLM usage and costs
  </Card>

  <Card title="Architecture" icon="sitemap" href="/getting-started/architecture">
    Understand LLM integration
  </Card>

  <Card title="Production Checklist" icon="clipboard-check" href="/deployment/production-checklist">
    LLM production requirements
  </Card>

  <Card title="API Reference" icon="code" href="/api-reference/introduction">
    Agent API endpoints
  </Card>
</CardGroup>

***

<Check>
  **Flexible & Resilient**: Multi-LLM support with automatic fallback ensures high availability and cost optimization!
</Check>