In my last post, we talked about key cache invalidation — the silent production killer that turns your gateway into a 502 factory. Today I want to talk about something equally dangerous but far more insidious: cost traps.
These aren't bugs. They're not crashes. Your gateway runs fine. Your users are happy. Then finance sends you a Slack message: "Why did our OpenAI bill jump 4x last month?"
I've been running LiteLLM Proxy in production for multiple teams across three companies. Here are the five cost traps I've personally been burned by — each with the config that would have saved me thousands of dollars.
Trap #1: The Retry Spiral — When num_retries=3 Actually Means 15
The Problem
LiteLLM's retry logic is smart. Too smart. When a request fails, it retries. When the retried request hits a fallback model and that fails, it retries again. If you've configured a fallback chain of 5 models with 3 retries each, a single user request can trigger up to 15 upstream API calls — and you pay for every single one, including the ones that errored out after consuming tokens.
Why It Happens
The default num_retries in LiteLLM is 3. Most teams set it and forget it. But retries multiply across your fallback chain. Here's the math:
Request → Model A fails → Retry A (1) → Retry A (2) → Retry A (3)
→ Fallback to Model B → Fails → Retry B (1) → Retry B (2) → Retry B (3)
→ Fallback to Model C → Fails → Retry C (1) → Retry C (2) → Retry C (3)
Total upstream calls: 9 retries + 3 initial = 12 billable calls for 1 user request
If Model C is GPT-4o and each retry consumes 2K input tokens before timing out, that's 24K tokens on a single failed request.
The Fix
Cap total attempts across the entire chain, not just per-model:
# litellm_config.yaml
litellm_settings:
num_retries: 2 # per-model retries
max_fallbacks: 2 # hard cap on fallback chain depth
retry_after: 5 # respect 429 Retry-After headers
allowed_fails: 3 # circuit breaker: after 3 fails, stop entirely
model_list:
- model_name: gpt-4o
litellm_params:
model: gpt-4o
max_retries: 2 # override: fewer retries on expensive models
- model_name: gpt-4o-fallback
litellm_params:
model: gpt-4o-mini # cheap fallback, not another expensive model
The key insight: your fallback should be cheaper than your primary, not equally expensive. If GPT-4o fails, fall back to GPT-4o-mini, not to Claude Opus.
Trap #2: Fallback Chains That Funnel Money Into Premium Models
The Problem
This is the trap that cost me $2,300 in a single weekend. A well-meaning engineer configured a fallback chain that looked like this:
# THE EXPENSIVE WAY — do not do this
router_settings:
fallbacks:
- "gpt-4o-mini": ["gpt-4o"]
- "gpt-4o": ["claude-3-5-sonnet"]
- "claude-3-5-sonnet": ["claude-3-opus"]
The logic seemed sound: "If the cheap model fails, try the better one." But here's what actually happened: GPT-4o-mini was rate-limited during a traffic spike (429s everywhere), so every single request fell through to GPT-4o and then to Claude 3.5 Sonnet. For 6 hours, we were running 100% of our traffic on the most expensive models in the chain.
Why It Happens
Rate limits are per-model, not per-gateway. When you hit OpenAI's TPM limit on gpt-4o-mini, LiteLLM dutifully falls back. But if the traffic spike is caused by overall volume (not a model-specific outage), the fallback model gets the same volume that caused the 429 in the first place. You're not solving the problem — you're just paying 10x more to have it on a different model.
The Fix
Structure fallbacks by cost tier, not by capability tier:
# THE SMART WAY — fallback within price tier, not up
router_settings:
fallbacks:
# Tier 1: Cheap models (fallback to other cheap models)
- "gpt-4o-mini": ["gemini-1.5-flash", "claude-3-haiku"]
# Tier 2: Mid-tier models (fallback to other mid-tier)
- "gpt-4o": ["claude-3-5-sonnet", "gemini-1.5-pro"]
# NEVER fall up from cheap to expensive
# If all cheap models fail, return an error, don't escalate
# Add a cooldown so the same model isn't retried immediately
cooldown_time: 60
Also add alerting. If your fallback rate exceeds 5% of total traffic, something is structurally wrong:
# prometheus metric in your LiteLLM custom callback
from litellm.integrations.custom_logger import CustomLogger
import litellm
class FallbackAlertLogger(CustomLogger):
def log_pre_api_call(self, model, messages, kwargs):
if kwargs.get("metadata", {}).get("fallback_idx", 0) > 0:
# This is a fallback call, not the primary
self.fallback_counter.inc()
# Alert if fallback rate > 5%
if self.fallback_counter._value.get() / self.total_counter._value.get() > 0.05:
self.alert_webhook.send(
"⚠️ Fallback rate >5% — check rate limits on primary models"
)
Trap #3: Zero Caching = Paying for the Same Answer 1,000 Times
The Problem
Most teams don't enable LiteLLM's built-in caching because "our prompts are dynamic." But in practice, a huge percentage of your traffic is near-identical: system prompts are the same, the first 500 tokens of user messages are often boilerplate, and many users ask the exact same questions.
I audited one team's traffic and found that 34% of their requests were exact duplicates of requests made in the last hour. They were paying OpenAI ~$400/day for identical completions.
Why It Happens
LiteLLM has Redis caching built in. But it's disabled by default, and the documentation buries it under "Advanced Settings." Most engineers set up the proxy, test it, ship it, and never circle back.
The Fix
Enable Redis caching with a sensible TTL. This is a 30-second config change that can cut your bill by 30-50%:
litellm_settings:
cache: true
cache_params:
type: "redis"
host: "your-redis-host"
port: 6379
namespace: "litellm_cache"
# Cache settings
ttl: 3600 # 1 hour for exact matches
# For semantic caching (similar but not identical prompts):
# semantic_cache: true
# similarity_threshold: 0.8
# Cache based on messages content, not just the full request
cache_key_include_models: true # don't share cache across models
model_list:
- model_name: gpt-4o
litellm_params:
model: gpt-4o
cache: true # enable per-model
For even bigger savings, use prompt caching with providers that support it (Claude, GPT-4o). LiteLLM supports this natively:
import litellm
# Enable prompt caching for Claude
response = litellm.completion(
model="claude-3-5-sonnet",
messages=[
{"role": "user", "content": [
{"type": "text", "text": "<long_system_prompt>", "cache_control": {"type": "ephemeral"}},
{"type": "text", "text": user_input}
]}
]
)
# Claude charges 90% less for cached input tokens
Real numbers from my audit: After enabling Redis cache with a 1-hour TTL, that team went from $400/day to $180/day. A 55% reduction for a config change that took less than a minute.
Trap #4: No Per-Key Budget Limits — One Runaway Loop Can Bankrupt You
The Problem
An intern pushes a while True loop to a staging environment. It doesn't crash — it just calls your gateway 4,000 times per minute with a 4K-token prompt. By the time PagerDuty fires, you've spent $847 in 12 minutes.
This isn't hypothetical. This is a Tuesday.
Why It Happens
LiteLLM's default configuration has no budget enforcement. The max_budget field exists but most teams never configure it because they're focused on getting the gateway working, not on constraining it.
The Fix
Set budgets at three levels: per-key, per-team, and global:
# Per-virtual-key budget (when creating keys via /key/generate)
# This is your first line of defense
litellm_settings:
# Global budget — emergency brake
max_budget: 500 # $500/day global cap
budget_duration: "1d"
# Rate limiting
rpm_limit: 1000 # requests per minute, global
general_settings:
master_key: sk-1234
database_url: "postgresql://..."
# Enable budget tracking
alerting: ["slack"]
alerting_threshold: 0.8 # alert at 80% of budget
When creating virtual keys for teams or individual developers:
# Create a key with a $50 daily budget and 100 RPM
curl -X POST http://localhost:4000/key/generate \
-H "Authorization: Bearer sk-1234" \
-H "Content-Type: application/json" \
-d '{
"max_budget": 50,
"budget_duration": "1d",
"rpm_limit": 100,
"tpm_limit": 50000,
"models": ["gpt-4o-mini", "gpt-4o"],
"metadata": {"team": "frontend"}
}'
And set up a webhook to catch budget breaches:
# In your LiteLLM proxy config
litellm_settings:
proxy_budget_respecting_alerting:
- webhook_url: "https://hooks.slack.com/services/..."
# This fires BEFORE the request is sent when a key is over budget
# LiteLLM will return a 429 to the client, not forward to the provider
The intern's loop? With a $50/day key budget and 100 RPM limit, it would have been throttled after 100 calls and blocked entirely after $50. Total damage: about $0.80.
Trap #5: The Streaming Tax — You're Paying for Tokens You Never See
The Problem
Streaming mode (stream=True) is great for UX. Users see tokens appear in real-time. But here's what most teams don't realize: when a streaming request is interrupted mid-stream, you still pay for the entire generation.
User starts a request → GPT-4o begins streaming a 2,000-token response → user navigates away after 50 tokens → the client connection drops → but the upstream API call completes fully → you pay for all 2,000 tokens.
At scale, this is devastating. I've seen teams where 23% of their token spend was on tokens that no user ever saw because the client disconnected early.
Why It Happens
LiteLLM (and most API gateways) doesn't automatically cancel the upstream request when the client disconnects during streaming. The gateway is acting as a proxy — it's happily receiving tokens from OpenAI and trying to forward them, even though nobody's listening.
The Fix
Enable client disconnect detection and upstream cancellation:
litellm_settings:
# Cancel upstream request when client disconnects during streaming
stream_options:
include_usage: true # get token counts in the final chunk
# Custom callback to track abandoned streams
callbacks: stream_cost_logger
router_settings:
# Close upstream connection when client disconnects
streaming_client_disconnect: true # LiteLLM 1.40+
If you're on an older version or need more control, add a custom middleware:
from litellm.proxy.custom_proxy_admin_logic import CustomProxyAdminLogic
class StreamCancellationMiddleware(CustomProxyAdminLogic):
async def async_pre_call(self, user_api_key_dict, cache, data, call_type):
if data.get("stream"):
# Mark the start time
data["metadata"] = data.get("metadata", {})
data["metadata"]["stream_start_time"] = time.time()
return data
async def async_log_stream_event(self, logging_obj, response, start_time, end_time):
# Log how many tokens were actually consumed vs delivered
if hasattr(response, 'usage'):
total_tokens = response.usage.get('completion_tokens', 0)
# If stream ended early (client disconnect), log it
if logging_obj.stream_connection_broken:
self.metrics.abandoned_stream_tokens.inc(total_tokens)
self.alert(
f"Abandoned stream: {total_tokens} tokens paid but undelivered"
)
Also, consider setting max_tokens conservatively for streaming endpoints:
model_list:
- model_name: gpt-4o-stream
litellm_params:
model: gpt-4o
max_tokens: 1000 # cap generation length
stream: true
stream_options:
include_usage: true
After implementing stream cancellation, that 23% wasted spend dropped to under 2%.
The Pattern Behind All Five Traps
Notice the theme: every one of these traps is a sensible default that becomes dangerous at scale. Retries are good — until they multiply across fallbacks. Fallbacks are good — until they funnel traffic to premium models. Caching is optional — until it's costing you 30% of your bill.
The fix is never "disable the feature." It's always "add constraints." Budgets, caps, cooldowns, TTLs. The gateway works for you, not the other way around.
If you're deploying LiteLLM or any AI API gateway, do a quick audit:
- What's your effective retry multiplier? (num_retries × fallback_depth)
- Does your fallback chain ever fall UP in price?
- Is caching enabled? (If you can't answer this in 5 seconds, it's probably not)
- Does every API key have a budget cap?
- What percentage of your streaming tokens are actually delivered?
If any of these questions made you nervous, you might want to check out the AI API Gateway Pitfall Map — a one-page production survival guide I put together that covers these traps (and a few more) in a format you can print and pin above your desk. It's the checklist I wish I'd had before I learned these lessons the expensive way.
Have you hit any of these traps in production? Or found others I missed? Drop a comment — I'm collecting war stories for a follow-up post.
Tags: #litellm #ai #devops #costoptimization
🎁 Free: AI API Gateway Pre-Deployment Checklist
Before you ship, run through this 43-point checklist covering auth, cost control, caching, fallbacks, security, monitoring, and production readiness. It's free — grab it here:
👉 Free Pre-Deployment Checklist (PDF)
And if you want the full pitfall map with detailed fixes for each trap above, that's here: AI API Gateway Pitfall Map ($9)
