Every AI API has rate limits. Hit them in development and it is annoying. Hit them in production and your users see errors, partial streams, and timeouts that look random until you inspect the pattern.
The key mistake is treating rate limiting as a single problem. It is usually four different problems hiding behind the same 429:
- requests per minute
- tokens per minute
- concurrent in-flight requests
- account-level or project-level quota exhaustion
If you build for only one of those, the others still bite you.
If you are still in the provider migration stage, read the migration guide first. If you are evaluating whether a gateway helps with fallback and operational overhead, the OpenRouter comparison is the best companion read.
What Rate Limits Actually Mean
Request limits
This is the obvious one. You sent too many requests in a short time window.
Token limits
This is the one teams underestimate. A single long prompt can burn as much budget as many small requests. If you suddenly add a 20 KB system prompt, the request count may look healthy while the token budget is already gone.
Concurrency limits
Some providers and gateways are perfectly happy with your per-minute average until you open fifty streams at once. The rate plan is fine. The burst shape is not.
Quota or balance exhaustion
This often surfaces as a “rate limit” symptom in dashboards because the operational result is the same: calls stop succeeding. But the remediation is different. Backoff is useless if the problem is zero balance.
How Providers Commonly Enforce Limits
The exact numbers change over time, which is why hardcoding a public pricing-table-style chart into your application docs ages badly. The stable pattern is this:
- OpenAI-style providers usually expose request and token headers, and they adjust your ceiling based on account history or usage tier.
- Anthropic-style providers usually enforce both minute-level throughput and broader project limits, especially on high-end models.
- Google-style providers often separate free-tier behavior from paid-tier behavior and may vary limits sharply by model family.
- Aggregators add one more limit layer on top of upstream constraints, but in return they can route to other channels when one upstream is temporarily saturated.
Treat provider limits as live configuration, not constants.
Reading Rate Limit Headers
All major providers return rate limit information in response headers:
x-ratelimit-limit-requests: 500
x-ratelimit-remaining-requests: 499
x-ratelimit-reset-requests: 60s
x-ratelimit-limit-tokens: 200000
x-ratelimit-remaining-tokens: 199500
Use these headers proactively. Don't wait for a 429 error to slow down.
The operational habit you want is simple:
- Log the headers on success, not just on failure.
- Alert when remaining capacity falls below a threshold.
- Shape traffic before the next request crosses the line.
If you only look at headers after a failure, you are already behind.
Building Retry Logic
The Wrong Way
# Don't do this
import time
def call_api(messages):
while True:
try:
return client.chat.completions.create(
model="gpt-4.1",
messages=messages
)
except Exception:
time.sleep(1) # Fixed delay, no backoff, catches everything
Problems: no exponential backoff, catches non-retryable errors, no max retry limit, no jitter.
The Right Way
import time
import random
from openai import RateLimitError, APIError, APIConnectionError
def call_with_retry(messages, model="gpt-4.1", max_retries=3):
"""Retry with exponential backoff and jitter."""
for attempt in range(max_retries + 1):
try:
return client.chat.completions.create(
model=model,
messages=messages
)
except RateLimitError as e:
if attempt == max_retries:
raise
# Use retry_after from response if available
wait = getattr(e, 'retry_after', None)
if wait is None:
wait = (2 ** attempt) + random.uniform(0, 1)
print(f"Rate limited. Waiting {wait:.1f}s (attempt {attempt + 1})")
time.sleep(wait)
except APIConnectionError:
if attempt == max_retries:
raise
wait = (2 ** attempt) + random.uniform(0, 1)
time.sleep(wait)
except APIError as e:
# Don't retry client errors (400, 401, 403)
if e.status_code and 400 <= e.status_code < 500:
raise
if attempt == max_retries:
raise
time.sleep((2 ** attempt) + random.uniform(0, 1))
Key principles:
- Exponential backoff: 1s, 2s, 4s, 8s
- Jitter: random 0-1s added to prevent thundering herd
- Respect
retry_afterheader when provided - Don't retry client errors (bad request, auth failure)
- Set a max retry count
Two extra production rules matter:
- never retry forever on a streaming endpoint
- never retry a request that is already tied to user-visible side effects unless the operation is idempotent
Chat completions are usually safe to retry. Tool-triggered side effects often are not.
Async Version
import asyncio
import random
from openai import AsyncOpenAI, RateLimitError
async_client = AsyncOpenAI(
api_key="sk-lemon-xxx",
base_url="https://api.lemondata.cc/v1"
)
async def call_with_retry_async(messages, model="gpt-4.1", max_retries=3):
for attempt in range(max_retries + 1):
try:
return await async_client.chat.completions.create(
model=model,
messages=messages
)
except RateLimitError:
if attempt == max_retries:
raise
wait = (2 ** attempt) + random.uniform(0, 1)
await asyncio.sleep(wait)
Shape Traffic Before It Becomes a Retry Storm
Retry logic is only half the solution. If your upstream is already overloaded, retries can turn one burst into a self-inflicted outage.
Three controls make the difference:
1. Queue by tenant or user
If one customer starts a massive batch job, you do not want every other customer to inherit the blast radius.
2. Cap concurrent streams
Streaming endpoints are easy to underestimate because each request “looks” cheap while it stays open for a long time.
3. Trim prompts before they hit the wire
Token limits are often the real ceiling. A prompt that is twice as long cuts the safe throughput roughly in half.
Client-Side Token Bucket
For high-throughput applications, implement client-side rate limiting to avoid hitting server limits:
import time
import asyncio
class TokenBucket:
def __init__(self, rate: float, capacity: int):
self.rate = rate # tokens per second
self.capacity = capacity # max burst size
self.tokens = capacity
self.last_refill = time.monotonic()
async def acquire(self, tokens: int = 1):
while True:
now = time.monotonic()
elapsed = now - self.last_refill
self.tokens = min(
self.capacity,
self.tokens + elapsed * self.rate
)
self.last_refill = now
if self.tokens >= tokens:
self.tokens -= tokens
return
# Wait for enough tokens
wait = (tokens - self.tokens) / self.rate
await asyncio.sleep(wait)
# 500 requests per minute = ~8.3 per second
limiter = TokenBucket(rate=8.0, capacity=20)
async def rate_limited_call(messages, model="gpt-4.1"):
await limiter.acquire()
return await async_client.chat.completions.create(
model=model,
messages=messages
)
Token buckets are good when you know your ceiling. They are even better when you tune them from observed header data instead of a hardcoded guess.
Model Fallback on Rate Limits
When your primary model is rate-limited, fall back to an alternative:
FALLBACK_CHAIN = [
"claude-sonnet-4-6",
"gpt-4.1",
"gpt-4.1-mini",
]
async def call_with_fallback(messages):
for model in FALLBACK_CHAIN:
try:
return await async_client.chat.completions.create(
model=model,
messages=messages
)
except RateLimitError:
continue
raise Exception("All models rate limited")
This is where model gateways help, but only if the fallback is deliberate. Do not silently jump from a premium reasoning model to a tiny budget model without thinking about the user impact.
A reasonable fallback chain is:
- same provider, smaller sibling model
- equivalent model family from another provider
- only then a cheaper or lower-context model
If you mix “fallback for availability” with “fallback for cost” in one step, debugging gets messy fast.
Monitoring Rate Limit Usage
Track your rate limit consumption to catch issues before they affect users:
import logging
def log_rate_limits(response):
headers = response.headers
remaining = headers.get("x-ratelimit-remaining-requests")
limit = headers.get("x-ratelimit-limit-requests")
if remaining and int(remaining) < int(limit) * 0.1:
logging.warning(
f"Rate limit warning: {remaining}/{limit} requests remaining"
)
Set alerts when remaining capacity drops below 10%. This gives you time to implement throttling before users see 429 errors.
You should also log:
- request ID
- model
- input size estimate
- stream duration
- retry count
- final outcome (
success,rate_limited,network_error,quota_exhausted)
Without those fields, rate-limit incidents become guesswork.
A Simple Production Checklist
Before you call your chatbot or agent “rate-limit safe,” verify these five items:
- A bounded retry policy exists for both sync and async paths.
- You log rate-limit headers on successful responses.
- Per-user or per-tenant shaping exists before the upstream call.
- At least one validated fallback model exists.
- The frontend gets a clean error state instead of a hung stream.
If you are building the full application rather than just the retry primitive, the one-key chatbot guide shows how these pieces fit into a real FastAPI service.
Summary
Rate limiting is not a corner case. It is a normal operating condition for any AI product with real usage. The teams that handle it well do not have magical higher limits. They treat throughput, retries, and fallback as part of application design from the start.
Create an API key at LemonData, test your retry path before production traffic, and build for the next 429 before it arrives.
| Strategy | When to Use |
|---|---|
| Exponential backoff | Always (baseline) |
| Client-side rate limiter | High-throughput apps (>100 RPM) |
| Model fallback | Production apps with SLA requirements |
| Proactive monitoring | Any production deployment |
| Batch API | Non-real-time workloads |
The goal isn't to avoid rate limits entirely. It's to handle them gracefully so your users never notice.
Build resilient AI applications: lemondata.cc provides multi-channel routing that automatically handles upstream rate limits. One API key, 300+ models.