Working with LLM APIs: retries, rate limits, timeouts and streaming

What an API is

API stands for application programming interface. You can think of it as a function that lives on a server. When you call it, the server executes the code inside it and returns something to you: a success message, data from a database, a web page, or, in the case of ChatGPT, Gemini or Claude, text generated by a model.

The code behind an API can be written in Python, JavaScript, Java or any other server-side language, and that doesn't matter to you as the caller. What matters is the contract: what you send, where you send it, and what comes back. That is why APIs are everywhere in software development, and why you can call a model from any language that can send an HTTP request.

What happens when you open a URL

There is an interview question I like for understanding APIs: what happens when you type amazon.com into your browser and press enter? The user's answer is "the website loads." The engineer's answer goes through every step in between, and those steps are the same ones your code goes through when it calls an LLM API.

Take a URL like https://api.example.com/v1/chat?stream=true apart:

• Scheme: https tells the browser which set of rules, or protocol, to follow. HTTP is the hypertext transfer protocol, and the S means the connection is encrypted with TLS (the successor of SSL), so nobody in the middle can read or change what you send. Other schemes exist, like ftp for file transfer and file for files on your own computer.
• Domain: api.example.com is a name for humans. Every server on the internet has an IP address, like 142.250.183.14, and the domain name system (DNS) translates the name into that address, because nobody remembers the numbers.
• Path: /v1/chat tells the server which route, or which function, you want.
• Query: ?stream=true passes extra parameters.

Then your request travels across the network to that server, the server runs the code for that route, and the response travels back. Every one of those steps can be slow or can fail, which is why the rest of this page exists.

Requests and responses

An HTTP request has four parts you need to know:

• Method: GET to read something, POST to send data or ask for something to be created (LLM calls are POST), PUT or PATCH to update, and DELETE to remove.
• Headers: metadata about the request, like the content type and your API key.
• Body: the data you send, usually JSON.
• URL: where it goes.

A response comes back with a status code, headers and a body. The status codes tell you what happened, and you will handle these ones constantly:

Code	Meaning	What to do
200	Success	Use the response
400	Bad request, something in your input is wrong	Fix the request, don't retry
401, 403	Missing or invalid key, or no permission	Check your key and access, don't retry
404	Wrong URL or model name	Fix it, don't retry
408	Request timeout	Retry with backoff
429	Too many requests, you hit a rate limit	Wait, then retry with backoff
500, 502, 503, 504	Something failed on the provider's side	Retry with backoff, then fall back

An API that follows these conventions, with resources at URLs and standard methods and status codes, is called a REST API, and most LLM APIs work this way.

What an LLM API call looks like

Every major LLM API follows the same basic shape, even if the names differ. You send the model name, the messages or the prompt, and settings like temperature and the maximum number of output tokens. You get back the generated content and a usage section that tells you how many input and output tokens the call used, which is what you pay for.

Here is a call with Google's Gen AI SDK for Gemini, which has a free tier that is enough for learning.

import os

from dotenv import load_dotenv
from google import genai

load_dotenv()

client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])

response = client.models.generate_content(
    model=os.environ.get("MODEL_NAME", "gemini-2.5-flash-lite"),
    contents="Explain what an API is in two sentences.",
)

print(response.text)
print(response.usage_metadata.prompt_token_count, response.usage_metadata.candidates_token_count)

The key comes from a .env file, never from the code, which the secrets page explains. Model names change often, so keep the name in configuration too, and check the provider's model list when you start a project.

Log every call

Before you add anything else, log every call. In Build30, the first thing we build on day 1, right after the first call, is a token tracker that writes one line per call to a JSONL file.

Why on day 1? Free tiers have limits, and paid APIs charge by the token, so your cost depends completely on how many tokens you use. If you track tokens from the first call, you see immediately which prompts are expensive, and you build the habit of designing for cost, which is one of the first things a company will ask you about.

A useful log line has these fields:

{"ts": "2026-10-16T10:42:07Z", "model": "gemini-2.5-flash-lite", "feature": "summarise", "input_tokens": 1180, "output_tokens": 212, "latency_ms": 1430, "status": "ok", "cost_usd": 0.00021}

import json
import time
from datetime import datetime, timezone
from pathlib import Path

LOG = Path("logs/token_log.jsonl")


def log_call(model: str, feature: str, input_tokens: int, output_tokens: int, latency_ms: int, status: str, cost_usd: float) -> None:
    LOG.parent.mkdir(parents=True, exist_ok=True)
    record = {
        "ts": datetime.now(timezone.utc).isoformat(timespec="seconds"),
        "model": model,
        "feature": feature,
        "input_tokens": input_tokens,
        "output_tokens": output_tokens,
        "latency_ms": latency_ms,
        "status": status,
        "cost_usd": round(cost_usd, 6),
    }
    with LOG.open("a") as f:
        f.write(json.dumps(record) + "\n")

The feature field matters more than it looks, because later you will want to know which part of your application is spending the money, not only how much in total.

Timeouts

A model call can take anywhere from under a second to several minutes, depending on the model, the length of the output and how busy the provider is. If you don't set a timeout, a stuck request can hang your application, and your user sits in front of a spinner forever.

Some SDKs set a very long default. The OpenAI Python SDK, for example, times out after 10 minutes by default, which is fine for a batch job and far too long for a chat interface.

So decide the timeout per feature:

• For an interactive chat, a short timeout for the first token, and a longer one for the full response if you stream.
• For a background job, a longer timeout, with the job marked failed and retried later.
• For an agent step, a timeout per tool call and a total budget for the whole run.

Retries with exponential backoff

Many failures are temporary: the network drops, the provider is overloaded, or you hit a rate limit for a few seconds. For those, the right response is to wait and try again.

But don't retry immediately and don't retry forever. Exponential backoff means each retry waits longer than the last one, for example 1 second, then 2, then 4, and jitter adds a small random amount to each wait, so that a thousand clients that failed at the same moment don't all retry at the same moment too.

Only retry errors that can succeed on a second try: timeouts, connection errors, 429 and 5xx. Never retry a 400 or a 401, because the same request will fail the same way and you only waste time and quota.

Most official SDKs already retry for you. The OpenAI Python SDK retries connection errors, 408, 409, 429 and 5xx errors twice by default with a short exponential backoff, and you can change that with a max_retries setting. Know what your SDK does, so you don't add your own retries on top and end up retrying nine times.

If you need to write it yourself, it looks like this:

import random
import time


class TemporaryError(Exception):
    pass


def with_retries(call, max_attempts: int = 4, base_delay: float = 1.0, max_delay: float = 20.0):
    for attempt in range(1, max_attempts + 1):
        try:
            return call()
        except TemporaryError:
            if attempt == max_attempts:
                raise
            delay = min(max_delay, base_delay * 2 ** (attempt - 1))
            time.sleep(delay + random.uniform(0, delay / 2))

In a real project, map the SDK's exceptions to "temporary" and "permanent" in one place, and libraries like tenacity give you the same pattern with less code.

When a 429 response includes a Retry-After header, wait at least that long, because the provider is telling you exactly when you can try again.

Rate limits

Every provider limits how much you can use in a period of time. The common limits are requests per minute (RPM), tokens per minute (TPM) and sometimes requests or tokens per day, and they depend on your account tier and the model. Free tiers have much lower limits than paid ones.

When you hit a limit, you get a 429 error. Retrying with backoff handles short bursts, but if your application regularly sends more than the limit allows, retries only make it worse. For that, you need to control how fast you send requests: process large jobs in batches, limit how many calls run at the same time (the async page shows how), and spread non-urgent work over time.

Your own APIs need limits too. When you put an AI feature behind your API, a gateway in front of it usually checks authentication and applies a rate limit per user, so one user, or one bug in a loop, can't spend your whole monthly budget in an afternoon.

Streaming

Without streaming, your application waits until the model has generated the whole response and then shows everything at once. For a long answer, that can be many seconds of nothing. With streaming, the API sends the response token by token as it is generated, usually through server-sent events, and you show each piece as soon as it arrives.

The total time is the same, but the time to first token, the moment the user sees something happening, is much shorter, and for chat interfaces that is what makes the application feel fast.

for chunk in client.models.generate_content_stream(
    model=os.environ.get("MODEL_NAME", "gemini-2.5-flash-lite"),
    contents="Write a short note on API timeouts.",
):
    print(chunk.text, end="", flush=True)

Streaming makes some things harder. You can't validate the whole JSON output until the stream ends, errors can arrive halfway through, and token usage usually comes in the final chunk, so your logging has to wait for it. For background jobs and structured outputs, you usually don't need streaming at all.

Fallbacks

Providers have bad days. A model can be overloaded, a region can go down, or a model can be retired with a few months' notice. If your application depends on a single model, it goes down with it.

A fallback is a second option you switch to when the first one keeps failing after its retries: a different model from the same provider, or a similar model from another provider. Test your prompts on the fallback model too, because a prompt tuned for one model can behave differently on another, and log which model answered each request, so you can see how often the fallback is being used.

Building your own API

So far we have been calling other people's APIs. Soon you will need to build your own, so that a frontend, another service or another person can use what you built.

In Python, there are three common frameworks. Django is a full framework that follows a fixed architecture (model, view, template) and gives you a lot built in, which is useful for big web applications. Flask is a micro framework, which means it doesn't force an architecture on you, so it is quick to start with and easy to understand. FastAPI is also lightweight, is built for APIs, uses type hints to validate requests, supports async code, and generates interactive documentation for your API automatically, which is why I use it for most AI projects.

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()


class Question(BaseModel):
    text: str


@app.get("/health")
def health() -> dict:
    return {"status": "ok"}


@app.post("/ask")
def ask(question: Question) -> dict:
    return {"answer": f"You asked: {question.text}"}

Run it with uvicorn main:app --reload and open http://127.0.0.1:8000/docs to try both endpoints in the browser. Replace the body of ask with your model call, and you have the first version of an AI API.

A checklist for every LLM call in your project

• The API key comes from the environment, never from the code.
• The model name comes from configuration.
• Every call has a timeout that fits its feature.
• Temporary errors are retried with backoff, permanent errors are not.
• You know what retries your SDK already does.
• There is a fallback for the features that can't go down.
• Every call is logged with tokens, latency, status, model and feature.
• Outputs are validated before the rest of your code uses them, with Pydantic.

From the floor

In one AI Engineer HQ session, a builder who came from a product management background told me they were zero in coding and would only be listening if I went at the usual pace. So before building the API, I went back to the beginning and explained what a URL is, part by part, starting from the scheme and the domain. Once the URL made sense, the idea that every URL is an API endpoint that returns something, a profile page, a video or a model's answer, made sense too.

In the Build30 day 1 session, the first LLM client we wrote already had a timeout and a retry setting with 3 attempts, the key loaded from a .env file, and a separate tracker module that logged tokens to a JSONL file. None of it was complicated, and later builds could reuse that client instead of starting from a bare API call.

Related

• Tokens and tokenization
• Async Python for LLM apps
• Environment variables and secrets
• LLMOps

Sources

• Google Gen AI SDK for Python
• OpenAI Python SDK README: retries and timeouts
• HTTP response status codes, MDN
• FastAPI documentation