APIs & services

An application programming interface, or API, is a set of rules that lets two programs talk. One program sends a request asking for something. Another program sends back a response with the result or a clear error.

Why this matters

AI applications rarely contain only a model. A website may call an AI service, an AI service may call a model provider, and both may read data from other systems. APIs are the contracts that keep these programs working together.

They also protect real users. A careful API rejects invalid data, limits overload, retries temporary failures safely, and avoids charging or processing the same request twice.

The intuition

A client is the program making the request. A server is the program receiving it. An endpoint is one available destination, such as POST /orders. A schema describes the fields and data types that are allowed.

A status code is a three-digit result label. For example, 200 means the request succeeded, 422 means the submitted data did not match the schema, and 429 means the client sent too many requests.

Read the diagram from left to right: the client sends JSON, Pydantic checks it, the handler performs the work, and FastAPI turns the result into an HTTP response. Essential information is repeated in the caption and surrounding text, so the flow does not depend on colour alone.

How it works

FastAPI + Pydantic v2 are how the contract in the diagram above gets enforced without you writing the checking code yourself. You declare the shape you expect as a class, and FastAPI wires it into the request path automatically:

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()

class Order(BaseModel):
    item: str
    quantity: int
    price: float

@app.post("/orders")
async def create_order(order: Order):
    return {"received": order.model_dump(), "total": round(order.quantity * order.price, 2)}

order: Order is doing two jobs at once: it tells FastAPI to parse the request body as JSON, validate every field against the Order schema, and only then call create_order with a real, type-correct Order instance — and it tells FastAPI’s auto-generated OpenAPI docs exactly what this endpoint expects, for free. If quantity arrives as "two" instead of 2, none of that code runs; FastAPI has already returned a 422 with a message naming the exact field and the exact problem.

Auth: OAuth2 and JWTs. Most real APIs need to know who is asking, not just what they’re asking for. The common shape: a client logs in once against an auth endpoint and gets back a token — commonly a JWTJWT (JSON Web Token)A compact, signed token — three base64url segments joined by dots (header.payload.signature) — that a server issues after login and a client sends back on later requests to prove who it is, without the server storing session state. The payload (claims, e.g. user id, role, expiry) is only base64-encoded, not encrypted, so it is readable by anyone who has the token; the signature only proves the payload hasn't been tampered with, and only holds if the server's signing secret stays secret. Never put a real password or secret inside the payload. — then attaches it to every subsequent request as Authorization: Bearer <token>, instead of re-sending a password on every call. FastAPI’s OAuth2PasswordBearer dependency handles the mechanical part of this — pulling that header out and making it available to your code — but validating the token is still your job.

A JWT is three base64url-encoded segments joined by dots — header.payload.signature — and the important, easy-to-miss fact is in the middle segment:

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ1c2VyLTQyIiwicm9sZSI6ImN1c3RvbWVyIn0.6QdXq_OV6av2eCHGSrF4ETy0YwmRkG4fKcLVHMNid9E
└──────────── header ────────────┘ └──────────────── payload ────────────────┘ └──────────── signature ────────────┘

The payload is encoded, not encrypted — anyone holding the token can decode the middle segment and read the claims inside it with nothing more than a base64 decoder, no secret required. What the signature (the third segment) actually guarantees is narrower than people assume: it proves the payload hasn’t been altered since the server signed it, and only if the server’s signing secret has stayed secret. It says nothing about whether the payload should be trusted for a particular action — that’s a separate check your code still has to make (e.g. does this token’s role claim actually permit this operation?). Never put a real secret or password inside a JWT payload — it is exactly as readable as if you’d sent it in plain text.

Rate limiting protects a service from being asked to do more work than it can handle, whether that’s abuse or just a legitimate traffic spike. A common, simple approach is the token bucketToken bucketA rate-limiting algorithm where a bucket holds up to N tokens, refills at a fixed rate (e.g. 5 tokens/second), and each request must take one token to proceed. If the bucket is empty, the request is rejected (typically with `429 Too Many Requests`) until refill catches up. Unlike a simple fixed-window counter, it allows short bursts up to the bucket's capacity while still capping the long-run average rate.: a bucket holds up to capacity tokens, refills at a fixed rate, and every request has to take one token to proceed. Run dry, and the server replies 429 Too Many Requests — usually with a Retry-After header — until the bucket refills. The useful property, compared to a plain “N requests per second” counter, is that it still allows a short burst up to the bucket’s capacity while capping the average rate over time — a real client doesn’t call at a perfectly even cadence, and a token bucket doesn’t punish it for that.

Retries with exponential backoff. A request can fail for reasons that have nothing to do with whether it was well-formed: the server is momentarily overloaded, a network hop drops a packet, a connection times out. The correct client response to a transient failure is usually to retry — but retrying immediately, in a tight loop, is close to the worst thing a client can do to a struggling server, because it adds load at exactly the moment the server has the least of it to spare. Exponential backoffExponential backoffA client retry strategy where the wait between attempts grows geometrically (e.g. 0.5s, 1s, 2s, 4s, 8s) instead of retrying immediately, so a struggling server gets progressively more room to recover instead of being hit by an instant flood of retries. Real implementations add jitter (a small random amount added to each delay) so that many clients who failed at the same moment don't all retry in lockstep and re-create the exact spike that caused the failure. waits progressively longer between attempts — 0.5s, 1s, 2s, 4s, ... — instead of retrying instantly, and real implementations add jitter, a random amount added to (or substituted for) each delay, so that many clients who failed at the same moment don’t all retry in lockstep and recreate the exact spike that caused the failure.

Idempotency is what makes a retry safe rather than merely polite. An operation is idempotent if doing it twice has the same effect as doing it once — PUT /users/42 {"name": "Alex"} is idempotent by definition: setting the name to “Alex” a second time changes nothing further. POST /charge {"amount": 50} is not: doing it twice charges the card twice. For non-idempotent operations, the fix is an idempotency keyIdempotency keyA unique value the client generates and attaches to a request (e.g. an `Idempotency-Key` header), so the server can recognize a retried request as the same one instead of a new one. The server stores the first response against the key; if the same key arrives again — because a retry followed a timeout, not an actual failure — it returns the cached response instead of processing the request twice. Without one, retrying a non-idempotent request (like "charge this card") can repeat its side effect. — a value the client generates once and sends on the original request and every retry of it. The server, on first seeing that key, does the work and stores the result against it; on seeing it again, it returns the stored result instead of doing the work a second time.

Streaming: polling, SSE, and WebSockets. Not every response is ready instantly, and not every client interaction fits a single request/response pair. Three shapes cover almost every case:

Server-Sent EventsServer-Sent Events (SSE)A one-way streaming protocol over a plain HTTP response (`Content-Type: text/event-stream`) where the server keeps the connection open and pushes `data: ...` lines as they become available, instead of the client waiting for one complete response. It only flows server-to-client — no messages back on the same connection — which is exactly the shape of an LLM streaming its answer token by token. Simpler than a WebSocket when the client never needs to send more than the original request. (SSE) run over a plain HTTP response with Content-Type: text/event-stream — the server keeps the connection open and writes data: ...\n\n lines as they become available, and the client reads them as a stream instead of waiting for one complete body. It only flows one way, server to client, which is precisely the shape of an LLM producing an answer token by token: the client never needs to send anything back mid-stream. A WebSocket, by contrast, is a genuinely bidirectional connection — needed when the client has to send more messages on the same connection during the exchange, like a chat interface where either side can speak at any time.

A worked example

Trace backoff_delay(attempt) = min(cap, base * 2^(attempt - 1)) with base=0.5s, cap=4.0s, across five attempts, and see where jitter changes the picture:

Without jitter, a thousand clients that all failed at t=0 because a server briefly choked would all retry again at exactly t=0.5s, then all again at exactly t=1.5s, and so on — a thousand-request spike, on a schedule, hitting a server that just proved it couldn’t handle the load a moment ago. With jitter, those thousand retries land at any point across each window instead of one instant, which is the entire reason jitter is worth the extra line of code.

Now trace the double-charge bug itself as a small ledger. The client sends POST /charge with Idempotency-Key: order-abc; the response is lost in transit; the client retries with the same key:

Drop the idempotency key from that trace and the second row reads “not present → server charges the card again, stores {charge_id: 2}” — two real charges for one purchase, exactly the support ticket this chapter opened with.

Try it

First, a real FastAPI service — Pydantic validation, a rejected request, and a real SSE stream — running end to end with no server anywhere except an in-memory ASGI transport (httpx.ASGITransport), which calls the app directly without opening a real socket:

Next, a JWT issued, read without a secret, verified with one, and rejected after tampering:

Now the bug from the top of this chapter, reproduced directly — the same retry logic, run once with an idempotency key and once without:

Last, a token bucket rate limiter — tune the capacity and refill rate and watch when 429 starts:

Production reality

Every FastAPI service the fixtures in this course describe — and every real LLM API you’ll call in Layer 4 — is built from exactly these pieces, under real constraints:

• Idempotency keys cost storage and a design decision, not just a header. Stripe’s real idempotency implementation (see Resources) keeps a stored key for 24 hours and explicitly documents what happens if you reuse a key with a different request body — it’s treated as a client bug, not honoured, because the whole guarantee depends on the key meaning “the same request,” not “some request.”
• A JWT’s short expiry is doing real work. Because a JWT can’t be revoked the way a server-side session can — the server has no record of it beyond the fact that its signature checks out — a leaked token stays valid until it expires. Real systems keep expiries short (minutes to a few hours) and pair a long-lived JWT-issuing flow with a separate, revocable refresh token, precisely so a leak has a small blast radius.
• 429 without Retry-After just moves the guessing to the client. A well-behaved rate limiter tells the client how long to wait, not just that it should; a well-behaved client reads that header instead of guessing its own backoff schedule against a server it knows nothing about.
• Retries interact badly with things that aren’t idempotent, in ways that are easy to ship by accident. A generic “retry on any 5xx” middleware, bolted onto a client library after the fact, will retry a POST exactly as happily as a GET unless someone explicitly taught it the difference — the bug in this chapter’s hook is usually not a missing feature, it’s a default nobody thought to turn off for the one endpoint where it matters.
• Streaming changes your error handling, not just your response type. A normal request either succeeds or fails as a whole; a stream can fail after it’s already sent the client half an answer. Production SSE endpoints need a defined way to signal “the rest of this didn’t arrive” mid-stream, not just a clean success/failure split.

Common mistakes

Exercises

import random

def backoff_delay(attempt, base=0.5, cap=8.0):
    return min(cap, base * 2 ** (attempt - 1))

def backoff_delay_jittered(attempt, base=0.5, cap=8.0):
    return random.uniform(0, backoff_delay(attempt, base, cap))

Knowledge check

Practice activity

def backoff_delay(attempt: int, base: float = 0.5, cap: float = 4.0) -> float:
    # Replace this line.
    return 0.0


for attempt in range(1, 7):
    print(backoff_delay(attempt))

def backoff_delay(attempt: int, base: float = 0.5, cap: float = 4.0) -> float:
    delay = base * 2 ** (attempt - 1)
    return min(delay, cap)


for attempt in range(1, 7):
    print(backoff_delay(attempt))

Key takeaways

Flashcards

One-page sketchnote summary

@app.post("/orders")
async def create_order(order: Order):
  return {"total": order.quantity * order.price}

Resources