Async Python

AI applications spend a lot of time waiting.

They wait for a model, a database, a file, or another service to reply. Async Python lets the program use that waiting time to make progress on other work.

It does not make the model faster. It stops one wait from unnecessarily blocking every other independent wait.

Why this matters

Imagine a support assistant that needs a customer’s plan, recent tickets, and a classification of their new message. Those three requests are independent. Waiting for each one before starting the next creates a slower answer.

Async code can start all three waits together while still limiting traffic and handling slow services safely.

The intuition

A coroutineCoroutineA function defined with `async def`. Calling it does not run the body — it returns a coroutine object, which only starts executing once you `await` it or schedule it as a task. A coroutine can pause at an `await` and hand control back to the event loop, which is what lets many coroutines make progress on one thread without any of them blocking the others. is a Python function that can pause and resume. It is written with async def.

Concurrency means several tasks make progress during the same period. It does not mean they execute Python instructions at exactly the same instant.

How it works

An async def function is a coroutineCoroutineA function defined with `async def`. Calling it does not run the body — it returns a coroutine object, which only starts executing once you `await` it or schedule it as a task. A coroutine can pause at an `await` and hand control back to the event loop, which is what lets many coroutines make progress on one thread without any of them blocking the others. function. Calling it does not run its body:

async def call_api(name, seconds):
    await asyncio.sleep(seconds)
    return f"{name}: done"

call_api("plan", 1)   # this alone does nothing — it just builds a coroutine object

You get back a coroutine object — a paused, not-yet-started computation — and nothing has happened yet. To actually run it, you await it, or hand it to the scheduler as a task. await is the key word in this whole chapter: it means “pause me here until this is ready, and let something else run in the meantime.” That “something else” is scheduled by the event loopEvent loopThe single-threaded scheduler that runs your coroutines. It keeps a queue of tasks that are ready to make progress; it runs one until it hits an `await` on something not yet finished, then parks it and runs the next ready task. Nothing here runs in parallel — the loop only ever gives the CPU to one task at a time — but no task sits idle waiting for I/O while another task could be making progress. — the single-threaded loop that keeps a list of coroutines that are ready to make progress, runs one until it hits an await on something not yet finished, parks it, and moves on to the next ready one.

In a normal Python script, you start that loop yourself:

import asyncio

async def main():
    result = await call_api("plan", 1)
    print(result)

asyncio.run(main())     # creates the loop, runs main() until it finishes, closes the loop

To run several coroutines concurrently instead of one after another, hand them to asyncio.gather:

results = await asyncio.gather(
    call_api("plan", 1),
    call_api("history", 1),
    call_api("classify", 1),
)

gather schedules all three as tasks right away, then waits for all of them to finish. Because each one spends its second waiting, not computing, the loop interleaves them — all three waits overlap — so the wall-clock total is close to the slowest one (about 1s), not the sum of all three (3s). Results come back in the order you passed the coroutines in, regardless of which one actually finished first.

A small complete example

import asyncio


async def fetch(name, seconds):
    await asyncio.sleep(seconds)
    return f"{name}: ready"


async def main():
    results = await asyncio.gather(
        fetch("plan", 1),
        fetch("history", 1),
        fetch("classification", 1),
    )
    print(results)


asyncio.run(main())

Common modifications include adding another fetch(...) call, giving calls different delays, or wrapping each call in a timeout.

Real APIs come with limits, so unlimited concurrency is not always safe. A semaphoreSemaphoreA counter, shared across coroutines, that limits how many of them may be inside a section of code at once. `asyncio.Semaphore(n)` lets up to `n` coroutines past `acquire()` at the same time; the rest wait until one of the `n` finishes and releases it. Used to cap concurrent calls to a rate-limited API without giving up concurrency altogether. caps how many coroutines may be past a certain point at once:

sem = asyncio.Semaphore(3)          # at most 3 "inside" at a time

async def call_api(name, seconds):
    async with sem:                 # waits here if 3 are already inside
        await asyncio.sleep(seconds)
        return f"{name}: done"

Ten calls can still all be scheduled together with gather; the semaphore just makes the 4th, 5th, … wait their turn until one of the first 3 finishes and frees a slot. You get bounded, not unlimited, concurrency.

Waiting also needs an upper bound in time, not just in count. asyncio.wait_for(coro, timeout=...) raises asyncio.TimeoutError if coro has not finished in time — and it also cancels coro for you:

try:
    result = await asyncio.wait_for(slow_call(), timeout=2)
except asyncio.TimeoutError:
    result = None

CancellationCancellationAsking a running asyncio task to stop via `task.cancel()`. Cancellation is cooperative, not immediate — it raises `asyncio.CancelledError` inside the task at its next `await` point, so the task keeps running until it reaches one. Code that needs to release a resource on cancellation must do it in a `finally` block, since the exception path still runs it. is cooperative, not immediate: calling .cancel() on a task (or timing it out) raises asyncio.CancelledError inside it at its next await point — the task does not stop mid-instruction, it stops the next time it would have paused anyway. Anything that must run no matter what — closing a connection, releasing a lock — belongs in a finally block, because finally still runs as that CancelledError propagates out.

A worked example

Three calls need durations of 1s, 1s, and 2s. You bound concurrency to 2 with a semaphore, and give each call a 1.5s timeout. Trace it by hand:

Two things happen at once and it is worth separating them. First, C cannot even start its network call until t = 1.0, because both semaphore slots are held by A and B until then — that is the semaphore doing its job, not a bug. Second, once C does start, its own 2.0s of work is longer than its 1.5s timeout budget, so it is cancelled at t = 2.5 regardless of the semaphore. Total wall-clock time for the whole batch: 2.5s — worse than the 2.0s you’d get with no semaphore at all, but with the guarantee that never more than 2 calls are in flight, which is often the actual constraint a real API forces on you.

Try it

Three live cells. The first two use asyncio.sleep to stand in for real network waits — long enough to see the effect, short enough not to be annoying.

First, the hook’s sequential loop next to the concurrent version, so you can see both totals from the same run:

Next, a semaphore capping concurrency at 2, with start/finish timestamps so you can see task C wait its turn:

Finally, a timeout that cancels a slow call — watch the finally block still run:

Production reality

Nearly every LLM client and web framework you will use is built on this. FastAPI request handlers, AsyncOpenAI-style SDK clients, and agent frameworks are all async underneath, for the same reason: a server handling many users at once needs exactly this pattern — lots of I/O-bound waiting, happening concurrently, without spinning up a thread per request.

A few things this chapter’s toy examples hide, which matter the moment you ship:

• Unbounded concurrency gets you rate-limited. asyncio.gather on ten thousand coroutines will fire ten thousand requests at once. Real providers respond with 429 Too Many Requests long before that, and you have gained nothing over a much smaller, well-chosen concurrency limit. A semaphore (or a small pool of workers pulling from a queue) is not an optimisation here — it is what keeps you under the provider’s actual limits.
• Timeouts are not optional. A dependency that hangs without a timeout can hold a connection open indefinitely. Enough hung requests exhaust your server’s connection pool, and one slow dependency takes the whole service down with it — not just the request that triggered it.
• A silently-failing background task is a real failure mode. If you asyncio.create_task(...) something and never await or check on it, an exception inside it can vanish with only a "Task exception was never retrieved" warning in the logs — easy to miss until the behaviour it was supposed to produce just never happens. If you want several calls to run concurrently but let some fail without cancelling the rest, use asyncio.gather(..., return_exceptions=True) and check each result explicitly, rather than firing tasks you never look at again.
• Async does not help CPU-bound work. Tokenising a huge corpus or running a local model’s forward pass does not get faster inside an async def — there is still only one thread, and no amount of waiting-reorganisation helps code that is actually computing the whole time. That needs multiprocessing, or moving the work to a service designed for it (Layer 7 covers serving-level concurrency properly).

That last point has a sharper, more dangerous version: mixing a genuinely blocking call into async code.

Common mistakes

Exercises

import asyncio, time

async def fetch_a():
    await asyncio.sleep(1)
    return 'a'

async def fetch_b():
    await asyncio.sleep(1)
    return 'b'

async def fetch_c():
    await asyncio.sleep(1)
    return 'c'

start = time.perf_counter()
results = await asyncio.gather(fetch_a(), fetch_b(), fetch_c())
print(results)                                        # ['a', 'b', 'c']
print(f'{time.perf_counter() - start:.1f}s')          # ~1.0s, not 3.0s

import asyncio

sem = asyncio.Semaphore(3)

async def call_api(url):
    async with sem:                 # blocks here if 3 are already inside
        return await do_request(url)

async def main(urls):
    return await asyncio.gather(*(call_api(u) for u in urls))

results = await main(urls)

import asyncio

async def fetch_with_timeout(coro, seconds):
    try:
        return await asyncio.wait_for(coro, timeout=seconds)
    except asyncio.TimeoutError:
        return None

Knowledge check

Practice activity

import asyncio


async def fetch(name, delay):
    # Pause without blocking, then return "{name}: ready"
    pass


async def main():
    # Run the three fetch calls concurrently.
    pass


asyncio.run(main())

import asyncio


async def fetch(name, delay):
    await asyncio.sleep(delay)
    return f"{name}: ready"


async def main():
    results = await asyncio.gather(
        fetch("profile", 1),
        fetch("history", 1),
        fetch("policy", 1),
    )
    print(results)


asyncio.run(main())

Key takeaways

Flashcards

One-page sketchnote summary

results = await asyncio.gather(
  fetch("profile"),
  fetch("history"),
)

Resources