Async vs Non-Blocking Operations for Responsive FastAPI Endpoints
Introduction
You’ve probably heard this principle before: “don’t put blocking operations on the main thread”.
Until recently, I’d been working only with synchronous python for the best part of 3 years. Using an outdated version of Django will do that to you… So when I finally started working with asynchronous python, I failed, predictably, to apply that principle in my code.
Thankfully I’ve seen the error of my ways. This post is an attempt to pass along (some of) my understanding. On we go! 🤓
In search of responsive endpoints
FastAPI encourages the use of async endpoints. A simple FastAPI server with a basic endpoint that processes files may look like this:
I made the mistake of thinking that this server would be able to handle a new request while the process_files endpoint function is paused on the await upload_to_s3() step. The endpoint function is async after all, and upload_to_s3 is also an async function, even though the sleep operation inside it is not. Subconsciously, I expected Python to just handle whatever was necessary to run the sync operation without blocking the main thread.
To my surprise, the server would block and not process any new requests sent within each 5s span. The logs looked like this:
The server waited for 5 seconds while sleeping and didn’t start the new request until the first one was completed, and similarly remained blocked during the second request. Why did this happen?
It’s because python’s async-await implementation uses an event loop that processes only one task at a time. So when a blocking operation is on that event loop, nothing else will run on the loop until that operation is complete. Wrapping the operation in an async function won’t change that. An async function is not necessarily a non-blocking function.
Unblocking the server… the wrong way ❌
Before I fully understood what I wrote in the paragraph above, my first thought was to simply defer the upload operation until after the client receives the server response. FastAPI’s background tasks is a good way to try to achieve that. A background task can schedule upload_to_s3 to be executed after the server responds to the client, as follows:
background_tasks.add_task accepts the function to be executed as the first argument, followed by the args and/or kwargs that the function should be invoked with. As for how to pass background_tasks into the endpoint function, simply defining a parameter with the BackgroundTasks type is enough. FastAPI does the work of invoking the function with the appropriate argument.
Repeating the test with that code shows that the blocking problem didn’t actually go away 😭.
Even though the request finished processing before the upload started, the server still remained blocked on the upload and didn’t pick up the new request until afterwards. Whuttt?
The blocking task actually prevented the event loop from picking up the next scheduled task (i.e. the new request). More so, the client that made the initial request was actually kept waiting for the server response while the server was occupied with the blocking background task. So, for all intents and purposes, nothing changed 😞.
Okay then, what’s the correct way to go about this?
Unblocking the server… the right way ✅
Either make everything async, or push the sync stuff into a separate thread.
Double down on async 😎
This route requires finding or creating a version of the blocking operation that properly frees up the event loop when the operation is not executing python code. “Properly” means yielding control back to the event loop appropriately. That can be tricky to get right, so I generally prefer to search for a widely used async implementation of an operation rather than rolling my own. A lot of libraries and packages expose both async and sync versions of their available methods, so I’ve mostly not needed to look very far. For the sleep function used in this writeup, asyncio has a drop-in replacement for that:
After testing again with a few simultaneous requests, they all got handled without the server blocking.
Success! 🎉
If you can’t find an async alternative to your blocking operation, and you’re a cool kid feeling up to the task, you can always build your own async implementation of that operation. As a token of appreciation for reading this far, I recommend this guide as a place to start 🙂.
For any number of valid reasons, the async path may not be the right one to follow in a particular context. Not a problem though!
Give up and stay sync 🤷🏾♂️
Python’s concurrency implementation allows the execution of synchronous functions in a separate thread. This frees up the main thread to keep the event loop running unblocked. FastAPI’s concurrency module exposes a run_in_threadpool helper for achieving this. Just like background_tasks.add_task, run_in_threadpool accepts the function to be executed as the first argument, followed by the args and/or kwargs that the function should be invoked with. In Code This Means:
Notice that upload_to_s3 is no longer an async function, and it’s executed in a separate thread using run_in_threadpool. The pre-upload operations are also now performed synchronously in prepare_for_upload_synchronously. Lastly, background_tasks.add_task is still used to ensure that the execution happens after the endpoint returns a response.
Repeating the test of quick-fire requests showed similar behavior to the fully async technique:
The goodies 🛍️ don’t stop there though. There’s an even simpler way to get this same behavior that doesn’t involve calling run_in_threadpool: just pass the regular non-async function to background_tasks.add_task. Internally it figures out if the function is async, and runs it in a thread pool if so. Convenient! 🙌🏾
A note on thread safety
Executing logic in parallel using multiple threads introduces the possibility that more than one thread will access and try to modify the same resources at almost the same time, which can create some serious problems. Personally, I favor using multiple threads in scenarios where the thread operations are sufficiently isolated for my taste.
For example, I recently used the run_in_threadpool method for performing file conversion without saving the converted bytes to the filesystem. They were uploaded straight to s3 instead. The function that did this upload had no other side-effects, and the filenames in s3 were freshly generated UUIDs. Ergo, very small chance of a name collision leading to a mishap. Make a similar assessment for your use case and take any necessary precautions. Tread carefully when working with multiple threads!
Closing thoughts
A quick recap of the essentials I learned:
defining a function with
asyncdoes not make its operations non-blocking; you must make sure the operations are actually non-blockingwhen possible, avoid mixing blocking and non-blocking operations inside the same function; use threads for synchronous blocking work if necessary
when using FastAPI, background tasks are a good way to defer non-essential chunks of work until after an endpoint has responded to the client
Lastly, it’s important to note that this writeup focuses on scenarios in which the background tasks are small enough to be safely executed within the same server as the FastAPI process, yet long enough to make a bad experience for the client hitting the endpoint. This writeup also contains an implicit assumption that custom sequencing of background tasks is not a requirement. Some or all of these may not be true for your use case. For larger workloads or workloads that require customized or consistent scheduling of tasks (eg FIFO task queues), tools like Celery and RQ are more appropriate.
Thanks for your time!
Questions? Feedback? Nice words, or mean ones? Feel free to reach out to @CodeWithOz on all the socials, or on LinkedIn.
