May 2001: PEP 255 was created, Simple Generators
October 2002: Twisted (Python network programming framework uses ioloop and futures) released
May 2005: PEP 342 was created, generator functions are coroutines
September 2012: Python 3.3 with
yield from statement (PEP 380).
December 2012: asyncio (formerly tulip) was proposed as an enhancement of Python in order to add asynchronous I/O support.
October 2013: asyncio 0.1.1 released
October 2013: aiohttp 0.1 released
March 2014: Python 3.4 with asyncio in the standard library
September 2015: Python 3.5 with async/await statements (PEP 492)
New syntax for generators/coroutines
There is a great talk by David Beazley on PyCon 2015 that shows how to use generators to simplify asynchronous code.
My notes on generators and coroutines.
The syntax is proposed for a generator to delegate part of its operations to another generator.
def subgenerator(): for i in (0, 1, 2, 3): yield i def generator(): # For Python < 3.3: # for i in subgenerator(): # yield i yield from subgenerator() if __name__ == '__main__': for i in generator(): print(i) # Python 2.7: # SyntaxError: invalid syntax # Python 3.3: # 0 # 1 # 2 # 3
In case of coroutine:
def subcoroutine(): while True: i = (yield) print(i) def coroutine(): sc = subcoroutine() # For Python < 3.3 # sc.send(None) # while True: # try: # i = (yield) # sc.send(i) # except StopIteration: # pass yield from sc if __name__ == '__main__': c = coroutine() c.send(None) for i in (0, 1, 2, 3): c.send(i)
Subgenerator is allowed to return with a value, and the value is made available to the delegating generator.
def subgenerator(): a = (0, 1, 2, 3) for i in a: yield i # raises # SyntaxError: 'return' with argument inside generator # in Python < 3.3 (must use raise StopIteration() instead) return len(a) def generator(): # subgenerator return value is available value = yield from subgenerator() print(value) if __name__ == '__main__': for i in subgenerator(): print(i) # 0 # 1 # 2 # 3 for i in generator(): print(i) # 0 # 1 # 2 # 3 # 4
asyncio required that all generators meant to be used as a coroutine had to be decorated with asyncio.coroutine.
@asyncio.coroutine def py34_coroutine(): yield from avaitable()
Since Python 3.5 native coroutines are their own completely distinct type, before they were just enhanced generators.
async syntax makes coroutines a native Python language feature, and clearly separates them from generators.
async def py35_coroutine(): await avaitable()
- async def functions are always coroutines, even if they do not contain await expressions
- It is a
SyntaxErrorto have yield or yield from expressions in an async function
- It is a
SyntaxErrorto use await outside of an async def function (like it is a
SyntaxErrorto use yield outside of def function)
- It is a
TypeErrorto pass anything other than an awaitable object to an await expression
- await expressions do not require parentheses around them most of the times
async for, async with
Support for asynchronous calls is limited to expressions where yield is allowed syntactically, limiting the usefulness of syntactic features, such as with and for statements (PEP 492).
The new async with statement lets Python programs perform asynchronous calls when entering and exiting a runtime context, and the new async for statement makes it possible to perform asynchronous calls in iterators. An asynchronous context manager is a context manager that is able to suspend execution in its enter and exit methods.
class AsyncContextManager: async def __aenter__(self): await log('entering context') async def __aexit__(self, exc_type, exc, tb): await log('exiting context')
An asynchronous iterable is able to call asynchronous code in its iter implementation, and asynchronous iterator can call asynchronous code in its next method.
class AsyncIterable: def __aiter__(self): return self async def __anext__(self): data = await self.fetch_data() if data: return data else: raise StopAsyncIteration async def fetch_data(self): ...
Tasks and coroutines
See Python Tasks and coroutines documentation
In debug mode many additional checks are enabled.
import gc gc.set_debug(gc.DEBUG_UNCOLLECTABLE)
When a native coroutine is garbage collected, a RuntimeWarning is raised if it was never awaited on.
aiohttp aiohttp.web aiohttp session aiohttp debugtoolbar aiopg aioredis / asyncio_redis aioes aiozmq aio-s3 ...
from aiohttp import web def index(request): return web.Response(text="Welcome home!") my_web_app = web.Application() my_web_app.router.add_route('GET', '/', index)
Static files serving while development
app.router.add_static('/static', '/path/to/static', name='static')
pip install gunicorn gunicorn -b 0.0.0.0:8000 -k aiohttp.worker.GunicornWebWorker -w 9 -t 60 project.app:app
--reload- reload on code change
-b) - server's socket address (e.g.
-k) - set custom worker subclass (e.g.
-w) - number of workers to use for handling requests (
(2 x $num_cores) + 1)
#!/bin/bash exec .env/bin/gunicorn -b 0.0.0.0:9001 -k aiohttp.worker.GunicornWebWorker -w 2 -t 60 app:app --env=APP_EMAIL=... --env SMTP_PORT=...
How to Deploy Python WSGI Apps Using Gunicorn HTTP Server Behind Nginx
Semaphore is a synchronization tool that can be used to limit the number of coroutines that do something at some point.
sem = asyncio.Semaphore(5) with (yield from sem): page = yield from get(url, compress=True)
- Less overhead
- GIL still there
- Passes arbitrary arguments
- Based on threading
Use for networking, if no async client available.
- More overhead
- No GIL
- Passes only picklable arguments
- Based on multiprocessing
Use for CPU heavy jobs.
uvloop is a fast, drop-in replacement of the built-in asyncio event loop. uvloop is implemented in Cython and uses libuv under the hood.
There is a gunicorn worker for it:
SQLAlchemy Object Relational Tutorial
"Think in coroutines" talk on PyCon Ukraine 2017 by Lukash Langa
«Asyncio stack для веб разработчика» Ігор Давиденко LvivPy#4 at YouTube, slides
«Продвинутый async/await в Python 3.5» Igor Davydenko LvivPy#5 at YouTube
Введение в aiohttp. Андрей Светлов at YouTube
Yury Selivanov - async/await in Python 3.5 and why it is awesome at YouTube
Łukasz Langa - Thinking In Coroutines - PyCon 2016 at YouTube
How the heck does async/await work in Python 3.5? by Brett Cannon
Building Apps with Asyncio on PyCon Ukraine 2017 by Nikolay Novik
PEP 342 - Coroutines via Enhanced Generators
PEP 380 - Syntax for Delegating to a Subgenerator
PEP 492 - Coroutines with async and await syntax
PEP 3156 - Asynchronous IO Support Rebooted: the "asyncio" Module