From iterators to coroutines and beyond

Presenter Notes

About me

  • I'm Nsukami
  • Freelance (FOSS) software engineer, currently at LogiLab
  • You'll find more about me on my website
  • I'm not on Twitter, I'm on Mastodon

Presenter Notes

Agenda

  1. Python iterators
  2. Python generators
  3. Python coroutines
  4. Beyond?

Presenter Notes

Disclaimers:

  1. I'm still learning about Asyncio.
  2. This talk won't cover the whole topic.
  3. It's fine if you do not understand everything that will be said.
  4. Snippets written using Python3.7

Presenter Notes

Iterable? Iterators?

An iterable is an object capable of returning its members one at a time. An iterable can be used to feed a for loop. The built-in function iter takes an iterable object and returns an iterator. A call to the next method on the iterator gives us the next element if any, and raises a StopIteration exception otherwise.

Presenter Notes

Iterator example:

 1 >>> x = iter("foobar")
 2 >>> next(x)
 3 'f'
 4 >>> next(x)
 5 'o'
 6 >>> next(x)
 7 'o'
 8 >>> next(x)
 9 'b'
10 >>> next(x)
11 'a'
12 >>> next(x)
13 'r'
14 >>> next(x)
15 Traceback (most recent call last):
16   File "<stdin>", line 1, in <module>
17 StopIteration
18 >>>

Presenter Notes

How to build my own iterator?

Iterator objects are required to support two methods, the __iter__ method that returns the iterator object itself, used in for and in statements. And the __next__ method that returns the next value from the iterator. If there is no more items to return then the __next__ method should raise StopIteration exception.

Presenter Notes

 1 class range_5:
 2     """A custom iterator producing integer up to 5."""
 3 
 4     def __init__(self):
 5         self.i = 0
 6 
 7     def __iter__(self):
 8         return self
 9 
10     def __next__(self):
11         if self.i < 5:
12             i = self.i
13             self.i += 1
14             return i
15         else:
16             raise StopIteration()

Presenter Notes

 1 >>> y = range_5()
 2 >>> y.next()
 3 0
 4 >>> y.next()
 5 1
 6 >>> y.next()
 7 2
 8 >>> y.next()
 9 3
10 >>> y.next()
11 4
12 >>> y.next()
13 Traceback (most recent call last):
14   File "<stdin>", line 1, in <module>
15   File "test.py", line 15, in next
16     raise StopIteration()
17 StopIteration
18 >>>

Presenter Notes

Other examples?

Python lists, tuples, dicts and sets are all examples of inbuilt iterators.

Presenter Notes

Generators?

Presenter Notes

If we browse the PEP 255, we can read somewhere:

A function that contains a yield statement is called a generator function. A generator function is an ordinary function object in all respects [...]

Anything that can be done with a generator can also be done with an iterator. However, generators are really nice to express certain ideas in a very clean and concise fashion.

Presenter Notes

1 # The previous iterator, written as a generator
2 def range_5():
3     index = 0
4     while index < 5:
5         yield index
6         index += 1

Presenter Notes

 1 >>> foo = range_5()
 2 >>> foo
 3 <generator object range_5 at 0x7f68abf13840>
 4 >>> next(foo)
 5 0
 6 >>> next(foo)
 7 1
 8 >>> next(foo)
 9 2
10 >>> next(foo)
11 3
12 >>> next(foo)
13 4
14 >>> next(foo)
15 Traceback (most recent call last):
16   File "<stdin>", line 1, in <module>
17 StopIteration
18 >>>

Presenter Notes

A generator function does not return any values. When called, a generator object is returned:

1 >>> y = range_5()
2 >>> y
3 <generator object range_5 at 0x7fb93181c3b8>
4 >>>

Presenter Notes

Our generator object is indeed an iterator, as we can see, it implements the __iter__ and the __next__ methods:

1 >>> print(hasattr(range_5(), '__iter__'))
2 True
3 >>> print(hasattr(range_5(), '__next__'))
4 True

Presenter Notes

Why generators are useful?

Because, sometimes, to deal with a huge amount of data, you just need to deal with one of them at a time, one after the other.

Presenter Notes

What is yield?

Presenter Notes

Yield

Again, somewhere inside the PEP 255, we can read:

If a yield statement is encountered, the state of the function is frozen, and the value of expression_list is returned to .next()'s caller. [...] the next time .next() is invoked, the function can proceed exactly as if the yield statement were just another external call. [...] One can think of yield as causing only a temporary interruption in the executions of a function.

Presenter Notes

Generators send() method

Another PEP, this time, the 342, introduced the send() method on generators.

However, if it were possible to pass values or exceptions *into* a generator at the point where it was suspended, a simple [...] scheduler [...] would let coroutines "call" each other without blocking .

This allowed one to pause a generator, but also to send a value back into the (already started) generator where it paused. Now, your generator can talk to you, but it can also listen yo you.

Presenter Notes

Let's write a range funtion that can jump on demand:

1 def jump_range(up_to):
2     index = 0
3     while index < up_to:
4         jump = yield index
5         if jump is None:
6             jump = 1
7         index += jump

Presenter Notes

 1 >>> y = jump_range(10)
 2 >>> next(y)
 3 0
 4 >>> next(y)
 5 1
 6 >>> y.send(3) # Let's send 3 back into the paused generator
 7 4
 8 >>> next(y)
 9 5
10 >>> y.send(-4)
11 1
12 >>> next(y)
13 2
14 >>> next(y)
15 3
16 >>> next(y)
17 4
18 >>> y.send(5)
19 9
20 >>> next(y)
21 Traceback (most recent call last):
22   File "<stdin>", line 1, in <module>
23 StopIteration
24 >>>

Presenter Notes

Yield from?

The yield from syntax was added in the PEP 380. Being able to refactor a generator was the main reason:

[...] It should be possible to take a section of code containing one or more yield expressions, move it into a separate function [...], and call the new function using a yield from expression.

Simply said, yield from allow you to yield from another generator, while suspending/pausing the one currently running. Consequence: you can now chain generators.

Presenter Notes

Chaining generators:

 1 def foo(up_to=5):
 2     index = 0
 3     while index < up_to:
 4         jump = yield index
 5         if jump is None:
 6             jump = 1
 7         index += jump
 8 
 9 def bar():
10     yield "foobar"
11     yield from foo()
12     yield "qux"

Presenter Notes

 1 >>> y = bar()
 2 >>> next(y) # same as y.send(None)
 3 'foobar'
 4 >>> next(y)
 5 0
 6 >>> next(y)
 7 1
 8 >>> y.send(2)
 9 3
10 >>> next(y)
11 4
12 >>> next(y)
13 'qux'
14 >>> next(y)
15 Traceback (most recent call last):
16   File "<stdin>", line 1, in <module>
17 StopIteration
18 >>>

Presenter Notes

Where are we now?:

  • yield, used like return, allowing an interruption in the execution of a generator.
  • send, allowing us to send a value back into, to resume, a paused generator.

Presenter Notes

Let's have an experiment:

Do you you know how your OS is working?

Presenter Notes

Hands on

Coroutines, Tasks, Scheduler without Asyncio

Presenter Notes

Hands on

Please follow this link

Presenter Notes

Coroutines

Coroutines declared with async/await syntax is the preferred way of writing asyncio applications.

Presenter Notes

Coroutines

  • Similar to generators, they can be started, stopped, restarted.
  • Similar to generators, they need to be driven (We say, you await on coroutines objects).
  • The difference being, generators generate values while coroutines consume values.

Presenter Notes

Coroutines

With Asyncio, you can structure your code as subtasks (as coroutines). Then, Asyncio allows you to schedule those subtasks as you want.

Presenter Notes

TL;DR:

  • Multithreading is an hard topic.
  • Sequential programs waste resources waiting for I/O.
  • Asyncio is suitable for I/O bound applications, not CPU bound ones.
  • Asyncio makes your concurrency visible, you see when there is a context switch.

Asyncio is not:

  • an alternative to thread programming --> see ThreadPoolExecutor
  • an alternative to callback programming. --> see attaching cb to Futures
  • a way to make code faster --> asyncio will always wait for CPU bound tasks

Presenter Notes

Hey, I want to know more \o/

If you want to explore more on this topic, I strongly recommend the following links:

Presenter Notes

Thanks a lot o/

Presenter Notes

Presenter Notes