Lecture 19 Advanced Python Topics I

1. Iterators

Iterators are everywhere in Python. They are elegantly implemented within for loops, comprehensions, generators etc. but hidden in plain sight. Iterator in Python is simply an object that can be iterated upon. An object which will return data, one element at a time. Technically speaking, Python iterator object must implement two special methods, __iter__() and __next__(), collectively called the iterator protocol. An object is called iterable if we can get an iterator from it. Most of built-in containers in Python like: list, tuple, string etc. are iterables. The iter() function (which in turn calls the __iter__() method) returns an iterator from them.

We use the next() function to manually iterate through all the items of an iterator. When we reach the end and there is no more data to be returned, it will raise StopIteration. Following is an example.



A more elegant way of automatically iterating is by using the for loop. Using this, we can iterate over any object that can return an iterator, for example list, string, file etc.



1.1 How for loop actually works?

As we see in the above example, the for loop was able to iterate automatically through the list. In fact the for loop can iterate over any iterable. Let's take a closer look at how the for loop is actually implemented in Python.



Is actually implemented as.



So internally, the for loop creates an iterator object, iter_obj by calling iter() on the iterable. Ironically, this for loop is actually an infinite while loop. Inside the loop, it calls next() to get the next element and executes the body of the for loop with this value. After all the items exhaust, StopIteration is raised which is internally caught and the loop ends. Note that any other kind of exception will pass through.

1.2 Building Your Own Iterator in Python

Building an iterator from scratch is easy in Python. We just have to implement the methods __iter__() and __next__(). The __iter__() method returns the iterator object itself. If required, some initialization can be performed. The __next__() method must return the next item in the sequence. On reaching the end, and in subsequent calls, it must raise StopIteration. Here, we show an example that will give us next power of 2 in each iteration. Power exponent starts from zero up to a user set number.



We can also use a for loop to iterate over our iterator class.



2. Python Generators

You'll learn how to create iterations easily using Python generators, how is it different from iterators and normal functions, and why you should use it.

What are generators in Python? There is a lot of overhead in building an iterator in Python; we have to implement a class with __iter__() and __next__() method, keep track of internal states, raise StopIteration when there was no values to be returned etc. This is both lengthy and counter intuitive. Generator comes into rescue in such situations. Python generators are a simple way of creating iterators. All the overhead we mentioned above are automatically handled by generators in Python. Simply speaking, a generator is a function that returns an object (iterator) which we can iterate over (one value at a time).

How to create a generator in Python? It is fairly simple to create a generator in Python. It is as easy as defining a normal function with yield statement instead of a return statement. If a function contains at least one yield statement (it may contain other yield or return statements), it becomes a generator function. Both yield and return will return some value from a function. The difference is that, while a return statement terminates a function entirely, yield statement pauses the function saving all its states and later continues from there on successive calls.

Differences between Generator function and a Normal function
Here is how a generator function differs from a normal function.

Generator function contains one or more yield statement.

* When called, it returns an object (iterator) but does not start execution immediately.

* Methods like __iter__() and __next__() are implemented automatically. So we can iterate through the items using next().

* Once the function yields, the function is paused and the control is transferred to the caller.

* Local variables and their states are remembered between successive calls.

* Finally, when the function terminates, StopIteration is raised automatically on further calls. Here is an example to illustrate all of the points stated above. We have a generator function named my_gen() with several yield statements.



One interesting thing to note in the above example is that, the value of variable n is remembered between each call. Unlike normal functions, the local variables are not destroyed when the function yields. Furthermore, the generator object can be iterated only once. To restart the process we need to create another generator object using something like a = my_gen(). Note: One final thing to note is that we can use generators with for loops directly. This is because, a for loop takes an iterator and iterates over it using next() function. It automatically ends when StopIteration is raised. Check here to know how a for loop is actually implemented in Python.



Python Generators with a Loop

The above example is of less use and we studied it just to get an idea of what was happening in the background. Normally, generator functions are implemented with a loop having a suitable terminating condition. Let's take an example of a generator that reverses a string.



Generator expression produces one item at a time. They are kind of lazy, producing items only when asked for. For this reason, a generator expression is much more memory efficient than an equivalent list comprehension.

We havne done the following example using a list and a for loop:



However, using the 'Generator Expression as follows, the generator expression did not produce the required result immediately. Instead, it returned a generator object with produces items on demand.

Take note of the parens on either side of the second line denoting a generator expression, which, for the most part, does the same thing that a list comprehension does, but does it lazily:



Why generators are used in Python?
There are several reasons which make generators an attractive implementation to go for.
1. Easy to Implement Generators can be implemented in a clear and concise way as compared to their iterator class counterpart. Following is an example to implement a sequence of power of 2's using iterator class.



This was lengthy. Now lets do the same using a generator function.



Since, generators keep track of details automatically, it was concise and much cleaner in implementation.

3. Python Closure

Before we do Closure, we need to know Python nested functions:



Defining a Closure Function

In the example above, what would happen if the last line of the function print_msg() returned the printer() function instead of calling it? This means the function was defined as follows.



That's unusual.

The print_msg() function was called with the string "Hello" and the returned function was bound to the name another. On calling another(), the message was still remembered although we had already finished executing the print_msg() function. This technique by which some data ("Hello") gets attached to the code is called closure in Python.

This value in the enclosing scope is remembered even when the variable goes out of scope or the function itself is removed from the current namespace. Try running the following in the Python shell to see the output.



Here is a simple example of a Closure:



4. Python Decorators

Decorators in Python make an extensive use of closures as well. Next, we will learn Decorators:

What are decorators in Python? Python has an interesting feature called decorators to add functionality to an existing code.

Here is an example:



When you run the code, both functions first and second gives same output. Here, the names first and second refer to the same function object. Now things start getting weirder. Functions can be passed as arguments to another function.

Such function that take other functions as arguments are also called higher order functions. Here is an example of such a function.



Even more weird, a function can return another function.



Here, is_returned() is a nested function which is defined and returned, each time we call is_called(). Finally, we must know about closures in Python. We learned Closure from the previous section.

Getting back to Decorators Functions and methods are called callable as they can be called. In fact, any object which implements the special method __call__() is termed callable. So, in the most basic sense, a decorator is a callable that returns a callable. Basically, a decorator takes in a function, adds some functionality and returns it.



In the example shown above, make_pretty() is a decorator. In the assignment step: pretty = make_pretty(ordinary), The function ordinary() got decorated and the returned function was given the name pretty.

We can see that the decorator function added some new functionality to the original function. This is similar to packing a gift. The decorator acts as a wrapper. The nature of the object that got decorated (actual gift inside) does not alter. But now, it looks pretty (since it got decorated). Generally, we decorate a function and reassign it as,:

ordinary = make_pretty(ordinary)

This is a common construct and for this reason, Python has a syntax to simplify this. We can use the @ symbol along with the name of the decorator function and place it above the definition of the function to be decorated. For example,



This didn't work since I called the functions from the decorator. I should call the inner function instead.



The wrapper (the decorator) will never be the most important part but the gift (inner function) is.

One more example:



5. Chaining Decorators in Python

Multiple decorators can be chained in Python. This is to say, a function can be decorated multiple times with different (or same) decorators. We simply place the decorators above the desired function.



Swap it:






Tasks:

1. What are the method(s) that iterator object must implement?

A. __iter__()
B. __iter__() and __next__()
C. __iter__() and __super__() __
D. iter__(), __super__() and __next__()

2. How can you create an iterator object from a list?

A. By passing the given list to the iter() function.
B. By using a for loop.
C. By using a while loop.
D. You cannot create an iterable object from the list.

3. If a function contains at least one yield statement, it becomes :

A. an iterable
B. a generator function
C. an anonymous function
D. None of the above

4. Use the 'one-liner' for loop and the 'next()' function to print out the square of the first two elements of the following list.



5. What are the criterias that must be met to create closure in Python?

A. Program Must have a function inside a function.
B. The nested function must refer to a value defined in the enclosing function.
C. The enclosing function must return the nested function.
D. All of the above.

6. What is the output of the following code?



7. What is the output of the following code?



8. Which of the following statement is true?

A. You cannot chain multiple decorators in Python.
B. Decorators doesn’t work with functions that take parameters.
C. The @ symbol doesn’t have any use while using decorators.
D. None of the above

9. The following Closure can generate the results as shown below:



Of course, you don't have to assign 'nf1=f(1), nf2=f(3)' but just do this:



Convert this into the format that using the '@' symbol to decorate function g() using the function f(). (as long as you use the '@' symbol to decorate the function and get the same result, it if flexible how you do this)

10. Design a function 'G(x)' to do the following calculation. This function only passes the parameter 'x'. Use a closure function F(a,b,c) to pass 'a, b, and c' to the function. Print the results.