Python functions beyond basics and a deep look at closure in Python.

All examples in this article use Python 3.3+.

1 Python functions beyond basics

Let's introduce several important features of Python functions first.

1.1 Python function as a variable

Python function, essentially is also object, like a common Python variable, can be assigned to a variable.

Let's see a simple demo. First of all, define a random string. Then call print and lenfunction and pass the defined string.

>>> slogan = "Life is short, I use Python">>> print>>> <built-in function print>>>> print(slogan)>>> Life is short, I use Python>>> len>>> <built-in function len>>>> len(slogan)>>> 27

These work fine, nothing special.

Next step:

• assign print to a new variable original_print
• then assign len to print.
  

>>> original_print = print    # assign print to original_print>>> original_print    # original_print now becomes print>>> <built-in function print>>>> original_print(slogan)>>> Life is short, I use Python>>> print = len    # assign len to print>>> print    # print now becomes len>>> <built-in function len>>>> print(slogan)>>> 27

The conclusion here is straightforward: Python function can be assigned to a variable.

1.2 Python function as function argument

A Python function can be passed as a argument to a function.

1.2.1 Custom example

Let's give a smart_add function as an example. It takes three arguments and the third argument is a function.

def smart_add(x, y, f):    return f(x) + f(y)

Try to call smart_add and pass abs function as the third argument to it.

>>> smart_add(-3, 7, abs)    # abs(-3) + abs(7)>>> 10

Again, try to pass another one: math.sqrt.

>>> import math>>> smart_add(4, 9, math.sqrt)    # math.sqrt(4) + math.sqrt(9)>>> 5.01.2.2 Built-in example: map>>> help(map)# ...map(func, *iterables) --> map object# ...

According to the document: map function will make an iterator that computes the function using arguments from each of the iterables. Stops when the shortest iterable is exhausted.

In short, map will take each item x in iterables and map it to func(x).

x in iterables |--map to--> func(x)

The first parameter of map is a function, and the second one is an iterable collection. For example, pass len function as the first argument and map string to its length.

>>> names = ["Tom", "Jerry", "Bugs Bunny"]>>> mapped_obj = map(len, names)>>> mapped_obj>>> <map object at 0x102629320>>>> print(list(mapped_obj))[3, 5, 10]

Many other functions in Python are similar to map, which takes a function as argument, such as reduce, filter.

You may have noticed that map function can be replaced with list comprehensions.

# same effect as the map function[func(item) for item in iterables]

In fact, the reduce function was demoted from built-in in Python 2.x to the functoolsmodule in Python 3 on that account. But the map and filter functions are still built-ins in Python 3.

Anyway, what we learned from this part is that Python function can be passed as an argument to a function.

1.3 Return a function in a Python function

Let's define an inner function within an outer function and then return this innerfunction from outer.

def outer():    print('call outer() ...')    # define an inner function within the outer function    def inner():        print('call inner() ...')    # return the inner function     return inner

Call the outer function and notice that the returned result is a function.

>>> r = outer()    # call outer()call outer() ...>>> r    # Returned result by calling outer() is a function>>> <function outer.<locals>.inner at 0x1089c6d08>>>> r()    # Call the returned functioncall inner() ...

One important thing to remember is not to confuse "return a function" with "return a data value".

import mathdef demo_one():    return math.sqrt    # return a functiondef demo_two(x):    return math.sqrt(x)    # return a data value

Look at another example below.

# pow_later.pydef pow_later(x):    y = 2    def lazy_pow():        print('calculate pow({}, {})...'.format(x, y))        return pow(x, y)    # Use Python built-in function: pow    return lazy_pow

Try it in Python shell.

>>> from pow_later import pow_later>>> my_pow = pow_later(3)>>> my_pow>>> <function pow_later.<locals>.lazy_pow at 0x10a043d08>

pow_later returns a function that will actually calculate the result of pow(3, 2) in the future.

So call it when you need, and you will get the real calculated result:

>>> my_pow()calculate pow(3, 2)...91.4 Bonus: higher-order function and first-class function

A function that meet at least one of the following criteria is called a higher-order function.

• takes one or more functions as arguments
• returns a function as its result
  

In fact, A Python function is not only a higher-order function, but also a first-class function, which satisfies following four criteria:

• can be created at runtime
• can be assigned to a variable
• can be passed as a argument to a function
• can be returned as the result of a function
  

2 Python closure

Now take a deeper look at the latest example mentioned above.

def pow_later(x):    y = 2    def lazy_pow():        print('calculate pow({}, {})...'.format(x, y))        return pow(x, y)    return lazy_pow

We called pow_later(3) and it returned a function object.

>>> my_pow = pow_later(3)>>> my_pow>>> <function pow_later.<locals>.lazy_pow at 0x10a043d08>

then we invoked the returned function object.

>>> my_pow()calculate pow(3, 2)...9

Obviously, the variable y and the parameter x are local variables of pow_later function. So when my_pow() was called, the pow_later function had already returned, and its local variables also had gone. But in fact my_pow() still remembered the vaules of x and yeven the outer scope pow_later was long gone. How did this happen?

2.1 Free variable

If a variable in a function is neither a local variable nor a parameter of that function, this variable is called a free variable of that function.

In short, free variables are variables that are used locally, but defined in an enclosing scope.

In our case, x is a parameter of pow_later and y is a local variable of pow_later. But within lazy_pow, x and y are free variables.

2.2 Closure2.2.1 What is closure

Specifically speaking, my_pow, actually the function object returned by callingpow_later(x), is a closure.

Note that the closure for lazy_pow extends the scope of lazy_pow function to include the binding for the free variables: x and y.

Generally speaking, a closure is a structure (code blocks, function object, callable object, etc.) storing a function together with an environment. The environment here means information about free variables that function bounded, especially values or storage locations of free variables.

For example, a closure is created, returned and assigned to my_pow after following function call.

>>> my_pow = pow_later(3)

Essentially, this closure is the codes of function lazy_pow together with free variables xand y.

2.2.2 Inspect closure

You can see that the closure keeps names of free variables by inspecting __code__attribute of my_pow function which represents the compiled body of the function.

>>> my_pow.__code__.co_freevars>>> ('x', 'y')

Meanwhile, pow_later will also keep names of local variables that are referenced by its nested functions in co_cellvars attribute of its code object.

>>> pow_later.__code__.co_cellvars>>> ('x', 'y')

However, where is the values of free variables?

>>> dir(my_pow)>>> my_pow.__closure__>>> (<cell at 0x10a428348: int object at 0x109e06b60>, <cell at 0x10a428378: int object at 0x109e06b40>)

Note that my_pow has an attribute named __closure__ and it's a tuple with two elements.

>>> dir(my_pow.__closure__[0])>>> ['__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'cell_contents']>>> my_pow.__closure__[0].cell_contents>>> 3>>> my_pow.__closure__[1].cell_contents>>> 2

So __closure__ is a tuple of cells that contain bounded values of free variables.

If your Python version is 3.3+, you can use inspect module to inspect. The nonlocalsdictionary in inspecting result is exactly the bounded free variables and their values.

>>> import inspect>>> inspect.getclosurevars(my_pow)ClosureVars(nonlocals={'x': 3, 'y': 2}, globals={}, builtins={'print': <built-in function print>, 'pow': <built-in function pow>, 'format': <built-in function format>}, unbound=set())2.2.3 __closure__

Functions without free variables are not closures.

def f(x):    def g():        pass    return g

Note that returned function g has no free variable. And its __closure__ is None.

>>> h=f(1)>>> h>>> <function f.<locals>.g at 0x10f650158>>>> h.__code__.co_freevars>>> ()>>> print(h.__closure__)>>> None

Global variables are not free variables in Python. So global functions are not closures.

>>> data=200    # global>>> def d():    # global>>>     print(data)... ... >>> d()>>> 200>>> d.__code__.co_freevars>>> ()>>> print(d.__closure__)>>> None

__closure__ attribute of global functions is None.

2.2.4 nonlocal declaration

Let's review our pow_later(x) function.

• pass a number x to function pow_later;
• pow_later will return a function object;
• the returned function object my_pow will calculate x**2 (y=2) each time it is called.
  

Now I'd like to change above behavior, let y increase 1 automatically each time my_pow is called. That is:

• the firt time call, calculate x**2;
• the second time call, calculate x**3;
• the third time call, calculate x**4;
• ....
  

The updated source codes are as follows.

# pow_later.pydef pow_later(x):    y = 2    def lazy_pow():        print('calculate pow({}, {})...'.format(x, y))        result = pow(x, y)        y = y + 1      # increase y        return result    return lazy_pow

Try it in Python shell.

>>> from pow_later import pow_later>>> my_pow = pow_later(3)>>> my_pow >>> <function pow_later.<locals>.lazy_pow at 0x108e020d0>

So far so good, let's call my_pow to see result.

>>> my_pow()>>> Traceback (most recent call last):...UnboundLocalError: local variable 'y' referenced before assignment

The error message is clear enough.

• It's a UnboundLocalError
• y is a local variable
• local variable y referenced before assignment
  

The problem happens in this line: y = y + 1.

We are actually assigning to y in lazy_pow scope, and that makes y becomes local tolazy_pow scope. So Python considers y a local variable of lazy_pow. Before assigning to that local variable, Python will first read the local variable y. But y is a free variable as mentioned eariler and there is no local variable named y in lazy_pow scope at all.

You may think, OK, we don't assign! How about use y += 1 instead of y = y + 1? The +=operation is performed in-place, meaning that rather than creating and assigning a new value to the variable, the old variable is modified instead.

The answer is: no change here. Because y is a number, which is an immutable type. +=will also create a new number object with new value behind the scene and assign the reference of the new object to y.

To deal with this situation, a nonlocal declaration was introduced in Python 3. It marks a variable as a free variable even though it is assigned a new value within the function.

# pow_later.pydef pow_later(x):    y = 2    def lazy_pow():        nonlocal y    # nonlocal declaration        print('calculate pow({}, {})...'.format(x, y))        result = pow(x, y)        y = y + 1        return result    return lazy_pow

Now the closure works well.

>>> from pow_later import pow_later>>> my_pow = pow_later(3)>>> my_pow()>>> calculate pow(3, 2)...9>>> my_pow()>>> calculate pow(3, 3)...27>>> my_pow()>>> calculate pow(3, 4)...813 Summary

Two topics were discussed in this article.

First, Python functions are first-class functions.

Second, what is closure and how it works in Python.

【转载】

https://juejin.im/entry/5b02755ef265da0b776ffdc3

ヾ(◍°∇°◍)ﾉﾞ