Python, cPickle, pickling lambda functions

Posted on

Question :

Python, cPickle, pickling lambda functions

I have to pickle an array of objects like this:

import cPickle as pickle
from numpy import sin, cos, array
tmp = lambda x: sin(x)+cos(x)
test = array([[tmp,tmp],[tmp,tmp]],dtype=object)
pickle.dump( test, open('test.lambda','w') )

and it gives the following error:

TypeError: can't pickle function objects

Is there a way around that?

Answer #1:

The built-in pickle module is unable to serialize several kinds of python objects (including lambda functions, nested functions, and functions defined at the command line).

The picloud package includes a more robust pickler, that can pickle lambda functions.

from pickle import dumps
f = lambda x: x * 5
dumps(f) # error
from cloud.serialization.cloudpickle import dumps
dumps(f) # works

PiCloud-serialized objects can be de-serialized using the normal pickle/cPickle load and loads functions.

Dill also provides similar functionality

>>> import dill           
>>> f = lambda x: x * 5
>>> dill.dumps(f)
'x80x02cdill.dilln_create_functionnqx00(cdill.dilln_unmarshalnqx01Uecx01x00x00x00x01x00x00x00x02x00x00x00Cx00x00x00sx08x00x00x00|x00x00dx01x00x14S(x02x00x00x00Nix05x00x00x00(x00x00x00x00(x01x00x00x00tx01x00x00x00x(x00x00x00x00(x00x00x00x00sx07x00x00x00<stdin>tx08x00x00x00<lambda>x01x00x00x00sx00x00x00x00qx02x85qx03Rqx04c__builtin__n__main__nUx08<lambda>qx05NN}qx06tqx07Rqx08.'
Answered By: ChrisB

Answer #2:

You’ll have to use an actual function instead, one that is importable (not nested inside another function):

import cPickle as pickle
from numpy import sin, cos, array
def tmp(x):
    return sin(x)+cos(x)
test = array([[tmp,tmp],[tmp,tmp]],dtype=object)
pickle.dump( test, open('test.lambda','w') )

The function object could still be produced by a lambda expression, but only if you subsequently give the resulting function object the same name:

tmp = lambda x: sin(x)+cos(x)
tmp.__name__ = 'tmp'
test = array([[tmp, tmp], [tmp, tmp]], dtype=object)

because pickle stores only the module and name for a function object; in the above example, tmp.__module__ and tmp.__name__ now point right back at the location where the same object can be found again when unpickling.

Answered By: Martijn Pieters

Answer #3:

There is another solution: define you functions as strings, pickle/un-pickle then use eval, ex:

import cPickle as pickle
from numpy import sin, cos, array
tmp = "lambda x: sin(x)+cos(x)"
test = array([[tmp,tmp],[tmp,tmp]],dtype=object)
pickle.dump( test, open('test.lambda','w') )
mytmp = array([[eval(x) for x in l] for l in pickle.load(open('test.lambda','r'))])
print mytmp
# yields : [[<function <lambda> at 0x00000000033D4DD8>
#            <function <lambda> at 0x00000000033D4E48>]
#           [<function <lambda> at 0x00000000033D4EB8>
#            <function <lambda> at 0x00000000033D4F28>]]

This could be more convenient for other solutions because the pickled representation would be completely self contained without having to use external dependencies.

Answered By: Rabih Kodeih

Leave a Reply

Your email address will not be published.